si 
tSHlfc.' 
s»ëf-. COMMISSION OF THE EUROPEAN COMMUNITIES 
Brussels, 07.01.1999 
COM(1998)782 final 
REPORT FROM THE COMMISSION 
MID-TERM REVIEW OF 
STRUCTURAL INTERVENTIONS 
OBJECTIVES 1 AND 6 (1994-99) 
mm 
i$fja£ DEVELOPING A MANAGEMENT CULTURE 
THROUGH EVALUATION: TOWARDS BEST PRACTICE 
m 
•5 
&£$ïk 

EXECUTIVE SUMMARY 
The Mid-Term Review: raising effectiveness 
1. The Mid-Term Review of Objective 1 and 6 programmes is a process informed 
primarily by the various Mid-Term Evaluations carried out at programme level. It 
also takes account of other elements including national policy contexts, the EU's 
own priorities and other relevant strategic analyses. The principal aim of the Mid-
Term Review is therefore a thorough and systematic assessment of the 
programmes, leading where necessary to the adjustment of priorities and 
reallocation of budget for each programme. 
2. The present Mid-Term Review was able to draw upon a large number of 
evaluation exercises (more than a hundred) and was greatly facilitated by 
discussions in partnership with national authorities. 
3. The Mid-Term Review covered virtually all Objective 1 and 6 CSFs and SPDs. 
The importance of this process varied from one Member State to another. In 
general, it has enabled a more effective deployment of resources and provided 
support for specific strategic priorities. 
Mid-Term Evaluations have proved useful to inform programme adjustments 
4. Mid-Term Evaluations are not an end in themselves but a means to improve the 
quality and relevance of programming. They involve substantial amounts of 
human and financial resources as well as, in some cases, considerable preparation 
and discussion involving programme partners. 
5. These evaluations were not implemented simply so as to comply with existing 
Community programming rules. They were also introduced because of a growing 
interest in evaluation on the part of national or regional authorities. Increasingly, 
these authorities are coming to see evaluation as a means of improving the 
management of Funds and maximizing the results obtained. 
6. Mid-Term Evaluations offer an important means of assessing the desirability of 
particular changes to original programme objectives. Programming of structural 
interventions can be influenced by changing external factors and by changes in the 
general policy framework. Factors such as these can necessitate a review of 
strategic priorities and lead to more systematic and effective programming. 
7 Benefits arising from the present evaluation include enhanced value for money 
and a revised balance of priorities matched to changing needs and circumstances. 
These should be seen as part of a more fundamental learning process, one which 
includes benefits which might not be fully recognised in the short term such as the 
increase of existing evaluation capabilities and improved co-ordination between 
national and regional authorities. 
Assessing performance is relatively straightforward 
8. Compared with previous exercises in respect of earlier programming periods, the 
Mid-Term Evaluations revealed significant developments in the use of 
quantitative indicators as a means of monitoring the implementation of 
interventions, assessing their impact and determining progress towards their stated 
objectives. Although the development of indicators has been uneven to date, 
elements of 'good practice' identified in certain programmes suggest that 
measuring and assessing performance is relatively straightforward, and does not 
necessarily require supplementary work on the part of national administrations. 
9. The use of formal performance indicators for measuring progress in relation to the 
main objectives and priorities of a programme can be of greater use than has 
hitherto been recognised. This should lead to more efficient management capable 
of delivering demonstrable improvements in terms of actual outputs and results. 
10. Assuming that Member States will have to carry out ex-ante evaluations of future 
programmes, it is essential that programming documents contain quantified 
objectives and indicators relevant to the Mid-Term Review process. The 
availability of this type of information can be genuinely useful for the 
management of the programmes in question and can serve as an objective basis for 
judging performance. It will, following proposals by the Commission in the draft 
Structural Fund Regulations, determine the allocation of a reserve to programmes 
according to their own internal performance. 
11 As regards the operational framework, monitoring systems will have to ensure that 
relevant indicators are measurable in a short time period and at low cost, using 
adequate existing information sources, and that they are subject to control by 
national and regional administrations. In addition, realistic targets should be 
established in a transparent way and initial objectives maintained during the 
implementation phase. These should be adjusted downwards only where this is 
justified by changing circumstances. 
The Mid-Term Review has been flexible enough in meeting changing needs and 
priorities 
12. The current approach to Mid Term Review seems, in general, to be both 
appropriate and sufficiently flexible in its response to changing circumstances and 
the need to ensure that current interventions are made more efficient. Both of 
these objectives were able to be pursued within the same operational framework. 
13. Strategic priorities are designed in a particular economic and institutional context 
and are subject to both external and internal influences. Although continuity of 
policy is desirable, there is always scope for further improvements, even in the 
case of the most successful interventions. In rapidly growing economies, (e.g. the 
Irish economy), economic performance enables structural changes to be made to 
the programming framework so as to address major problem areas such as 
inadequate physical infrastructure or skills shortages. 
14. The Mid-Term Review carried out for Objective 1 and 6 programmes did not 
radically affect their overall strategic orientations. The extent to which these 
interventions have been adjusted varies between Member States and regions, as 
well as between CSFs and SPDs. These latter, although smaller in scale, involved 
transfer between the different measures which were proportionately more 
substantial. 
15. In advancing its own priorities for adjustments to be made in the context of the 
Mid-Term Review, the Commission gave particular emphasis to sharpening the 
focus on employment, especially by supporting more job-intensive interventions. 
Although more could have been done to support this overriding priority, 
encouraging signs such as the results of the Mid-Term Review have emerged from 
the current experience. For example, there has been some reorientation of major 
aid schemes so as to benefit initiatives favouring SMEs and employment. In 
addition, Territorial Employment Pacts have been included within the mainstream 
programmes and specific actions introduced with regard to Information Society 
and equal opportunities between men and women. 
Notable progress in evaluation, but further improvements are needed 
16. Lessons drawn from the present Mid-Term Review show that it is not a purely 
academic exercise, but operationally and strategically oriented. In many Member 
States, programme managers and policy makers took a similar view and asked, on 
the basis of the evaluation findings, what kind of adjustments needed to be made, 
particularly with regard to the current interventions. 
17. Notable progress has been made regarding the quality and content of the Mid-
Term Evaluations and their incorporation into the overall Mid-Term Process. In 
time, this will lead to an improved management culture. In identifying good 
practice, factors such as process management (e.g. identifying key issues, reaching 
consensus, etc.) and more careful planning of the Mid-Term Review process have 
been noted. These will be included in any future guidance on best practice. 
18. As a result of the Mid-Term Review, national and regional authorities have taken 
steps to improve further the effectiveness of their interventions, especially with 
regard to their own delivery systems for Structural Funds. In particular, significant 
improvements will be made to the monitoring systems, notably by raising the 
quality and relevance of the indicators set out in the programmes. Other 
arrangements are currently being sought to simplify the management of small-
scale programmes and establish more coherent selection criteria in line with 
programming objectives. 
19. With regard to the next programming period, the main issue will be to consolidate 
evaluation practice further, taking stock of the results achieved so far. Reinforcing 
evaluation as a genuine tool for decision making, strengthening monitoring and 
indicator systems, supporting evaluation methodologies and promoting a 
multilateral exchange of experience across the European Union appear, in this 
perspective, as challenging areas for further raising the effectiveness of structural 
interventions. 
in 
EXECUTIVE SUMMARY 
INTRODUCTION 3 
/. Key elements of the Mid-Term Review process 4 
1.1. Policy context and evolving needs 4 
1.2. Commission guidelines 5 
1.3. Mid-Term Evaluations 5 
1.4. Financial implementation 6 
//. Lessons from the Mid-Term Evaluations 6 
2.1. Implementing the Mid-Term Evaluations 6 
2.2. Principal findings 11 
2.3. Utilisation of evaluation results 21 
///. Main outcome and adjustments 22 
3.1. Link between evaluation and reprogramming 22 
3.2. Overall reallocations of Structural Funds ...24 
3.3. Inclusion of EU priorities 26 
IV. Policy implications and future challenges 29 
4.1. Short-term implications for current delivery systems 29 
4.2. Initial reflections for future programmes 30 
4.3. The challenge of evaluation 30 
2-
INTRODUCTION 
The Mid-Term Review1 aims to ensure a more effective and efficient implementation of 
structural interventions delivered through Community Support Frameworks (CSFs) and 
Single Programming Documents (SPDs). This process is particularly important given 
public budgetary constraints. These latter make it increasingly necessary for Structural 
Funds resources to be fully used and for Structural Funds interventions to be of high 
quality in terms of their individual content. 
Evaluation represents an important means by which these goals might be realised. An 
important innovation of the current Structural Regulations is the requirement for 
systematic evaluation to be introduced for all EU programmes. In the context of the SEM 
(Sound and Efficient Management) 2000 initiative, the Commission regards evaluation as 
a key element for improving the management culture in which Community programmes 
operate2. More recently, the Commission's policy document Agenda 20003 covering the 
period 2000-2006 has aimed for greater cost-effectiveness from all structural 
interventions through reinforcing monitoring, financial control and evaluation in the 
context of decentralised management. 
The present report summarises the principal elements of the Mid-Term Reviews carried 
out in the Member States and analyses, in particular the role of the Mid-Term Evaluations 
as well as the extent to which EU priorities feature in all Objective 1 and 6 programmes. 
The Mid-Term Evaluations carried out by the individual Member States are the main 
source of information for this Report. It attempts to establish the role of the evaluation 
process in the development of the present Mid-Term Review and to identify best 
evaluation practice as a guide for the future programming period. This reflects a 
management culture which is emerging through the development of evaluation practice. 
By identifying good practice (Annex 1), this report contributes to a better understanding 
of how evaluations were implemented and used tp inform the Mid-Term Reviews 
particularly with regard to how they will be used in future in the context of the new 
Structural Fund regulations. It thus offers some experimental basis for actions to be taken 
after 1999. 
The following four areas are examined in the Report: 
1. the overall Mid-Term Review process; 
2. the principal findings of the Mid-Term Evaluations; 
Throughout this report a distinction is made between Mid-Term Evaluation and Mid-Term Review. 
Although the two terms are sometimes used interchangeably, Mid-term Review more properly 
describes the complete process comprising the evaluation exercise and the resulting adjustment of 
priorities and programme budget reallocations. Mid-term Evaluation is simply the actual evaluation 
exercise which informs the Mid-Term Review. 
SEM 2000 - Communication on Evaluation, 8 May 1996. 
Agenda 2000, For a stronger and wider Union, 1997. 
-3-
3. the main outcomes and adjustments made; 
4. the implications for programme delivery and priorities as well as future challenges 
I. KEY ELEMENTS OF THE MiD-TERM REVIEW PROCESS 
The Mid-Term Review process draws on four main elements: the policy context; the 
Commission guidelines; the Mid-Term Evaluations; and the state of financial 
implementation of Objective 1 and 6 programmes. 
/. /. Policy context and evolving needs 
The main aim of CSFs and SFDs is to reduce disparities between Member States in terms 
of their respective per capita income. These interventions seek to effect a long-term 
change in the structure and the performance of recipient economies. They are driven by 
specific national priorities and strategies to tackle basic structural problems, such as 
differences in infrastructure and human capital. These are important factors influencing 
regional competitiveness and hence growth performance. 
The four poorest Member States - the "Cohesion countries"- have seen a notable 
improvement in their per capita income levels. However, continuing disparities still exist 
among the European regions: in 1995, the 25 most prosperous regions had an average per 
capita income three times as high as that of the least-developed regions. The 
unemployment gap is even. more acute, affecting, in particular, the most socially 
disadvantaged groups4. 
Changes in the economic, social or political context in the Member States have affected 
their respective needs and problems, and thereby influenced the way in which they have 
adjusted their programmes. For example, Ireland's recent economic success has altered 
the need for certain types of intervention, notably interventions which aim to assist the 
productive sector. The Mid-Term Evaluation carried out in respect of that Member State 
has shown that there is a need to redirect resources towards public infrastructure (e.g. 
non-urban roads) to keep pace with the rapid economic growth. 
Similarly, the acknowledgement of employment as an overriding EU priority has also led 
to a number of programme adjustment such as the formal introduction of Territorial 
Employment Pacts. In addition, changes in national employment policies, such as 
structural reforms in the labour market have contributed to the refocusing of some 
training actions. 
Unexpected developments were also addressed in the context of the Structural Funds. For 
example, the earthquake emergency in the Umbria and Marche regions of Italy resulted in 
those regions benefiting from CSF resources an example of mutual support between 
Italian regions. 
See First Report on Economic and Social Cohesion (1996), COM (96) 542 final of 6 November 
1996. 
1.2. Commission guidelines 
The changing economic and social context at Member State level has led the 
Commission to adopt general guidelines5 to foster the value-added of Community 
interventions. These aimed to provide a general policy and priority framework within 
which adjustments to their current 1994-1999 Objective 1 and 6 programmes could be 
made as part of the Mid-Term Review process. 
The guidelines set out a number of thematic priorities in support of the main aim of 
Structural Funds interventions: helping to establish the conditions for sustainable 
economic development, growth and competitiveness and, thereby, lasting-jobs. This 
overall aim was underpinned by the following specific priorities: Basic Infrastructure, 
Productive Environment (all types of measures to improve the growth and 
competitiveness of business and industry), Research and Technological Development, 
Environment and Sustainable Development, Human Resources Development and Equal 
Opportunities. 
1.3. Mid-Term Evaluations 
In accordance with Structural Funds regulations and the programmes' standard clauses, 
all Member States (with the exception of Italy6) have managed to carry out Mid-Term 
Evaluations to inform the review of their structural interventions. These evaluations were 
designed to contribute to the process of improving the execution and management of the 
programmes for the remainder of the period. 
The guidance document7 published by the Commission stated that Mid-Term Evaluations 
should involve: a critical analysis of all of the data collected (particularly monitoring 
data); some measurement of the extent to which objectives were being achieved; an 
explanation of any discrepancies between the actual and expected results of the 
intervention; an assessment of the rationale for the intervention and the continued 
relevance of its objectives. 
' In addition, evaluations were expected to provide the data necessary to assist Monitoring 
Committees to form an opinion and propose corrective measures to the relevant 
authorities so as to enable adjustments to be made to the programmes. 
As a general rule, in the case of programmes lasting longer than three years, a Mid-Term 
Evaluation is carried out at the end of the third year of implementation. These 
7 Priorities for the adjustment of Structural Funds programmes to the end of 1999* - Commission 
Guidelines, May 1997. The formulation of guidelines had been endorsed by the Member States at an 
informal meeting of Ministers responsible for regional policy and spatial planning held in 
Ballyconnell, Ireland on 14-15 November 1996. The Member States formally asked the Commission 
to "formulate policy guidelines established in accordance with the principle of subidiarity, aimed at 
adapting the current Objective 1 and 6 programmes towards increased job-intensiveness for the 
period to the end of 1999 ". 
The Mid-Term Evaluation was undertaken on an interim basis by the Italian authorities (Evaluation 
unit of Budget Ministry) in the first half of 1997. 
Common Guide on Monitoring and Interim Evaluations (1996). 
-5-
programmes are also subject to ex-post evaluation at the end of the reference period. 
Evaluation arrangements should recognise this timescale. 
1.4. Financial implementation 
Evaluations deal normally with assessing impacts and progress in relation to objectives 
and priorities. They rely on financial data relating to the implementation of programmes 
which are supplied by the relevant monitoring systems. In addition, budget 
implementation at EU level provides an overview of commitments and payments. 
Progress here is summarised in the next paragraph for programmes under Objectives 1 
and 6 (see Annex 2). 
Since 1994, there has been continuous and regular progress in the implementation of the 
Objective 1 programmes. This was sustained during the first half of 1998. By the end of 
1997, 64% of the funds had been committed (compared with 46% at the end of 1996). 
Similarly, payments had been made covering 48% of the funds (compared with 33% at 
the end of 1996). Progress was particularly marked in Spain and Portugal where, over the 
1994-1997 period, respectively 69% and 70% of funds were committed and 53% and 
56% of funds were paid. Other Member States reported significant delays in 
implementation in terms of commitments, (particularly Netherlands, France, Belgium), 
while for Italy and Austria, delays related mainly to payments made. With regard to 
Objective 6, the two Member States offer contrasting situations. Implementation is more 
advanced in Finland than in Sweden. In Finland, by the end of 1997, 59% of funds had 
been committed compared with 32% in Sweden. 
II. LESSONS FROM THE MiD-TERM EVALUATIONS 
Mid-Term Evaluation has been an important development in the management of 
Structural Funds. It offers a valuable framework for reflection on the overall efficiency of 
EU interventions. The main lessons to be drawn concern the implementation process and 
the value of the main findings in terms of policy. 
2.1. Implementing the Mid-Term Evaluations 
a) Mid-Term Evaluation: a shared responsibility 
One innovation in the current Regulations governing Structural Funds is the 
strengthening of requirements regarding the evaluation of efficiency and effectiveness of 
the interventions. These requirements are clearly set out in the Regulations.8 
The Commission and the Member States have worked together to define the operational 
procedures to be followed. Evaluation is thus a shared responsibility. During 1995, the 
Member States and the Commission were actively involved in trying to reach a consensus 
According to art. 26.2 of the Co-ordination Regulation, "appraisal and evaluation shall, according to 
the circumstances, be carried out by contrasting the goals with the results obtained, where 
applicable, and by reference to macroeconomic and sectoral objectives and indicators based on 
national and regional statistics, to information yielded by descriptive and analytical studies and to 
qualitative analyses". The Standard Clauses in the CSFs and SPDs, agreed with the Member States, 
indicate that "monitoring [...] is backed up by interim evaluations so that any adjustments required 
to the CSFs and to operations in progress may be made". 
on the content of such evaluations and on the operational procedures to be followed in 
each case. While the operational framework proposed by the Commission was discussed 
multilaterally, the task of implementing the evaluation has, in general, been carried out at 
the level of each Monitoring Committee. 
By and large, despite the problems which accompany any new procedures and the 
difficulties some Member States had in defining and initiating an evaluation process, 
there have been no major differences of opinion between the Member States and the 
Commission9. 
b) Setting up the Mid-Term Evaluations 
Beyond the regulations - Improving management 
Most of the forms of intervention (CSFs, SPDs, Operational Programmes, Global Grants, 
etc.) programmed in each Member State were subject to a specific evaluation. Some 
Member States also instigated additional evaluations to complement their Mid Term 
Evaluations. These were either thematic (examining, inter alia, SMEs and Research 
related measures) or operational (e.g. selection criteria, procedures, etc.). The high 
number of Mid-Term Evaluations (more than a hundred for Objective 1 & 6 
programmes10) indicates that there is now a widespread recognition of their importance 
by the Member States. This is shown, not just through their compliance with the 
regulatory provisions, but also through their perception of evaluation as an instrument for 
improving the management of funds and optimising the results obtained thereby. 
The cost of the evaluations 
In the SEM 2000 Communication on Evaluation (1996), it was recommended that the 
overall budget for evaluation activities should not exceed 0.5% of the programme budget. 
Evaluation costs were generally financed under the "Technical Assistance" budget of 
each programme. The percentage of funds allocated to this activity has been in general 
below 0.1% of total cost of the programmes. In any case, fixed percentages would have 
been inappropriate since the cost of an evaluation is not necessarily proportionate to the 
cost of the programme in question. 
The evaluation procedure 
The starting point for the Mid-Term Evaluation process is to analyse the extent to which 
the programme in question lends itself to evaluation (the practicability of carrying out an 
evaluation or "evaluability ") and to establish a methodology to apply in the subsequent 
phase. An assessment of how readily a given prograrnme can be evaluated is necessary in 
order to address some gaps or inconsistencies in the information contained in the 
programming documents such as an insufficient quantification of targets for certain 
measures. Important to this process is the recognition that, before a successful evaluation 
can be undertaken, there is a need to ensure that sufficient arrangements have been put in 
10 The Madrid Informal meeting of Ministers of regional policy and spatial planning on 30th 
November-1st December 1995 marked an increased awareness among the Member States to the 
importance of Mid-Term Evaluation and its operational aims. 
See list in Annex 3. 
-7-
place to allow the programme ir^question can be evaluated. In particular, the evaluation 
of a programme generally requires that relevant indicators and targets have been set in 
advance and monitored over the programme's lifetime (Box 1). 
Box 1: Assessing the practicability of evaluation - ERDF Andalusia OP (Obj. 1) 
This evaluation aimed to describe the rationale for the programmed actions, objectives and goals, the 
degree of synergy between them and the adequacy of programme management. It focused on three main 
elements: quality of planning; information systems; management capacity. The approach adopted involved 
an analysis of the programming documents (using a Logical Framework Scheme) and interviews with key 
programme managers. 
The application of this methodology led to some practical conclusions. With regard to the quality of the 
programme, the evaluator observed some deficiencies in the SWOT (Strengths-Weaknesses-Opportunities 
and Threats) analysis as well as problems with some of the objectives (e.g. Tourism) which were poorly 
targeted. What is more, the information base was deemed to be too reliant on output as opposed to impact 
indicators. An assessment of the management system was also undertaken in order to bring about possible 
improvements, with particular regard to the division of tasks and responsibilities within the regional 
administration. 
Although not all of the recommendations made were taken on board, this first assessment raised an 
increased awareness among programme managers of the need to promote an evaluation culture and 
establish a better understanding of their own programme. 
The Mid-Term Evaluation itself is carried out halfway through a programme's 
implementation. It is intended to assess the degree to which the programme has met its 
objectives, determine the initial impact of the interventions and, where necessary, 
propose recommendations in order to improve the management of the programme and 
optimise its results. Mid-Term Evaluations should, in general, take as a reference point 
the outcomes of the ex-ante evaluations conducted on the basis of the plans submitted by 
the Member States. 
Independence of the evaluators 
The Member States agreed that it was particularly important to ensure that evaluation 
reports were drawn up independent of both the authorities responsible for managing the 
Structural Funds and the executive bodies with responsibility for co-financed actions. 
Mechanisms have therefore been established for appointing external evaluators 
(consultants, specialist academic teams, etc.). In those Member States in which 
evaluation is a well-established part of public expenditure programmes, external 
evaluators have tended to benefit from continuous advice and support from the relevant 
authorities. 
Internal evaluations can have some benefits (e.g. promoting "learning by doing") since 
managing authorities are closely involved in questioning the 'how' and the 'why' of their 
activities. However, for mid-term and ex-post evaluations, internal evaluations may not 
be practical, cost effective or even desirable. A notable exception was Italy where the 
Mid-Term Evaluation, performed by the Evaluation Unit of the Budget Ministry, raised a 
number of critical issues, especially on the quality of the programming documents and 
the lack of appropriate targets and indicators. 
-8 
c) Building up evaluation capacities in the Members States 
Establishing management structures 
Responsibility for the implementation of Mid Term Evaluations usually rests with the 
Monitoring Committees. Typically, these are involved in defining the content of 
evaluation reports and establishing a work programme for the evaluators. More 
importantly, they have analysed and discussed the completed evaluation reports, and, 
where applicable, proposed or advised on possible programme changes to maximise the 
efficiency and effectiveness of the Structural Funds allocated. 
In a significant number of cases, Monitoring Committees have established technical 
groups or sub-committees (see Box 2) in charge of ensuring ongoing liaison between the 
Committees themselves and the evaluators. These have then drawn up reports and 
proposals (based on evaluation results), for discussion and clearance by the Committee as 
a whole. 
Box 2: Managing evaluations - the creation of ad-hoc structures. 
In Ireland, the CSF as a whole and its three largest component Operational Programmes were each served 
by a dedicated evaluation unit providing on going advice and expertise. External evaluators supplied a 
similar on-call service to five other Operational Programmes. 
The Monitoring Committees have discussed the Mid-Term Evaluation reports at two separate meetings. 
Their conclusions and recommendations, along with the content of individual Operational Programme 
evaluations, have provided input to the overall CSF Mid-Term Evaluation. In addition, a regional 
evaluation report informed the overall CSF evaluation and account has also been taken of the 
Commission's guidelines. 
In Portugal, technical evaluation groups were set up to orient and control the evaluation process. Their 
main tasks included, inter alia, validating terms of reference, assisting in the selection of evaluators and 
discussing final reports prior to submission to the Monitoring Committees. These groups consisted of 
Commission officials and representatives of the relevant national and regional administrations. They 
normally met at least twice a year in advance of the Monitoring Committee meetings. Constructive 
relations between the partners within these groups have contributed to an improvement in the quality of 
evaluation work. 
Role of the partnerships 
In general, programme partners have shown a positive attitude towards Mid-Term 
Evaluations and willingly accepted responsibility for taking these forward. However, 
regional authorities vary in terms of their specific involvement in the tasks associated 
with Mid-Term Evaluation. These differences reflect characteristics of the political and 
institutional systems (i.e. the degree of decentralisation), the different forms of Structural 
Fund interventions (national, sectoral, regional and sub-regional programmes) and 
differing practices and experiences at the different Member State level. 
In some cases, regional authorities were responsible for setting up and managing the 
evaluation process. They were involved in selecting the evaluators, defining and 
facilitating their work and ensuring that the Monitoring Committees discussed and 
analysed the outcome of the evaluations. In addition, they ensured that evaluation 
recommendations were applied at programme level. In other cases, responsibilities and 
work were taken on by both national and regional authorities. But normally, where there 
were Community Support Frameworks (as in the case of some Objective 1 regions), the 
national authorities were responsible for the evaluation. 
-9-
A further situation is that of Member States where the regional authorities do not have 
executive responsibilities for the programming and management of Structural Funds. 
Their participation in the process, consisted of facilitating the provision of the data and 
information needed to draw up evaluation reports, co-operating in the analysis of the 
impact that policy measures had made at local level, and, to some extent, participating in 
the redefinition of strategic and operational guidelines stemming from the 
recommendations of the evaluation reports. Social partners were also associated with this 
process to the extent of their participation in Monitoring Committees. In general, 
partnerships have been developed in a pragmatic and flexible manner. 
Creating professional skills in evaluation 
Over the past four years, the development of evaluation activities across all Member 
States has been notable. Structural Funds regulatory requirements have been a main 
factor in this development. The provision of evaluation services remains largely outside 
of the various administrations, although Ireland and Italy are notable exceptions, having 
created their own specialised teams. The development of evaluation as a specialised 
service has led to growing professionalism and expertise. Although the evaluation teams 
appointed for different programmes tend to have multi-disciplinary expertise and include 
both consultancy companies and academic research centres, their main strengths are 
usually in the fields of economics and management. 
d) Improving the quality of evaluations 
The average quality of Mid-Term Evaluations carried out during the 1994-99 
programming period has undoubtedly improved compared with the previous period 
(1989-93) although factors such as delays in the launch of certain programmes or the 
adoption of inappropriate methodologies meant that some exercises were below standard. 
Evaluation reports were subject to assessments according to the eight criteria set out 
under the MEANS Programme11. These assessments suggest that most reports were of 
reasonably good quality (Box 3). The evaluations frequently provided updated 
information and analysis of the programmes in relation to their objectives and targets and 
some evaluations (e.g. Northern Ireland) have applied their own quality assessment in 
terms of strengths and weaknesses. Experienced evaluators and more extensive 
knowledge and research on the practicalities of evaluation have also contributed to 
enhancing the quality of the reports. 
11 Quality Assessment of Evaluation Reports: A Framework. MEANS (Méthodes d'Evaluation des 
Actions de Nature Structurelle) is a European Commission programme, which aims to improve the 
methodological tools available for assessing structural actions. They were given by assessing the 
evaluation reports against the following 8 criteria: meeting needs: does the evaluations adequately 
address the information needs of the MC and fit the Terms of Reference?; relevant scope: is the 
programme's rationale, outputs, and impacts fully covered, including unexpected outcomes?; 
defensible design: is the evaluation design appropriate to answer the questions asked?; data: is the 
data used/collected appropriate and is its reliability duly taken account of?; sound analysis: is the 
information available subjected to appropriate analysis?; credible findings: do the findings follow 
logically from, and are justified, by the analysis?; impartial conclusions: are conclusions fair, 
unbiased by stakeholder views, and operational?; and clarity: is the report written in a way that is 
easy to understand? 
-10-
Box 3: Assessing the quality of an evaluation: the Irish CSF evaluation 
The evaluation report fulfils the following criteria 
information needs were adequately addressed; 
the rationale of the programme and its objectives were comprehensively examined; 
a specific methodological framework was developed to assess the effectiveness of measures; 
an adequate amount of reliable data were collected (although gaps and deficiencies still exist); 
a sound analysis of CSF's performance was provided; 
credible findings were produced and had practical effects on the progrgmme's structure 
objective conclusions were proposed and were sufficiently detailed for operational use; 
presented to stakeholders in an accessible and clearly written formats. 
The overall quality rating is therefore very high, due to a well designed evaluation framework. The report 
was published and made available to the public. 
The assessments made of rural development programmes, based on the interim 
evaluation, vary somewhat. In some cases this assessment is considered acceptable 
(Spain), and in others satisfactory (various German Lander) since it has achieved its aim 
of drawing attention to good results. Elsewhere, it is regarded as being of a very high 
level in view of the experience and quality of the independent assessors and a more 
precise definition of aims and specifications (Ireland). Sometimes, the scope of the 
evaluations has been more limited because of their academic or excessively general 
nature or because they looked only at implementation and management rather than at 
results and impact (France, Belgium). 
Some evaluations were also criticised as being insufficiently independent of the 
administration which commissioned them. In other cases, constraints were imposed on 
the assessor's work by lack of cooperation from the administrations concerned. 
The methodologies applied by evaluators have varied in nature and quality from one 
programme to another. Although the Commission has promoted a number of methods 
and tools through the MEANS programme, their implementation is still at an early stage. 
For example, impacts on employment are often seen as being difficult to measure, 
especially when deadweight and displacement effects have been taken into account12. 
Despite a number of qualitative improvements, there remains a need to reinforce, through 
appropriate guidance, the use of sound evaluation methods over the next evaluation 
stages. 
2.2. Principal findings 
Amongst their main goals, the Mid-Term Evaluations attempted to assess progress on the 
attainment of the various programme objectives, as well as their likely impacts on, for 
example, job creation. They also sought to examine the process by which the programmes 
had been managed and their component projects selected and monitored. 
See European Commission document, "Counting the jobs", 1997 and the European Job Challenge 
(1998). 
-11-
The purpose of this section is to review, on the basis of a representative cross section of 
evaluations, significant results relating to mainstream issues of concern and to suggest 
areas where improvement is needed. It will highlight some key features of the analyses 
carried out concerning three main themes: 
- macroeconomic impacts for the largest programmes, notably with regard to growth 
and employment; 
- microeconomic assessment, focusing on effectiveness issues; 
- the quality of delivery systems, particularly monitoring and indicators systems and 
project selection criteria. 
From this analysis, some general conclusions can be drawn in terms of how these results 
might be used and integrated into the Mid Term Review process. 
a) Macroeconomic impacts: growth and employment effects 
Most evaluations of Objective 1 CSFs generally include a macroeconomic assessment 
providing a number of valuable insights into the overall Mid-Term Review process. 
These illustrate the importance of taking into account supply-side, as well as demand 
side, impacts and provide a first quantification of the likely scale of these impacts. 
For large-scale interventions such as major Objective 1 CSFs, macro-economic effects in 
terms of economic growth and employment are likely to be visible. However, these 
effects are difficult to quantify due to the need to identify the counterfactual situation, i.e. 
the situation without Structural Funds. One option is the use of simulation techniques 
based on models which take into account the macro-economic interdependence of 
variables. Although there are several models quantifying the short-term demand side 
effects (e.g. input-output models), only a few take into account the long-term effects of 
supply side conditions the improvement of which is the main objective of Structural 
Funds13. 
In the context of the Mid-Term Evaluations, the supply-side effects of adding 
infrastructure, human capital and productive investment to an economy are estimated in 
different ways of varying sophistication. For Ireland and Spain, HERMIN14-type models 
have been applied, while for Greece, Portugal and Italy more general econometric models 
have been used. Many variables are endogenous but "external" influences include interest 
rates, exchange rates and economic growth among trade partners. Due to the non­
availability of a macro-model for East Germany, the German evaluation took an ad hoc 
approach, applying, step by step, estimates for the various parameters influencing GDP 
13 The most recent attempts to quantify the combined demand and supply side effects of Structural 
Funds have been made using the QUEST II model, the results of which were presented in the First 
Cohesion report (1996), the HERMIN model used for the Single Market Review (1996). 
14 Hermin is a macroeconometric model specifically designed to quantifiy the medium to long-term 
impact of the CSFs. It provides a common framework focusing on the most important structural 
features of the 'Cohesion countries'and explains the mechanisms through which the CSFs will affect 
the supply-side of their economies. 
- 12-
and employment; the authors themselves note in the report that the results carrying a high 
degree of uncertainty. 
Two studies did not provide an integrated supply side in their model and simply made 
additional calculations on supply side effects based on average values available in the 
literature. The Portuguese evaluation calculates an additional productivity effect of 
between 1.17% and 2.35%. The study on East Germany assumes long-term supply side 
effects on GDP to be 20% of the short-term demand side effect. For this reason, the 
results presented below for these two countries (and, at this stage, for Greece) feature 
demand side effects only. 
A specific problem of macro-level modelling is given for those countries whose territory 
is not fully eligible for Objective 1 assistance (D, E, I). Each of the evaluations has dealt 
with this problem in a different way: the German evaluation analysed the impact on East 
Germany alone whereas the Spanish evaluation only considered the impact on Spain as* a 
whole. The dualistic structure of the Italian macro-model enabled at the impact on both 
Southern Italy and Italy as a whole to be assessed. 
The results of the different Mid-Term Evaluations (see Graph 1) should be treated with 
caution regarding their comparability since the methods used varied across countries. 
Nevertheless, there can be no doubt on the positive growth and employment effects of 
Objective 1 programmes. Mainly due to the impact of increased productivity, effects on 
employment are usually estimated to be much lower than those on growth. Graph 1 
shows, for example, in Spain, that the CSF is likely to increase GDP by 5.1% which 
means an additional annual growth of roughly 0,8% average; and employment by 2.4% 
by 1999 relative to the baseline position (i.e. the situation without CSF). 
The challenge for future evaluations of the macroeconomic impact of Structural Funds 
will be to make more use of methodologies which represent the state of the art. Macro-
models should have integrated demand and supply sides, the latter allowing to distinguish 
the main categories of interventions such as infrastructure, human capital and productive 
investment. Main variables to give results for are GDP, investment, employment, 
consumer prices, budget deficit, imports and exports. The most interesting additionality 
scenarios are EU funding alone as well as EU funding and national funding together, both 
excluding private co-financing. Account should also be taken of opportunity costs of 
public spending, i.e. the effects of an alternative use of EU and national funds. Finally, 
the sensitivity of results to changes in CSF spending and in economic policy can provide 
more concrete results in terms of policy conclusions. 
-13-
Graph 1: Mid-Term Evaluation results -percentage deviations from baseline in 1999 
% 
• GDP 
• employment 
Germany Greece Spain Ireland Italy Portugal 
(Obj. 1) (Obj. 1) 
Note: The employment figure for Portugal is the reduction of the unemployment rate in percentage 
points. 
Sources: Ifo Institut (D), KEPE (EL), Quasar (E), ESRI (IRL), Ministero del Tesoro-Nucleo di 
Valutazione (I), CISEP (P). 
Assessing the macroeconomic impact in small areas 
For small areas, specific models were designed to estimate the long term impact on 
growth, investment and employment. An interesting methodology was developed for the 
Belgian Hainaut region (Objective 1), notwithstanding the difficulties in collating 
regional data at a very disaggregated level (Box 4). 
Box 4: Hainaut (Obj.l) : the HELM model 
HELM (Hainaut Lead-in Model) is an econometric model assuming a key relationship between output 
growth and productivity growth. This model is extended to integrate different components of productivity 
growth (research and development, physical capital, skills). The results are presented for three situations: 
the baseline situation (A), the SPD without constraint (B) (measuring the additional impact of the SPD), 
the SPD with constraints (C) (measuring the additional impact but integrating, for the aid schemes, the 
rate of assistance and selection criteria adopted). The comparison between these situations allows for an 
estimate of the additional impact of the SPD against the baseline. 
The results of the simulations run for the SPD Mid-Term Evaluation relate to three main variables 
(investment, value-added, employment) in the manufacturing and business services sector. According to 
this model, the SPD will generate an additional investment growth of 0,7% per year for the 1994-2005 
period and additional employment growth of 0,3% per year, amounting to 5100 net jobs in 1999 and 
15800 net jobs in 2005. 
The massive absorption of funds at the end of the period will reduce the efficiency of implementing the 
SPD. Delays in execution may also have the effect that impacts are more visible at the end of the period 
and thereafter. The model shows that any increase in the rate of assistance will have only a moderate effect 
on investments, but these will produce some beneficial (indirect) effects within the area. Therefore, the risk 
of returning to slower growth compared with the EU average after the completion of the SPD is 
significant. 
14 
b) Microeconomic assessment 
Assessing effectiveness 
The evaluators were asked to provide an analysis of the effectiveness of the interventions. 
Most evaluations took a view on the extent to which the programmes were meeting their 
stated objectives15. As part of their work, they had to review the existing indicators, and 
in some cases proposed a number of new or revived core and key indicators, which had to 
be agreed with the regional authorities. This assessment has often required a substantial 
review of the indicator systems, in terms of both financial results and the monitoring of 
the agreed physical and impact indicators. 
A systematic approach to effectiveness was adopted for the Irish CSF evaluation, using a 
microeconomic model to assess whether the measures were designed properly and were 
able to meet their objectives. This approach implies that Structural Funds cannot only be 
judged on spending but rather on the social benefits they will bring to the economy. The 
methodology adopted by the evaluator aims to compare the effectiveness of the measures 
in order to ensure an optimal allocation of funds (i.e. comparable value of money across 
the programmes) and identify best practice. A further more detailed description is 
presented in Box 11. 
At a more operational level, evaluations should seek to measure the extent to which main 
programme objectives are being met. In the case of Merseyside, the evaluation contains 
some key indicators (SMEs, land improved, floor space), including output data and 
corresponding results of job creation and private sector leverage. Quantified targets are 
available for 1999 and the Mid-Term Evaluation presented figures measuring progress to 
date. These indicators provided in an aggregate fashion at the level of priorities. 
Subsequent to the Mid-Term Evaluation, a process of revision of the Merseyside 
baseline, target and output data was initiated. 
Box 5: Assessing effectiveness through programme indicators-the case of Mersey side-Objective 1 (1994-
1999) -actual outputs/results for ERDF interventions ^^^ 
Driver 
1 
2 
3 
4 
5 
All ERDF 
expenditure 
£m %of 
SPD 
28.8 21 
44.4 54 
15.4 42 
4.8 16 
22.1 26 
115.5 31 Total Gross Jobs 
(direct & indirect) 
000s % of 
SPD 
Target 
5.4 15 
9.2 142 
0.8 16 
0.7 64 
3.0 454 
19.2 39 Total Floorspace 
000m3 
114 
116 
1 
9 
240 %of 
SPD 
Target 
20 
70 
0 
15 
n/a 
24 Total land 
mproved 
(Hec) % of 
SPD 
Target 
255 52 
17 12 
32 n/a 
210 300 
513 73 Total Private 
Investment 
£m %of 
SPD 
Target 
67 40 
76 90 
0 
8 28 
23 n/a 
176 77 
15 Efficiency issues were rarely addressed, being one of the most difficult aspects of evaluation. 
Analysing efficiency involves comparing programme inputs (financial resources) with outputs (the 
goods and services it provides) and results (the initial effects) to estimate if the same benefits could 
have been produced using fewer inputs or, alternatively, if the same inputs could have produced 
greater benefits. Discussion of efficiency necessarily entails comparisons with various counterfactual 
positions. The main difficulty in this area is therefore the choice of appropriate benchmarks. 
-15-
Note: Drivers (or priorities) for the Merseyside SPD are the following; 1) inward investment, 2) 
indigenous enterprise, 3) knowledge based measures, 4) cultural, media and leisure, 5) actions 
for Merseyside. 
Source: Mid- Term Evaluation report (Nov. 1996). 
Indicators defined at OP or SPD level provide a basis on which effectiveness can be 
monitored and assessecl. However, they do not in themselves allow definitive conclusions 
to be drawn, particularly on impact, because of the difficulty in establishing causation in 
some areas (i.e.: the indicators may be affected by external factors). They often need to be 
supplemented by qualitative analysis and by appropriate evaluation work. 
Overview of main outputs and results 
Microeconomic data on outputs and results arising from the evaluation reports are not 
easy to present in an aggregated way because of their specificity and lack of 
comparability across programmes. It is also recognised that the extent to which these data 
can be quantified at the Mid-Term stage will vary across both the areas of interventions 
(infrastructure, SMEs, human resources, etc. ...) and regions concerned. The standard 
and coverage in the evaluation reports is inevitably somewhat variable as is the extent to 
which key elements such as outputs, results and changes in marked disparities could be 
quantified. In Box 6, some achievements relating to Objective 1 and 6 programmes are 
presented. 
Basic infrastructure - reducing disparities 
Basic infrastructure is the area where the impact of Community action is most visible and 
measurable. The incidence of these major investments is sometimes high in structural 
terms. For example, in Spain, CSF resources supporting transport, communication and 
energy networks represent on average more than 25% of national infrastructure. Notable 
progress in reducing disparities with the rest of the Union has been made in most sectors 
(see box 6). 
Box 6: Important achievements in basic infrastructure. 
• Substantial effort is continuing to support the development of the strategic road networks in the 
Cohesion countries through, for example, the improvement of four major corridors in Ireland (reaching 
49% in 1996 against a target level set for 1999 of 53%) and the completion of the Athens-Thessaloniki 
motorway (400 km built to date). In Portugal, the construction of motorways and primary roads to date 
represents 74% of the 1999 target. In Spain, the stock of physical infrastructure for motorways 
increased by 13% between 1993 and 1995 (latest data available), benefiting in particular to the less 
developed regions like Andalusia. 
• In all Objective 1 regions, major investment has gone into the telecommunication sector, providing 
more modern systems such as digital exchanges and fibre optic links. This has contributed to a 
significant reduction of disparities in provision compared with the rest of the Union. In Portugal, the 
average density of telephone lines (per 100 inhabitants) is aligning line with the EU level; this 
objective was expected to be attained by 1999. Similarly, in Italy, the rate of digitalisation in the 
Southern regions has reached the same level (85%) as that of the Centre North. In Spain, digitalised 
networks increased by 36% in absolute terms between 1993 and 1995. 
• Progress in energy diversification, notably the reduction of dependence on oil has also been made. 
Deliveries from the new natural gas distribution should begin in Greece. The high pressure natural gas 
pipeline (513 km) has already been completed and another major project (1000 km of low pressure 
natural gas network) is nearing completion (84%). In Portugal, more than half of the gas network (600 
km) had been laid by 1996 and this will account, by 1999, for 7.5% of total energy consumption. 
Greece, Spain and Portugal have all seen significant reductions in their energy use relative to GDP in 
recent years. _^ 
-16-
As regards the environmental sector, support is being provided to improve systems of water supply 
and increase the capacity of wastewater treatment facilities. In Greece, water supply, sewerage and 
wastewater treatment systems, serving about 5 million inhabitants (half of the population) have already 
been completed. Investment has also gone into smaller Objective 1 regions, like the Northern Ireland 
Sub-programme where the construction of water supply and waste water facilities have generated some 
2270 jobs in the area. Significant environmental benefits have also arisen from a whole range of 
specific interventions, including the clearance of major contaminated industrial sites, environmental 
audits ofSMEs, assistance in the development ofeco products and reduction of emissions due to the 
diversification of energy supply. 
Productive environment - supporting employment 
In terms of the productive sector, structural assistance has been directed at improving 
conditions for existing firms, thereby encouraging new activities. It is largely recognised 
that support, especially for SMEs is a major driver for job creation, and this contributes to 
bridging structural gaps in productivity and income levels. Some examples may highlight 
recent trends in Structural Funds impacts in this area (see box 7). 
Box 7: Some achievements in the productive environment 
• Major aid schemes have emphasised job creation, partly offsetting the negative consequences of 
industrial restructuring. 
In Hainaut, almost all funds available to aid schemes, notably SME schemes had been used before the 
end of the period; it has been estimated that job creation targets for those measures have already been 
exceeded to date (with more than 2200 jobs created compared with 1800 jobs expected). 
In Germany (Obj. I), Structural Funds assistance contributed actively to support employment in, for 
example, Sachsen-Anhalt where 18.500 jobs were assisted in the chemical industry and a further 
20.000jobs in SMEs. 
In Italy, the industry Programme is likely to have created or safeguarded more than 75.000 jobs over 
the 1994-96 period. 
• In the tourism sector, increased benefits will derive from new capacity being made available in, for 
example, Sachsen (additional capacity of 11500 new hotel beds) and in Ireland, where 18 large 
projects which have already been completed providing additional facilities to increase the quality and 
standard of tourist attractions. New employment opportunities will be generated by these Structural 
Funds assisted improvements 
• Physical regeneration of industrial sites will also contribute to the improvement of the economic 
environment and attract new business. This was a main priority in some German Lander: in East 
Berlin, for example, where 51000 m2 were refurbished, creating potentially 600 jobs or in Sachsen 
where 620 ha of land were transformed into business parks. In Nord-Pas de Calais, approximately 40-
45% of industrial land located in the Objective 1 area will have been improved. 
Human resources: quantifying impacts 
Specific evaluations conducted in the area of human resources included, as part of their 
workplan, an impact analysis of training measures on beneficiaries. The main results of 
these studies are presented below in box 8. 
Box 8: Impact of training measures on beneficiaries 
In Spain the evaluator made a comparative analysis of the placement of trainees by target group, using the 
INEM (The National Employment Institute) database as a quasi control group. For this analysis only the 
occupational training measures were taken into account. The main conclusions are in line with the results 
of the 1989-93 ex-post evaluation: in terms of gross impact, the placement rate of trainees is on average 
20% higher than the control group; placement is higher for young people, men and those holding a 
secondary education diploma. However, if net impact is estimated, taking into account the characteristics 
of the trainees, the differences vary substantially. For example, in terms of age, trainees over 45 years old 
perform 32% better in finding a job than those in the control group. Training also seems to increase the 
-17-
chances of women finding jobs: the net impact is 27.1%, whilst for men it is 14.9%. If qualification levels 
are taken into account, the effect of training on those holding a primary education diploma is striking: 
going through a training action increases their chances of finding a job to 48.8%. 
Germany: impact of the QfQsj (Gerneinsçhafi Initiative - Qst) 
In Germany, it has been estimated that the Bund programme in support ofGIOst (i.e. apprentice schools 
for the young) has brought about a reduction in youth unemployment of three percentage points in 1995. 
At Lander level, very few placement indicators have so far been collected. The combination of training 
measures with grants generally leads to higher performance, but this is the exception rather than the rule. 
Some regional results can be mentioned. For Berlin-Ost, the evaluator concluded that, for continuous 
training and unemployed training taken together, 60% of trainees had found or kept jobs after six months. 
Training programmes in Meckelenburg-Vorpommern achieved placement rates of 40-45% (21-32% in the 
first labour market). A programme for business starters, which basically tries to help people ensure that 
they start a viable business, resulted in about 50% of actual "successful" starters. Various continued 
training and education programmes for employees (Objective 4 type measures) in Mecklenburg-
Vorpommern, Thuringen and Sachsen-Anhalt resulted in increases of 35 to 77% in the perception of job-
security by participants. 
Portugal: impact on the education system 
Impacts in terms of strengthening the Portuguese education system (through PRODEP) have been found to 
be linked to the diversification of education pathways and to a focus on both quantity and quality issues. 
Training teachers and other agents appears to be one of the major contributions of the programme, both 
at 1st and 2nd level (representing 49.3% of teaching staff) and at university level (provision of 
scholarships for Masters and PhD programmes). As regards the expansion and diversification of education 
/ training pathways, major changes have included enlarging secondary education, bringing it closer to the 
workplace and increasing guidance and counselling in schools. In fact, support to middle level technician 
training has set up a whole sub-system providing technical skills directly relevant to the labour market and 
an alternative to mainstream education. 
Agriculture and rural development 
Since agriculture and rural development constitute separate programmes at national level, 
they have been assessed separately. In the case of SPDs and the agricultural sections of 
regional programmes, evaluation formed part of a more general evaluation. 
Some Member States such as Ireland carried out detailed evaluations of the main 
priorities of the programmes (the food industry, establishment of young farmers, 
compensatory allowances, equal opportunities, agricultural training, etc.). In such cases 
evaluation proved a useful tool for obtaining better knowledge of the situation, analysing 
problems and finding the best ways of solving them. 
Box 9: Evaluation of agricultural interventions - the case of PAMAF (modernisation of agricultural 
structures in Portugal 
The evaluation report found that financial and physical implementation had been slow in getting started, 
mainly because of delays in drafting the relevant legislation and management changes. However, the 
initial delay has been made up and the assessor is optimistic about the chances of some adjustments in 
management and the allocation of funding between measures permitting programme implementation to get 
back on schedule so that the original goals can be achieved. 
The operational assistance measures concern primarily support for investment on agricultural holdings 
where the rate of implementation is high, infrastructure such as irrigation, rural roads, drainage and soil 
conservation, forestry, the processing and marketing of agricultural and forestry products, compensatory 
allowances in less-favoured areas, training and research. Measures of these types are intended to make 
agricultural holdings more competitive. Most of them represent a continuation of the earlier programme, 
hich had proved successful. They also form part of structural policy at Union level. 
The assessor stressed in particular that measures were to some extent scattered, which affected their real 
impcct as well as management and implementation. He suggested that the funding available should be 
rea!" seated to the measures regarded as priorities, particularly aid to agricultural holdings and irrigation, 
-18-
and insisted on the need to improve vocational training. He also noted the effort made to concentrate 
support on the private sector and a non-uniform regional distribution of projects. The report notes some of 
the results of the agriculture programme in 1994-96, including the financing of some 20 000 projects, 
supportfor 11 500 agricultural holdings and the establishment of 2 800 young farmers. 
Fisheries 
The FTFG provides support in five main areas of assistance: adjustment of the fishing 
effort (26% of the budget), renovation and modernisation of the fleet (24%), the 
processing and marketing of products (23%), the development of aquaculture (11%) and 
facilities in fishing ports (7%). It currently includes 31 programmes, of which 17 are 
under Objective 1, 12 under Objective 5(a) and two under Objective 6. 65% of the budget 
goes to Objective 1 regions. 
The Mid-Term Evaluations of these programmes provided interesting information about 
the effectiveness of assistance and reprogramming (the reallocation of funds between 
areas of assistance or between measures). They revealed some shortcomings in 
monitoring, particularly in the provision of socio-economic data. In some cases proposals 
to improve the indicators were implemented and initial analyses of the impact on 
employment carried out. 
Box 10: An example: the Spanish Objective 1 prqgramme 
The operational programme for fisheries, which has funding of about ECU 1 billion, is one of the priorities 
in the CSF. The measures in the programme are mainly concerned with adjusting the fishing effort, 
modernising the fleet, the processing of products, aquaculture and facilities in fishing ports. The 
evaluation report found that the average implementation rate was satisfactory at 57% but varied widely 
depending on the area of intervention: it was low for modernisation of the fleet and aquaculture (10%-
20%) and high for port facilities and processing of products (60%-100%). 
Effectiveness was lowest in aquaculture (2%): this measure has been affected by the cumbersome 
administrative procedures required to implement projects, poor returns which discourage potential 
investors and environmental constraints on project selection. Effectiveness was highest in the case of 
processing (65%), where beneficiaries often achieve high growth rates and so good investment capacity. 
The assessor proposes to reallocate funds from adjustment of the fishing effort and aquaculture to the 
modernisation of vessels and the processing of products. 
A survey of beneficiaries showed that 35% of the projects receiving grants under the programme had 
created or preserved jobs (initial figures suggest a total of 780 jobs), principally in the area of product 
processing. Proposals have also been made to improve the indictors of achievement and results, to reduce 
the number of specific indicators for aquaculture, establish impact indicators for the fleet and employment 
indicators for all areas of intervention. The assessor noted that aquaculture paid strict attention to 
environmental protection and recommended that greater account should be taken of this principle in other 
areas (processing of products and port facilities). 
c) Quality of delivery mechanisms 
Mid-Term Evaluations were also required to assess the way in which the Funds are 
managed, particularly in terms of monitoring systems and project selection criteria. 
Monitoring and indicators systems 
Compared with the previous programming period, the current programmes show 
significant developments in the use of quantitative indicators as a means of monitoring 
their implementation, assessing their impact and determining progress towards meeting 
overall objectives. 
-19 
The Commission prepared a Common Guide for Monitoring and Evaluation with a view 
to ensuring consistency of approach and uniform standards16. This has been a primary 
impetus in bringing about these improvements. However, experience across programmes 
is mixed. In a number of programmes, the information available by way of indicators is 
often inadequate in terms of measuring performance or verifying if satisfactory progress 
is being achieved. Other principal shortcomings relate to the determination of appropriate 
and practical targets in relation to the measures in question as well as the lack of a 
consistent, regular system of data collection relating to physical outputs, results or 
impacts. 
Notwithstanding many improvements, the definition and quantification of appropriate 
indicators remains an unsatisfactory area of programme management. Whilst further 
work is required on indicator systems to improve monitoring and assist evaluation work, 
this should begin by identifying examples of good practice in this area. Interesting 
attempts at building appropriate management information systems can be found in a 
number of programmes. Some examples are provided in Box 11. 
Box 11: Enhancing the monitoring systems 
A well managed information system is a necessary (although not sufficient) condition for supporting the 
evaluation process. The case of Nord-Pas-de Calais (Objective 1) is an example of an integrated 
monitoring system, which couples financial and physical indicators at project level. In Northern Ireland, 
an important part of the Mid-Term Evaluation was an intensive scrutiny ofNIQUID (the database of all 
the indicators agreed for the SPD) to transform it into an effective management information system for the 
SPD. The evaluator proposed monitoring on a regular basis the progress of 52 indicators, rather than the 
330 indicators identified in the SPD. In addition, for each sub-programme, a set of key indicators was 
developed to capture its specific impact. 
Significant work has also been done in Italy, where a standardised system of indicators was established for 
the ESF interventions. In Portugal, indicators have improved in almost all areas of intervention, even if 
there is still scope for further improvements in the quality of data. 
Project selection criteria 
The first two or three years of programme implementation have, in some cases, seen the 
development of relatively sophisticated scoring systems for co-financed projects. The 
systems put in place (e.g. in Merseyside) take account of a wide range of factors 
simultaneously and in addition to assessing eligibility they look at selection criteria both 
at priority and measure level. For example, priority criteria which are common to all 
projects include project outputs, value for money, private sector leverage and more 
recently environmental impact. 
Many Mid-Term Evaluations have questioned the effectiveness of such systems. Having 
transparent and objective criteria, which in itself is considered as good practice, may not 
be a sufficient condition to ensure that the best projects are selected. Further checks will 
16 The Commission Guide distinguishes between indicators of output, results and impact. Output 
indicators refer to financial and physical implementation (e.g. number of kilometres of road built and 
cost, number of training courses provided. Results indicators refer to the immediate effects of an 
intervention (for example, time savings in a road project; the number of people who successfully 
complete a training course). Impact indicators refer to the outcomes of the interventions. A 
distinction can be made between the specific or immediate impact of an intervention (e.g. number of 
people placed into jobs) and the general or final impact, i.e. the socio-economic effects (e.g. increase 
in employment or GDP). 
-20-
have to be made, for example on the degree of credibility attached to the information and 
output forecast provided by the applicants. 
Box 12: Merseyside - scoring systems for ESF7ERDFprojects 
In the Merseyside SPD, the system put in place for selecting projects has three different aspects: eligibility 
or "core criteria", priority selection criteria common to all projects and measure selection criteria (which 
may be different from the priority criteria). 
The aim,was to develop sets of criteria which were understandable, transparent and which could be 
applied to a large number of heterogeneous projects. Main priority criteria include the expected outputs of 
the projects; value for money, the nature of beneficiaries (for ESF); labour market issues; evidence of 
partnership and linkage; private sector leverage; and environmental impact. 
2.3. Utilisation of evaluation results 
As described in the previous section, Mid-Term Evaluations represent an important 
source of knowledge for programme managers and decision makers. They inform them 
on key issues such as the relevance of the aims and objectives of their interventions and 
the effectiveness of the wider economic effects on the area concerned. An authoritative 
evaluation should be supported by rigourous analysis and be sufficiently operational to be 
fully utilised for policy purposes. 
However, the quality of the evaluation results does not guarantee that they will 
necessarily be utilised. In other words, the evaluations cannot be considered as the only 
input for carrying out the Mid-Term Review. Institutions and policy actors are 
continuously involved in exchanges of information. Other factors relating to the political 
context of the macroeconomic framework may also influence directly the final 
reprogramming decisions. Bearing in mind this wider framework, three different levels of 
utilisation should be distinguished in the Mid-Term Evaluation process. 
The first level of utilisation involves evaluation as a feedback mechanism relating to the 
effectiveness of policy measures. Evaluation results and recommendations have, in a 
significant number of cases, been largely followed and incorporated in the changes made 
to the programmes. The best example is the Irish CSF Mid-Term Evaluation which 
produced credible and useful conclusions leading to the reallocation of funds between 
measures or sub-programmes (see box 12). 
Secondly, evaluation played a major role in providing new sources of information or on­
going advice to programme managers and policy makers . For example, the evaluation of 
the Hainaut Objective 1 programme, which included a specific macroeconomic model 
(see Box 3), was also recognised as useful for informing the reprogramming actions. 
Conclusions here opened longer-term perspectives on the convergence trend of the 
regional economy. Other evaluations, having a narrower focus, stressed the importance of 
certain critical aspects of programme management, rather than being exhaustive with 
regard to the whole implementation cycle. 
Finally, the experience also showed that substantial input from the programme's 
partnerships is key to the success and effectiveness of the evaluation process. One 
important result is that decision makers were involved in the discussion of evaluation 
issues and a number of lessons learned in terms of improving their interventions. 
Although this learning process was not always formally structured, it had a significant 
impact on the final outcome of the Mid-Term Reviews. 
-21-
III. MAIN OUTCOMES AND ADJUSTMENTS 
The evaluations informed the Mid-Term Review process by providing an overall 
assessment of what had been achieved during the first half of the programming period. 
Taking stock of these elements, this part of the report assesses the main outcomes of this 
process priority in terms of the adjustments made to programmes. Account is also taken 
of the extent to which priorities laid down by the Commission have been incorporated in 
the programmes. 
3.1. Link between evaluation and reprogramming 
The evaluations were delivered within the agreed time period enabling the results to 
influence the Mid-Term Reviews carried out between 1997 and 199817. A special 
reference should be made to the three new Member States. Although their programmes 
started a year later than for those of the other Member States, all three made formal 
arrangements to launch Mid-Term Evaluations. The Commission promoted an exchange 
of experience between evaluators which brought new ideas into the conduct of the 
evaluations. A key feature of this process was that no significant adjustments were made 
prior to the evaluations and the formal review of programmes. 
Evaluations have, in general, proved useful in addressing major issues or highlighting the 
kinds of decision to be made. Their ability to do so has depended on the intrinsic quality 
of the analysis as well as on the operational focus of their recommendations. 
In the case of rural development, interim evaluations produced specific recommendations 
and proposals as requested both by the Commission and the managing authorities. These 
included the reallocation of resources. In Spain and Germany they coincided with the 
proposals by the responsible authorities since the grounds (progress of measures as 
shown by the monitoring data) were the same. In other cases, only some were taken into 
consideration (Portugal). Sometimes, evaluations were carried out after programmes had 
been amended and it was subsequently found that there had been no contradictions 
(France). In Italy the assessor concentrated on analyses of problems and obstacles to 
implementation and those responsible for the programmes proposed the reallocation of 
resources. In the case of Ireland, it has already been noted that the assessor's 
recommendations concerning the strengthening of certain measures could not be 
implemented because of lack of finance. 
The example of Ireland and Merseyside 
Two examples serve to highlight the entire Mid-Term Review process, from the formal 
evaluation exercise to the implementation of the financial reallocation decisions . These 
relate to two different levels of intervention: a large CSF (Ireland) and an SPD 
(Merseyside) (see Box 13 and 14). 
* ' There has been some delay in the negotiations for the Mid Term Review in France-Obj. 1 (especially 
la Réunion) and for Greece (reprogramming of ERDF interventions; ESF modifications scheduled 
for the second half of '98). In Italy, a number of reprogramming actions were decided in April '98 
and the Mid Term Review will be finalised in October, on the basis of actual progress of some slow 
spending programmes. 
-22-
Box 13: Mid-Term Review (I): the case of the Irish CSF 
The report states that the Irish CSF represents a notable success and that the funds have been deployed 
effectively to support and enhance what has been a remarkable economic recovery. Under the CSF, 
medium term planning of public expenditure has allowed more systematic and effective programming in 
many areas. 
The CSF evaluation raised questions about the value of devoting public resources to a number of specific 
measures, notably free or subsidised provision of services for which firms or individuals would be willing 
to pay. A core assumption here was that markets are the most effective and efficient means of achieving 
economic and social objectives so that public intervention is justifiable only where the market is not 
working properly and the intervention in question creates no economic distortions of its own. In the light of 
these theoretical assumptions, four situations were identified, where in the opinion of the evaluators, 
public intervention in a market economy could be justified: 
the provision of public goods and services (such as transport infrastructure) which would not be 
provided in the absence of public intervention; 
the introduction of corrective subsidies designed to alter the price of goods and services where the 
market price does not adequately reflect their wider social benefit (e.g. the cost of loans to SMEs) 
the management of targeted schemes aimed at changing behaviour through correcting a lack of 
knowledge, awareness or understanding (e.g. introducing business owners or employees to new 
technology) 
the redistribution of wealth through subsidies or welfare benefits in pursuit of broadly social aims 
The CSF evaluation categorised the 166 CSF measures using the above headings. For each category, one 
or two "anchor measures "- measures which were large scale or which addressed long-established and 
well-understood elements of public policy - were selected and used as comparators for the other measures 
in that same category. On the basis of these comparisons, decisions were taken as to which measures were 
successful and which were not. 
Some 150 recommendations were in turn produced providing how CSF resources might be deployed more 
effectively, with particular focus on those measures poorly targeted or having undesirable effects. The 
evaluation pointed to a clear case for more public spending on physical infrastructure, especially non-
urban roads. In some demand-led measures it was suggested that spending targets should be revised in 
line with actual evolution, for example a reduction of the grant to the Food Industry Sub-programme. 
It was also recognised, in discussions within the Monitoring Committees, that more emphasis should be 
placed on youth and long-term unemployment and that allocations to specific measures for education and 
the disadvantaged should be increased. 
The CSF evaluation also contains a number of detailed recommendations in relation to the management 
and implementation of specific measures. It is argued, for example, that there is insufficient competition in 
the provision of subsidised services due to the predominant position of state agencies and the lack of 
competition. Regarding monitoring, the report suggests further improvements to performance indicators, 
while admitting that the current CSF represents the most advanced experiment in monitoring and 
indicators systems. 
The outcome of the Mid-Term Review largely followed these recommendations. Accordingly .the shifts 
were mainly away from the productive sector towards human resources development and economic 
infrastructure. Financial reallocations amounted to 163 Mecu, focusing on RTD (37 Mecu), Transport 
Infrastructure (36 Mecu) and a range of measures dealing with early school leavers and other 
marginalised groups (46 Mecu). 
Box 14: Mid-Term Review (II) : the case of Merseyside (Obj.l) 
Structural Funds assistance to Merseyside comprises 816 MECU delivered through the Merseyside Single 
Programme's five "Drivers for Change": Inward investment and the corporate sector; Indigenous 
enterprise and local business; Knowledge-based industry and advanced technology; Culture, media and 
leisure industries; Action for the people of Merseyside. The latter receives the largest proportion of 
Programme funds (more than two fifths). The inward investment and local business drivers received, 
respectively, a quarter and a fifth of Programme funds. 
23 
The advent of the Single Programme coincided with a cyclical upturn in the UK economy characterised by 
falling unemployment and economic growth, both at rates higher than the EU average. Merseyside has 
shared in this upturn with the result that Structural Funds receipts are being deployed in a situation more 
favourable than that envisaged when the Single Programme was being developed. Although improvements 
in terms of unemployment have been small relative to the UK as a whole, they should be seen in the context 
of a recent picture of decline in the principal social and economic indicators. 
The Mid Term Evaluation assessed the current and potential impact of the Single Programme on the 
people and business life of Merseyside, commented critically on existing monitoring arrangements and 
made recommendations as to how the Programme should proceed over the remainder of the programming 
period. Desk research was supplemented with a review of 100 separate projects along with extensive 
survey and interview research focusing on Programme beneficiaries. The volume and quality of the 
evaluation research was significant. It compensated significantly for resource constraints which limited the 
amount of in-depth analysis possible. 
The evaluator made a series of detailed and reasoned recommendations, proposing in particular that 
future monitoring and evaluation work should focus on the larger projects likely to deliver the greatest 
impact as well as on issues such as displacement and sustainability. It was also recommended that the 
existing set of indicators be expanded to enhance the assessment of Programme impact. Proposed 
indicators included, inter alia, survival rates for SMEs and their use of new technology. In addition, it was 
suggested that insufficient focus was being given to synergies within the five Programme Drivers and to 
themes which cut across several measures. 
Reprogramming changes at the Mid-Term Review stage broadly followed the evaluator's proposals and 
were much in line with the Commission's Regional Policy Priorities to the end of 1999. A total of 97 
MECU was reallocated, primarily to measures most likely to lead to direct job creation (SMEs, technology 
and social inclusion). New measures were created for fish processing and for the development and 
application of communications technologies. The Commission also set a deadline for a comprehensive 
review of baseline data and quantified targets, with particular regard to net employment impacts. 
3.2. Overall reallocations of Structural Funds 
Under Objectives 1 and 6, all CSFs, SPDs and other forms of intervention were subject to 
a Mid-Term Review. The only exception was the new German Lander, where 
adjustments to the programmes are being made on an on-going basis. All these reviews 
led to financial reallocations of varying significance in order to improve the overall 
efficiency of the interventions. 
Due to the financial amounts involved, the scope of these changes is more wide-ranging 
for the larger and more complex CSFs than the SPDs. But it is not directly proportionate 
to the success or failure of the programmes concerned. Even a highly successful 
programme (e.g. the Irish CSF) can be improved and necessary corrections made without 
altering the main priorities and objectives. 
The scope for reallocations varied widely among programmes. Financial progress has 
been the general focus for these actions, especially with regard to the under-spending 
programmes. In addition, the reallocations also reflect the degree to which the EU 
priorities have been taken into account (see 4.3). In general, limited adjustments (fine-
tuning) were made to the programmes without affecting strategic priorities. In some cases 
(e.g. Italy-Obj.l, Portugal CSF), these modifications involved a transfer of resources 
from the slower spending programmes or sub-programmes to the faster spending ones, 
with due consideration given to other qualitative criteria. In most cases, however, 
modifications were made within programmes and between measures. Additional 
.. uurces from the deflator were also used for refocusing certain actions, particularly in 
favour of employment creation. Graph 4 indicates the extent of the transfers of resources 
for each Member State. 
24-
The largest modifications were made to the Objective 1 CSF in Italy (around 700 Mecu, 
including non-programmed resources and CSF indexation for 1998), followed by Spain 
(more than 600 Mecu) and Greece (400 Mecu only for the national programmes). Despite 
the large financial amounts involved, these only represent between 2% (Spain) and 5% of 
the total Structural Funds allocations. In Ireland, financial transfers amounted to 
160 Mecu of Structural Funds, or 3% of EU Structural Funds for the 1994-99 period. In 
Northern Ireland, no transfer of funds between sub-programmes was made, aside from 
the allocation of resources deriving from the deflator. In Finland and Sweden, the most 
significant changes were made through the merger of a number of measures in order to 
simplify the programme's structure. 
Conversely, in some relatively small programmes, significant changes were made, in 
relative terms, i.e. as a share of total Structural allocations. In the Belgian Hainaut, about 
19% of available resources were shifted particularly towards the SME aid schemes. The 
case of Flevoland, despite a relatively high amount of Funds transfered (13%) involved 
several minor changes in the SPD measures, without affecting its overall aims and 
objectives. 
It is somewhat difficult to present a comprehensive overview of the nature of these 
adjustments. More qualitative insights relating to some key EU priorities only will 
therefore be presented in the next section. 
Graph 4: Mid-Term Review Reallocation of Structural Funds by Member State (in 
Mecu) 
Mecu 
D Arhount in Mecu • Percentage of S.F allocation 
1 of which Nord-Pas-de-Calais (19,6 Mecus), Corse (10,1 Mecus), Guyane (36,1 Mecus) and 
Martinique (22,4 Mecus). 
2 no Mid-Term Review 
3 only ERDF reallocation 
4 of which Northern Ireland (69,8 Mecus), Merseyside (97,0) and Highlands and Islands (21 Mecu). 
-25-
3.3. Inclusion of EU priorities 
The extent to which the EU priorities laid down in the Commission's Guidelines for the 
Mid-Term Review have been addressed is not always easy to assess. New measures 
added or existing measures receiving additional funding may provide some indication 
however. 
Basic infrastructure - reinforcing TENs 
Reducing disparities in infrastructure endowment in the areas of transport, energy, 
telecommunications and environment has been recognised as a long term commitment of 
Structural Funds. This is reflected, in particular, in the share of total funding (more than 
30%) allocated to this area in the current Objective 1 programming documents 1994-99. 
Although still a major priority for most regions, no significant resource was devoted to 
funding new infrastructure. In general, the emphasis was placed on specific TENs 
projects with the aim of establishing efficient connections and systems, particularly in the 
area of transport (box 15). 
Box 15: Airport programme - Italy Objective 1 
The 10 Mecu airport programme, recently approved by the Commission, aims to improve or upgrade 
existing airports in the South of Italy. Following the Mid-Term Review, it was agreed to provide additional 
funding (50 Mecu) to this programme, exclusively for TENs projects, in order to further enhance regional 
endowments in this area. 
Productive environment - a strengthened focus on employment 
The results of the Mid-Term Reviews show that there has been scope to increase the 
focus on employment through a variety of actions (reinforcement of aid schemes, 
Territorial Employment Pacts, local development initiatives, etc.). A number of decisions 
already taken (e.g. to include Territorial Employment Pacts within the mainstream 
programmes, Box 16) reveal a political aspiration and commitment to combat 
unemployment more effectively. The increased focus on the employment generating role 
of projects, particularly SME projects, is also in line with the direction of policy within 
most Member States. This is reflected in the CSFs and SPDs. 
In this context, the Mid-Term Review has questioned the effectiveness of policy 
instruments in terms of supporting job creation and responding to the need for more 
active employment policies through human resources and education measures. At a more 
operational level, emphasis was also given to the measurement of employment impacts 
(e.g. UK regions, Nord-Pas de Calais) and to take better account of employment issues in 
project selection criteria (e.g. Italy). 
Box 16: Territorial Employment Pacts 
In three Member States (Spain, Greece, Italy), the Monitoring Committees allocated CSF resources to a 
programme or a specific measure to finance action plans for Territorial Employment Pacts. In Spain 
(Objective I), priority was given to five pacts which should receive 55 Mecu of EU funding (deriving from 
the CSF deflator), for integration into a new multi-regional programme. In Greece, it is envisaged that the 
resources arising from the 1997 and 1998 deflator will be allocated to specific sub-programmes within 
regional programmes which are currently under review. In Italy, national authorities proposed, in the 
context of the Mid-Term Review, a multi-regional and multi-fund programme of 280 Mecu, of which 140 
Mecu from Structural Funds will provide assistance to 9 pacts. 
26-
National aid schemes, which are being cofinanced out of Structural Funds, have also been 
reviewed in order to increase their focus on employment creation as well as on SMEs 
(see Box 17). 
Box 17: SMEs : the case of the Greek CSF 
The Greek authorities have refocused their Industry Operational Programme towards SMEs, with 
additional funding of 136 Mecu. The national aid scheme has been modified in line with the Commission's 
guidelines by increasing support to business services and by providing indirect assistance rather than 
direct grants. Intermediary bodies have been set up to manage the SME actions and projects will be 
selected on a competitive basis. A prior appraisal of the SME projects to be funded (about 1450 projects 
over 4200 received) was also undertaken. 
Environment and sustainable development 
Environment, as an element of sustainable development is one of the wider EU priorities 
most frequently addressed in the context of the Mid-Term Review. The pursuit of this 
objective is considered both through the introduction of specific measures and more 
importantly, through horizontal integration with other programme priorities. Because of 
objective difficulties (lack of indicators, methodological aspects) the results of the Mid-
Term Review could not in all cases lead to direct operational conclusions for the 
adjustment of programmes. Specific attention was drawn to a lack of systematic 
environmental impact assessment of projects and to the gains in project quality that could 
be secured by systematising such assessments. The way has been prepared for 
improvements in environmental protection and general project selection procedures over 
the new programming period (box 18). 
Box 18: Mainstreaming environment 
Environment is a long-standing concern of most Member States. For the Structural Funds to contribute 
properly to environmental protection, they must take account of this issue in all project decisions, so as to 
minimise damage to the environment and to maximise positive benefits, by for example giving preference 
to projects with positive environmental impact (preventive action, eco-products, ...). The Commission 
services are actively involved in Environmental Impact Assessment actions (specific thematic study already 
launched, a Handbook on Environmental Assessment of Regional Development plans and EU Structural 
Funds Programmes prepared, etc) with a view to improving the integration aspect. 
Progress has been made in operationalising environment as a "core criterion" for selection of projects, in 
particular in Finland and Sweden. Recently, this criterion has also been introduced in Italy within the 
industry programme, for the selection of new industrial projects. 
Research and Technological Development 
While research and development facilities are, in general, already in place, the use of 
existing assets needed to be optimised. Some improvements have been made in 
particular, in increasing RTD investment towards the local business and industry (see 
Box 19). 
Box 19: Research and Technological Development: the case of Ireland 
The importance of RTD was recognised in the original CSF and its position has been enhanced further in 
the context of the Mid-Term Review. Some 37 Mecu was devoted to additional RTD measures within the 
Industry operational programme. Elements of competitive bidding and reduced aid rates will be 
introduced. The Monitoring Committee also agreed that certain services directed towards the industrial 
sector could be provided on a cost recovery basis. 
27-
Information Society 
With regard to Information society, further resources have been directed to this priority in 
recognition of its importance in helping to reduce the effects of peripherality, and thereby 
increasing investment and productivity (Box 20). 
Box 20: Information Society (Portugml, Spain) 
In Spmm, a new sub-axis (6.6.) on Information Society was created within the CSF, following a specific 
recommendation by the evaluator. An initial funding of 46 Meat has been earmarked to support this 
prierity. ERDF measures will be devoted to the provision ef professional and interactive services for 
universities, tourism and businesses. ESF measures wM focus on specific training actions for citizem as 
well as for local authorities. 
A hey innovation brought about in the Portuguese CSF has been the decision to introduce a new measure 
to support Information Society within the Telecommunications sub-programme. This measure has received 
some 7 Mecufrom ERDF. In addition, significant efforts have also been made in other CSF interventions, 
in particular in the Local Development Programme (PDDR) where all the local development agencies will 
be connected or in the Education programme, with the introduction of Internet access to secondary 
schools. 
Human Resources Development 
A refocusing on the unemployed was deemed necessary with regard to interventions 
targeting human resources development following three years of strong emphasis on 
young people. This emphasis on the unemployed, in line with the preventive approach 
adopted in the Employment guidelines, took various forms: strengthening integrated 
interventions (guidance and counselling, training, employment aids), boosting micro-
enterprise creation subsidies, developing new forms of training, and local employment 
services. Within interventions targeting young people, support was given to moves away 
from conventional education and training programmes towards interventions in favour of 
young people at risk of exclusion and early-school leavers. In this regard, a closer 
integration between mainstream education systems and training and employment systems 
has been supported, especially with regard to upper-secondary level professional training 
(box 21). 
Box 21: Human resources development - Ireland Objective 1 
The Mid-Term Review of the Human Resources Development Operational Programme (HRDOP) in 
Ireland was approved by the Monitoring Committee (MC) in October 1997. The Mid-Term Review was 
informed not only by the HRDOP, the CSF and the ESF Programme Evaluation Unit reports but also by 
national and Community policy orientations, such as Government White Papers and the EU Employment 
Strategy. 
The Mid-Term Review package adopted by the MC consisted of: a broad policy framework, financial re­
allocations with a revised financial plan (there was approximately 40 MECU in changes within the 
programme, with a net addition of 14 MECU to the OP) and the identification of qualitative 
recommendations of the OP and CSF evaluations requiring action. 
The major reallocation within the OP has been to the Early-school leavers measure (reinforced with 26 
MECU) aiming to provide a minimum of 1000 additional places (with capacity within the financial 
envelope to provide 1400 plus, if necessary)y. Accompanying measures were strengthened, allowing for a 
more flexible approach. Provision was also made for the re-allocation and ring-fencing of additional 
progression places for early-school-leavers in other measures. Qualitative recommendations were 
adopted by the MC to refocus on long-term unemployed; increase the responsiveness of Industry training 
to employers needs; and reinforce investment in childcare infrastructure. 
-28-
Equal opportunities 
The theme of equal opportunities has been strengthened in many programmes, although 
this priority often appears as a horizontal objective and not systematically at measure 
level. However, significant efforts were made to refocus some measures towards this 
priority, especially under Human Resources interventions. Under ERDF interventions, 
specific measures for female entrepreneurship were introduced, especially in Italy and 
Greece, within the respective industrial programmes of those Member States(see 
Box 22). 
Box 22: Equal opportunities: reinforcing female entrepreneurship 
Access for women to enterprise creation schemes has been encouraged in a number of programmes. In 
Italy and Greece, a specific measure on female entrepreneurship (supported by national legislation) was 
introduced within their industrial Operational Programmes. In Finland, it is envisaged to guarantee 
access to a 'soft loan ' scheme, which is currently under scrutiny. 
IV. POLICY IMPLICATIONS AND FUTURE CHALLENGES 
The elements of the Mid-Term Review process presented earlier in this report had 
various implications for policy purposes. They drew attention to certain implementation 
and management issues, which could already be addressed during the remainder of the 
programming period. In addition together with an indication of future challenges arising 
from current evaluation practice, they provided a basis for reflection on strategic 
priorities for future programmes. 
4.1. Short-term implications for current delivery systems 
In discussing the Mid-Term Reviews, the point was frequently made that monitoring 
procedures tend to place greater emphasis on financial execution than on physical 
achievements. The majority of Mid-Term Evaluations have pointed to the need for new 
sets of indicators enabling better measurement of programme results and impacts. The 
identification of such indicators - as well as ensuring reliability and proper 
quantification - will increase the quality and scope of the next round of Mid-Term 
Evaluations. 
Some Member states have already taken steps to improve a number of elements in their 
current delivery systems, inter alia: 
- fixing clear quantified targets and indicators for monitoring and evaluation (e.g. 
Austria, Portugal, Italy, United Kingdom); 
- setting more appropriate selection criteria for projects (including employment and 
environmental criteria (e.g. Italy); 
simplifying management procedures, in particular for small programmes (e.g. 
Finland, Sweden); 
- securing synergies between different Structural Fund interventions (e.g. Portugal, 
Spain). 
29 
4.2. Initial reflections for future programmes 
The Mid-Term Review was concerned with making necessary changes to the 
programmes to reflect changing circumstances and needs arising in the course of the 
implementation phase. In parallel with the Mid-Term Evaluations, both the Commission 
and the Member States have conducted a number of strategic analyses which look ahead 
to the period beyond the current interventions. These were designed to prepare for future 
Structural Funds interventions by considering longer term issues which could not be 
addressed during the remainder of the existing programmes. For example, the 
Commission has supplemented the limited information basis offered by the interim 
evaluation in Italy with a set of thematic studies relating to key CSF priorities: industry-
related aid schemes, RTD, environment, transport infrastructure and human resources 
development. 
These studies set out to address certain key policy issues and also to provide 
recommendations for amending future CSFs. Whilst necessarily broad in outlook, these 
exercises have nonetheless usefully informed the Mid-Term Review process on issues 
such as the future re-orientation of spending priorities. 
The process of re-examining funding priorities has already commenced in the context of 
the Mid-Term Review in some Member States. Informal discussions and seminars have 
also taken place with national authorities to address some key issues for the future. The 
Commission has launched four thematic studies (RTD, SMEs, environment and the 
partnership principle) to help identify the broad directions of policy and facilitate the 
introduction of new arrangements for the period after 1999. 
4.3. The challenge of evaluation 
This report has shown that notable progress has been achieved in evaluation practice 
throughout the various Objectives 1 and 6 programmes. Important good practice features 
of the Mid-Term Review have been highlighted throughout the report including: 
- The soundness of evaluation (methodology and quality criteria); 
- The involvement of programme partnerships in the evaluation process; 
- The organisation and timing of the Mid-Term Review process; 
- The feedback role of evaluation in supporting reprogramming decisions as well as 
anticipating key issues. 
Lessons arising from this experience may represent a basis for guidance on best practice 
for the next programmes in order to improve further the management of the funds. The 
crucial issue is, thus, how to take stock of the results to promote a more systematic 
dissemination of good practice both within and between regions. In the light of 
experience to date, the Mid-Term Review shows that there is still a need for continuing 
improvements to the quality of the process in view of the challenges to be faced in the 
context of future programmes. 
Consolidating evaluation as a tool for decision making 
Evaluation is increasingly seen as a management tool, which should assist policy makers 
and programme managers in their tasks by providing them with important insights as to 
-30-
the effectiveness of measures and programmes. In its organisation, this function should 
link more closely the production of reliable and operational results to their integration in 
the decision making process. This would imply the creation, where unavailable, of 
adequate structures for managing the evaluation activities in the context of partnership. 
Improving monitoring systems and indicators 
Some significant actions have already been taken by national and regional authorities to 
address some gaps or deficiencies in their monitoring systems over the remainder of the 
programming period. Improvements in the definition of indicators and data collection 
will be necessary to enable these to perform their management tasks more effectively. 
Better integration between financial and physical indicators is a further, notable concern 
for programme managers. 
For future programmes, there is a need to ensure adequate quantification of baseline and 
target indicators and provide benchmark data to allow for better comparison within and 
between programmes. In this respect, the Commission will provide, in due course, a 
guidance document on methodological issues, including an indicative list of relevant 
indicators for the main areas of Structural Funds interventions. 
Supporting evaluation methodologies 
Substantial progress has been achieved in the area of evaluation methodology. The 
MEANS Programme, initiated by the Commission in 1994, has contributed to the 
development of that organisation's expertise, drawing on first hand experience of 
Structural Funds. There is no single and standardised evaluation methodology but rather a 
wide range of methods which should be customised so that they match the particular 
needs and situation of each programme. Efforts will be made, however, to ensure a better 
knowledge of evaluation techniques, not only among evaluators but also programme 
managers, who will have to appraise the quality of evaluation results. 
Promoting multilateral exchange of experience 
Since evaluation has been recognised as a major instrument for decision making, national 
and regional authorities are generally keen to have informal meetings to exchange 
experience and opinions on this matter. 
The Technical Evaluation Group at EU level is made up of representatives of Member 
States and Commission officials involved in the management of Structural Funds and 
might be considered a suitable forum in which to discuss evaluation issues and 
disseminate best practice. The main issues of common interest include the type and 
relevance of evaluation methodologies (e.g. measuring employment effects) and the 
deployment of evaluation results for policy purposes. 
Overall effectiveness of Structural Funds interventions will also be influenced by the 
extent to which national and regional authorities take necessary steps to improve their 
monitoring systems and are able to adopt good practice features to conduct the next Mid-
Term Review process. 
-31 
Annex I: List of boxes 
Assessing the practicability of evaluation - ERDF Andalusia OP (Obj.l) 
Managing evaluations - the creation of ad-hoc structures 
Assessing the quality of an evaluation: the Irish CSF evaluation 
Hainaut (Obj. 1 ) : the HELM model Box 1 
Box 2 
Box 3 
Box 4 
Box 5 
Box 6 
Box 7 
Box 8 
Box 9 Assessing effectiveness through programme indicators-the case of Merseyside-
Objective 1 (4994-99) -actual outputs/results for ERDF interventions 
Important achievements in basic infrastructure. 
Some key achievements in the productive environment. 
Impact of training measures on beneficiaries 
Evaluation of agricultural interventions the case of PAMAF (modernisation of 
agricultural structures) 
Box 10: An example: the Spanish Objective 1 programme 
Box 11 : Enhancing the monitoring systems 
Box 12: Merseyside - scoring systems for ESF / ERDF projects 
Box 13: Mid-Term Review (I): the case of the Irish CSF 
Box 14: Mid-Term Review (II) : the case of Merseyside (Obj.l) 
Box 15: Airport programme - Italy Objective 1 
Box 16: Territorial Employment Pacts 
Box 17: SMEs : the case of the Greek CSF 
Box 18: Mainstreaming environment 
Box 19: Research and Technological Development: the case of Ireland 
Box 20: Information Society (Portugal, Spain) 
Box 21: Human resources development - Ireland (Objective 1) 
Box 22: Equal opportunities: reinforcing female entrepreneurship 
32-
Annex 2 : Financial implementation of Objective I programmes 
Commitments/ total SF assistance under 
Objective 1 
i._..: 1994 1995 
I annual 1996 1997 
• cumulated I 1998 (1st 
semester) Payments/ total SF assistance under 
Objective 1 
1994 1995 
I annual 1996 1997 
•cumulated ! 1998 (1st 
semester) 
% Financial implementation of Objective 1 programmes by Member States (31.12.97) 
Financial implementation of Objective 1 Programmes bv Member-State (31/12/97) 
75 
70 
65 ] 
60 
55 
50 
45 
40 
35 
30 
25 
20 
15 
10 
5 
0 47 61 61 69 
44 63 
59 
51 70 
62 64 
Belgium Germany Greece Spain France Ireland Itaiy Netherlands Austria Portugal United- Total 
Kingdom 
IE Commitments/total SF assistance 1994-97 £3 Payments/ Total SF assistance 1994-97 
(Austria: SPD approuvée! in 1995) 
33 
Financial implementation of Objective 6 programmes Finland 
Commitments/total SF assistance under 
Objective 6 Finland 
57 57 
1994 
I annual 1995 1996 
-^--curnjteted 1997 1998(1st 
L; Payments/ total SFassistance under 
Objective 6 Finland 
1997 1998 (1st 
semester) 
Financial implementation of Objective 6 programmes Sweden 
Commitments/ total SFassistance under 
Objective 6 Sweden 
1994 1995 1996 1997 1998 (1st 
• annual —•— currulated ~] semester) Payments/total SFassistance under 
Objective 6 Sweden 
I annual 1996 
•cumulated 1997 1998 (1st 
semester) 
34 
Annex 3: List of reports 
Objective 1 reports 
Austria 
Belgium 
Germany 
Germany 
Germany 
Germany 
Germany 
Germany 
Germany 
Spain 
Spain 
Spain 
Spain 
Spain 
Spain 
Spain BURGENLAND : ZWISCHENEVALUATION DES 
ZIEL1-PROGRAMMS BURGENLAND: Endbericht 
HAINAUT : EVALUATION INTERMEDIATE DU 
DOCUP OBJECTIF 1 HAINAUT - RAPPORT FINAL -
REGION WALLONNE 
BERLIN: ZWISCHENBEWERTUNG DER EU-
STRUKTURFONDSINTERVENTIONEN IM LAND 
BERLIN IM ZEITRAUM 1994 BIS 1996 
BRANDENBURG : ZWISCHENBEWERTUNG ZUM 
EINSATE DES EFRE, DES ESF UND DES EAGFL-A IM 
LAND BRANDENBURG 1994-1996 
MECKLENBURG-VORPOMMERN : 
ZWISCHENBEWERTUNG DES EINSATZES DER 
EUROPAISCHEN STRUKTURFONDS (EAGFL, EFRE, 
ESF) FUR DIE PERIODE 1994 BIS 1996 IM 
BUNDESLAND MECKLENBURG-VORPOMMERN 
SACHSEN : ZWISCHENEVALUIERUNG DES 
EINSATZES DER EUROPAISCHEN STRUKTURFONDS 
IM FREISTAAT SACHSEN 1994-1996 
SACHSEN-ANHALT: ZWISCHENBEWERTUNG DES 
EINSATZES DER EU-STRUKTURFONDS IN 
SACHSEN-ANHALT FUR DEN ZEITRAUM 1994 BIS 
1996 
THURINGEN : DIE EUROPAISCHEN 
STRUKTURFONDS IN THURINGEN : 
ZWISCHENBEWERTUNG DES 
MITTELEINSATZESVON 1994 BIS 1996 
NEUE BUNDESLANDER : DIE EUROPAISCHEN 
STRUKTURFONDS IN DEN NEUEN 
BUNDESLANDERN - ZWISCHENBEWERTUNG DES 
MITTELEINSATZES VON 1994 BIS 1996 (BERLIN/ 
BRANDENBURG/MECKLENBURG-
VORPOMMERN/SACHSEN/SACHSEN-
ANHALT/THURINGEN) 
PO CANARIES : EVALUACION INTERMEDIA 
PROGRAMA OPERATIVO DE CANARIAS (FEDER) 
1994-1999 
OP INCENTIVOS REGIONALES : EVALUACION 
INTERMEDIA DEL PROGRAMA OPERATIVO DE 
INCENTIVOS REGIONALES FEDER (1994-1999) PARA 
LAS REGIONES ESPANOLAS INCLUIDAS EN EL 
OBJETIVO N°l DELOS FONDOS ESTRUCTURALES 
EUROPEOS 
CANTABRIA : ESTUDIO DE EVALUACION 
INTERMEDIA DEL PROGRAMA OPERATIVO DE 
CANTABRIA (FEDER) 1994-1999 
CASTILLA Y LEON : EVALUACION DE LOS 
PROGRAMAS OPERATIVOS DEL FEDER Y DEL FSE 
1994-1999 DE CASTILLO Y LEON 
EVALUACION INTERMEDIA DE LA SUBVENCION 
GLOBAL FEDER-CDTI EN LAS REGIONES 
OBJETIVO 1 (1994-1999) 
ESTUDIO DE EVALUACION INTERMEDIA DEL 
PROGRAMA OPERATIVO DE INFRAESTRUCTURA 
CIENTIFICA DEI PERIODO 1994-1999, EN REGIONES 
ESPANOLAS INCLUIDAS EN EL OBJ. 1, 
COFINANCIADO CON FEDER Y COORDINADO POR 
CICYT 
EVALUACI6N INTERMEDIA DEL MARCO 
COMUNITARIO DE APOYO-REGIONES OBJETIVO 1-
1994-1999-ESPANA Apr-98 
Apr-97 
May-97 
May-97 
Apr-97 
May-97 
May-97 
May-97 
Sep^97 
Jun-97 
May-97 
Jun-97 
Jan-97 
Jun-97 
Jul-97 
Sep-97 OIR 
SEMA GROUP/RIDER-
UCL 
PROGNOS BERLIN 
TROJE BERENTUNG 
HANN, MUNCHEN 
BUSTRO GmbH 
ROSTOCK 
DIW BERLIN 
ISW HALLEFLEIPZIG 
IFO INSTITUT FUR 
WIRTSCHAFTSFOR-
SCHUNG DRESDEN 
IFO INSTITUT FOR 
WIRTSCHAFTSFOR-
SCHUNG DRESDEN 
SERVICIOS OMICRON, 
S.A. 
SERVICIOS OMICRON 
SA 
INFYDE 
(UNIV.VALLADOLID) 
INFYDE/CDTI 
PRICE WATERHOUSE 
QUASAR 
35-
Spain 
Spain 
Spain 
Spain 
Spain 
Spain 
Spain 
Spain 
Spain 
Spain 
Spain 
France 
France 
France 
France CASTILLA Y LEON : EVALUACION DE LOS 
PROGRAMAS OPERATIVOS DEL FEDER Y DEL FSE 
1994-1999 DE CASTILLO Y LEON 
CEUTA : 2e ESTUDIO DE EVALUACION INTERMEDIA 
DEL MARCO DE APOYO COMUNITARIO DE LAS 
REGIONES OBJETIVO N°l PARA CEUTA (PO CEUTA 
1994-1999) 
GALICIA: EVALUACION INTERMEDIA DEL 
PROGRAMA OPERATIVO DE GALICIA 1994-1999 
cofinanciado per el FEDER (PO GALICIA) 
PO MELILLA : EVALUACION INTERMEDIA DEL 
PROGRAMA OPERATIVO FEDER DE MELILLA 1994-
1999 
ASTURIAS : EVALUACION DEL PROGRAMA 
OPERATIVO DEL PRINCIPADO DE ASTURIAS 
(FEDER) 1994-1999 - INFORME PREPARATORIO 
OP LOCAL: ESTUDIO DE EVALUACION 
INTERMEDIA DEL PROGRAMA OPERATIVO LOCAL 
(FEDER) 1994-1999 PARA LAS REGIONES 
ESPANOLAS INCLUIDAS EN EL OBJETIVO N° 1 DE 
LOS FONDOS ESTRUCTURALES EUROPEOS 
OP MEDIO AMBIENTE: EVALUACION INTERMEDIA 
DEL PROGRAMA OPERATIVO DEL MEDIO 
AMBIENTE LOCAL (FEDER) 1994-1999 EN REGIONES 
ESPANOLAS INCLUIDAS EN EL OBJETIVO NO 1 DE 
LOS FONDOS ESTRUCTURALES EUROPEOS 
OP CASTILLA-LA-MANCHA:SEGUNDO INFORME DE 
EVALUACION-PROGRAMA OPERATIVO FEDER DE 
CASTILLA-LA MANCHA 1994-1999 
ANDALUCf A : EVALUACION INTERMEDIA DEL P.O. 
ANDALUCIA 1994-1999 
ANDALUCIA : SUBVENCI6N GLOBAL DE 
ANDALUCfA 1994-1999 : INFORME DE EVALUACION 
INTERMEDIA 
DONANA : EVALUACI6N INTERMEDIA DEL P.O. 
DONANA Ha PHASE (OBJ.l) 
CORSE : EVALUATION INTERMEDIAIRE DOCUP 
OBJECTIF 1 CORSE 1994-1999 
GUYANE : EVALUATION INTERMEDIAIRE DE LA 
MISE EN OEUVRE EN GUYANE DES 
PROGRAMMATIONS REGIONALES (1994-1999) 
COFINANCEES PAR L'UNION EUROPEENNE (DOCUP 
ET REGIS II) - SYNTHESE DU RAPPORT 
NORD-PAS-DE-CALAIS (AVESNES / DOUAI / 
VALENCIENNES): PROGRAMME OBJECTIF 1 (1994-
1999): EVALUATION INTERMEDIAIRE, 
EVALUATION DENSEMBLE DU PROGRAMME, 
ASPECTS SECTORIELS EVALUATION DES MESURES 
GUADELOUPE.EVALUABILITE DU DOCUP 
GAUDELOUPE : RAPPORT D'ETUDE (DOCUP 1994-
1999) Jul-97 
Jun-97 
May-97 
May-97 
Jul-96 
Dec-97 
Aug-97 
Jul-97 
Mar-98 
Mar-98 
May-97 
Jun-97 
Feb-98 
Dec-97 
Feb-96 (UNIV.VALLADOLID) 
GEDESA 
AGRO CONSULTING 
INTERNATIONAL 
ARANDA Y BELTRAN 
S.L. ESTUDIOS Y 
ANALISIS SOCIO 
ECONOMICOS 
UNIVERSIDAD DE 
OVIEDO 
ANDERSON ARTHUR 
ECOTEC 
UNIVERSIDAD DE 
CASTILLA-LA 
MANCHA 
INSTITUTO DE 
DESARROLLO 
REGIONAL, 
FUNDACI6N 
UNIVERSITARIA 
ARENAL GRUPO 
CONSULTOR SL 
INSTITUTO DE 
DESARROLLO 
REGIONAL, 
FUNDACION 
UNIVERSITARIA 
FERE CONSULTANTS 
ERNST & YOUNG 
ACT 
CODE/ATHOS 
36 
Greece 
Greece 
Greece 
Greece 
Greece 
Greece 
Greece 
Greece 
Greece 
Greece 
Greece 
Greece 
Ireland 
Ireland 
Ireland 
Ireland 
Ireland 
Ireland 
Ireland 
Ireland 
Ireland 
Ireland 
Ireland 
Italy 
Netherlands 
Portugal 
Portugal PELOPONNESE : INTERMEDIATE EVALUATION 
OBJECTIVE 1 - OPERATIONAL PROGRAMME 1994-
1999 (a) VOL. A (b) VOL. B 
GRECE OCCIDENTALE (GRECE OUEST): 
INTERMEDIATE EVALUATION OBJECTIVE 1 -
OPERATIONAL PROGRAMME 
CDS P.O. EPIRIUS : INTERMEDIATE EVALUATION 
1994-1999 OBJECTIVE 1 - OP 
CRETE : INTERMEDIATE EVALUATION OBJECTIVE 1 
- OPERATIONAL PROGRAMME 1994-1999 
THESSALIE: INTERMEDIATE EVALUATION 
OBJECTIVE 1 -O.P. 1994-1999 
ILES IONIENNES : INTERMEDIATE EVALUATION 
OBJECTIVE 1 - O.P. 1994-1999 
MACEDOINE CENTRALE + THRACE : 
INTERMEDIATE EVALUATION OBJ. 1 -O.P. 1994-99 
MACEDOINE OCCIDENTALE: INTERMEDIATE 
EVALUATION OBJECTIVE 1 O.P. 1994-1999 
MACfEDOINE CENTRALE: INTERMEDIATE 
EVALUATION OBJECTIVE 1 -O.P. 1994-1999 
PO GRECE CONTINENTALE (=CENTR.): 
INTERMEDIATE EVALUATION O.P. OBJ. 1 - 1994-1999 
EGEE DU SUD : INTERMEDIATE EVALUATION 
OBJECTIVE 1 ILES DE LA MER EGEE SUD -
OPERATIONAL PROGRAMME 1994-1999 
CSFGRECE : OBJ.l - 1994-1999 
EXTERNAL EVALUATION OF THE OPERATIONAL 
PROGRAMME FOR INDUSTRIAL DEVELOPMENT 
1994-1999 : MID-TERM REVIEW 
CSF MID-TERM EVALUATION 
MID TERM EVALUATION OF OPERATIONAL 
PROGRAMME FOR ECONOMIC INFRASTRUCTURE 
MID-TERM EVALUATION OF THE LOCAL, URBAN 
AND RURAL DEVELOPMENT O.P. 
MID-TERM EVALUATION OF THE HUMAN 
RESOURCES DEVELOPMENT-O.P. 1994-1999 
MID TERM EVALUATION OF THE O.P. FOR 
ENVIRONMENTAL SERVICES 1994-1999 
OPERATONAL PROGRAMME FOR TRANSPORT - MID 
TERM EVALUATION 
MID-TERM EVALUATION OF THE OPERATIONAL 
PROGRAMME FOR TOURISM 1994-1999 
OPERATIONAL PROGRAMME FOR AGRICULTURE, 
RURAL DEVELOPMENT AND FORESTRY 1994-1999 
MID-TERM EVALUATION OF OPERATIONAL 
PROGRAMME FOR FISHERIES 1994-1999 
MID-TERM EVALUATION : REGIONAL IMPACT OF 
THE COMMUNITY SUPPORT FRAMEWORK FOR 
IRELAND 1994-1999 
VALUTAZIONE DI MEDIO TERMINE QUADRO 
COMUNITARIO DI SOSTEGNO 1994-99 delle REGIONI 
ITALIANE DELL'OBIETTIVO 1 
FLEVOLAND: ON-GOING EVALUATIE 
DOELSTELLING 1 PROGRAMMA 
AMBIENTE:AVALIACAO INTERCALAR DO 
PROGRAMA AMBIENTE - EVALUATION DU SOUS-
PROGRAMME ENVIRONNEMENT DU CCA II PT 
ALENTEJO:AVALIACAO INTERCALAR DO 
PROGRAMA OPERACIONAL DA REGIAO DO 
ALENTEJO - CCA 1994-1999 May-97 
Jun-97 
Apr-97 
May-97 
May-97 
May-97 
Apr-97 
May-97 
May-97 
May-97 
May-97 
Mar-98 
Feb-97 
Apr-97 
Feb-97 
Jan-97 
Feb-97 
Jan-97 
Feb-97 
Feb-97 
Jan-97 
Feb-97 
Feb-97 
Jul-97 
Jan-97 
Feb-97 
Feb-97 OMAS LTD 
EEO GROUP SA 
PROPTIKI SA. -
METPON 
ASTIKI DIACHIRISI 
S.A. 
YPODOMI LTD 
PROOVADISMA LTD 
INDECO S.A. 
EXANTAS LRD 
EUROTEC LTD 
LDK 
ENVIPLAN 
REMACO 
ERNST & YOUNG 
ESRI 
INDECON 
GOODBODY 
GOODBODY 
ERM 
DKM ECONOMIC 
CONSULTANTS 
FITZPATRICK 
ASSOCIATES 
FITZPATRICK 
ASSOCIATES 
INDECON 
FITZPATRICK 
ASSOCIATES 
NUCLEO DI 
VALUTAZIONE/INEA E 
ISFOL 
RESEARCH VOOR 
BELEID 
QUATERNAIRE 
(PINHO PAULO) 
CESO I&D 
-37-
Portugal 
Portugal 
Portugal 
Portugal 
Portugal 
Portugal 
Portugal 
Portugal 
Portugal 
Portugal 
Portugal 
Portugal 
Portugal 
Portugal 
Portugal 
Portugal 
Portugal 
Portugal AMBIENTE:ESTUDO DE AVALIACAO INTERCALAR 
DA INTERVENCAO OPERACIONAL-RENOVACAO 
URBANA DO PROGRAMA'AMBIENTE E 
REVITALIZACAO URBANA DO QUADRO 
COMUNITARIO DE APOIO 1994-1999 
SOCIAL:SUB-PROGRAMA INTEGRAR AVALIACAO 
INTERCALAR 
LISBOA E VALE DO TEJO:ESTUDO DE AVALIACAO 
INTERCALAR DO PROGRAMA OPERACIONAL DA 
REGIAO DE LISBOA E VALE DO TEJO 
PPDR:ESTUDO DE AVALIACAO INTERCALAR 
PROGRAMA OPERACIONAL PROMOCAO DO 
POTENCIAL DE DESENVOLVIMENTO REGIONAL -
QUADRO COMUNITARIO DE APOIO 1994-1999 
SAUDE:AVALIACAO INTERCALAR DA 
INTER VENCAO OPERACIONAL DA SAUDE 
ENERGIA:AVALIACAO INTERCALAR DO 
PROGRAMA ENERGIA E DO PROJECTO DE GAS 
NATURAL (INTERREG II - CONCLUSAO DAS REDES 
DE ENERGIA) DO QCA II (1994-1999) 
PO CENTRO:ESTUDO DE AVALIACAO INTERCALAR 
DO P. 0. REGIAO CENTRO 
PO NORTE:ESTUDO DE AVALIACAO INTERCALAR 
DO PRONORTE 
PEDIP II: AVALIACAO INTERCALAR DO PEDIP II: 
RELATORIO FINAL INTERCALAR 
PESCA : AVALIACAO INTERCALAR DA 
INTER VENCAO OPERACIONAL DAS PESCAS 
TURISMO: ESTUDO DE AVALIACAO INTERCALAR 
DA INTER VENCAO OPERACIONAL TURISMO E 
PATRIMONII CULTURAL DO PROGRAMA 
MODERNIZACAO DO TECIDO ECONOMICO 
AGRICULTURA : QCA II: ESTUDO DE AVALIACAO 
INTERCALAR DA INTERVENCAO OPERACIONAL 
AGRICULTURA : PROGRAMA MODERNIZACAO DO 
TECIDO ECONOMICO - PERIODO 1994-1996 
PROFAP : PROGRAMA INTEGRADO DE FORMACAO 
PARA A MODERNIZACAO DA ADMINISTRACAO 
PUBLICA - PROFAP 2 : ESTUDO DE AVALIACAO 
INTERCALAR 
ALGARVE : ESTUDO DE AVALIACAO INTERCALAR 
DO PROGRAMA OPERACIONAL DO ALGARVE 
TELECOMUNICACOES : ESTUDO DE AVALIACAO 
DA INTERVENCAO OPERACIONAL DAS 
TELECOMUNICACOES DO PROGRAMA INFRA-
ESTRUTURAS DE APOIO AO DESENVOVIMENTO DO 
QCA 
SUB-PROGRAMME TRANSPORTES: AVALIACAO 
INTERCALAR DA INTERVENCAO OPERACIONAL 
DOS TRANSPORTES DO PROGRAMA INFRA-
ESTRUTURAS DE APOIA AO DESENVOLVIMENTO 
DO QUADRO COMUNITARI DE APOIO 1994-1999, NO 
PERIODO ENTRE 1994 E 1996 
SUB-PROGRAMME COMERCIO E SERVICOS 
:ESTUDO DE AVALIACAO INTERCALAR DA 
INTERVENCAO OPERACIONAL COMERCIO E 
SERVICOS 
ACORES=ESTUDO DE AVALIACAO INTERCALAR DO 
PROGRAMA OPERACIONAL DA REGIAO 
AUTONOMA DOS ACORES DO QUADRO 
COMUNITARIO DE APOIO 1994-1999 Mar-97 
Jan-97 
Jan-97 
Jan-97 
Feb-97 
Jan-97 
Jan-97 
Jan-97 
Mar-97 
Apr-97 
Feb-97 
Jun-97 
Mar-97 
Jan-97 
Mar-97 
Apr-97 
Nov-97 
Feb-97 PARTEX-INSTITUTO 
DE GESTAO E 
ALIENACAO DO 
PATRIMONIO 
HABITACIONAL DO 
ESTADO 
CIES 
CEDRU 
CEETA 
CESO I&D 
IESE 
CEDRU 
QUATERNAIRE 
UNIVERSIDADE 
CATOLICA 
PORTUGUESA 
CESO I&D 
DELOITTE & TOUCHE 
AGRO.GES 
QUATERNAIRE 
CEDRU 
SILICON 
CISED 
CESO I&D 
CESO I&D 
38-
Portugal 
Portugal 
Portugal 
Portugal 
United 
Kingdom 
(IRN) 
United 
Kingdom 
(IRN) 
United 
Kingdom 
(IRN) 
United 
Kingdom 
(IRN) 
United 
Kingdom 
(IRN) 
United 
Kingdom 
(IRN) 
United 
Kingdom 
(IRN) 
United 
Kingdom 
(IRN) 
United 
Kingdom 
(IRN) 
United 
Kingdom 
United 
Kingdom SUB-PROGRAMME EDUCACAO:AVALIACAO 
INTERCALAR DE PRODEP II: EDUCACAO 
CIENCIA E TECNOLOGIA:ESTUDO DE AVALIACAO 
INTERCALAR DO SUB-PROGRAMA CIENCIA E 
TECNOLOGIA 
OP PESSOA: ESTUDO DE AVALIACAO INTERCALAR 
PROGRAMA OPERACIONAL EMPREGO E 
FORMACAO PROFISSIONAL DO QCA II 1994-1999 
(PROGRAMMA PESSOA) 
CSF PORTUGAL-OBJ.l:AVALIACAO INTERCALAR 
DO QUADRO COMUNITARIO DE APOIO II 
NORTHERN IRELAND SINGLE PROGRAMME 1994-
1999 - MID TERM REVIEW - EXTERNAL 
EVALUATION 
MID-TERM REVIEW OF THE ENVIRONMENTAL 
SERVICES AND PROTECTION SUB-PROGRAMME 
MEASURES 4.1.8 AND 4.19. OF THE SUB-
PROGRAMME FOR AGRICULTURE AND RURAL 
DEVELOPMENT : MID TERM EVALUATION 
MID-TERM EVALUATION OF THE NORTHERN 
IRELAND SINGLE PROGRAMME FOR AGRICULTURE 
AND RURAL DEVELOPMENT (SPARD) 
ENERGY SUB-PROGRAMME 1994-1999 : MID-TERM 
REVIEW 
MID TERM EVALUATION OF THE 
TRANSPORTATION SUB-PROGRAMME, 1994-1999 
MID-TERM EVALUATION OF THE INVESTMENT IN 
THE DEVELOPMENT OF PEOPLE SUB-PROGRAMME, 
NORTHERN IRELAND SINGLE PROGRAMME 1995-99 
MID TERM REVIEW OF PHYSICAL & SOCIAL 
ENVIRONMENT PROGRAMME (EU S. F.) 
MID-TERM EVALUATION OF THE TOURISM 
SUBPROGRAMME 1994-1999 
MERSEYSIDE. MID-TERM EVALUATION OF 
MERSEYSIDE OBJECTIVE ONE PROGRAMME 
HIGHLANDS AND ISLANDS - OBJECTIVE 1 
PROGRAMME INTERMEDIATE ASSESSMENT Scp-97 
Feb-98 
Mar-97 
jun-98 
May-97 
Dec-96 
Feb-97 
Mar-97 
Jan-97 
Jan-97 
Dec-96 
Jan-97 
Fcb-97 
Nov-96 
Jun-97 QUATERNAIRE 
IESE/GEOIDEIA 
IESE (INSTITUTO DE 
ESTUDOS SOCIAIS E 
ECONOMICOS 
CISEP 
COLIN STUTT 
CONSULTING 
PIEDA CONSULTANTS 
COOPERS & LYBRAND 
ERM 
ERM 
ERM 
LRDP 
COOPERS & LYBRAND 
NIERC 
PIEDA PLC 
SQW 
Objective 6 report 
Finland 
Sweden SUOMEN TAVOITE 6 - OHJELMAN ARVIOINNIN 
VALIRAPORTTI [FI] 
HALVTIDSUTVARDERING AV SVERIGES NAL-6-
PROGRAM Jan-98 
Nov-97 HELSINGIN 
YLIOPISTON 
TIETOPALVELUT 
OY/ETLATIETO OY/OY 
FINNAGRO 
AB/INT.DEVEL.IRL/PE 
LLERVON 
NORD-REGIO-EPRC 
-39 
ISSN 0254-1475 
COM(98) 782 final 
DOCUMENTS 
EN 04 13 09 16 
Catalogue number : CB-CO-98-786-EN-C 
Office for Official Publications of the European Communities 
L-2985 Luxembourg 
ÙO