Evaluating socio-economic programmes. Volume 5, Transversal evaluation of impacts on the environment, employment and other intervention priorities.
Date: 1999-07-14
Institutions: Directorate-General for Regional and Urban Policy
Subjects: EU programme, Structural Funds, economic and social cohesion, economic development, project evaluation, social development, structural policy

EC Structural Funds 

iluating socio­economic programmes 

Transversal evaluation of impacts on the 

¡bther* 
■¡ties 

MEANS 
Collection 

Volume 5 

opean Commission 

EC Structural Funds 

Evaluating socio-economic programmes 

Transversal  evaluation  of  impacts 
on the  environment,  employment 
and  other  intervention  priorities 

Volume 5 

European Commission 

MEANS is a programme  of  the European Commission. 
It is aimed at improving  Methods  of Evaluating Structural  Policies. 
The MEANS Programme and  Collection  was supervised by 
Phillipe  Goybet and Miguel Angel Benito  Alonso, 
Unit for  the Coordination  of  Evaluation in the Directorate  for  Coordination 
and Evaluation of  Operations of DG XVI 
Coordination-evaluation@fmb.dg16.cec.be 

The realisation of  the Programme  was entrusted  to the 
Centre for European Evaluation Expertise  (C3E) 
under  the direction  of Eric Monnier and Jacques Toulemonde. 
Responsibility  for  the opinions  expressed in this 
book  lies solely with  the authors of  the text: 
C3E  -  13 bis Place Jules Ferry  -  F  -  69006 Lyon  -  France  -  c3e@c3e.fr 

A great deal of additional information on the European Union is  available 
on the Internet. 
It can be accessed through the Europa server (http://europa.eu.int) and 
on the Inforegio Website (http://inforegio.cec.eu.int). 

Cataloguing data can be found at the end of this  publication. 

Luxembourg: Office for Official Publications of the European 
Communities, 1999 

ISBN  92-828-6626-2 

©  European Communities, 1999 
Reproduction is authorized, except for commercial purposes,  provided 
the source is acknowledged. 

PRINTED ON WHITE CHLORINE-FREE PAPER 

Printed in Italy 

Foreword 

Developing  good  evaluation  practices 
has  become  a  priority  area  for 
the 
is  directly  relevant 
Commission  and 
to 
the  effective  use  of  Community 
resources. 
Indeed, the sound financial  management 
principles  which  the  Commission  seeks 
to  promote  via  its  SEM2000  (Sound  and 
Efficient  Management)  Programme  rely, 
specifically,  on  the  regular  evaluation  of 
the Community  activities. 
The  Directorates  General  responsible  for 
structural policy have considerable expe­
rience in this area.  Since  1988 they  have 
made  evaluation  an  integral  part  of  each  stage  in  the  process  of 
Community  Structural  intervention.  Thus,  the  programme  development 
stage  includes  a  prospective  evaluation;  the  implementation  stage  a 
strengthening of the monitoring systemsand, at the end of the intervention, 
evaluation  establishes  what  has  been  achieved  in  terms  of  programme 
impacts. 
However, the  capacity  of  evaluation  to  contribute  to the effectiveness  and 
transparency  of  structural  interventions  rests  ultimately  on the  integrity  of 
the  methods  adopted  to verify the credibility  of  results. 
This  is why, four  years  ago, on the initiative for  the  Directorate  General  for 
Regional  Policy  and  Cohesion  (DG  XVI),  the  MEANS  programme  was 
initiated as a response to the demand for  developing  evaluation  methodo­
logy.  This  also  include  practices  combining  both  general  scientific 
requirements  and  the  specific  requirements  of  Community  interventions 
whilst, at the same time,  remaining  accessible  to  a large audience. 
One of  the  more tangible  results  of  the  MEANS  programme  has  been  the 
publication of this six volume set  by a group  of independent  experts, focu­
sing  on the  principal  elements  of  socio-economic  programme  evaluation. 
A short summary  of this work can be found at the beginning  of the  present 
volume. This collection covers the various methodological  approaches  and 
innovations  that  have  taken  place  in  the  context  of  structural  funds  eva­
luation.  It  will  enable  those  responsible  for  evaluation  to  develop  their 
approach  in  a  flexible  and  independent  manner  and  for  the  various 

programme  partners  to  exercise  greater  control  over  the  evaluation  pro­
cess.  Attending  the  conferences  of  evaluation  specialists  and  "clients" 
convened  in  Brussels  (1995),  Berlin  (1996)  and  Seville  (1998),  I  gained 
some  impression of the  interest that  new evaluation techniques  can gene­
rate.  I would  stress that  the  development  of  evaluation  needs to  progress 
in  tandem  with  efforts  to  incorporate  evaluation  findings  in  the  decision 
making  process. This  implies  evaluation  as a  means  of  establishing  whe­
ther  policies  are equitable,  effective and  relevant to the  identified  needs. 
In satisfying  these  requirements,  evaluation  should  be seen  as  a  valuable 
instrument  affording  both  Member  State  and  Community  authorities  the 
necessary  guarantees  that  Community  funds  are  being  deployed  in  the 
best  interests of the EU citizen. This is a cost worth  paying when the goals 
of  Economic  and  Social  Cohesion  and  the  European  Union  enlargement 
are at stake. 

Finally,  I wish  to  extend  my  thanks  to  the  members  of  the  Committee  of 
Independent  Experts  established  at  the  launch  of  the  programme.  Their 
expertise and standing have contributed  greatly to the quality and relevan­
ce  of  the  MEANS  programme.  I therefore  thank:  E.  Chelimsky  (USA),  ex-
President  of  the  "American  Evaluation  Association";  K.  Kennedy  (IRL), 
Director  of  the  "Economic  and  Social  Research  Institute"  (ESRI);  J.  R. 
Cuadrado  Roura  (SP),  President  of  the  "European  Regional  Science 
Association"; C. Seibel (F), Director of the "Statistiques démographiques et 
sociales de l'Institut  National de la Statistique et des Etudes  Economiques" 
(INSEE);  L.  Senn  (I),  Director  of  the  Department  of  Economic  Regional  of 
the  University  Bocconi;  E. Stern  (U.K.), Director of the  unit of evaluation  of 
the  Tavistock  Institute;  A.  Sorber  (t)  (NL),  Head  of  the  "Policy  Analysis 
Department of the Ministry of  Finance"; L. I. Strom (SV), Statens  Institut för 
Regionalforsking  (SIR);  L.  Tsoukalis  (GR.),  professor  at  the  University  of 
Athens;  H. Wollmann  (D), professor  at the  University  of  Humboldt  -  Berlin 
and founder  member  of the "European  Evaluation  Society". 

E. Landaburu 

The  MEANS  Collection: 
"Evaluating socio-economic programmes" 

The  evaluation  of  European  Union  structural  interventions  has 
entered into a phase of maturity, attested  by the extension  of  work 
carried out by both the Member States and the Commission. Regu­
lations based on decisions  by the Council  have largely  contributed 
to  the  development  of  evaluation  practice  and  have  stimulated 
methodological research. 

Over the past ten years, the Commission has striven to promote and 
improve methods for evaluating Structural Funds. The main metho­
dological conclusions have been debated and disseminated as part 
of  the  MEANS  programme  initiated  by the  Commission  in 1991. 
Achievements  to date  provide  enough  material for a complete  and 
accessible  publication  intended  for  a broad  public:  a six  volume 
Collection covering all aspects of the evaluation of  socio-economic 
programmes. These Volumes are written  with  a perspective that  is 
broader  than the current  administrative  framework,  for  this  is  likely 
to evolve more rapidly than professional standards. 

In general, the Collection Is aimed at a readership of  non-specialists 
seeking  essential  information on the  evaluation  of  socio-economic 
development  programmes. These programmes have multiple objec­
tives, a partnership dimension and a multi-sectoral content - all cha­
racteristics which call for suitable evaluation methods and solutions 
not  yet  provided  by  the  international  technical  literature.  The 
complete Collection consists of 6 volumes  designed to  be comple­
mentary and to provide essential information and  recommendations 
for professionals wanting to independently develop their own evalua­
tion practices.  Each volume can, however,  be read and  understood 
without having to refer to the other volumes in the Collection. 

The  first  Volume  in  the  MEANS  Collection  entitled  "Evaluation 
design and management"  serves as an introduction both to the Col­
lection  and  to  the  evaluation  of  socio-economic  programmes.  It 
enables  readers to understand the implications of evaluation and to 
know  what  they  can  and  cannot  expect  from  an  evaluation  at a 
given time in a particular context.  It covers  all the elements  needed 
to steer an evaluation project from A to Z: sharing of responsibilities, 
defining  of  the  aims  and  method  of  an evaluation,  establishing  a 

schedule  and  budget,  assessing  the  quality  of  the work  performed 

and, finally, managing the use of conclusions. 

The second Volume focuses on the  "Selection  and use of  indicators 

for monitoring  and evaluation".  It deals with the subject  of  indicators 

used for monitoring  and evaluation. It shows  how indicators can  be 

used  to  support  policy  decisions,  improve  management  and 

measure  the  overall  performance  of  programmes.  The  volume 

proposes a methodological framework  and a practical  guide for  the 

use of indicators in these different situations. 

The third Volume presents a selection of  "Principal  evaluation  tech 

niques  and  tools"  applicable  to  the  evaluation  of  socio-economic 

programmes.  These  tools  are taken  from  relevant  scientific  disci­

plines:  economics,  sociology,  management,  geography,  etc.  Each 

one of the  23 tools  Is described  in the form of an information  sheet 

featuring a general description, the principles of  its implementation, 

a list of its strengths and weaknesses for the evaluation of Structural 

Funds, an example of its application and a list of references. 

A number  of methods adapted to the evaluation  of  socio-economic 

programmes  are the  object  of  a  more  detailed  description  in  the 

fourth  Volume  entitled  "Technical  solutions  for  evaluation  within  a 

partnership  framework".  Several  pilot  evaluations  were  implemen­

ted  within the  MEANS  programme  to design  and test  the  following 

four  innovations,  the  application  of  which  is  described  in  detail  in 

this Volume: impact  mapping, cross-impact  matrix, score cards and 

multi-criteria analysis. 

The fifth Volume deals with  "Transversal evaluation of impacts  on the 

environment,  employment  and  other  intervention  priorities"  of  the 

European  Union  and  Member  States.  Among  the  questions 

addressed are impacts  in terms of equal opportunities between  men 

and women and the competitiveness of SMEs. The Volume provides 

the elements required for an analysis of those impacts which are not 

necessarily  part of the explicit objectives of all the actions evaluated. 

Finally,  the  last  Volume  is  a  "Glossary  of  concepts  and 

technical 

terms".  The  compilation  of  the  Collection  has  necessitated  the 

rigorous  standardisation  and  definition  of  the  terms  used.  Each  of 

the  first  five  volumes  has  an  index  of  terms  which,  along  with 

hundreds of definitions, are brought together  in the glossary. 

Contents 

Introduction 

I  Technical considerations relating to the assessment 

of thematic impacts 

II  Thematic analysis of impacts on political priorities 

Evaluating the environmental impact of structural interventions 
Evaluating the impact of structural  interventions 
on the competitiveness of Small and Medium-Sized Enterprises 
Evaluating the impact of structural interventions on employment 
Evaluating the impact of structural interventions 
on equal opportunities 

III  Thematic evaluations of domains of interventions 

Evaluating impacts arising from interventions 
in the domain of Research and Technological Development 
Evaluating impacts arising from interventions 
in the domains of transport, telecommunications and tourism 

IV  Framework for performing thematic evaluations 

A step-by-step approach for thematic analysis 
Cross programme thematic evaluation 

Conclusion 

Appendix  1  - Evaluation tools and corresponding themes 
Appendix 2 - Extracts from a Terms of Reference 
for a programme evaluation incorporating thematic elements 
Appendix 3 - An example of terms of Reference 
for a transnational thematic evaluation 
Appendix 4 - List of boxes 

Index 

7 

13 

27 

29 

59 
77 

107 

129 

131 

151 

167 

169 
179 

189 

195 

197 

199 
209 

213 

Introduction 

Economic  and  social  cohesion  is  the  global  objective  of  programmes 
financed by the Structural Funds. The objective has three elements:  catching 
up of lesser-developed regions, social and  economic  reconversion of urban 
and  rural areas and  reintegration  of social groups  in difficulty.  Interventions 
implemented by way of these main objectives must try to satisfy four political 
priorities common to the European Union and the Member States (see Box 1). 

These priorities  are  most  often  recalled  in  the preamble  of  programmes. 
However, axes entirely devoted  to them within a programme  are not neces 
sarily found. A  sub-programme,  an environmental  priority,  or even  specific 
measures  concerning  the  competitiveness  of  SMEs are sometimes  to  be 
found.  But  countries  or regions have rarely programmed  specific  measures 
focusing on equal  opportunities. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Box  1 -  Priorities, domains  and  themes 

The political priorities defined for the 1994-99  programming  period of the 
Structural Funds comprise: 
•  the fight against unemployment and the creation of new jobs; 
•  increasing the competitiveness  of  productive structures, more particu­
larly SMEs; 
•  protecting the environment; 
•  and finally, reducing inequality between men and women. 
The notion of  "domain" refers to specific  domains in which  interventions 
take  place and, therefore, outputs  are programmed. Thus, for  example, 
impacts  in terms  of  "competitiveness"  are the consequence of  interven­
tions  in the  domains  of  research  and  development,  transport,  telecom­
munications, etc. 
The notion of "theme" is used in this book to cover both political  priorities 
and  domains  of  intervention.  A  thematic  evaluation  might  therefore 
examine if a priority  has been respected in all of the domains of interven­
tion. On the other hand, another thematic  evaluation might  be  interested 
in a given domain and examine all of the possible impacts (notably for the 
four priorities mentioned above). 

The four  priorities,  which  differ  in  importance  according  to  the  context,  are 
transversal  in relation to the programmes  financed  by the  Structural  Funds. 
All managers and operators are supposed to apply them in the implementa­
tion of  measures, in project  selection, in the  definition of target  publics, etc. 

Most  often, the  evaluation  of  a programme  favours  an analysis  of  impacts, 
which  are  explicitly  described  by  the  principal  objectives  of  priorities  and 
measures. It tends to  neglect  transversal  priorities, which  concern, in a way 
that  is  less  precise,  the  entirety  of  programme  measures.  Progress  is 
therefore necessary  if reports are to contain  reliable thematic  conclusions  at 
a reasonable cost. These conclusions  must  be presented  in a  standardised 
and sufficiently  synthesised form in order that the Commission Services  can 
themselves use them to produce a European-level synthesis. 

Moreover,  in accordance  with  the  principle  of  subsidiarity, the  Commission 
has to periodically  evaluate the different  modes of intervention applied in the 
Member  States  in order to  identify  best  practices  and  organise the transfer 

Introduction 

of know­how in Europe. This is done through evaluations performed at the 
European level, with a domain of intervention (transport infrastructure, 
support for research activities, professional training, etc.) being subject to an 
in­depth analysis. These domains of intervention are generally found, to 
greater or lesser degree depending on the case, in the majority of regional 
programmes. 

■ A Volume for Community and programme level 

commissioners of evaluations 

This Volume is destined for those who, at the programme or European Com­
mission level, want to undertake a transversal examination of impacts cor­
responding with a priority or domain of intervention. 

These "thematic" evaluations are designed differently according to whether 
they are performed at the programme level itself or the European level. At the 
programme level, thematic evaluation covers both the direct impacts of 
measures devoted to a given priority (a measure in favour of the environ­
ment, for example) and the indirect consequences of other measures (envi­
ronmental consequences of road building, for example). 

In relation to a given theme, this first type of evaluation is a synthesis of the 
entirety of possible impacts, both direct and indirect, and generally verifies 
the overall coherence (and the mutual reinforcement) of the different 
measures contained in a programme. The conclusions of this type of evalua­
tion are often placed in a series of chapters dedicated to each of the themes 
at the end of an evaluation report. 

The second type of thematic evaluation, for which the Commission Services 
are responsible, is to produce conclusions relevant for a number of regions 
or countries of the European Union. These evaluations commence with a 
"trans­programme "analysis of the impacts of similar interventions in order 
to identify the most efficient, and to distinguish, in the case of a successful 
intervention, between what is proper to the economic, administrative or ins­
titutional context and lessons that can be applied to other territorial, regional 
or national units. This type of evaluation highlights the relative efficiency of 
interventions and plays a central role in the identification of best practices. 

This Volume contains advice and solutions targeted at public officials who 
want to include these themes in their terms of reference and evaluation 

Transversal evaluation of impacts on intervention priorities 

teams wanting to design a work programme (see examples of terms of 
reference in the appendix). 

In order to take on the challenge of obtaining reliable thematic conclusions at 
a reasonable cost, in the following pages analytical grids and methodologi­
cal solutions can be found for: 
• identifying potential impacts of a programme in relation to a theme ; 

• transversally taking into account impacts within the same programme or 
within group regional or national programmes; 

• identifying and defining suitable indicators for monitoring and evaluating 
these impacts; 

• collecting and processing appropriate information; 

• and finally, presenting a synthesis of the conclusions of an evaluation. 

■ T he structure of t he Volume 

This Volume does not claim to deal with specialist concepts and methods 
relating to the themes tackled. It could easily be imagined, for example, that 
experts in employment policy would consider that the subject has not been 
sufficiently developed, or that environmental specialists would judge that the 
analysis of ecological impacts is highly superficial. Indeed, the raison d'être 
of this volume arises from the unrealistic demands of experts of different 
themes concerned by structural programmes. 

The challenge for programme evaluation is to analyse a programme's merits 
in respect to its own objectives and in relation to the four political priorities 
common to the European Union and the Member States. For reasons of time 
and budget, an approach which simplifies each theme has be taken. In order 
to do this, concepts and solutions specific to each area of expertise have to 
be selected, and a trade­off has to be made between reliability and feasibility. 

The text is organised in four parts. General technical considerations appli­
cable to all thematic evaluations are brought together in the first part of the 
Volume, so as to highlight the points common to all themes. It should be 
noted that the development of certain points is limited and these points are 
more fully developed in the first and third Volumes of the Collection. Volume 
1 of the MEANS Collection describes the entire evaluation procedure and 
Volume 3 presents a set of tools for the collection and processing of informa­
tion. 

Introduction 

The second part of the book is given over to the principle political priorities: 
environment,  competitiveness  of  SMEs, employment  and equal opportuni­
ties.  Methodological  requirements  vary from  one  priority  to  another. Thus, 
the taking  of  environmental  impacts  into  account  in an evaluation  is  based 
on methods adapted to the estimation of impacts that are diffuse and hete­
rogeneous.  On the  other  hand, employment  impacts  are  by  nature  more 
homogeneous  since they  relate  to  people and  enterprises.  For  example, a 
survey  of  direct  addressees  is suitable for  studying  employment  impacts, 
but not environmental impacts. 

The  creation  and  maintenance  of  employment  has, since the  start  of  the 
1980s,  been  one  of  the  preoccupations  of  the  European  Union. This  is 
concretely  demonstrated  by the  numerous  measures  devoted  to  employ­
ment  within  programmes  financed  by the  Structural  Funds.  Moreover,  a 
large amount of conceptual and technical work has been undertaken on this 
theme by  research centres, allowing a substantial  quantity  of  know-how  to 
be  accumulated.  Within  the  framework  of  the  MEANS  Programme, 
numerous  workshops  and  reports  have  been  given  over  to  the  theme  of 
employment. This Volume emphasises the selection of realistic and uncom­
plicated solutions within the framework of thematic  evaluations and cannot 
claim to be situated at the same level of sophistication  as an actual evalua­
tion of employment policy 

A theme such as equality of opportunity between men and women has given 
rise to  important  conceptual  advances, for  example, the  notion  of  gender 
mainstreaming (Box 2). 

Box 2 -  Gender  mainstreaming: a  definition 

Gender  mainstreaming  is the  systematic  inclusion  of the  priorities  and 
needs  specific  to  women  and  to  men  in  all  Community  policies  and 
actions, with the unique aim of achieving equality. This is to be achieved 
by actively and visibly taking account, at the planning stage, of the effect 
of interventions on the specific situations of women and men during their 
implementation,  follow-up  and  evaluation.  Gender  mainstreaming 
requires the development of measures aimed at adapting the way society 
is  organised  to  ensure  the  more  equal  distribution  of  roles  between 
women and men. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

However,  evaluations that tackle this theme are still few,  reflecting the  fact 
that  this  preoccupation  is relatively  new  for the  majority  of  countries. This 
type  of  thematic  evaluation  is  more  often  ex  ante  than  ex  post.  The 
examples  of  evaluations  presented  are therefore  aimed  at  prospective 
rather  than  retrospective  analysis.  In order  to  prepare  this  chapter  it  has 
therefore been necessary to go beyond existing practices. 

The  third  part  of  the  text  deals  with  thematic  evaluations  of  domains  of 
intervention. Four domains have been selected not for any strategic reason, 
but  more  prosaically  because  they  have  recently  been  the  object  of 
European level thematic  evaluations within the framework  of the  Structural 
Funds. The domains  of research  and  development, transport,  telecommu­
nications and tourism are examined. The first of these domains is tackled in 
some  detail, while the  others  are  more  briefly  examined. The  Volume  2 of 
the  Collection  provides  a  more  complete  view  of  all the  domains  of  inter­
vention from the  point  of  view  of  indicators  for  monitoring  and evaluation. 

Part  IV of the Volume focuses  on  methodological  issues which  cut  across 
thematic  boundaries  and  which  are  specific  to  evaluations  conducted  at 
the  programme  or  EU  level.  Firstly,  a standardised  procedure  is  provided 
which  is designed to structure the implementation of programme  level eva­
luations in such a way as to ensure that conclusions can be aggregated and 
synthesised  at  a  supra-national  level.  Secondly,  the  basic  principles  of 
transnational  thematic  evaluation, organisational  issues  associated  with 
this  type  of  evaluation  and  its  use  in  highlighting  best  practice  are 
examined.  Finally,  a  possible  future  direction  for  transnational  evaluation 
practice is mapped out. 

I 
Technical considerations relating 
to the assessment of thematic impacts 

The evaluation of the impacts  of Structural  Fund interventions  is  structured 
around a number of basic, but very important,  tasks which need to be under 
taken, whether the evaluation is focused on a theme, a political  priority  or an 
entire programme.  Some of these key tasks include:  the clarifying of explicit 
and  implicit  objectives,  selecting  indicators,  gathering  data, and  estimating 
net impacts.  The objective  of this chapter is to present  how  these four tasks 
can be approached.  This presentation  is not theme specific  and is illustrated 
by  examples  drawn  from  the  individual  themes  presented  further  in  this 
book.  The issues tackled  are, of  course,  applicable  to  themes  that  are not 
presented  here,  but  which  may  be  taken  into  consideration  in  or  be  the 
object of future evaluations. 

Technical considerations 

■ Clarifying the explicit and implicit objectives 

of t h e me focused interventions 

An early step in an evaluation is the clarification of a programme's objectives. 
During the ex ante evaluation of a programme it is important to clarify not 
only its central or global objectives, but also those objectives related to 
specific priorities, such as those mentioned in this book. In intermediate eva­
luations, additional objectives that might have emerged in the course of the 
programme should be clarified, along with the contribution of projects to 
attaining objectives. 

Even if there is a single overall priority objective (e.g. the promotion of envi­
ronmentally sustainable development), attaining that objective is likely to 
depend on attaining specific and intermediate objectives that may be 
expressed differently. It is therefore necessary to draw up a clear hierarchy of 
the objectives and the relationships between them. 

In order to draw up the hierarchy of objectives an "objectives tree" can be 
constructed. This tool allows the degree of internal coherence between the 
measures of a programme in relation to overall priority objectives to be 
examined (both vertically ­ between different levels of objectives ­ and hori­
zontally ­ between the objectives of different components of the programme 
which are at the same level). It also facilitates the examination of the external 
coherence of a programme with other public policies. For example, Structu­
ral Fund intervention in favour of SMEs might target the development of 
innovatory capacity, increased networking and technology transfer, while 
many Member states are seeking to create a general business environment 
which is conducive to the development of small firms. An examination of 
external coherence would focus on whether the objectives of these interven­
tions were compatible or not. 

While the hierarchy of objectives is usually constructed on the basis of 
official documents, in the case of thematic evaluations it can also be useful 
to incorporate implicit objectives into the analysis. An objectives tree 
constructed in this way can therefore be considered as a thematic objectives 
tree. Although the incorporation of implicit objectives into an analysis allows 
a better understanding of the logic of a programme, it is important to be 
careful when using implicit objectives as evaluation criteria. When the work 
of programme managers and operators has to be judged, this should only be 
done in relation to explicit objectives of an intervention. 

Transversal evaluation of impacts on intervention priorities 

The principle of a thematic objectives tree is illustrated in Box 3. Actions 
financed by the Structural Funds are designed to attain the explicit global 
objective of social and economic cohesion. A hierarchy of the explicit 
objectives linking actions to the global objective can be constructed in the 
form of an objectives tree. At the same time the promotion of equality between 
men and women is a priority concern for Structural Fund programmes, but 
objectives related to this political priority are largely implicit. While global 
objectives are stated in official texts, working documents and studies, this is 
usually not the case for specific and intermediate objectives. These objectives 
are often constructed implicitly by the various actors of the programme and 
serve to regulate and direct their operational decisions. The hierarchy of 
implicit objectives is presented in a thematic objectives tree which allows the 
relations between actions and the global objectives of a political priority to be 
structured. 

It should be remembered that the objectives trees presented in the following 
chapters are only illustrative, objectives trees should be constructed by the 
evaluation team in consultation with the steering committee of the evalua­
tion on the basis of the specific programme to be evaluated. Moreover, an 

Box 3 ­ Using objectives trees for clarifying implicit and explicit 

objectives related to a given theme 

Global objectives 

Intermediate and specitic objectives 

Actions 

■* 

Objectives tree 
(of a programme) 

Social and 
economic 
cohesion 

Balanced 
development 
for men 
and women 

Thematic objectives 
tree 

Explicit objectives 

Implicit objectives 

Technical considerations 

objectives tree is neither exhaustive nor definitive and it will have to be 
modified over time to take into account the evolution of political priorities 
and the socio­economic context. 

For more information on the clarification of objectives using an objectives 
tree, see Volumes 1 and 4 of the MEANS Collection. 

■ Selecting indicators for thematic evaluations 

Volume 2 of the MEANS Collection is devoted to the subject of indicators for 
monitoring and evaluating Structural Fund interventions. Only essential 
elements are developed hereafter. 

In order to assess the impacts related to a given theme, indicators need to 
be defined which provide an estimation of the phenomenon to be measured. 
The selection of several complementary indicators may be necessary to 
estimate an impact that is considered as important. For example, in order to 
estimate air quality, a number of indicators relating to different types of 
atmospheric emissions would have to be employed. 

Given the multiplicity of objectives, the number of components of a 
programme (sub­programmes, measures, and actions) and the need to take 
into account transversal political priorities, it is difficult to imagine an opera­
tional system of indicators which would include indicators for all compo­
nents at all levels. It is therefore necessary to select indicators that meet the 
most important needs of managers and officials and can be collected at a 
reasonable cost. In the case of thematic evaluations, since the role of indica­
tors is to allow an intervention to be judged in relation to the political priori­
ties to be taken into account, they should be related to the most important 
impacts of the theme concerned. The indicators retained should be reliable, 
giving the same indication irrespective of the person quantifying them and 
providing up­to­date information on the effects being measured. 

The construction of a system of indicators for performing thematic evalua­
tions requires, first of all, that the relevant evaluatory questions are establi­
shed. These questions will usually encompass the following issues: "To what 
extent have changes taken place?", "To what extent are these changes attri­
butable to the programme?" "Are the results and impacts that have occurred 
satisfactory?" Secondly, those evaluation criteria by which an intervention 
will be judged are agreed. Thirdly, that context indicators, which provide 
information on the reference situation in which an intervention occurs, and 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

programme  indicators, which  focus  on  the  direct  and  indirect  effects  of  an 
intervention  its  addressees,  are  defined  in  the  light  of  the  evaluatory 
questions and criteria selected. 

The  in-depth  analysis  of  impacts  also  requires  that  exogenous  factors  are 
taken  into  account,  i.e.  those  factors  over  which  the  programme  has  no 
control  and  which  have  an  influence  on  impacts.  It  is therefore  useful  to 
identify the main effects of confounding factors by means of indicators. 

A  system  of  indicators  is designed  to  allow  the tracking  of  an  intervention 
from the resources employed during  its implementation  to  its impacts as far 
as possible along the chain of causality. In order to do this indicators furnish 
information on the human and financial resources employed in implementing 
an  intervention  (resource  indicators),  the  material  and  immaterial  outputs 
produced  (output  indicators),  the  immediate  impacts  on  the  addressees 
resulting  from  an  intervention  (result  indicators),  and  the  subsequent 
impacts on direct and indirect addressees  (impact  indicators). 

When  referring to  different  types of  indicators  and the concepts  associated 
with them, it is highly desirable that a standardised terminology  be used. It is 
therefore strongly  recommended  that the  definitions figuring  in Volume 2 of 
the MEANS Collection are systematically employed. 

Example:  Indicators  for  measuring  the  effects  of  interventions  for  improving 
water quality and  resources 

In order to improve and safeguard the quality of local water supplies, a programme of 
investments  is  planned  with  the  objective  of  increasing  the  amount  of  waste  water 
treated and reducing the amount  of treated water  lost through  leakage. At the  output 
level indicators would  be chosen that measure the physical achievements of the inter­
ventions, for example: kilometres  of  mains supply  pipes  modernised, the capacity  of 
new waste water treatment  plants constructed and the capacity of waste water  plants 
to  undertake  primary  /  secondary  treatment.  An  indicator  at  the  result  level  is  the 
number of households or businesses  connected to the plant. At the impact  level indi­
cators  would  include  the  volume  of  water  consumed  via  the  modernised  system, 
number  of days of  when  supply  is restricted  and volume  of  water  lost through  leaks. 
Impact indicators also reflect improvements  in the quality and quantity of  underground 
water  resources and river quality, for example: withdrawal rate from sources, withdra­
wal  rate as a  %  of  annual  availability,  %  of  waste  water  samples  complying  with  EC 
directives, % of length of rivers monitored subject to slight or moderate pollution. 

It  is  necessary  to  take  the  timescale  of  effects  into  account,  otherwise 
judging  them  will  be  distorted  by  incomplete  information. This will  result  in 

Technical considerations 

the underestimation of expected effects that have yet to occur and / or the 
overestimation of temporary effects that have not come to an end. 

Example: The timing of employment effects arising from the promotion of 
research collaboration 
A project to improve collaboration between SMEs and public research bodies would 
produce different employment effects at different time scales. In the short­term, 
employment effects would be mainly temporary with employment being maintained or 
created during implementation (e.g. advisors, consultants, etc.). In the more medium­
term the recruitment of technicians, researchers and scientists to take part In the 
project would lead to permanent employment of individuals associated with the per­
forming of research. In a longer­term perspective employment Impacts will occur 
within the SMEs which are involved in research projects as the resulting innovations 
are marketed and levels of production increase. 

Four basic features of the indicators should not be forgotten in the interpre­
tation of data. First, the information provided might require additional pro­
cessing, since it is not always usable or communicable in its initial state. 
Secondly, the changes measured may not be entirely caused by the inter­
vention being evaluated and in any event, the analysis of an indicator is not 
enough to establish relations of cause and effect. 

Thirdly, when comparing information provided by an indicator, it is important 
to ensure that the comparison is relevant (i.e. that the context of the 
reference situation is the same). Finally, indicators should also provide infor­
mation on confounding factors which influence the evolution of the reference 
situation (e.g. when evaluating the impact of a job creation programme on 
unemployment, changes in the size of the working population should be 
taken into account). 

■ Gathering data about a t h e me 
When the evaluation of a theme is performed within the framework of a 
programme, it is recommended that this is based on documentary research 
and existing data (e.g. monitoring data, statistical surveys and existing eva­
luation reports) supplemented by minimal case studies, interviews and focus 
groups (see the "step by step", approach presented in Part IV). 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Box 4  - 

In-depth  analysis  of impacts: appropriate  tools  and 
constructing  an  evaluation  method 

When  an  in-depth  analysis  of  impacts  is  required,  it  is  necessary  to 
construct  an  evaluation  method  composed  of  a  number  of  steps  (a 
"technical chain") which  includes  a combination of  interdependent  tools 
for primary data collection and analysis. The "step-by-step"  approach for 
thematic evaluations, proposed in Part IV of this Volume, while remaining 
faithful to the "technical chain" concept favours the use of a limited range 
of tools that facilitate a "quick and dirty" approach to primary data collec­
tion.  Some  tools,  more  often  associated  with  in-depth  analysis,  are, 
however,  better  suited  to  certain  types  of  thematic  evaluations  than 
others. It is with the objective of highlighting the suitability of certain eva­
luation  tools  to  different  themes  that  tools  for  in-depth  analysis  are 
presented in each of the chapters devoted to specific themes. 

Volume  3 of the  MEANS  Collection  provides  a full  discussion  about  the 
choice and use of evaluation tools within the framework of the evaluation 
of Structural Fund interventions. 

The  monitoring  of  the  implementation  of  interventions  makes  most  use  of 
resource,  output  and  result  indicators.  These  are  largely  quantified  by 
programme  managers  and their  services  which  issue  payments  and  create 
or collect information on: number of outputs and / or level of physical output; 
expenditure;  descriptions  of  the  projects;  reports  on  activities  of  manage­
ment  services.  Several  services  are  usually  involved  in this  activity,  so  it  is 
generally  necessary  to  centralise  these  data  in advance,  concentrating  on 
the most  useful data for  managing and evaluating interventions.  If this  basic 
information is to be suitable and complete, the collection of monitoring  infor­
mation should be planned from the outset, bearing evaluation needs in mind. 

It  is  not  always  possible,  however,  to  supply  data  for  indicators  through 
existing  information  systems. This  is, in fact,  the  case  for  the  majority  of 
indicators  focusing  on the short  or  medium term  impacts  of  structural  inter­
ventions. 

The use of statistical time series data, while relatively cheap and quick,  has a 
number  of  major  drawbacks  which  limit  their  use  in evaluation: the  time-lag 
inherent in the production of data for standard statistical surveys; the absence 
of a direct  link between an evaluation criterion and available statistics, the dif-

Technical  considerations 

ference between the geographical area covered by an intervention and the ter­
ritories  or groups of territories  used to  compile  statistics. Regional  statistical 
observatories have also been created but, even when statistics are presented 
at  the  level  of  territorial  sub-divisions  of  the  Member  States, these  sub­
divisions do not always correspond with zones eligible for Structural Funds. In 
these conditions, the trends highlighted by the statistics may not allow direct 
conclusions to be drawn on the effects of a particular programme. 

Statistical  information  systems  are, however,  being  developed  to  remedy 
this  shortcoming.  Eurostat  is co-ordinating  the  collection  of  data  used  to 
provide information for indicators of the activity of 35 industrial sectors in all 
the areas eligible for support to promote development, structural adjustment 
and  industrial  reconversion. These data are available for the  period  1985-
1993 and will be available for 1994-1999. 

The  different  information  sources  have various  strengths  and  weaknesses 
that are summarised in Box 5. 

Box 5 -  Typical strengths and weaknesses  of  sources 

of existing  information 

Strengths 

Weaknesses 

Monitoring  systems 

Produced at frequent  intervals 
Relatively  up to date 
Generally  readily  available 
Significance  Is usually  apparent 
Relatively  low  cost 

Permanent  statistical 
surveys 

Low  cost 
Reliability 
Richness 
Comparability 
Ease with which  information  Is obtained 

Other research and 
evaluations 

Low  cost 
Data already  analysed 

Sometimes  unreliable  statements 
Rarely available for all 
measures /  actions 
Data on physical  outputs  and  results 
of various components  not  easily 
aggregated 
Not easily  comparable 

Often  not directly  linked to  evaluation 
criteria 
Geographical  level not always suitable 
Information  not up to  date 

Information not up to  date 
ften too  restricted to  provide a reliable 
answer or one that  can be generalised 

Given their strengths and weaknesses, the sources of information are com­
plementary  rather  than  alternative.  For  example,  permanent  statistical 
surveys  provide  data  for  comparison  (e.g.  evolution  of  employment  at 
regional  and  national  level), and  indications  of the  evolution  of the  context 

Transversal evaluation of impacts on intervention priorities 

(e.g. economic situation, evolution of workforce, growth and decline of 
certain sectors) which allow for a more in­depth analysis of impact indica­
tors, and in some cases, relevant impact indicators at a reasonable cost. On 
the other hand, evaluations and existing research provide elements for com­
parison (effects of similar interventions), elements for estimating deadweight 
or displacement coefficients allowing the calculation of net impacts, and 
hypotheses to be postulated on ways in which effects are produced. 

■ Estimating net impacts 

The need to estimate the net impact of an intervention applies to all evalua­
tion themes, however, Indicators are only capable of providing information on 
the gross impact of an intervention (i.e. the observed change in the reference 
situation or the situation of direct and indirect addressees). The gross impact 
of an intervention is not a proxy for its net impact and, as a consequence, the 
latter has to be estimated for the evaluation to provide accurate information. 
The estimation of gross impacts is therefore only an initial step in the evalua­
tion of impacts that can be genuinely attributed to an intervention. While the 
factors that have to be taken into account when calculating net impacts 
depend to some extent on the theme of the evaluation and the indicators 
employed, the following convention represents a minimal definition: 

Net impacts = 
Gross impacts ­ Deadweight ­ Displacement (Substitution)* 

('Unintended or unwanted) 

Deadweight is the term given to the change that would have been experien­
ced even in the absence of an intervention. Assessing displacement is 
relevant for interventions targeted at geographical zones. On the other hand, 
assessing substitution is relevant in the case of social group­targeted inter­
ventions. Only when displacement and substitution are not objectives of an 
intervention are they subtracted from gross impacts. 

In order to assess the significance of deadweight, it is necessary to construct 
the counterfactual situation or policy­off scenario based on detailed projec­
tions of the likely evolution of key variables if no intervention had occurred. 
The reconstruction of the counterfactual situation is of critical importance for 
the accurate assessment of the impacts of structural interventions. It is, 
however, a difficult task that becomes even more complex at programme 

Technical  considerations 

level  when  a variety  of  inter-related  interventions  has to  be considered. In 
practice, there are very few occasions when the counterfactual situation can 
be reconstructed  with  certainty;  an open, reasoned and  informed  effort  to 
make a best estimate is frequently the best that can be done. 

An  illustration  is  given  in  Box  6  of  the  principle  of  deadweight.  In  this 
example the change in the number of jobs during the period of an interven­
tion  (from  baseline to  the  moment  of the  evaluation)  is j0n less jo, in  other 
words  the  gross  employment  impact.  During  this  period,  however,  it  is 
estimated  that  in the  absence  of  the  intervention, the  policy-off  situation, 
employment would have increased to j0ff- The impact of the intervention net 
of displacement  is in the difference between j0n and j0ff. 

Box 6 - Estimating impacts net of deadweight 

1  on 

J  off 

Pollcy-on 

</ 

A 
, 

A 

;/ 

^^-^^ 

Policy-off 

Impact 
net of 
deadweight 

y 
k 

Deadweight 

Change of 
situation 
(gross 
impact) 

γ 
τ 

y 

Baseline 

Evaluation 

Τ 

When  displacement  occurs,  impacts  on  addressees  are accompanied  by 
effects on non-addressees within the programme area. An example of this is 
when  a  reduction  in  traffic-related  atmospheric  emissions  due  to  the 
construction  of  a new  road  leads to the  unintended  displacement  of  road 
traffic and exhaust emissions to another part of the programme area. In this 
instance  the  effect  is  unwanted  and  displacement  should  be  subtracted 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

from  gross  impacts.  If,  on  the  other  hand, displacement  is  sought  as  an 
impact  of  an intervention  (e.g. the  movement  of  small-scale  industrial  firms 
causing  noise and atmospheric  pollution from a heavily  populated  city  area 
to a greenfield site), displacement  is not subtracted from gross  impact. 

When an intervention  leads to the unintended  substitution of addressees for 
non-addressees, the effect  should  be considered  as negative  and  subtrac­
ted from gross impacts (e.g. the retraining of unemployed  males who  replace 
female  workers  in local enterprises)  If,  however,  these  effects  are  expected 
or  sought  (e.g. the  insertion  of  long-term  unemployed  which  occurs  at  the 
expense  of  the  short-term  employed), the  effect  should  not  be  subtracted 
from gross impacts. 

Displacement  should  not  be put on an equal footing with deadweight  since 
the  latter  is much  more crucial  to the estimation  of the  impact  of  Structural 
Fund interventions. When deadweight  is substantial, the  impact  of an  inter­
vention  is correspondingly  reduced.  On the  other  hand, displacement  only 
implies  that  impacts  have  occurred  in one  part  of  the  assisted  area  at  the 
expense of another  part. 

The homogeneity of impacts has repercussions on how deadweight, displace­
ment and substitution are assessed. In the case of employment,  impacts are 
homogenous and the estimation of deadweight and displacement focuses on 
a single evaluation criterion: the number of jobs created or maintained. 

Environmental  impacts  are,  however,  heterogeneous  (e.g. air  quality,  water 
quality, bio-diversity, etc.) and the net impact  of an intervention may have to 
be  estimated  using  a  number  of  relevant  indicators.  Similarly,  impacts  of 
interventions  on  SME  competitiveness  (e.g.  in  terms  of  innovations, 
products,  turnover,  value  added,  etc.)  can  be  estimated  using  an  array  of 
indicators  and the  estimation  of  deadweight  and displacement  will  depend 
on the impacts. 

The leverage effect, defined as the amount of private sector finance invested 
in an assisted  project  that  can  be directly  attributed  to  Structural  Fund  co-
financing  of  projects  (see  Box  7). This  is  another  effect  that  should  be 
estimated  whenever  an  objective  of  interventions  is  to  encourage  the 
financial participation of the private sector. 

In addition to these effects, there are a number effects which, depending  on 
the theme of the evaluation, will need to  be taken  into account.  In the case, 
for  example,  of  interventions  in favour  of  employment,  income  multiplier 
effect  (i.e. the employment  created or maintained indirectly  by the additional 

Technical  considerations 

income generated by the direct employment effects of an intervention) and 
supplier effects (i.e. the additional employment created or maintained  indi­
rectly  in suppliers to  assisted firms)  may  need to  be estimated  in order  to 
have a more accurate picture of the employment  impacts. 

It  is  important  that  these  effects  some  times  referred  to  as  additional 
effects, are not confused with the notion of additionally  as it is used by the 
European Commission. This concept  is illustrated in Box 7. If the  principle 
of additionality  is fully respected, Structural Fund expenditure is fully addi­
tional  to  national  government  expenditure  (lower  left-hand  figure). The 
opposite case is presented in the lower right-hand figure. This concept  is a 
concern  for  financial  monitoring  at  a  national  level  rather  than  for 
programme evaluation. 

Box 7 - Additionality, deadweight  and  leverage  effects 

Programme  level 

SF 
expenditure 

Private 
expenditure 

expenditure 

Govt 
expenditure 

Govt 
expenditure 

100% deadweight  effect 

Leverage  effect 

National  level 

SF 
expenditure 

Private 
expenditure 

Private 
expenditure 

Private 
expenditure 

Govt 
expenditure 

Govt 
expenditure 

Govt 
expenditure 

SF 
expenditure 

Private 
expenditure 

Govt 
expenditure 

100%  additionality 

0%  additionality 

Il 
Thematic analysis of impacts 
on political priorities 

Structural  Fund interventions are intended  to promote  economic  and social 
cohesion and are complementary  to the policy  efforts of individual  member 
states.  To achieve this objective  they selectively address the political  priori 
ties  of  the  European  Union  and  the  major  preoccupations  of  European 
citizens.  There are currently  four political  priorities  which  are  specifically 
addressed  by  interventions:  environmental  protection, 
competitiveness, 
employment and the promotion  of equality between men and women. 

The evaluation of political  priorities  can  take place  within  a programme  or 
across programmes  (transnational or  transregional  thematic  evaluation). In 
the  first  instance  the  objective  is  to provide  conclusions  on  the  extent  to 
which  a programme  impacts  on one or more  policy  priorities.  The second 
scenario is that of an evaluation that is specifically centred  on a given theme 
covering a number  of different  regions across a range of countries. A major 
objective  of this type of evaluation is to highlight  examples of best  practice. 

The following  chapters  are  structured  around  the  key  points  that  were 
presented  in Part I,  these key points  being addressed and developed  within 
the context  of each of the four policy priorities.  In addition  to these points,  a 
number  of evaluation tools are presented  that can be employed  when an in-
depth assessments of impacts is required. 

Evaluating the environmental impact of structural 
interventions 

■ Why do environmental impacts need to be evaluated? 

• The regulatory framework 

For at least the past three decades growing international concern with the 
state of the environment has resulted in increasing provision for environmen­
tal matters in legislation and public policy. The European Union's response 
to this concern can be traced back to the Paris Declaration on the Environ­
ment in 1972. The Commission published its First Action Programme on the 
Environment in the following year and has commissioned research on envi­
ronmental impact assessment (EIA) since 1975. In 1977 work was begun on 
an EIA Directive which, after extensive consultation, was approved by 
Council in 1980. 

In 1985, Directive 85 / 337 / EC made it compulsory to assess the effects of a 
wide range of public and private projects, on the basis that "the best envi­
ronmental policy consists in preventing the creation of pollution or nuisances 
at source, rather than subsequently trying to counteract their effects". 

In 1992, the Treaty on European Union reflected an important shift towards a 
more active approach by requiring that environmental considerations be 
integrated into the formulation and implementation of all types of policy. 

A strategy for achieving sustainable development (i.e. development that 
meets the needs of the present without compromising the ability of future 
generations to meet their needs) in such an integrated fashion was spelt out 
in further detail in the Fifth Action Programme on the Environment which was 
adopted by Council in 1993. In this document it is stated that "The word 
'sustainable' is intended to reflect a policy and strategy for continued 
economic and social development without detriment to the environment and 
the natural resources on which continued human activity and further deve­
lopment depend." 

The economic development policies of the European Union thus increasin­
gly recognise the connection of development to the environment and ack­
nowledge that increasing the cohesion of the Union will demand a path that 
is sustainable in both economic and environmental terms. Since the reform 
of the Structural Funds in 1993, it has been required that explicit considera­

Transversal  evaluation  of  impacts  on  intervention 

priorities 

tion be given to the environmental  impact of all the activities assisted by the 
Funds: plans submitted  by Member States seeking assistance under  objec­
tives  1, 2  or  5b  have  since  been  expected  to  include  "an  appraisal  of  the 
environmental  situation of the area concerned and an evaluation of the envi­
ronmental  impact  of  the  strategy  and  operations..."1. This  requirement  was 
underlined  with the  creation, in  1993, of  the Cohesion  Fund that  devotes a 
substantial  portion  of  its resources to  projects  with  primarily  environmental 
objectives. However, the primary objective of many interventions  is not  envi­
ronmental  and there is often  a tension  between  the type of  activity  which  is 
typically  deemed  to  be  necessary  to  promote  economic  development,  on 
the one hand, and environmental concerns, on the other. 

•  The relationship between the environment and cohesion 

There  is,  of  course,  no  immediate  or  automatic  correlation  between  a 
region's  level  of  economic  development  and  the  state  of  its  environment. 
Both economically  developed  and  underdeveloped  regions can either  have 
a healthy  environment  or experience  severe environmental  problems. In the 
past, controlling environmental  impact was widely viewed as entailing  costs 
and therefore  as  inevitably  having  a constraining  effect  on  economic  deve­
lopment. 

High  growth,  low  environmental  protection  strategies  are  not  sustainable 
over the medium to long-term. They need, therefore, to be replaced by more 
sustainable  strategies  that  take  advantage  of  a  high  quality  environment  in 
order to attract investment and use activities related to environmental impro­
vement  as a vehicle for economic  growth. Responding  to  relatively  specific 
environmental  problems  may  also  offer  short  to  medium-term  employment 
opportunities  and furnish  necessary  elements for wider  economic  develop­
ment strategies. 

The growth  of  "eco-industries"  provides  what  is perhaps the  most  obvious 
example of the complementarity  of environmental  protection and  economic 
development. These industries  have developed to  meet  increasing  demand 
from consumers  for environmentally  friendly  products. They  have also  been 
a  response  to  increased  controls  on  pollution  and  the  generally  enhanced 
awareness of the importance of the environmental  impact of human activity. 

1. Council Regulation No 2081 /93, part III. 

Evaluating  environmental 

impacts 

•  The need for evaluation 

Since cohesion can develop only through activity that is sustainable In both 
economic  and environmental terms, monitoring and evaluation  need syste­
matically  to  take  account  of  both  economic  and  environmental  factors. 
Indeed, the Commission  is now obliged to respect environmental  concerns 
in the design, monitoring and evaluation of structural interventions. 

In the case of  structural  interventions  in lagging  regions  and  rural zones, a 
sizeable  number  of  measures  have clear  environmental  objectives  and  the 
implications of supported  activities for the environment  are relatively direct, 
even if the measurement of the actual effects of such activity may be techni­
cally very difficult.  In the  case of  interventions  in other  regions the environ­
mental  consequences  of  supported  activity  are often  less direct  and  more 
difficult to evaluate. 

While techniques  for the  evaluation  of  environmental  impact  have  become 
increasingly sophisticated over the past few decades, for the most  part they 
have concentrated  on providing ex ante projections  of  likely  impact  so that 
informed choices might be made between alternative  policies and projects. 
Ex ante environmental impact  assessment  is an increasingly significant  and 
systematic  element  in the formulation  and  planning  of  structural  interven­
tions. However, evaluation of the actual impact of these interventions  (either 
individually or in total) is not yet widespread. 

Nevertheless, the Commission  needs to ensure that  methods  are  available 
for  evaluating the actual  environmental  impact  of the heterogeneous  set of 
activities supported by the Structural Funds for three main reasons: 

•  first, the  overall  orientation  of  the  structural  policies  and  the  degree  to 
which  the  efforts  they  devote  to  promoting  cohesion  through  sustainable 
development  have borne fruit, need to  be checked  by obtaining some indi­
cation of the environmental impact of structural policies as a whole; 

•  second, whether  the  environmental  priorities, which  are often  set  out  in 
supported programmes, have in fact been achieved needs to be verified; 

•  finally, in relation to  individual projects, project  selection criteria related to 
the  environment  need to  be validated  by  comparing  predicted  with  actual 
outcomes. 

The EU's structural  policies are delivered through fairly complex and hetero­
geneous  programmes.  Because  an appreciation  of  the  overall  impact  of 

Transversal evaluation of impacts on intervention priorities 

these policies is required, it is necessary to evaluate not only the cumulative 
consequences of particular projects on the environment with explicit envi­
ronmental objectives, but also the environmental implications of both 
projects and broader, macro­level policies which may have neither explicit 
environmental objectives or direct environmental impacts. 

Unfortunately, established methodologies for the evaluation of environmen­
tal impact have been developed almost exclusively for application to 
specific projects rather than to programmes or broader policies. It is true 
that, in recent years, strategic environmental assessment (SEA) has 
emerged as an approach to the evaluation of the environmental impact of 
broader policies. However, it appears that this approach is yet to be fully 
established technically and is, in any event, most useful in relation to ex ante 
evaluation. 

The approach presented here is by necessity less ambitious than a full eva­
luation of environmental impacts. This chapter is therefore oriented towards 
the evaluation of direct environmental impacts, focusing on those tech­
niques that in a simplified and operational form seem to be particularly well 
suited to ex post evaluation of environmental impacts. 

■ Defining what needs to be evaluated 

• The environment and its influences. 

The concept of the environment can cover a vast range of topics that are inter­
related in complex ways. It is therefore important to clearly identify the main 
features of the environment and the most fundamental mechanisms shaping it. 

Natural elements may simply be consumed directly as resources. This, in 
itself, has an impact on the environment and can cause dramatic changes in 
eco­systems ­ changes which may be viewed as either positive or negative. 
In practice, human intervention in the environment always goes beyond 
simply consuming natural resources, to entail the establishment of infra­
structures and settlements, the use of energy, the transport of people and 
material, and the output of waste from each of these activities ­ all of which 
impact on the environment. All of these elements are interconnected and 
changes which occur may be either positive or negative. 

Evaluating  environmental 

impacts 

Example: The environmental effects of agricultural activity 
Agricultural  activity  is  highly  varied  in terms  of  both  products  produced  and  tech­
niques  used.  In the  case  of  the  use  of  intensive  farming  techniques,  it  can  have 
negative environmental effects leading, for example, to a deterioration in the quality of 
soil, water  and  air, as well as the  loss of  habitats  and bio-diversity.  But  agricultural 
activity  can  also  have  positive  effects  with  crops  acting  as a sink  for  greenhouse 
gases, and with  appropriate  land  management  contributing  to the conservation  and 
enhancement  of  bio-diversity  and the  country  landscape,  and to  the  prevention  of 
flooding and landslides. 

The basic determinants of environmental change are set out schematically in 
an adaptation  of  the  "Driving  Force-State-Response  (DF-S-R)  framework 
(See  Box 8). This  model  represents  the environment  as comprising essen­
tially  non-human elements that evolve in interaction with each other to form 
complex  and  delicately  balanced  eco-systems.  It  is one  of  a  number  of 
models  that  have been developed  to  explain the  determinants  of  environ­
mental  change  (other  more  recent  examples  include  the  "Driving  Force-
State-Impact-Response"  model of the European Environmental Agency and 
the  OECD's  adaptation  of  the  DF-S-R  framework  to  include  agri-environ­
mental linkages and sustainable agriculture). 

Box 8 -  An adaptation  of the  Driving  Force-State-Response 
(DF-S-R)  model of sustainable  development 

Economy 

1 
transport 

energy 

industry 

agriculture 

trade 

tourism 

t_ 

State of the 
environment 

\ 
Wealth and welfare 

(economic) 
pressures  or 
driving  forces 

(economic) 
use of 
resources 

air 

water 

land 

wildlife 

natural 
resources 

responses 

(public  and 
private) 
welfare 

(public 
action) 
responses 

Actors 

4 

administrations 

households 

enterprises 

international 
agencies 

Source: Adapted from Department  of the Environment. (1993), Environmental Appraisal of Deve 
lopment  Plans, A good practice  guide,  London: HMSO. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

The environment  can be  broadly  defined  as the set of  natural elements  that 
surround  an  individual  (human, animal  or  vegetal)  or  its  species. Within  the 
framework  of the DF-S-R  model this set of natural elements is comprised of 
air, water, land, wildlife and natural resources. 

This framework allows environmental change to be situated in the context of 
development  in  specific  economic  sectors  to  identify  more  accurately  in 
terms of: 
•  those  driving  forces,  which  can  be  defined  as  the  elements  causing 
changes  (both  positive and  negative)  in the state  of the environment,  these 
include:  natural  environmental  process  and  factors,  biophysical  inputs  and 
outputs, and socio-economic  pressures; 
•  the  precise  changes  in  the  state,  which  reflects  the  changes  in  the 
condition  of  the  environment,  which  are  attributable  to  the  various  driving 
forces; 
•  the  policy  and  behavioural  responses  which  are  being  made  to  those 
changes in the state of the environment. 

While this framework allows the issues involved to be reviewed in a clear and 
systematic  way,  and  fosters  a  balanced  consideration  of  environmental, 
economic  and  policy considerations,  it does  not  help with the ultimate  need 
to  reconcile  different  types  of  criteria. This  is  particularly  necessary  in  the 
matter  of  sustainable  development  since  there  is  an  element  of  tension 
between economic activity and environmental concerns. The very concept of 
sustainable  development  is  based  on  the  premise  that  economic  growth 
should,  at  the  very  least,  not  entail  unacceptable  environmental  cost.  Fre­
quently  it is difficult  to define criteria which allow us to make judgements that 
are equally  valid  in relation  to  both economic  and  environmental  considera­
tions. 

•  Environmental impacts arising from the pursuit of  non-environmental 

objectives 

Whilst the  DF-S-R framework  is unquestionably a useful tool for  conceptua­
lising the  determinants  of  environmental  change,  it  is structured  around  a 
chain of causality which  is circular  in nature and therefore of limited use as a 
basis  for  evaluating  the  impacts  of  structural  interventions  on the  environ­
ment.  A  more  operational  approach  is to  construct  a typology  of  environ­
mental  impacts,  which  combines  socio-economic  and  environmental 
concerns and is generally relevant across the European Union. 

Evaluating  environmental 

impacts 

For example, such a typology might consist of the following categories: 

•  Water quality and resources 
•  Atmosphere and material flows 
•  Environmental management and innovation in enterprises 
•  Land use and landscapes 
•  Flora and fauna 
•  Environmental behaviour of individuals 

The  categories  of  this  sort  of  typology  are  not  intended  to  be  mutually 
exclusive. For example, an action to increase public transport, which  might 
have a direct  impact on the  "environmental  behaviour of individuals",  could 
also  have the  indirect  impact  on  "atmosphere  and  material flows"  through 
the diminished use of private vehicles leading to reduced exhaust emissions 
and demand for non-renewable energy resources. Likewise, impacts that fall 
within the category of  "environmental management and innovation  in enter­
prises" will tend to be upstream of impacts in other categories. However, it is 
unlikely  that  all  of these  levels  of  impact  could  be accurately  traced. This 
type of approach allows the most direct impact to be specified while leaving 
open the potential to identify indirect impacts. It also allows specific themes 
to be analysed and reported on separately. 

Examples  of  effects  (impacts  and  results)  are classified  according  to  the 
typology of impacts (see Box 9). 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Box 9  -  Detailed typology  of environmental  impacts 

of structural  interventions 

Categories of impact 

Examples of impacts and results 

Water quality and resources 

•  Increased proportion of Industrial and toxic waste water treated 
•  Increased proportion of population connected to treatment facilities 
•  Reduced use of nitrates in agriculture 
•  Improvement in the quality of underground, river and marine water 
•  Reduced leakage of treated water from mains system 
•  Increased recycling of used water 

Atmosphere and material flows 

•  Improved collection, treatment and recycling of solid industrial 

waste 

•  Increased re-use, recovery and recycling of urban waste 
• Reduced size and number of landfill sites, and reduced pollution of 

underground water 

• Reduced Industrial emissions of CO2, sulphur and nitrogen oxides 

and unburned hydrocarbons 

Environmental management 
and innovation in enterprises 

•  New patents in the field of environment related technologies 
•  New enterprises specialising in environmental products and services 
•  Increased investment in "clean technologies" 

Land use and landscapes 

• Treatment and reclamation of derelict industrial sites and abando­

ned urban areas 

• Increased number of areas designated for their landscape signifi­

cance 

• Natural areas protected from urban and industrial development, and 

development projects 

• Protection of areas and buildings of outstanding historical and archi­

tectural interest 

Flora and fauna 

•  Reduced use of pesticides and herbicides in agriculture 
•  Maintenance of bio-diversity 
• Cleaning up coastal areas, beach and  rivers 
•  Better management of timber and agricultural resources 

Environmental behaviour 
of individuals 

•  Increased proportion of commuters using public transport 
•  Increased proportion of tourists visiting off-peak 
•  Reduction in use of private vehicles 

•  Clarifying environmental  objectives 

The object of evaluating the environmental  impact  of structural  interventions 
is to  obtain  a  measure  of  the  effects  of  a  mix  of  activities,  all of  which  are 
intended to promote economic  and social cohesion, but only some of  which 
have environmental objectives. When environmental objectives are explicitly 
incorporated  into  an  intervention  they  are obviously  designed  to  result  in a 
positive impact. However, other environmental impacts that result from  inter­
ventions may be either positive or negative. Consequently, a complete  envi­
ronmental  impact  assessment  of  these  activities  would  need  to  take  into 
account the full range of these effects. 

Evaluating  environmental 

impacts 

Since the objectives  of  each component  of  a programme  and the  connec­
tions between different levels of a programme are rarely articulated as clearly 
and as explicitly  as they might  be, and since the environmental  implications 
of  particular  interventions  are frequently  unclear,  it  is usually  necessary  to 
begin  by clarifying  the  objectives.  One approach  to  the clarification  of  the 
objectives  of Structural  Fund Interventions  is to develop an objectives tree. 

Box 10 -  Environmental objectives of Structural Fund Interventions 

presented  in the form of a thematic  objectives  tree 

Global  objectives 

Examples of specific  objectives 

< 

Promote 
environmentally 
sustainable 
development 

Improved water  quality 

-  and  resource 
management 

-  Cleaner 

atmosphere 
and  reduced 
material  flows 

< 

- 

I 

•  Increase  the number  of  households 
and industrial  plants connected to  waste 
water  facilities 

> 

•  Improve the collection, treatment  and 
recycling of industrial  waste 

* - - -. 

1 

I 
1 

Better  environmental 
management 
and  innovation 
in  enterprises 

•  Increase  environmentally  oriented  R &D 

•  Increase  energy  efficiency  of SMEs 

Better  land  use and 
landscape  protection 

Better  conditions 
for flora and fauna 

| 

. 
<-, 

I 

1 

•  Make  more  industrial  sites  available 
in developed  areas 
•  Improve  protection for  areas  designated 
for their  landscape  significance 

•  Improve  protection  of  marine  areas  and 
coastal zones 

More  responsible 
behaviour  in  respect 
to the  environment 

•  Facilitating  the use of  public  transport 
•  Promote  less environmentally  damaging 
tourist  activity 

By way of an illustration, the typology of potential impacts has been integra­
ted  into  an objectives tree specific  to environmental  impacts  resulting  from 
structural  interventions. The global  objective, which  is to  promote environ­
mentally  sustainable  development,  is linked, via intermediate  objectives,  to 
examples  of  specific  objectives  relating  to  possible  measures  of  a 
programme and other interventions. This example is largely based on explicit 
environmental  objectives  since the  implicit  objectives  of  a programme  only 
become  apparent  when  programme  managers  and  decision-makers  are 
interviewed. Relations between intermediate objectives are indicated by their 
relative position and the links between them (see Box 10). 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

The drawing up of an objectives tree facilitates: the specification  of environ­
mental objectives  of a programme, the clarification of  relationships  and any 
priorities  that  may  exist  between  them,  and  better  judgement  of  both  the 
internal  coherence  of  elements  of  a  programme  in  relation  to  the  environ­
ment and their external coherence with other public  policies. 

•  Gross and net environmental  impacts 

The environment  is subject to a complex  combination of pressures arising from 
economic activity and the effects of individual and collective behaviour, as well 
as the effects of a range of public policy interventions. As a result, an observed 
environmental  impact  may  not  be wholly  attributable  to  an  intervention.  It is 
therefore  important  to  calculate the deadweight  of  intervention, so allowing  its 
net  effects  to  be estimated. A  definition  of  deadweight  in this context  is the 
environmental  change  which  may  reflect the  environmental  objectives  of  an 
intervention but which would have occurred even without the intervention. 

Displacement  in this  context  refers  to  the  geographical  shifting  of  negative 
environmental  impacts  rather than to their decline. This  has emerged  as an 
issue of  concern  in relation  to the environment,  for example,  with  regard  to 
practices  such  as exporting  toxic  and  radioactive  waste  from  their  sites  of 
production  to  other  locations  that  are  no  better  equipped  to  cope  with  the 
presence of such substances. 

Example: Estimating the net Impacts of a town by-pass on the environment 
Assuming that after the completion of a town by-pass project, vehicle exhaust emissions 
in the town centre are estimated as having fallen by around 20% (the gross impact of the 
intervention). Yet, over the same period, the growth of private car use in the rest of the 
country  has led to a 5%  increase  in exhaust  emissions  (counterfactual  situation). 
However, although there is an obvious positive effect, there might be negative impacts 
on a number of residential areas on the outskirts of the town that are located near to the 
by-pass because of an increase in traffic levels as travel patterns change. The resulting 
increase in vehicle emissions, corresponding with the displacement effect of the inter­
vention, might be estimated at 10%. To estimate the gross impact of this hypothetical 
intervention the deadweight effect (counterfactual situation minus baseline) and the dis­
placement effect would have to be subtracted from the gross effect. 

It  is  also  important  to  verify  that  none  of  the  measures  contained  in  a 
programme  merely duplicate  existing  regional policies. A case of this  would 
be if existing support  mechanisms for general R&D projects were capable of 

Evaluating environmental impacts 

increasing environmentally oriented R&D. In that case, the environmental 
impact of environmentally oriented R&D may be positive, but it would not be 
entirely attributable to the measure being evaluated. A proportion of the 
impact observed would therefore have to be subtracted from gross impacts. 

■ Examples of impacts by domain of intervention 

Three broad types of intervention are commonly supported by the Structural 
Funds: the provision of basic infrastructure, support for the productive envi­
ronment, and the development of human resources. All these types of inter­
vention may have possible environmental impacts and an example of each is 
dealt with separately below. 

• Basic infrastructure 

Support for the provision of basic infrastructure is usually intended to indi­
rectly promote cohesion by relaxing specific constraints on economic 
activity (e.g. by providing a high quality water supply which is suitable for 
specific production processes) or generally reducing production costs (e.g. 
by upgrading the road network). Infrastructure projects may have explicit 
environmental objectives and, whether or not this is the case, significant 
environmental impacts. In either case, evaluation of the economic impact of 
an infrastructure project needs to be complemented by an evaluation of its 
environmental impact if the sustainability of development based on this type 
of intervention is to be assessed. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Example:  Light  rail s c h e m es  in t he  United  Kingdom 

The  S o u th  Yorkshire  Supertram  a nd  Midland  Metro  were  b o th  f u n d ed  under  objective 

2  in  the  United  K i n g d o m.  Both  of  these  projects  should  result  in  the  creation  of  new 

e m p l o y m e nt  not  only  in  the  construction  and  operation  of  the  new  infrastructure  but 

also  b e c a u se  of  the  Increased  competitiveness  of  the  region  d ue  to  its  improved  links 

to  transport  corridors  and  decreased  journey  times.  Both  projects  should  also  lead  to 

reductions  in  noise  a nd  air  pollution  and  enhanced  values  of  investments  in  b o th  resi­

dential  a nd  c o m m e r c i al  properties. 

Outputs  ΐ> 

Construction  of 
a  new  light 
rail  system 

Decreased 
volume  of  private 
vehicle  use 

Effects 

Reduced  noise 
levels 

> 

Reduced  exhaust 
emissions 

Improved 
air  quality 

Improved 
public  transport 
links 

Reduced 
journey  times 

Improved 
competitiveness 

Priority 

•  Productive  environment 

Various  types  of  investment  subsidy  have  traditionally  been  a  major 
component  of  most  regional  development  strategies,  supporting  the deve­
lopment  of  increased  capacity  for  production,  increased  productivity,  other 
types of direct support to individual enterprises (e.g. in relation to marketing, 
R&D or  specifically  environmentally  friendly  business  practices),  or  indirect 
support  by  promoting  complementary  activities which may  include environ­
mental services. Whether  the environmental  impact  of this type of  interven­
tion  is positive  or  negative, any  individual case will depend  on the  nature of 
the supported  activity. 

Evaluating  environmental 

impacts 

Example: Energy audits and  advice for S M Es  (United  Kingdom) 

In  an  effort  to  e n h a n ce  t he  c o m p e t i t i v e n e ss  of  SMEs  in  the  Knowsley  area  of 

Liverpool,  the  Powersave  project  supported  experts  from  the  local  electricity  supply 

c o m p a ny  in  visiting  c o m p a n i e s,  assessing  their  energy  efficiency  and  p r o p o s i ng 

improvement  plans.  Putting  these  plans  into  action  was  c o - f u n d ed  equally  by  the 

ERDF  and  the  electricity  company.  For  a total  investment  of  3.7  MECU,  cost  savings 

of  over  1  MECU  were  achieved  annually  and  C 02  emissions  were  cut  substantially. 

Outputs 

^>  [ 

Effects 

O 

Energy 
management 
plan for  SMEs 
TJ 

Reduced fuel 
consumption 

Lower  emissions 
of  CO, 

Improved air 
quality 

Reduced 
demand  for 
electric  power 

Reduced 
production  costs 

Improved 
competitiveness 

Priority 

•  Human resources 

Interventions  in support  of  human  resource  development  are intended  to 
strengthen  and  enhance  the  skills  and  abilities  of  beneficiaries  and  so 
improve their employability. The objective is also to make the local economy 
in which  they  work  more  competitive, owing  to  improved  adaptability  and 
productivity  levels. To the  extent that  such  interventions  increase the  envi­
ronmental awareness of beneficiaries or their capacity to engage in activities 
which are not detrimental to the environment  or which have a positive envi­
ronmental  impact, they  increase the  potential sustainability  of the  resulting 
development.  A  positive  environmental  impact  is  likely  to  result  from  the 
supported activity if the human resource development  itself, or the employ­
ment that its beneficiaries eventually obtain, has an environmental objective. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Example: Developing recycling activities In Germany 
Assistance was  provided from the Structural  Funds to a training company  in Geissen 
(Germany)  which  aimed  to  provide jobs  and  qualifications  for the  local  unemployed, 
training  for  local  employees  and  services  with  a  positive  environmental  impact.  The 
company  was established  in 1988. By 1996 it employed 370 people, was training  150 
and had recycled thousands of domestic  appliances (refrigerators, washing  machines 
and  dryers).  This  had  also  led  to  a  significant  reduction  of  CFC  emissions  and  oil 
wastage. 

Outputs  C> 

Effects 

ΐ> 

Provision  of 
training  places 

Recycling  of  domestic 
appliances 

Increased  employment 

Increased  skill  levels 
in the  area of  recycling 

Reduced  CFC 
emissions 

Reduced  oil 
wastage 

Diversification  of 
economy  into 
eco-industries 

Priority 

It is generally difficult to say whether  economic  development will be accom­
panied  by  positive  environmental  spin-offs  and  consequently  follow  a sus­
tainable pattern over the longer term. However, it is important to bear in mind 
that  a project that  gives rise to negative environmental  effects  is not  neces­
sarily a case of environmentally  unsustainable economic  development.  Miti­
gation measures (i.e. measures that reduce the adverse effects of an activity 
on  the environment)  can  be  introduced  which  avoid, reduce  or  remedy  the 
negative  environmental  impact  of  an  intervention.  These  measures  may 
succeed  in stopping  the  deterioration  of the  environment  or  even  lead to  a 
net improvement, therefore achieving  sustainability. 

Defining  a nd  selecting  indicators 

•  The need for and limitations of indicators 

Volume 2 of the MEANS Collection  is devoted to the subject of  indicators for 
monitoring  and  evaluating  Structural  Fund  interventions. The above  Volume 
includes an information sheet on indicators for the priority of the environment. 

Evaluating  environmental 

impacts 

The overall  environmental  impacts  of  the  EU's structural  policies  are  the 
cumulative consequence of a wide range of different types of activity taking 
place  in a similarly  diverse  set of  local  environmental  contexts. To be  in a 
position  to  obtain  an  accurate  overview  of  the  nature  and  scale  of  these 
impacts, access must be available to information that accurately reflects the 
most important aspects of change in the environment over time. In order for 
environmental  change to  be  monitored, indicators  should  be able  to  show 
trends by incorporating a description of conditions which exist at a specified 
point  in time  (a baseline  value) against  which  subsequent  changes  can  be 
measured  through  monitoring.  The  number  of  indicators  selected  and 
defined should be limited enough to be both feasibly attainable and focused 
on the  most  relevant  issues, but  also  comprehensive  enough to  reflect  the 
most significant developments. For example, it is probable that many general 
levels of atmospheric or water contaminants would not be noticeably influen­
ced for a considerable time by even relatively substantial individual  projects. 
In this  case, for  point  sources  of  contamination  (specific  industries,  major 
outlets)  a simple  measure of  decreased  output  of  specified  pollutants  from 
the source  in question  is more likely to  be useful, while for  diffuse  sources 
(such as sectors of production - e.g. levels of use of artificial fertiliser, or broad 
patterns of behaviour - e.g. levels of private vehicle usage) a reasoned calcu­
lation of expected reductions in such outputs would be acceptable. 

•  Recent efforts to develop indicators of sustainable development 

Considerable  efforts  to  develop  indicators  of  sustainable  development  on 
the  international  level  have been made since  1992, notably  by the  UN, the 
OECD, and Eurostat. The United Nations Commission on Sustainable Deve­
lopment has produced individual methodological  specifications for the defi­
nition, collection and interpretation of 132 different  indicators. These indica­
tors  are  developed  within  the  DF-S-R  framework  for  four  categories  of 
information:  economic,  social,  environmental  and  institutional.  Eurostat 
produces  a variety of publications which supply information on the environ­
ment (often prepared jointly with the OECD) and two on-line databases (New 
Cronos and Envstat). Eurostat has also been engaged  in a pressure indices 
project  which  is  intended  to  both  establish  a conceptual  framework  for 
measuring the impact of economic  activity on the environment, and develop 
basic  indicators  for  this  purpose.  The  OECD  has  been  engaged  in  a 
programme  to  develop  policy-relevant  environmental  indicators  and  has 
working groups, for example, on environmental indicators for agriculture and 
for forestry.  Finally, the  European  Environment  Agency  was  established  in 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

1993 to produce information on the environment, including a comprehensive 
triennial  state  of  the  environment  report  on  Europe, the  first  of  which  was 
published in 1995. 

The  European  Commission  is also  developing  environmental  indicators, 
including  the  establishment  of  a  European  System  of  Environmental 
Pressure  Indices,  comprising  50-100  physical  pressure  indicators  which 
show  trends  for  ten  key  "policy  fields".  Once  these  data  series  have  been 
established, they will be to combine them into indices by attributing  different 
weights to them, making the information they provide more manageable and 
better tailored to particular policy needs. Such indices are essentially  proxies 
for more direct measurements of the impact of policies which are less readily 
available.  For  example,  in the  case of  deforestation,  an  index would  ideally 
not  simply  be  an  indicator  of  the  rate  of  deforestation  but  something  that 
combines  this  with  an  indicator  of  incentives  for  forest  clearing.  Similarly, 
instead of simply reporting the weight of herbicides  used each year, a useful 
index would reflect the subject of eco-toxicity more directly. 

It is also possible to construct  ad hoc regional level indicators for  evaluating 
environmental impacts, as the following example shows. 

Example: The use of ad hoc indicators  (France) 
The evaluation of the  Burgundy  objective 5b programme to support  the  development 
of  rural zones  required the  construction  of  regional  level  indicators  of  environmental 
impact.  The  percentage  of  hedgerows  destroyed  was  used  as  an  environmental 
impact indicator of land regrouping operations. Furthermore, this measure was refined 
by  categorising  the  destroyed  hedgerows  in terms  of  quality.  In the  area  of  public 
works, impacts were measured by the number of new beneficiaries of water treatment 
facilities, and  in terms of the  increase  in the equivalent  number  of  inhabitants  having 
access to the facilities. The impact  of waste management  projects was measured  by 
the increase In tonne /  months of waste collected at disposal sites and the increase in 
tonne / months of selective waste collected. 

•  Selecting indicators for a given programme 

Indicators are useful only if they reflect the changes which are to be evaluated. 
To an extent, the  nature of these changes  may  be self-evident.  Nonetheless, 
ensuring  that  they  are adequately  identified  and  interpreted  requires  some 
reflection. Regardless of whether the indicators of environmental impact which 
can  be obtained  relate to changes  in the state of the environment  itself or  to 
changes  in  the  forces  acting  on  the  environment,  evaluators  must  be 
conscious of the variety of ways in which the environment  may be influenced. 

Evaluating  environmental 

impacts 

They may also be both the direct consequences of assisted activity (e.g. the 
damage caused by the construction of a major infrastructure project such as 
a motorway) and the indirect consequences of these or other actions (e.g. an 
action which may even have had a positive direct impact on the environment 
- the establishment of a country park - but which has ultimately led to negative 
impacts - the pressures on the environment resulting from visitors travelling to 
and using the park). Environmental  impacts  may also be of differing  duration 
and be either discrete or cumulative. It is important to ensure that all of these 
types of environmental impact are considered in any evaluation. 

Once relevant changes in the environment have been identified, the interpre­
tation of how these changes were caused may be approached from two di­
rections. First, the outputs,  results and direct  impacts  of the  intervention  in 
question can be monitored  and reasonable  inferences drawn  regarding  the 
impact  that  the  outputs  have  on  the  overall  state  of  the  environment. 
Second, the state of the  environment  can be  monitored  and  any  changes 
noted  may  be  attributed  to  any  one  or  to  a  combination  of  a  range  of 
possible influences. 

Since the  environment  is open to  such  a variety  of  influences,  it  is  often 
impractical  to  establish  an  empirical  connection  between  programme 
outputs, on the one hand, and impacts which can  be attributed to them, on 
the other (at least if impacts in terms of the state of the environment are to be 
measured). In such circumstances, the first is the most practical one to take, 
although it should be crosschecked  when possible with the results of  some 
efforts that follow the second approach. 

Indicators of the state of the environment  have become increasingly sophis­
ticated in recent years. Examples of such indicators include: 

•  an index of evapo-transpiration which provides a powerful indication of the 
state and character of the vegetation of a region; 

•  an  index  of  the total  appropriation  of  net  primary  production  of  a  region 
which provides an equally powerful indication of the degree to which vegeta­
tion is being exploited. 

Yet it is almost  inconceivable that extensive practical use could  be made of 
such  indicators  in  evaluating  the  diverse  and  fragmented  operations 
assisted  by the  Structural  Funds. The strength  of  indicators  of this  kind  in 
reflecting the environment is largely due to their independence from individual 
influences, but in terms of Structural Fund interventions this type of indicator 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

is too all embracing. Consequently, the impacts of supported  projects  would 
be lost amidst the numerous influences arising from other sources. 

For this  reason, a  "bottom-up"  approach  is appropriate  for the  selection  of 
indicators  for the  ex-post  evaluation  of  environmental  impacts  arising  from 
Structural  Fund  Interventions.  This  approach  consists  of  establishing  a 
system of indicators, which allows the impacts of interventions to be tracked 
from the micro level towards the macro level. 

Evaluations  of  environmental  impacts  seek  to  answer  two  fundamentally 
different types of question requiring the use of different types of information. 
First,  in  relation  to  the  components  of  an  individual  programme  which  had 
explicit  environmental  objectives,  how  successful  were  they  in  reaching 
those objectives?  Second, in relation to all types of intervention, what  is the 
potential environmental  impact at the programme  level or in terms of groups 
of programmes (e.g. those aimed at a particular global objective of the struc­
tural policies or those in a particular  Member State)? 

In the first  instance, it  is often  difficult  to definitively  attribute changes  in the 
overall  state  of  even  particular  aspects  of  the  environment,  to  specific 
projects.  Reasonable  assumptions  have  to  be  made  on the  likely  impact  of 
assisted  activity  on the  basis  of  information  that  does  not  precisely  reflect 
the state of the environment.  In some cases this is perfectly satisfactory (e.g. 
in the case of clear  reductions  of discharges from  point  sources of water  or 
air  pollution).  However, the  more  indirect  the  relationship  between  interven­
tions and changes  in the state of the environment  (e.g. in the case of  human 
resource  measures  for  training  in environmentally  relevant  skills),  the  more 
difficult  it is to develop a convincing  measure of  impact. 

This  problem  is less taxing than it might  be due to the fact that the environ­
mental  objectives  of  a  structural  intervention  are  themselves  often 
expressed  in terms  of  change  in  driving  force  variables.  These  objectives 
make assumptions  about  the  relationships  of  such  variables  to  the state  of 
the environment.  Provided such assumptions are credible, it is best to  focus 
the evaluation of the environmental  impact of the programme in question  on 
the analysis of the driving forces concerned. 

In the second  instance,  it might  be expected  that  because the  scale  of  the 
overall effort  is larger,  impact  is easier to  identify  in terms of state variables. 
This  may  be  the  case  in  some  instances  but  usually  information  on  the 

Evaluating  environmental 

impacts 

impacts  of the overall  effort  is itself  also  available, at  best, only  in terms of 
driving-force variables. 

Moreover,  evaluation  of  impact  at this  level  is complicated  by the  need to 
aggregate a variety of different  measures of impact which, even if they refer 
to  the  same  variable,  may  have  been  expressed  for  good  reason  on  the 
programme  level in ways which are context-specific.  For this reason, when 
evaluating environmental impact on the multi-programme /  policy  level, it is 
necessary to  refer  more systematically  to  established  data-series  on  state 
variables, even if it is impossible to establish a direct link between these and 
the information which is available on the actual impacts of interventions. 

By way of an example, a system of indicators which could be used to track 
the  production  of  impacts, both  environmental  and socio-economic  arising 
from the construction of waste water treatment plants is presented in Box  11. 

Box 11 -  Example of a system  of indicators for the  construction 

of waste-water  treatment  plants 

Programme  Indicators 
(Related to the intervention and Its effects) 

Context  Indicators 
(Related to the assisted area) 

Output  Indicators 

Progress 

Capacity 

Result  Indicators 

Compliance  with  project  duration 
Rate of  completion 
Number of new waste water  treatment 
plants  constructed 
Maximum annual throughput (m1) 

Treatment  in relation  % of waste-water  undergoing 
to demand  and pro- 
cessing  standards 

!  primary  treatment 
% of waste  water 
undergoing  secondary  treatment 

Impact  indicators 

Consumption 

Indirect  economic 
effect 

Volume of waste-water  treated 
Number of households /  businesses 
connected to waste-water  treatment 
facilities  by  programme  after one year 

%  of waste water  treated 
%  of households /  businesses 
connected to waste water  treatment 
facilities 

Number  of economic  units (firms, 
farms, etc.) declaring  that the new 
water supply  system  has released a 
major 
constraint  for their  development 

Environment 

%  of waste water samples from  point 
sources which show a specified  level of 
decline of  Identified  pollutants 

%  of  km of rivers monitored on which 
pollution  overcome a specified 
standard 

Source: Volume 2 of the MEANS  Collection 

Transversal evaluation of impacts on intervention priorities 

Although the "top­down" approach to measuring environmental impacts is 
dominated by the use of quantitative data (e.g. the indices constructed by 
Eurostat), a "bottom­up" approach can be based on qualitative data. Fur­
thermore, this qualitative data can later be transformed into a quantitative 
indicator, should this be necessary. 

Example: Constructing an environmental impact indicator on qualitative data 

A French evaluation team had to evaluate the environmental impact of a motorway. 
Their method consisted of sending a group of people to photograph what they consi­
dered to be landscape blackspots in the area of the motorway before corrective 
measures had been undertaken. The same photographic survey was then carried out 
after the corrective measures had been undertaken. The change in the number of 
environmental blackspots was then recorded. 

To summarise, indicators will have to be selected which are neither simply 
the standard type of socio­economic indicators traditionally used in the eva­
luation of structural policies, nor the specialist technical indicators which are 
being developed to monitor change in the state of the environment. These 
indicators will tend to resemble the traditional systems of indicators used for 
the evaluation of structural interventions in relation to resource, output and 
result indicators. However, the impact indicators will be largely made up of 
indicators of changing driving forces on (rather than the changing state of) 
the environment. 

■ Principal sources of information 

Each programming document should contain the following information on 
the environment: 

• a description of the main environmental characteristics of the geographical 
area covered by the programme; 

• an identification of any sub­areas of particular environmental significance; 

• the nature, cause and location of any significant pollution or other environ­
mental driving force problems; 

• a description of existing national / regional environmental policies; 

• information on national / regional environmental authorities and procedures; 

• a projection of the expected environmental impact of the programme as a 
whole and of each constituent priority / measure; 

• details of any steps which are proposed to mitigate any negative environ­
mental impacts of the programme or its components. 

Evaluating environmental impacts 

Although all of these details are not always found in a programming 
document, these documents are still the obvious starting point for obtaining 
information. 

Programme monitoring systems should provide data on resource expendi­
ture and outputs needed to answer many evaluation questions, but are 
unlikely to provide data on the evolution of the environment itself in the 
region concerned. However, in cases where the preparation of an interven­
tion has shown that gaps exist in the availability of specific context data that 
are of particular importance in evaluating its impacts, provision may be 
made to obtain this information within the monitoring system. 

Established statistical time series of environmental data provide an increa­
singly rich source of information that can be used to complement the infor­
mation provided in programming documents. 

Information from programming documents and monitoring systems can be 
expected to provide very specific micro­level information on the allocation of 
resources and their impacts in terms of driving forces on the environment. 
More established data series tend to concentrate on macro­level state 
variables. One of the tasks of evaluation is to try to draw connections 
between these two levels of information, and in carrying out this task it may 
also need to draw on information from previous research and evaluations or 
engage in original data collection and / or production. 

■ S o me examples of current practice in the European Union 

While a more detailed treatment of evaluation tools is provided in Volume 3 
of the MEANS Collection, a number of examples in­depth analyses of envi­
ronmental impacts are presented below. 

A variety of tools has been used in the identification and analysis of the envi­
ronmental impact of public policy. Selecting the most appropriate evaluation 
method or technique to use in any instance depends on the particular eva­
luation requirements that prevail. It is important to note, however, that each 
technique has its own strengths and limitations and, even within the European 
Union, the choice made is likely to be influenced by relevant national legislation 
and established national practices in relation to: project planning; the series of 
pre­feasibility, feasibility and technical studies which are typically required; and, 
lastly, established implementation, monitoring and audit procedures. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

An  assessment  of  the  impact  of  interventions  on  the  environment  can  be 
undertaken  using the  "step-by-step"  approach  presented  in  Part  IV of  this 
Volume. The  choice  of  tools  used  in this  instance  would  focus  on  minimal 
case studies, individual  interviews and focus groups. On the other hand, for 
an  in-depth  evaluation  of  environmental  impact  analysis,  other  evaluation 
tools have to be employed. 

•  Strategic environmental assessment of programmes in the Netherlands 

Strategic  environmental  assessment  (SEA)  refers  to  the  environmental 
assessment  of  programmes  and  policies  as opposed  to  individual  projects 
(the  latter  is the  traditional  function  of  environmental  impact  assessment  -
EIA). SEA developed as a result of the realisation that EIA may occur too late 
in the planning process to allow adequate consideration to be given to  alter­
natives. SEA is a formalisation  of the  process  in which the other  techniques 
mentioned  in this  section  are  applied,  and  an  SEA  system  is expected  to 
include formal  guidelines  on the  use of  scenario techniques,  screening  and 
scoping, case studies, etc. 

SEA takes  place  in the  context  of  a tiered  forward  planning  process  where 
policies at the international /  national  level are seen as shaping  programmes 
at  the  regional  level  and  projects  at  the  local  level.  Within  the  EU  this 
approach is adopted with  increasing frequency  in relation to waste manage­
ment, land-use and energy policies. 

Within the EU, the Netherlands  is the country where the practice of  strategic 
environmental  assessment  (SEA) is most  developed. The Dutch  experience 
in this area dates  back to the early  1980s and SEA is now legally required for 
various activities  including electricity  and water  supply, nature  conservation 
and outdoor recreation, and provincial waste management  proposals. 

A  National  Programme  on  Waste  Management  has  been  in  place  in  the 
Netherlands since 1992. This programme is revised every three years and an 
integrated  set  of  ElAs  as  well  as  a  SEA  have  been  implemented  on  a 
voluntary  basis  by the Waste  Management  Council  -  a partnership  of local, 
regional and national authorities. The approach works on three levels: 

•  Decisions  are made  on the  national  level concerning  the  waste  treatment 
options /  technologies  that  might  be  used. These decisions  are subject to a 
SEA before a selection is made. 

•  Decisions are made on the regional level regarding the possible  location of 
treatment facilities; these are also subject to a SEA. 

Evaluating  environmental 

impacts 

•  Finally, individual  project  alternatives  are selected for  each of the  chosen 
locations; these are subject to ElAs. 

The SEA considered two alternative scenarios. First, a situation in which the 
objectives  of the  National  Environmental  Policy  Plan on  recycling  and  the 
production  of  waste  were  met.  Second, a situation  in which  these  broad 
policy  objectives  might  not  be  attained  but  individual  programmes  and 
measures  were still expected  to  be effective  in reaching  their  more  limited 
objectives.  Based  on  this  analysis,  a  strategy  was  formulated  for  the 
medium-term  development  of  waste  disposal  facilities.  Regular  meetings 
with  provincial  authorities  (responsible  for  implementing  the  policy)  and 
interest groups were an integral part of the development of the SEA. 

The policy was then implemented by each of the 12 provincial authorities, with 
each one developing  four-year  programmes.  In each case global  objectives 
were set (e.g. 400 factories should have a waste prevention plan by the end of 
the programme), needs calculated, the potential for improvement in relation to 
specific indicators estimated and project selection criteria established. 

Within the provincial programmes project selection is usually carried out on the 
basis of ElAs. In view of the fact the EIA method is more established, it is not 
surprising  that  work  on the  project  level  has  proved to  be  more  evidently 
methodical, with  a clear  focus  on  assessing  alternative  processing  tech­
niques, and  a definite choice of  indicators. The relationship  of  output  from 
this  project  level activity  to  the  SEA  is not  clear-cut  and  it  seems that  the 
latter exercise has an inherently poorer quality of information. With regard to 
intermediate  or  ex-post  evaluation, even though  Dutch  legislation  is  more 
demanding  in this  respect  than that  of  many  other  countries,  there  is  only 
advice (rather than a requirement) that this  be implemented. Even if there is 
no obligation to adopt technically sophisticated evaluation techniques within 
SEAs, experience  has shown that the formalisation  of the process itself has 
brought  a number  of  benefits to the design and  implementation  of  policies 
with an environmental  impact. The following advantages  of the formal  inte­
grated  approach  offered  by  SEAs  have  been  identified  by  administrators: 
•  environmental  issues  are given  a greater  degree  of  influence  on  general 
policy formulation; 
•  decision makers routinely  use those results of the SEA process which are 
clearly well designed; 
•  although test  case experiences  may  not  - strictly  speaking  -  be generali­
sed, an impression of their relevance is more readily obtained in the context 
of a SEA and their results are more likely to influence decision-making. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Further potential benefits of adopting a SEA approach are: 
•  it  encourages  consideration  of  alternatives  which  might  otherwise  be 
ignored 
•  it allows more effective analysis of the cumulative effects of  projects 
•  it facilitates both the planning and analysis of synergistic  effects 
•  it is better able to evaluate ancillary / secondary effects of  actions 
•  it is capable of taking delayed impacts into account 
•  it  permits  the  development  of  standard,  cross-measure  mitigation 
provision 
•  it  can  enhance  both  communication  between  those  responsible  and  the 
involvement of the public 
•  it provides a framework  in which cumulative and synergistic  environmental 
impacts can betaken  into account  in a systematic fashion. 

•  The screening and scoping of a programme in the UK 

The concept of screening refers to two different types of  process  depending 
on the level at which  it  is applied. Within a measure, screening  involves the 
selection of projects for assistance through the application  of a standard set 
of project selection criteria. An example of this process is supplied below. At 
the  measure  level  itself,  screening  refers  to  the  initial  examination  of 
measures  to  determine  the  degree  of  environmental  impact  assessment 
which is required before approval is given for their implementation. 

The  process  of  scoping,  on  the other  hand, is concerned  with  determining 
which  potential  impacts  of  alternative  projects  deserve  consideration  in an 
EIA. In practice, this  is a considerably  more complex  process as there  may 
not  be  agreement  among  interested  parties  as  to  precisely  which  impacts 
are relevant. 

Under  UK  legislation, specific  types  of  physical  project  are either  classified 
as  being subject  to  mandatory  environmental  impact  assessment  (typically 
very  large  capital  projects  with  a clear  and  direct  environmental  impact)  or 
checked against  a number of criteria (such as the  location, scale,  proposed 
method of dealing with waste, etc.) of the individual project to judge  whether 
a full EIA is needed. Scoping is carried out  in voluntary discussions  between 
project  developers  and  local planning  authorities  prior to the submission  of 
environmental statements on the proposed  projects. 

The  region  of  Merseyside  in  England  is  the  subject  of  a  programme  of 
assisted activity for the period  1994-1999  under objective  1 of the structural 

Evaluating  environmental 

impacts 

policies. Each project seeking assistance under this programme is screened 
in a two-stage process: 
•  an initial stage which verifies that the proposed project meets UK statutory 
controls; 
•  a second  stage  which  scores  proposals  against  environmental  criteria 
which have been separately established for each of the programme's priori­
ties  and  measures  and  also  takes  account  of the  precise  location  of  the 
proposed  project. 

A summarised  example of an environmental  screening form for  a particular 
measure of the Merseyside programme is provided in Box 12. This approach 
is of  particular  interest  as  an example  of  combining  economic  objectives 
with the other  objectives  pursued  under  a specific  programme  assisted  by 
the Structural Funds. 

Box 12 - Example of an environmental  screening form 

Priority  1: Measure  1: Quality  sites and premises for industrial  and commercial investors. 

Objective: To ensure an adequate  provision of quality  industrial  and commercial sites and  premises 
encourage  indigenous growth and Increased employment  opportunities. 

to 

Project: 

Reference: 

Yes 

N/A 

No 

1.  If the development  Is on a brownfield site, will It reduce the area of 
derelict  land or return buildings to productive  use? 

2. Does the development  respect designations  regarding  the natural 
environment  or built  heritage? 

3. Does the development  incorporate good  practice for energy conservation, 
water recycling  or waste reduction or involve innovative  environmental 
measures? 

4. Will the site have an environmental  management  plan or are measures 
In place for post-development  environmental  management? 

5. Have any  measures been taken to ensure that  the site is easily 
accessible by  public  transport? 

6. Is the development  designed to  Improve the local built  environment, 
for example by minimising the effects  of crime, vandalism  or litter? 

7. Does the development  project  contain  proposals for  environmental 
Improvements such as  landscaping? 

1 

1 

1 

1 

1 

0.5 

0.5 

0.5 

0.5 

0.5 

0.75 

0.38 

0.75 

0.38 

Total score: 

If the project  Is In a priority  area multiply  this  score by  1.5  and round to the nearest 1. 

0 

0 

0 

0 

0 

0 

0 

In those sub-areas of the Merseyside region which are judged to experience 
particularly  severe environmental  problems  or where environmental  impro­
vement could be expected to have an unusually  strong impact  on develop­
ment,  the  total  score  of  environmental  criteria  is  positively  weighted  in 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

relation  to  other  project  selection  criteria. These  particular  sub-areas  were 
themselves defined according to a number of criteria, including: the visibility 
and reputation of the site in question, its proximity to strategic  transportation 
corridors or locations regarded as assets, and whether it already qualified for 
assistance from other sources of funding. 

Council  Directive 85 / 337 /  EEC specifies all projects for which  assessment 
is mandatory  and  others  to  which  particular  thresholds  and  criteria  apply. 
However,  in the context  of  programmes  supported  by the Structural  Funds, 
Member  States  are  also  required  to  ensure that  any  individual  operation  is 
compatible  with  EU environmental  policy  and  legislation.  Some  form  of 
screening is therefore required of all the components of any prograrhme, but 
much  depends  on  the  quality  of  the  specific  criteria  developed  and  the 
design of the process adopted in each instance. 

Screening is obviously necessary at the ex ante phase and should at least be 
confirmed during subsequent  phases of evaluation. It may also be useful for 
ex-post  evaluations  to  adopt  a screening  technique  to  focus  analysis  on 
those measures likely to produce the greatest  impact. 

•  Case studies of projects in Scotland 

It is often useful to select  individual examples of a phenomenon for  in-depth 
study, for example: 
•  when  exploratory  fieldwork  is  required  due  to  the  absence  of  adequate 
information; 
•  to provide a detailed analysis of a particular process of environmental impact; 
•  to test a hypothesis or assertion about a particular  project. 

Case studies may be individual or multiple and in the latter case may involve 
sophisticated  techniques  to  ensure  the  replication  of  design  in  different 
contexts. The  approach  has four stages: design and  selection, data  collec­
tion, data analysis and report  preparation. Selection of the  case(s) for  study 
is particularly important for the relevance of findings. Cases may be selected 
on the basis of being either typical or representative of a situation which is of 
interest for other reasons (e.g. best / worst cases, particular  circumstances). 
Data collection may involve the use of surveys and interviews. 

In  an  attempt  to  develop  a  useful  method  for  appraising  environmental 
projects  intended  to  restore  or  protect  specific  sites,  the  Scottish  Office 
engaged  evaluators  to  review  methods  of  valuing  environmental  impacts. 

Evaluating  environmental 

impacts 

On the basis of the first stage of the study and in view of the type of  projects 
being considered, it was decided  that the contingent  valuation method (the 
various impacts are valued by the beneficiaries and losers through question­
naires), in particular, warranted further investigation. The second stage of the 
work  involved  applying  this  method  to  seven  projects. The cases  studied 
include the improvement  of an old industrial site, the restoration of an urban 
area,  the  protection  of  a mountain  area  and  woods,  and  support  for  an 
industrial heritage centre. In each case members of the public were intervie­
wed  or surveyed. The  results of this  case study  confirmed  the value of the 
technique  examined  and  gave  a clearer  indication  of  the circumstances  in 
which it could usefully be applied. 

Multiple case studies can be used to analyse the effects of interventions in a 
number  of contexts  across the programme  area. Provided  adequate provi­
sions were made for design replication, they could be used to better  identify 
impacts by comparing situations in which a measure was applied to those in 
which it was not. Another subject, which appears to offer particular scope for 
the  application  of  this  technique,  is the  relationship  between  changes  in 
pressure  variables  and  state  variables.  Coefficients  might  be  developed 
from case studies that would allow calculations of impact on the state of the 
environment to  be made  on the  basis of  indicators of  change in pressures. 

While the nature of effects can be determined inductively by using the case-
study  technique,  this  technique  cannot,  strictly  speaking,  demonstrate 
causality.  However,  since the  evaluation  of  environmental  impact  generally 
needs to  make an  inductive  leap  of faith  from  knowledge  about  change  in 
driving force variables to related changes in environmental state variables, a 
case-study approach is at no disadvantage in this respect. 

•  Combining cost-benefit & multi-criteria analysis to assess projects in 
Ireland 

Cost-benefit analysis attempts to measure in monetary terms all of the costs 
and benefits of a project  over the course of its lifetime. It is commonly  used 
at the ex ante stage of  project  evaluation, to choose  between a number  of 
alternative  projects  and to judge for which, if any, the calculated  economic 
benefits exceed the economic costs involved. 

In a more sophisticated version of cost-benefit  analysis, instead of attemp­
ting to express all benefits in terms of a single monetary value, a number of 
explicit  evaluation  criteria  are  established  which  can  be  weighted  by 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

decision  makers  to  allow  for  a  ranking  of  alternatives.  This  approach  is 
known  as  multi-criteria  analysis  (MCA)  and,  in  relation  to  our  present 
concerns, offers the advantage of permitting systematic account to be taken 
in the decision-making  process  of  both  environmental  costs  and  economic 
costs,  rather  than  attempting  to  translate  all  environmental  criteria  into 
monetary  terms.  In  Ireland, the  Department  of  the  Environment  engaged 
consultants  to  undertake evaluations  both to  meet  EU requirements  for  the 
selection of specific  environmental  infrastructure  projects  (co-funded  in this 
case  by  the  Cohesion  Fund)  and  to  establish  a  methodology  for  future 
decision-making.  Traditional  cost-benefit  analysis  was  rejected  by  the 
consultants  who  recognised  that  the  quantification  of  benefits  in  monetary 
terms  posed  too  many  problems  in this  case  (water  supply,  wastewater 
treatment  and water quality management  projects). 

One  potential  weakness  of  adopting  a  multi-criteria  approach  is that  the 
weightings  used  are  essentially  subjective  and  scores  are  sometimes 
arbitrary.  For this  reason, the  approach  adopted  combined  the  techniques 
for  identifying  criteria  used  in  MCA,  with  a  common  denominator  of 
monetary values derived from  relevant studies. In each case the basis of the 
weightings  used  was  explained. This  approach  was  applied  with  variations 
for each of the three types of project to be evaluated. 
In the  case  of  water  supply  projects,  the  approach  had  four  components: 
•  identification of all costs associated with each alternative project; 
•  quantification of all direct and indirect  benefits; 
•  ascribing values to benefits for each project; 
•  calculation of benefit / costs ratios. 

Information  was  collected  on  pre-  and  post-project  demand  levels  broken 
down  by sector,  population  served  and the  design  demand  and  population 
for  each  project.  Standard  operating  costs  were assumed  to  apply  to  each 
alternative and three types of benefit were identified: 
•  quality  improvements 
•  extension of supply to new  consumers 
•  wastage elimination and control. 

Each  of  these  three  benefits  was  assigned  a  specific  unit  value  and  a 
weighting  derived  from  analysis  of  relevant  studies.  The  basis  of  these 
values was  identified  and  both the  reasons for  using them  and the  limits  to 
their  applicability  were  explained. Competing  projects  were  then  analysed 
using  a  number  of  different  scenarios  of  likely  demand  evolution  and  a 
benefit / cost ratio was calculated for each one. A ranking of the projects was 

Evaluating environmental impacts 

made and the results of this ranking were clearly explained so that the impli­
cations for decision­makers were obvious. 

All of the variants on this approach are essentially tools for decision­making 
in the selection of alternative projects. Much depends on the professiona­
lism of evaluators in clearly indicating either the assumptions involved in 
establishing single indices for ranking projects or (in the case of MCA) the 
need for decision­makers to themselves arbitrate between criteria. 

To the extent that these techniques are applied rigorously, they can be a 
useful means not only of appraising the relative strengths and weaknesses 
of alternative projects but also of providing a synthetic score for their sustai­
nability which can then be checked against actual performance. 

■ Summary 

The quality of the environment and how it can be maintained in the face of 
economic pressure is an issue that has risen rapidly to the top of the political 
agenda. At the same time, there has been an increasing acknowledgement 
that economic development does not have to involve a trade­off in environ­
mental terms and that the two can, in fact, be mutually reinforcing. However, 
ensuring that economic development and cohesion are sustainable requires 
that both economic and environmental factors are taken into account when 
evaluating structural interventions. For evaluation, this means that interme­
diate and ex post evaluation of environmental impacts of projects and pro­
grammes need to be undertaken, in addition to the more common practice 
of assessing projects for their environmental effects. 

Although the standard DF­S­R framework may be a useful tool in other cir­
cumstances, it is of little use for the evaluation of the effects of Structural 
Fund interventions on the environment. A "bottom­up" approach is therefore 
proposed which, at the impact level, focuses on driving force variables, 
combining both socio­economic and environmental concerns. 

The predominance of ex ante project assessment means that there are few 
examples of good practice in the use of tools in intermediate and ex post 
evaluation. Nearly all the examples presented are thus of tools that have 
been used in circumstances of ex ante project assessment, but which can 
generally be adapted to other evaluation situations. 

Transversal evaluation of impacts on intervention priorities 

■ For m o re information see... 

Department of the Environment. (1993), Environmental Appraisal of Develop­
ment Plans, A good practice guide, London: HMSO. 

Ecotec Research and Consulting Ltd. (1997), Encouraging Sustainable 
Development through Objective 2 Programmes: Guidance for Desk Officers, 
Note of the European Commission (DGXVI). 

European Commission (1998), A Handbook on Environmental Assessment 
of Regional Development Plans and EU Structural Funds Programmes, 
European Commission, DG XVI. 

European Commission (1999), The Thematic Evaluation of the Impact of the 
Structural Funds on the Environment, European Commission, DG XVI. 

European Environment Agency. (1996), Guidelines for Data Collection for 
The Dobris+3 Report, Technical Report, Copenhagen: EEA. 

European Environment Agency. (1995), Environment in the European Union, 
Report for the Fifth Environmental Action Programme, Copenhagen: EEA. 

Fehily, Timoney, Weston. (1995), The Economic Appraisal of Environmental 
Projects Supported by the EU Cohesion Fund, Report for the Department of 
the Environment, Dublin. 

OCDE. (1996), La Cohérence dans l'Evaluation Environnementale, Orienta­
tions générales pratiques pour les projets de coopération pour le développe­
ment, Paris: OCDE. 

OECD. (1997), Environmental Indicators for Agriculture, Paris: Publications 
Service, OECD. 

Sadler B. (1996), Environmental Assessment in a Changing World : Evalua­
ting Practice to Improve Performance, International Study of the Effective­
ness of Environmental Assessment, Canadian Environmental Assessment 
Agency, Minister of Supply and Services Canada. 

Statistical Office of the European Community. (1997), Indicators of Sustai­
nable Development, Luxembourg: Office for Official Publications of the 
European Communities. 

Wood C. (1995), Environmental Impact Assessment, A comparative review, 
Harlow: Addison Wesley Longman Ltd. 

Evaluating the impact of structural interventions on the 
competitiveness of Small and Medium­Sized Enterprises 
SMES 

■ Why evaluate impacts on Small and Medium­Sized 

Enterprises? 

Small and Medium­Sized Enterprises (SMEs) make an important contribu­
tion to job creation, competitiveness and growth. Net job creation, particu­
larly in enterprises with fewer than 100 employees, more than compensated 
for job losses in large enterprises during the period 1988 to 1995. Despite 
this apparent dynamism, employment created by new enterprises tends to 
be somewhat unstable, with only half of new businesses created still in 
existence after five years. 

Box 13 ­ Definition of SMEs 

The European Commission has adopted a Recommendation concerning 
a standard definition of SMEs: Medium­sized enterprises have less than 
250 employees, an annual turnover of less than 40 MECU or balance 
sheet assets of less than 27 MECU and must be independent, which 
means having less than 25% of their capital owned by a large enterprise. 
Small enterprises have to have less than 50 employees, a turnover of less 
than 7 MECU or assets of less than 5 MECU and must be independent. 
Micro­enterprises are defined as firms with less than 10 employees (The 
recommendation on the definition of SMEs of 3 April 1996, can be found 
in the JOL 107 of 30.4.1996). 

The Madrid SME Policy Paper2 stresses that European SMEs' full potential 
for growth and employment creation has not yet been properly realised for a 
number of reasons. Firstly, it has been argued that the complex legal, fiscal 
and administrative environment faced by European enterprises has a relati­
vely higher impact on SMEs than on large companies. Secondly, SMEs often 
have problems accessing bank loans at reasonable rates or raising equity on 
the markets. In particular, small enterprises have difficulties in gaining 
access to parts of the market due to a lack of comprehensive market intelli­
gence or problems in identifying suitable partners. Others have argued that 
the business environment in Europe (in particular, the lack of sufficient risk 
capital, the existence of significant regulatory barriers and an aversion to risk 
taking) is not as conducive to the development of SMEs and, in particular, of 

2. Commission Communication of 29.11.1995 entitled Small and Medium Sized Enterprises 
­ A dynamic source of employment, growth and competitiveness in the European Union, 
Document CSE (95)2087. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

New  Technology  Based  Firms  (a restrictive  definition  of  New  Technology 
Based  Firms  -  NTBFs  -  is that  of  firms  active  in the  developing  industrial 
sectors  based  on  emerging  technological  paradigms  such  as  bio-techno­
logy or  laser technology).  For the purpose of this document,  the wider  term 
of  high-tech  firm  covering  all  enterprises  active  in  high-tech  sectors  or 
producing  high-tech  products  will  be  used  (a  typology  of  sectors  and 
products  has  been developed  by the  OECD). Where SMEs are  not  techno­
logy intensive, better  use needs to be made of readily available technologies 
in order to remain competitive.  However, very few  SMEs search patent  data 
banks in order to find out what technology  is available  "off-the-shelf". 

In  less  favoured  rural  areas  SMEs  account  for  a  relatively  high  share  of 
employees  in the  local  economy  but  also  face  particular  problems.  Whilst 
difficulties  vary  between  regions,  the  quality  of  infrastructure  and  the 
distance  from  main  markets  can  sometimes  be  a  problem,  as  can  poor 
access  to  information, poorly  developed  local  industrial  base and  a  lack  of 
suitably  qualified  workers.  Although  these  factors  pose  problems  for  the 
development  of  SMEs  in  general,  in  some  regions  self-employment  and 
entrepreneurship  are  values  that  are  recognised  and  encouraged.  Thus, 
some less favoured rural areas belong to the most dynamic regions, in terms 
of  new  and small  business development,  and  generate  more new jobs  than 
the national average. 

•  The Community  context 

Recent  European  Councils  have  given  a  positive  political  signal  towards 
SMEs,  notably  in Amsterdam  (June  1997),  Luxembourg  (November  1997) 
and  in Cardiff  (June  1998). As a result  of  the Amsterdam  Council, the  BEST 
task force was created and  presented  its report to the Commission  in 1998. 
This report  includes a number  of recommendations  to  improve the  business 
environment,  some  of  which  should  help  SMEs  to  grasp  the  opportunities 
offered  by  Community  programmes. The  Luxembourg  Special  Employment 
European  Council  recognised  the  promotion  of  entrepreneurship  as  a 
common guideline for Member States' employment policies within the Union. 
As  a follow  up, the  Commission  has  adopted  a  Communication  entitled 
"Fostering Entrepreneurship  in Europe - Priorities for the Future" setting out a 
number of priority  issues both at European and national levels. Moreover, the 
conclusions  of the Job Summit gave a positive signal towards SMEs by sup­
porting the further development of financial instruments and cross border part­
nerships. The Cardiff European Council has asked the Commission to present a 
report in Vienna on the follow up of the BEST recommendations. 

Evaluating  impacts  on  the  competitiveness  of SMEs 

Since SMEs play a primary role in balanced  local and  regional  development, 
and are a driving force behind  effective  adaptation  to  structural change and 
innovation, the  European  Commission  has recommended  that, within  the 
overall framework of interventions in industry and services, it was necessary to 
take the needs of SMEs systematically  into account  in the implementation of 
programmes. For this purpose, the Commission has developed the Integrated 
Programme, a global framework for  co-ordination  of  all activities for SMEs. 
This includes the Third  Multi-Annual  Programme for SMEs (1997-2000), the 
contributions that  other policies make to the development  of SMEs, and the 
Concerted  Actions  which  have the goal  of  promoting  the  co-ordination  of 
Member States' SME policies through the exchange of best practices. 

To accomplish this aim, the following five priorities are set out in the Integra­
ted Programme: 

•  simplify  and  improve the administrative  and  regulatory  business environ­
ment; 

•  improve the financial environment for enterprises; 

•  help SMEs to Européanise and internationalise their strategies, in particu­
lar through better information and services; 

•  enhance  SMEs'  competitiveness  and  improve  their  access  to  research, 
innovation and training; 

•  promote  entrepreneurship  and  support  special  target  groups  such  as 
women and young entrepreneurs. 

Structural  Funds  contribute  to  achieving  the  objectives  of  SME  policy 
through  their  support  for  the  creation  and  development  of  SMEs  in  the 
various  operational  programmes. The  European  Investment  Bank  and  the 
European Investment  Fund are also involved in providing support for SMEs. 

Although  the  main  objectives  of  Structural  Fund  interventions  in favour  of 
SMEs are similar across the EU, there are differences  in the specific  objec­
tives  from  one  region  to  another. Thus, in objective  1 regions  there  is  an 
emphasis  on  strengthening  the support  structure  for  SMEs.  In objective 2 
regions  where  large-scale  traditional  industries  have declined,  measures 
focus on promoting entrepreneurship and creating alternative job opportuni­
ties. In objective 5b regions the emphasis is on supporting the diversification 
and  modernisation  of  rural  SMEs and  in objective  6 areas the  priority  is to 
help  SMEs  in overcoming  problems  caused  by their  distance  from  major 
population centres. 

Transversal evaluation of impacts on intervention priorities 

The development of the potential of European SMEs in general, and in 
relation to the role they play in social and economic cohesion in particular, 
represents a major strategic aim of the European Union. The level of support 
for SMEs in terms of resources allocated from the Structural Funds for the 
period 1994­1999 reflects the importance that the Commission attaches to 
small and medium­sized firms. For instance, on average 10% of ERDF 
resources are devoted specifically to measures dealing with the promotion of 
SMEs and the improvement of the framework conditions in which they 
operate (Framework conditions correspond with the business environment in 
the widest sense, incorporating: the overall macroeconomic conditions, tax 
system, regulatory environment, bankruptcy laws, competition policy, etc.). 
This is almost double the level of support provided in the period 1989 ­1993. 

The purpose of evaluating the impacts of Structural Fund interventions with 
respect to the priority of SME competitiveness is defined by the European 
Commission as being to "provide a thorough and systematic analysis of the 
contribution and impact of Structural Funds' support to SMEs in the EU 
Member States and regions". Thus, evaluation has a major role to play in 
assessing the extent to which policy objectives have been or are being 
achieved, in providing insights into the impacts of Structural Funds on 
SMEs, in guiding the process of policy making and, consequently, in 
enhancing the effectiveness of future programmes in this domain. 

■ Defining w h at n e e ds to be evaluated 

Evaluations of Structural Fund interventions in favour of SMEs need to look 
at the strategies underpinning the implementation of programmes (i.e. how 
well­founded the strategies are in relation to the strengths and weaknesses 
of the assisted regions, to other public policies related to SMEs and to 
current ideas on SME support). They also need to examine the implementa­
tion of measures in favour of SMEs in order to assess them in terms of effi­
ciency and effectiveness. Impacts of measures not specifically targeted at 
SMEs should also be assessed to see if interventions are benefiting SMEs 
while meeting their stated objectives (i.e. increasing employment or growth). 

• Guidelines and key issues in SME evaluations 

The effectiveness of Structural Fund interventions in favour of SMEs 
depends as much on how efficiently support is delivered as on their design. 
For this reason it is important to examine the implementation of pro­
grammes. The following key issues can be used as guidelines: 

Evaluating  impacts  on  the  competitiveness  of SMEs 

•  How are support measures being delivered? Has the fragmentation of SME 
support  structures and schemes that exists in some countries been reduced 
by  the  introduction  of  innovations  such  as  "one  stop  shops"  (a  function 
performed,  for  example,  by  Chambers  of  Commerce  in  Germany  and 
Business  Links  in the  UK)? Are addressees  less  confused  about  support 
schemes?  Who  manages  SME  programmes  and  to  what  extent  are they 
implemented  through  partnerships  between  public  and  private  bodies?  In 
regions where institutional structures are relatively weak, to what extent have 
SME support  programme  delivery  mechanisms  been strengthened  through 
institutional  restructuring, capacity  building  or the  involvement  of  private 
agencies? 
•  How  are the different  types  of  SME  programmes  themselves  appraised? 
How  is  the  implementation  of  measures  monitored  and  evaluated  by 
different national and regional authorities? How will the results of the evalua­
tion be taken  into account  in the formulation  of future  policy  developments 
or the reorientation of existing programmes? 

There are a number of key issues related to the impact on regional develop­
ment of Structural Fund interventions in favour of SMEs: 
•  Interventions aim at both encouraging start-up of new businesses (e.g. by 
providing seed capital) and supporting existing SMEs. Issues of survival and 
performance of assisted enterprises are therefore both relevant. 
•  Interventions  also  aim  at  promoting the exchange  of  information  and the 
diffusion  of  best  practices  between  SMEs situated  in different  regions. The 
extent of transborder networking is therefore an important issue for evaluation. 
• To a certain extent, the creation of employment and the creation of wealth at 
the micro-level involve a trade-off  between these objectives. It may therefore 
be important  to examine to what  extent assisted enterprises  reinvest  profits 
to assure their survival over the medium and long-term, and the  potential of 
these investment  to engender job creation. Furthermore, when  employment 
creation or  maintenance  can be attributed  to the  contribution  of  Structural 
Fund interventions, the cost should, whenever possible, be estimated. 
•  Many  small firms  produce  a narrow  range of  products  and services. This 
means that when diversification  is the  objective  of  an intervention, it  is the 
emergence of new firms specialising in new market niches, rather than the reo­
rientation of existing ones, that will tend to be the main motor of this process. 
•  SMEs may also have a very limited client base, and in some cases they can 
find  themselves  in  a  position  where  they  have  only  one  buyer  of  their 
products  or  services.  In this  situation  close  relations  with  larger firms  can 
have  a positive  effect  as subcontractors  adopt  technological  and  perfor-

Transversal  evaluation  of  impacts  on  intervention 

priorities 

mance standards of the contracting  firm, or benefit from technology  transfer, 
technical training and assistance. On the other hand, the competitive  position 
of such firms is likely to be weak, and the economic fabric of the region will, as 
a consequence, be fragile. The place of assisted enterprises in supply  chains 
and clusters may in certain cases represent a key evaluation issue. 

When comparing different  programmes at the national or regional level, top-
down  techniques  can  be  employed  to  gather  relevant  contextual  informa­
tion. On the other hand, the use of bottom-up techniques allows data collec­
tion  about  programme  outputs  and  results,  such  as the  number  of  SMEs 
assisted,  the  number  of  new  businesses  assisted,  and  the  efficiency  and 
effectiveness  of  the  implementation  of  interventions. The  estimation  of  the 
impacts of  interventions  at the project  and measure  level also lends itself to 
the bottom-up approach. 

It  is  important,  when  looking  at  measures,  to  distinguish  between  those 
which are directly targeted at SMEs and which are easily identified  in Opera­
tional  Programmes,  and  those from  which  SMEs  benefit  only  indirectly  and 
which are therefore much more difficult to trace. 

Finally, it is essential to distinguish  between addressees in terms of their size 
by using the classification of SMEs, including micro-enterprises,  established 
by the Community. 

•  Clarifying the objectives of SME  interventions 

An early step in the evaluation of SME interventions is to develop a hierarchy 
of objectives  in order to be able to examine the strategic aims of the Structu­
ral Fund programmes at various  levels. This can be done by constructing  an 
objectives  tree, which  allows  specific  SME  objectives  to  be  linked  up  to 
more  global  socio-economic  objectives  envisaged  within  the  Structural 
Fund framework. An example  of  such an objectives  tree for Structural  Fund 
interventions  in the  domain  of  SMEs  is  presented  in  Box  14. The  interme­
diate objectives of increasing competitiveness and promoting entrepreneurship 
and, to a lesser extent, Europeanisation and internationalisation of SME opera­
tions, are those most closely associated with Structural Fund interventions. 

By integrating objectives that are not necessarily uniquely those of Structural 
Fund  interventions  (e.g. simplifying  start-up  procedures  and the  legislative 
framework, etc.), the objectives tree allows the coherence of objectives to be 
examined  and the existence of synergies  to  be explicitly taken into conside­
ration. 

Evaluating impacts on the competitiveness of SMEs 

Box 14 ­ Example of a thematic objectives tree for SME 

interventions 

Global objectives 

Examples of specific objectives 

Maximise the 
contribution of 
SMEs to social 
and economic 
cohesion 

Improve the business 
environment 

Européanise and 
internationalise the 
operations of SMEs 

Increase the 
competitiveness 
of SMEs 

Promote entrepreneurship 
within special target |­
groups 

Improve access to venture capital 
Reduce administrative burdens 

Increase the supply of equity capital 
Simplify start­up procedures 
Simplify the legislative framework 

Reduce costs In trading internationally 
Improve Information and co­operation strategies 

Reduce trade barriers to third countries 

• Improve access to R & TD Programmes 
• Adapt training programmes to the needs 

of SMEs 

Encourage entrepreneurship culture 
Increase the development of new SMEs 
Increase the number of women and 
young entrepreneurs 

Links between intermediate objectives <­­

Objectives outside the Structural Funds  Π 

• Net impacts arising from interventions in favour of SMEs 

Whenever possible, attempts should be made to assess the impacts of SME 
programmes on a net basis. This means that deadweight effects should be 
estimated so that the counterfactual situation can be established (e.g. the 
level of turnover, the amount of investment, the number of innovations, etc. 
that would have occurred in a policy­off situation). Furthermore, displace­
ment effects may have to be taken into account. Substitution effects may 
also occur and may even, in certain instances, be sought (e.g. an increase in 
the proportion of women and young entrepreneurs, diversification of the 
local productive activities towards more dynamic sectors). 

Transversal evaluation of impacts on intervention priorities 

Example: From gross to net impacts in the case of support for new business 
creation 
A scheme, part financed by the Structural Funds, to provide seed capital to finance 
the setting up of new businesses, has led to 60 new micro­enterprises being launched 
with a combined turnover of 1,36 million euro after the first year. Surveys of the 
addressees of the scheme found that 15 new enterprises with a turnover of 0,38 
million euro would have been launched in the absence of the scheme (deadweight). It 
was also found that two addressees had been running undeclared businesses with a 
combined annual turnover of 32 000 euro. However, because this was considered as a 
positive effect of the intervention (i.e. the substitution of registered actives for unde­
clared activities) it was not subtracted from the gross effects. The net effect of the 
intervention was thus estimated at 0,98 million euro. 

In the case of support for investments (e.g. in physical infrastructure, capital 
goods, research and development, etc.) the leverage of Structural Fund 
expenditure is often explicitly sought. The leverage effect, defined as the 
amount of private sector finance invested in an assisted project that can be 
directly attributed to Structural Fund co­financing of the project, should be 
estimated when relevant. 

■ Examples of i m p a c ts by d o m a in of intervention 

SMEs are directly affected by interventions aimed at supporting productive 
investment. Furthermore, interventions targeting the provision of basic infra­
structure and the development of human resources have direct and indirect 
effects on SMEs. 

Evaluating 

impacts  on  the  competitiveness 

of  SMEs 

•  Basic  infrastructure 

Infrastructure  investments  in  favour  of  SMEs  are  directly  targeted  at  the 
needs  of  businesses, for  example,  interventions  supporting  the  construction 
of  facilities  specially  designed  to  nurture small  high-tech  enterprises  or  new 
businesses. This type  of infrastructure  is often an integral element  of  broader 
interventions  in  favour  of  SMEs.  Other  infrastructure  interventions  have  an 
indirect  impact  on  industry,  usually  by  reducing  environmental  and  accessi­
bility  constraints  (transport,  energy supplies,  information). 

Example: A road improvement project in Ireland 
Within the framework  of Structural  Fund interventions  in the area of  basic  infrastruc­
ture, one of the prime objectives is to increase mobility within regions. The Athlone By-
Pass  project  involved  the construction  of  a  10  km dual  carriageway  by-pass  and a 
river  bridge. This  project  did  not  include  specific  measures  to  support  industry  or 
SMEs,  but  by  allowing  road freight  to  avoid  the town  centre, constraints  on  local 
industrial  development  and  access  to  ports,  and therefore to  export  markets,  was 
improved. In addition to these indirect effects, during the period of construction  local 
SMEs were directly  involved  in subcontracting to the main  contractor  of the  project. 

Outputs  > 

Effects 

> 

Creation of temporary 
employment 

Jobs created 
or maintained 

Construction of 
a new dual 
carriageway 

Reduction in 
delivery times 
to major markets 

Improved road 
accessibility for local SMEs 

Priority 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

•  Productive environment 

Interventions  in the  area  of  the  productive  environment  are those  with  the 
most direct effect on SMEs. These may include interventions  in the domains 
of  rural  development  and  agriculture,  tourism, fisheries,  etc.  which,  whilst 
not specifically aimed at SMEs, are intended to have an impact on entrepre­
neurship  and  small  businesses.  Other  interventions  explicitly  target  SMEs 
(e.g.  information  services,  provision  of  tailored  facilities  and  on-site  advice 
services, support for venture capital schemes, etc.). 

Example: T he  Ouest  D o u ze  Initiative  (France) 

The  Ouest  Douze  Initiative,  located  in  C a p d e n a c,  A u b in  et  Decazeville,  in  southern 

France,  is  c o - f i n a n c ed  by  the  ERDF,  the  French  g o v e r n m e n t,  local  b o d i es  a nd  the 

private  sector.  It  ¡s  a  non-profit  organisation  offering  technical  advice  a nd  financial 

s u p p o rt  (interest-free  loans)  to  potential  local  entrepreneurs.  It  also  runs  a  s c h e me 

w h e re  entrepreneurs  c an  be  t w i n n ed  w i th  e s t a b l i s h ed  c o m p a n i es  a nd  w h e re  past 

addressees  return  to  pass  on  their  experience  to  their  successors  on  the  s c h e m e. 

Outputs 

J^> 

Effects 

Support  and  advice 
sessions  with  potential 
entrepreneurs 

Arrangements  for 
soft  loans  for 
the  launching 
new  enterprises 

New  businesses 
formed 

ΐ> 

Increased 
entrepreneurial 
activity  in the  region 

Strengthening  of  SME 
base  in the  region 

Priority 

Q 

Evaluating impacts on the competitiveness of SMEs 

• Human resources 

Interventions concerning education, training and research in this domain 
may either be directed at industry as a whole or explicitly oriented towards 
supporting SMEs (e.g. training for individuals wishing to become self­
employed, assistance for research and development projects, brokering of 
research partnerships with other firms or institutes, etc.) 

Example: Furniture Technology and Design Project (Finland) 
This project, based in Päijät­Häme in the south of Finland, aims to help increase the 
competitiveness of businesses in the furniture industry. To do this it supports co­
operation and networking between manufacturers, suppliers and training organisa­
tions. Vocational training schemes are tailored to the needs of individual companies 
and advice is given on training strategies and on anticipating future training require­
ments. The project has succeeded in increasing the level of technology and design in 
participating firms, leading to increased competitiveness both at home and abroad. 

Outputs 

> 

Employees attending 
courses and improving 
their skills 

Effects 

\ t 

> 

Participating SMEs 
increasing export sales 

Training courses 
designed specifically 
for SMEs 

SMEs participating in 
the project and 
improving the quality 
of their workforce 

Participating SMEs 
expanding into new 
overseas markets 

Priority I I 

■ Defining and selecting indicators 

A more detailed treatment of indicators is provided in Volume 2 of the 
MEANS Collection devoted to indicators for monitoring and evaluation. The 
above Volume includes an information sheet on indicators for the priority of 
competitiveness of SMEs and enterprises. 

Interventions may be directly targeted at SMEs, others will affect SMEs indi­
rectly. The definition and selection of indicators will, to some extent, depend 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

on the type of intervention being studied. For interventions that do not focus 
uniquely on SMEs, but on the productive sector as a whole, a system of  indi­
cators should systematically  incorporate  the percentage of resources  being 
allocated to SMEs, % of  projects concerning SMEs, etc. This approach  cor­
responds  with  that  of  mainstreaming  used  to  systematically  highlight  the 
extent to which the promotion of equality  between men and women is being 
taken into account  in programmes (see Box 2). 

Demand  for  many  SME  support  services  depends  to  a  large extent  on  the 
economic  context.  Financial  consumption  will  consequently  tend  to 
fluctuate over time, depending  on the  position  of  national  economies  in the 
business  cycle.  It  is  also  interesting  to  examine  the  percentage  of  support 
being taken up by projects in the more dynamic or high-tech sectors. 

Output  indicators concern the quantity  of  physical  infrastructure  or  support 
services financed. In the case of  business support  services, which are of  an 
intangible  nature  (e.g. information  management,  training  or  consultancy), 
output  indicators  focus  on  the  scale  of  activity  actually  taking  place and  of 
services financed by the programme (e.g. hours of advice received by addres­
sees). They also focus on the number of addressees or projects assisted. 

Result  indicators  inform  on  how  the  supply  structure  of  a  region  has  been 
modified  by the  intervention  by, for  example,  looking  at the number  of  new 
business  start-ups. Other  indicators  may  be  more sophisticated; they  may, 
for example, look at the leverage effect that funding generates. 

Impact  indicators  of  interventions  in this  domain  reflect the  heterogeneous 
nature of  impacts  on  SMEs. They  generally  focus on the  increased  in value 
added  arising from  an intervention,  knock-on  effects  on suppliers  and sub­
sequent employment  impacts that eventually arise. In the case of new firms, 
indicators  focus  on  the  survival  rate  of  new  start-ups.  It  is,  however, 
important  to  be realistic  about the  impacts that  can  be observed during  the 
timescale  of  an  intervention.  Whilst  some  direct  impacts  on  employment 
may  manifest  themselves  in  a  relatively  short  period, indirect  employment 
impacts, knock-on  effects, and  impacts on product  and process  innovation 
usually  occur  only  over  the  medium  to  long-term. The  estimation  of  such 
impacts usually has to be based on the expectations of addressees. 

By way of an example, a system of indicators that could be used to track the 
production of impacts arising from a project  providing informational  support 
to promote exports by SMEs is presented in Boxi 5. 

Evaluating impacts on the competitiveness of SMEs 

Box 15 ­ Example of a system of indicators for an intervention 

providing informational support for SMEs to promote exports 

Programme Indicators 
(Related to the Intervention and its effects) 

Context Indicators 
(Related to the assisted area) 

Output Indicators 

Number of addresses 

Number of assisted SMEs 

Number of SMEs in the eligible area 

Result Indicators 

Satisfaction rate 

Geographical 
diversification 

Impact indicators 

Exports 

% of addressees who are satisfied or 
very satisfied with the support 
services provided 

Number of SMEs becoming 
new exporters 
Number of SMEs exporting 
to new markets 

% of exports sales in the turn over 
of assisted SMEs after 18 months 

Regional exports of SMEs 
related to GDP 

Value added 

Increase In value added generated 
after 18 months 

Average value added by employee 
in the region 

Direct employment 

Number of net jobs created / 
maintained (FTEs of which held by 
women) In firms In relation to export 
sales after 18 months 

Source: Volume 2 of the MEANS Collection 

■ Obtaining useful information 

Total number of 
unemployed in the assisted area 

The extent to which an intended approach to the evaluation of SME interven­
tions can be adopted will clearly depend on the quality and availability of data 
and the extent to which statistical data on regional economies can be disag­
gregated to the level of assisted regions. It needs to be emphasised that some 
impacts are not possible to quantify, and in this case a purely qualitative 
approach has to be adopted. The evaluation of Structural Fund interventions in 
the domain of SMEs will, to a large extent, be strongly dependent upon primary 
data, complemented by established statistical sources. 

One particular problem which arises when using the established definition 
based on employment levels of small (0 to 50 employees) and medium 
enterprises (51 to 250 employees) is that a medium­sized enterprise consis­
ting of 250 employees has very little in common with one of 50 employees. 
Similarly a small enterprise of 50 employees is usually very different to a firm 
of 10 or fewer employees. When primary data is to be collected, the hetero­

Transversal  evaluation  of  impacts  on  intervention 

priorities 

geneity within the overall category of SMEs can be dealt with through careful 
sampling  and  choice  of  case  studies.  However,  established  data  sources 
often do not collect data on enterprises of less than 20 employees - a popu­
lation which for  certain  types  of  intervention  (i.e. financial  support  aimed  at 
assisting small firms to break through the 20 employee barrier or the creation 
of  micro-enterprises)  may  be  of  particular  interest.  Another  problem  with 
secondary  data  is that  while the  "birth"  of  a new  enterprise  is  immediately 
registered  when the owner  goes through  the  relevant  administrative  proce­
dures, the  "death" of a firm may not  be recorded until a long time after  it has 
terminated  its  activity.  This  may  lead  to  an  over-estimation  of  overall  SME 
activity and to the inclusion of non-existent firms in analyses. 

Many  context  indicators, such  as business  start-up,  may  be available  from 
European  and  national  administrative  records,  statistics  and  surveys  (e.g. 
those  collected  for  taxation  purposes).  Eurostat  has  developed  a  special 
regional  database,  called  Regio,  of  regional  industrial  statistics  with  annual 
Europe-wide comparable data dating back to 1994. 

In conjunction  with  DG XXIII,  Eurostat  publishes  the  report  "Enterprises  in 
Europe"  at  regular  intervals. This  report  contains  detailed  information  by 
sector of economic  activity  and by payroll size classes on all the  enterprises 
in  the  European  Union  (on  the  basis  of  their  number,  workforce  and 
turnover),  as  well  as  thematic  analyses  (the  demography  of  enterprises, 
innovation in SMEs, labour costs and productivity  in SMEs, etc.). 

An  annual  publication  of  the  European  Observatory  for  SMEs  gives  an 
overview of the performance and perspectives of SMEs in all Member  States 
of  the  European  Economic  Area and  Switzerland  since  1993, and  provides 
structured data on SMEs in both quantitative and qualitative terms as well as 
a number of detailed thematic studies (tourism, environment, etc.). 

The BACH database, based on detailed balance sheet data and covering 23 
manufacturing  sectors  in  1987-1995  in  12  EU  countries,  contains  data  on 
SMEs. Data is broken down  by different size groups based on a harmonised 
definition.  The  information  is  provided  by  central  banks  and  statistical 
agencies. The database is particularly well suited to financial analysis. 

Evaluating impacts on the competitiveness of SMEs 

■ Appropriate tools for analysing impacts on SMEs 

While a detailed treatment of evaluation tools is provided in Volume 3 of the 
MEANS Collection, a number of tools particularly suited to evaluating 
impacts on SMEs are presented below. 

An assessment of the impact of interventions on SMEs can be undertaken 
using the "step­by­step" approach presented in Part IV of this Volume. 

Face­to­face interviews can be used to examine issues concerning the 
strategic aspects of Structural Fund interventions. They might be carried out 
with key actors in national authorities and managers who have overall res­
ponsibility for SME support programmes in the Member States. A disadvan­
tage of this technique is that it can be very time consuming when a large 
number of interviews have to be organised. When a large number of people 
sharing a collective experience (e.g. in the implementation of interventions) 
have to be interviewed, the use of focus groups may be less time consuming 
and more useful. However, it should be kept in mind that whilst the use of 
this tool may allow useful qualitative information about implementing inter­
ventions to be gathered, it is not suited to all situations (e.g. where a detailed 
description of processes is required, individual interviews are more likely to 
produce more useful information). 

Questionnaire surveys allow a basic assessment of the impacts of measures 
and give insight into the contribution of different schemes to SME competiti­
veness and growth as well as the implementation of programmes (i.e. the 
accessibility and delivery of EU­funded schemes). When a questionnaire is 
well designed, it will allow reasonably definite judgements to be made on the 
extent to which improved performance can be linked to assistance from EU­
funded schemes. 

Whilst it is less expensive and time consuming to rely on monitoring data for 
constituting samples, this presents a problem of sampling bias. Monitoring 
data can effectively include non­addressee firms (i.e. firms which have made 
contact with the operators but which have not subsequently become addres­
sees). However, using information on these enterprises to construct the coun­
terfactual situation ignores the behaviour of non­addressee firms that, for a 
number of reasons have not approached the authorities with regard to the 
programme (e.g. they may be unaware of a programme's existence, they may 
perceive that the cost of applying for assistance outweighs the benefits, etc.). 

Factor analysis is particularly useful for analysing the characteristics of large 
homogenous populations. Because SMEs are highly differentiated, this 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

technique can be used to generate clusters of firms according to their key cha­
racteristics. Impacts can then be analysed in relation to these characteristics. 
A  factor  analysis  is  usually  fed  with  data  from  questionnaire  surveys.  This 
technique  was successfully  used  in Spain when  evaluating the  employment 
impacts  of job  subsidies  in the Catalonian  co-operative  sector  (see Box  16). 

Box 16 - Example of a factor analysis of the Catalan co-operative sector 

The  use  of  factor  analysis  can  be  illustrated  with  respect  to  an  ex  post 
evaluation  project,  oriented  towards  the  employment  effects  (creation 
and maintenance  of jobs)  of  subsidies  in the sector of Catalan  co-opera­
tives. Evaluation in this case was complicated  by the diversity  of co-ope­
ratives and the consequent  heterogeneity of the population. The  problem 
was twofold: first, the  relevance  of  measures  taken  by the  region  had  to 
be  determined  in  view  of  this  situation;  secondly,  the  importance  of 
impacts  on  those  co-operatives  which  received  aid  had to  be assessed. 

Given the variety,  as well as the  lack  of  knowledge  of the population  and 
of  impacts,  factor  analysis  was  chosen  for  its  ability  to  reveal  the  main 
characteristics  of  the  population  under  consideration  (about  3,000  co­
operatives). 

The first step in the evaluation consisted of sending a questionnaire to all 
the  co-operatives.  This  was  completed  by  interviews.  The  data  thus 
gathered  were first  analysed  using traditional  statistical  methods  (search 
for  the  main  characteristics  of  the  population, drawing  up  of  cross-refe­
renced tables). The second step in the data analysis consisted of a factor 
analysis  with  a view  to  isolating  variables  likely to  explain  variations  in 
employment  in the  co-operatives:  including  the  size  of  the  entity,  age, 
sector of activity, technological level, etc. 

Source: Volume 3 of the MEANS Collection 

Surveys  can  be  prepared  or  complemented  by  a  number  of  case  studies 
which  are  able to  uncover  impact  causation  of  SME  measures  and to  test, 
justify  or qualify  results  obtained  by  surveys  or  other  methods. They  might 
focus on particular types of Structural Fund interventions (e.g. financial engi­
neering, human  resources,  business support  services, innovation and  tech­
nology, etc.). It is important, however, to carefully select case studies to allow 
useful in-depth analysis and comparison between  individual  projects. 

Although  the  selection  is  strongly  dependant  upon  the  willingness  of 
sponsors and partner  organisations  responsible for the action to  participate 
in the research, and on the availability of quality information (including moni-

Evaluating  impacts  on  employment 

toring  data),  reasonable  coverage  of  the  various  types  of  actions  is 
desirable.  The  selection  of  case  studies  can  be  guided  by  taking  into 
account  the SME initiatives that  involve  major  expenditure, and the  impor­
tance of  different  SME actions  from the  point  of  view  of  regional  develop­
ment strategies. For an example of the use of case studies see Box 17. 

Box 17 -  Example  of the  use of the case study technique  in the 

evaluation of the  NordTek programme,  Denmark 

This programme, intended for small and medium-sized firms, consisted of 
about  100 projects offering services to firms in the areas of training, consul­
tancy,  access  to  modern  computer  equipment,  etc.  It  was  therefore  a 
programme  that  could  be qualified  as  "soft",  as opposed to  "hard"  pro­
grammes  aimed  at  improving  infrastructures.  The  results  of  such  pro­
grammes  are difficult  to  evaluate  precisely  because they are not tangible. 

Six  large  projects  were selected  as cases to  be studied. In terms  of  the 
selection  criteria, the  projects  had to  be  representative  of  the  diversity 
existing  in intervention  strategies  and  locality. The selected  projects  also 
had to  have  benefited  from  significant  funding  in the framework  of  the 
NordTek Programme and be oriented exclusively towards firms (rather than 
research  institutions  or administrations). The six  projects  evaluated repre­
sented such a large portion of the total Programme that the analysis of the 
impacts  of these  interventions  could  be used  to  cover  all the  important 
phenomena. It was recognised without ambiguity that the selected projects 
represented the "best cases" of the programme. 

The data were collected at two operational levels: at the level of the project 
as such, and at that of the firms. As far as the  project was concerned, the 
evaluators studied each of the cases  in the field four  or five times  over a 
period of two  years. Each time the  managers were questioned  and each 
project  had to  keep  a record, in a  "register"  designed  especially  for  the 
purpose, of the services proposed and the firms concerned. The analysis of 
the  documents  thus  obtained  was  handed  to  the  project  leaders  and 
discussed  during visits. As for  the firms, the  data  were collected  during 
interviews with a sample of 20 SMEs which  had used the services offered. 
Finally, 40 SMEs who had not used the proposed services were questioned 
in the final stage of the evaluation. 

Source: Volume 3 of the MEANS  Collection 

Transversal evaluation of impacts on intervention priorities 

■ S u m m a ry 
The important role of small and medium­sized enterprises (SMEs) in suppor­
ting growth and job creation in the European Union has been brought into 
sharper focus over the past few years. The flexibility and innovative capacity 
of SMEs enable them to respond quickly to structural changes and adapt to 
changing demand. However, there are a number of factors that hamper the 
competitiveness of European SMEs and basic market and policy imperfec­
tions that inhibit growth and job creation. The evaluation of Structural Fund 
interventions in favour of SMEs therefore has a major role to play in 
assessing the extent to which stated policy objectives have contributed to 
overcoming these problems. 

Interventions may focus directly on SMEs or on SMEs as a sub­population of 
industry. SMEs are also implicitly targeted by interventions in other domains, 
especially those where diversification of productive structures is an 
objective (i.e. rural development, fisheries and tourism). 

As far as suitable empirical techniques for evaluating Structural Fund inter­
ventions in favour of SMEs are concerned, experience of their application to 
this domain is limited, but it is likely to accumulate rapidly as evaluations of 
this type of intervention become more commonplace. 

■ For m o re information see... 

EIM Small Business Research Consultancy (1997), European Observatory 
for SMEs: Fifth Annual Report, The Netherlands: EIM. 
European Commission. (1995), Activities in favour of SMEs and the craft 
sector, Luxembourg: Office for Official Publications of the European 
Community. 

European Commission. (1997), Report from the Commission to the 
European Parliament, the Council, and the Economic and Social Committee 
and the Committee of the Regions on the Coordination of Activities to Assist 
Small and Medium­sized Enterprises (SMEs) and the Craft Sector. 
Grant Thornton International Business Strategies Ltd. (1998), European 
Business Survey: Prospects and Issues for SMEs, Spring (6). 
OCDE. (1993), Les Petites et Moyennes Entreprises: technologie et compéti­
tivité, Paris: OCDE. 

OECD. (1998), Besf Practice Policies for Small and Medium­sized Enter­
prises, Paris: OECD. 

OECD, (unpublished), Assessment of SME Policies: An Analytical 
Framework, Working Document, DST/ IND/ PME (98)2. 

Evaluating the impact of structural interventions on 
employment 

■ Why do employment impacts need to be evaluated? 

• The Community context 

Structural unemployment is a persistent problem with which the European 
Union has to contend, even during periods of strong economic growth. 
Those who are unemployed experience a loss which is more than financial 
and the wider community has to bear the burden of supporting an economi­
cally inactive section of the population, the costs of increasingly established 
pockets of long­term unemployment and the associated deprivation which 
tends to be concentrated in both social and geographic terms. 

The response of the European Union to this situation is based on the 
premise that a Union­wide effort is needed to establish a both a micro and 
macro­economic context that favours the creation of employment. This 
effort must simultaneously co­ordinate and work in partnership with initia­
tives and action taken on a national, regional and local level. 

The Structural Funds are used to co­finance policy interventions which aim 
not only to promote a better economic and social balance across the Union 
by reducing regional and national disparities (cohesion), but also to assist 
the overall development of the Union's structural fabric, including that of the 
labour market. They are intended to stimulate economic growth and the 
creation of employment. Although some of the objectives of structural 
policies have always been explicitly related to employment, in recent years a 
growing concern with producing substantial employment effects has 
become increasingly evident both in principle and in practice. As a major ins­
trument for influencing economic development patterns and a major vehicle 
for the delivery of resources to local partnerships, it is evident that the use of 
the Structural Funds needs to reflect the increased priority which the Union 
has given to employment. 

The types of employment impacts likely to be produced by interventions 
assisted by the Structural Funds are critically influenced by the particular 
instruments used in each case. The choice of such instruments is influenced 
in turn by the objectives of the intervention in question. An intervention in the 
area of the productive environment or basic infrastructure may aim to 
generate employment effects by creating new jobs or maintaining existing 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

ones.  On  the  other  hand,  interventions  in the  area  of  human  resources 
focusing on training are targeted  at enabling  individuals to meet employabi­
lity  (i.e. the  ability  of  an  individual  to  meet  the  technical  and  organisational 
requirements  of employment  available at a given time and in a given  place). 
The  addressees  of  these  interventions  may  be for  people  already  in  work 
who  are adapting to changing work conditions, or the unemployed, particu­
larly  special  target  groups  such  as women, the  long term  unemployed  and 
young  people,  who  need  to  improve  their  employability  by  enhancing 
existing skills or developing  new ones. 

A  defining  characteristic  of  structural  interventions  is that  they  are concen­
trated  on  a limited  number  of  global  objectives,  and  each  programme  of 
interventions  assisted  by the Structural  Funds  is concentrated  on  achieving 
one of these objectives. Global objectives concern both specific regions and 
specific social groups and in either case they may be defined in relation to a 
specified  economic  sector.  The  relationship  between  the  different  global 
objectives  and  the  priority  of  promoting  employment  varies  considerably. 

•  Accurate evaluations of the effectiveness  of Structural Fund 
interventions 

The  potential  benefits arising from  effective  Structural  Fund  interventions  in 
the  area  of  employment  are  substantial:  a  reduction  in  the  poverty  and 
misery that  unemployment  causes, the creation of  labour supply  conditions 
that  promote  growth  and  competitiveness  in the  weaker  regions,  and  an 
increase  in social  and  economic  cohesion.  Ultimately  these  benefits  could 
also  include  increased  European-wide  competitiveness  and the  underpin­
ning of the European social model. However, if Structural Fund  interventions 
are  to  be  shown  to  be  effective,  evaluations  will  need  to  be  sufficiently 
sophisticated to measure employment  impacts accurately. 

If interventions  are to  be effective, the ex ante and ex-post  measurement  of 
employment  effects  has to  be as accurate as possible. On a more  strategic 
level, accurate  information  allows the  measurement  of the overall  impact  of 
interventions  on employment,  permitting  informed decisions to  be made by 
the  European Commission  and  Parliament  on the continuation  or  modifica­
tion of Structural Fund interventions. 

Since Structural Fund interventions combine  programmes of  measures  with 
a  variety  of  objectives,  there  is  a particular  need  to  evaluate  not  only  the 
overall  employment  effects  of  these  programmes,  but  also  the  impacts  of 

Evaluating  impacts  on  employment 

each  component  measure.  Moreover,  in  many  cases  it  is  necessary  to 
evaluate both the general impacts on employment  (e.g. how many new jobs 
have  been  created?)  and  impacts  related  to  more specific  objectives  (e.g. 
how many jobs are now concentrated in a particular region or held by young 
people? - regardless of whether the total number of jobs has increased). 

To fully appreciate the effectiveness of interventions it is of the utmost  impor­
tance that  net  impacts  are  estimated.  Consequently,  this  means  that  the 
creation or maintenance  of employment that would  have taken place in the 
absence  of  an  intervention  has to  be taken  into  account.  Similarly,  allo­
wances  must  be made for  employment  that  is displaced  elsewhere  in the 
programme area and for  individuals who are eligible for  assistance who are 
substituted for non-addressees. Finally, the wider repercussions of an inter­
vention on suppliers  of recipient firms and the increased purchasing  power 
of assisted individuals should be estimated when this is feasible. 

Continuous  interaction  between labour  markets and other types of  markets 
(i.e. goods, services  or financial  markets)  or  markets  outside the  defined 
geographical  area  also  has  to  be  taken  into  account.  For  example,  an 
increase in exports due to favourable economic  conditions overseas  would 
have a positive effect on employment creation. Another example could be a 
period  of  relatively  low  interest  rates  that  may  have  positive  effects  on 
investment  and  exports  with  consequent  benefits  for  employment.  These 
confounding factors should  be taken into account  if the effectiveness  of an 
intervention is to be accurately measured. 

Furthermore,  numerous  interventions  often  occur  at  a national  level. As a 
result, synergies may often be generated between Structural  Fund interven­
tions  and  national  initiatives. It  is therefore  useful  to  be able  to  identify  the 
existence of such synergies. 

Certain  interventions are specifically  focused on social groups and, in parti­
cular,  the  employability  of  individuals  in  employment  or  unemployed. 
Employability  is closely  linked to  the evolution  of the  skills of  individuals  to 
reflect  the  changing  needs  of  the  economy.  The  problem  of  maintaining 
employability  is particularly acute in former industrial regions where profound 
changes  in the  organisation  of  production  and corporate  strategies  have 
occurred  as a result  of  structural  change,  leading to the  obsolescence  of 
workers' skills. Box  18 presents several forms of skill obsolescence and cor­
responding training strategies for increasing the employability  of  individuals. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Box 18 - Different types of skill  obsolescence 

Form of 
obsolescence 

Related to 
the individual 

Related to 
the  conditions 
of the post 

Related to 
changes in 
the post 

Related to 
the firm 

Related to 
to the market 

Description 

Age, Illness, 
injury 

Insufficient 
use of skills 

Demands 
related to the 
type of job 
change 

Appearance 
of new compe­
titors undermi­
nes a business 

Employment 
in a given 
sector declines 

Response in terms 
of training 

Revision of 
skills already 
acquired 
or total 
re-quallfication 

Revision of 
skills already 
acquired 

Training for new 
requirements 
of the job 

Development 
of existing 
skills or total 
re-qualification 

Training for 
similar jobs in 
In a new sector 
or total 
re-quallfication 

In order to  be clear about the type of employment  effects  under discussion, 
reference will be made throughout this chapter either to job creation or main­
tenance effects  produced  by interventions  in non-human  resource  domains 
or to  employability  effects  produced  by  human  resource type  interventions 
(mainly  concerning  training).  The  essential  difference  between  these  two 
types  of  effect  is that  in the  first  instance  employment  effects  concern  the 
creation  or  maintenance  of  a  post  which  may  be  occupied  over  time  by  a 
number  of  individuals. In the second, employment  effects  concern  maintai­
ning  or  improving  the  employability  of  individuals  so  that  they  can  either 
remain in a post at a certain level of qualification (i.e. in the case of training of 
workers  in  employment),  or  have  a  better  chance  of  matching  the  skills 
requirements of  potential employers and therefore of finding a job (i.e. in the 
case  of  training  for  the  unemployed).  The  characteristics  of  these  different 
types of employment effect are summarised  in Box 19. 

Box  19 -  Breakdown  of different types  of employment  effects 

Employment  effects 

Job  creation  /  maintenance 
effects 
(Related  to a  post) 

Employment  created  or  maintained 
by  the  process  of  implementation 
of  SF interventions  or  as a  consequence 
of  interventions  where job  creation 
or  maintenance  is an  explicit 
or  implicit  objective 

Employability  e f f e c ts 

(Related to  an  individual) 

Employment  maintained  (existing 
employees)  or found  (unemployed 
persons)  as a consequence  of  SF 
interventions  targeted  specifically  at 
maintaining  or  improving  the  employability 
of  individuals  through  the  development 
of  skills  and  qualifications 

Evaluating impacts on employment 

■ Defining what needs to be measured 

How employment impacts are produced 

Many interventions are not explicitly targeted at the creation of employment 
effects; they are therefore unlikely to have a substantial direct effect on 
employment. Other interventions are targeted at the development of human 
resources, including training and research which produce only limited direct 
employment effects in the short term. Consequently, the most significant 
employment effects resulting from the use of the Structural Funds are often 
indirect and tend to arise in the medium to longer term. They can therefore 
be relatively difficult to isolate and measure. This situation is further compli­
cated by the fact that certain interventions may well create negative employ­
ment effects initially, before any positive longer term impacts occur. This is 
particularly the case with interventions targeted at increasing productivity 
and competitiveness. 

The ways in which structural interventions may influence employment 
patterns are listed in Box 20. While it may be argued that the key analytical 
distinction to be made in relation to employment effects is whether they 
result from changes on the supply­side or the demand­side of the economy, 
the types of effects listed serve as a more practical reference to ensure that 
no effects are overlooked. 

The employment effects presented in Box 20 are situated at different points 
along the chain of causality, it is important to understand the complexity of 
the causal mechanisms that link interventions with their eventual employ­
ment impacts. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Box 20 - Production of employment effects through structural interventions 

•  As  a direct  consequence  of the  implementation  of  an intervention (e.g. 
construction, administration and training jobs), 
•  As  a  direct  consequence  of  assistance  provided  i.e.  through  job 
creation  or  maintenance  in  companies  which  have  been  established  or 
which have grown, 
•  As a consequence of training  measures targeted  at the employability  of 
existing workers and the unemployed, 
•  As  a  consequence  of  interventions,  which  may  or  may  not  have  an 
explicit  and  immediate  employment  objective,  either  through  furnishing 
an intervention with goods and services (e.g. providing building materials, 
office  supplies,  etc.)  or  as  a  result  of  improved  efficiency  in the  local 
economy  (e.g. hauliers employed to ship goods, the export of which  have 
been  made  competitive  due to  decreased  transport  costs  resulting  from 
the use of new  infrastructure), 
•  In response to increased demand generated by any increase in income 
resulting  from  the  intervention  ("income multiplier  effects"-  e.g.  increase 
in  local  retail  employment  or  local  house  construction  to  meet  the 
increased demand from the newly employed). 

These mechanisms  are presented  in Box  21 and detailed  in the  resume that 
follows it. 

Box 21 - Structural Fund interventions and employment effect mechanisms 

Low complexity of causal mechanisms 

High complexity of causal mechanisms 

(a) 
Construction and 
implementation 

(b) 
Employment 
subsidies 

(c) 
Training 

(d) 
Support for 
productive 
activities 

H 
Infrastructure and 
increased 
competitiveness 

(a) Construction and implementation leading to job creation 
Almost  all  structural  interventions  will  have  employment  as  a side  effect  in 
terms  of  construction jobs  (e.g. workers  employed for the construction  of a 
motorway)  or  implementation  jobs  (e.g. teachers  recruited  for  training  pro­
grammes). These effects may range from the almost  insignificant (i.e. certain 
financial  support  measures), to the  major  part of effects  (i.e. construction  of 
physical  infrastructure).  In this  case the  link  between  the  interventions  and 
the employment effect is also fairly direct. 

(b) Employment  subsidies leading to job creation 
Employment  subsidies for the long-term  or young  unemployed  are a typical 

Evaluating  impacts  on  employment 

example of policies producing employment effects as an immediate result. It 
is quite straightforward  to  connect  the  recruitment  of  a new  worker  to  the 
subsidy given to the employer. 

(c) Employment effects as a consequence of the improvement of skills 

Training  policies  are an example of  measures that  can have a complex  link 
with employment  effects. The rationale for training  policies  is that there is a 
mismatch between the skills of the available labour force and those needed 
by sectors with good growth prospects. Training is then supposed to ensure 
the employability  of the unemployed  or of workers whose skills are likely to 
become outmoded. In the case of the training for existing workers, employ­
ment effects are relatively direct. Less directly, this may lead to an improve­
ment in the quality of the workforce causing both productivity and the supply 
of trained workers to increase, moderating pressure on wages and inducing 
firms to eventually increase their level of activity and employment. 

(d) Induced or facilitated employment effects through support for  productive 
activities 

This  category  encompasses  all  jobs  created  by  companies  or  other 
economic actors in response to the policy, through the creation or growth of 
productive activities supported directly (with investment grants or subsidies) 
or  indirectly  (with  services,  provision  of  sites  etc.).  It  is therefore  the  most 
important channel through which employment effects occur. However, along 
with  other  types  of  interventions  that  create  employment  effects  through 
complex  causal  mechanisms,  impacts  on  employment  are  difficult  to 
attribute and measure. This is partly due to the fact that as the impact  chain 
from intervention to employment effects becomes more complex, numerous 
factors, other than the support  scheme, will be considered  by an economic 
actor before taking a decision on employment. 

A significant  example to  consider  here is the employment  impact  of  invest­
ment  subsidies  to firms. The resulting  increase  in the  level  of  investment 
should  lead to  an increase  in the economic  activities  of the  firm  and  to  a 
consequent  increase  in  employment.  However,  if  demand  is  static  or 
declining, there might well be a loss of employment. This will happen because 
any  measure  directed  to  the  better  use of  capital  equipment  will tend  to 
increase the productivity  of fixed  capital, making  labour  less competitive. In 
conditions  of  static  demand  this  will  lead to  the  substitution  of  labour  by 
capital. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Support  for  productive  activities  also  has  indirect  effects  (income  multiplier 
and  supplier  effects)  with  consequent  employment  effects.  Further  steps 
have  to  be  taken  along  the  impact  chain  and  it  is thus  more  complex  to 
disentangle such effects. 

(e)  Employment  effects  as  a  consequence  of  increased  competitiveness 
arising from new  infrastructure 

It  is also  possible  that  employment  effects  might  arise from  the  increased 
competitiveness  of the assisted economy  owing to  material (e.g. roads, tele­
communications  etc.)  and  immaterial  (e.g. business  services,  research  and 
development  etc.)  infrastructure.  These  long-term  effects  are,  however, 
produced  by a complex and indirect  mechanism, situated close to the end of 
the impact chain, and are difficult to identify and connect to a specific interven­
tion. 

Not only do the Structural Funds produce employment  effects  in a variety of 
ways, but the types of employment  effect  produced  also vary. In addition  to 
the distinction  between job creation or maintenance  effects and employabi­
lity effects, two other important distinctions can be made. 

When carrying out  an evaluation  it is necessary to distinguish  between sus­
tainable  (or  durable)  employment  (i.e. jobs  resulting  from  an  intervention 
which will continue in the absence of public support) and employment  which 
are likely to  be of a transitory  nature (i.e. jobs which cease to exist when  the 
funding  stops).  Moreover,  while  employment  in the  implementation  of  an 
intervention  is usually temporary, experience  has shown that  only a  propor­
tion  of  so-called  permanent  jobs  can  be  expected  to  last,  even  in  the 
medium-term  (e.g. a significant  proportion  of jobs  created  in newly establi­
shed  enterprises  cannot  be expected  to  survive for  long). When  measuring 
employment  effects, it is important  to specify the  nature of the jobs  created 
or maintained and  not to give a false impression of the long-term  security of 
any jobs  which  are a consequence  of  the  intervention. Some jobs  are  only 
created for a short  duration  (temporary  employment),  and are often  related 
to  the  implementation  of  the  intervention.  If aggregated  figures  are  to  be 
produced  then,  rather  than  simply  counting  the  number  of  jobs  that  are 
created by an intervention, it is more accurate to use expected job-years as 
a unit of measurement. Time discounting  of expected job years can be used 
to estimate their present value. 

Evaluating  impacts  on  employment 

Furthermore, the  employment  effects  of  an  intervention  may  include  both 
part-time and full-time jobs. Once again this  distinction  needs to  be made. 
Consequently, two different types of figure are useful: first, full-time equiva­
lents (FTEs) which  is the  aggregation  of  part-time  employment  so that  it is 
expressed as the equivalent  number of full-time positions (e.g. one half-time 
and two  quarter-time jobs  together  being  considered  as the  equivalent  of 
one full-time job)  and, secondly, the simple number of jobs  generated (e.g. 
one half-time  and two  quarter-time jobs together  being  considered  as the 
equivalent  of  three jobs)  which  better  reflects  the  objective  of  combating 
unemployment. 

•  Clarifying employment objectives 

When evaluating  the employment  effects  of  interventions,  reference  needs 
to  be made to the objectives  of the  intervention  in question  and  how  they 
relate to employment.  In practice, neither the objectives of each component 
of  a  programme  nor  the  connections  between  different  levels  of  a 
programme are always articulated as clearly and explicitly as they might be. 
The  need  for  clarifying  the  multiple  levels  of  objectives  in  a  programme 
supported  by  the  Structural  Funds  is  all  the  more  necessary  when  the 
uncertain implications for employment of individual types of intervention are 
considered. 

For  example,  improvements  in  transport  infrastructure  serving  remote 
regions  might  lower  the  costs  for  exporters  from  such  regions  and  allow 
easier access to support services, but they might equally expose the regions 
concerned  to  even  stronger  competition  and  facilitate  the  seasonal  or 
permanent  emigration  of  the  labour  force.  Similarly,  assisted  training 
schemes should generally increase workforce qualification  levels, producti­
vity and the attractiveness of a particular location for inward investment, but 
in some circumstances they might lead to increased emigration of the newly 
qualified in search of better career prospects. 

If the nature of the employment effects expected from an intervention are to 
be assessed and appropriate techniques for their measurement applied, it is 
important  for the Commission to  work  closely with those partners  involved 
on the  national,  regional  and  local  level. Together  they  should  make  the 
specific  objectives  relating  to  employment  effects  at  all  levels  of  each 
programme as explicit and clear as possible. This task is of particular  impor­
tance during the ex ante phase of programming. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

However, it is commonly found, especially in the cases of programmes  which 
run  over  a  relatively  long  period,  that  absorption  problems  or  changing 
economic  situations  can  lead  to  a  need  for  refocusing  in order  to  better 
achieve their  desired  impact  on employment,  or for that  matter on any  other 
priority, and to refine objectives originally specified in individual  programmes. 
Objectives related to employment must therefore not only be clearly specified 
at the outset  of  programmes, they must also be open to  modification  in light 
of new detailed political priorities or changing economic  conditions. 

The construction  of an objectives tree, preferably  at the ex ante phase, can 
be used to clarify the employment  objectives  of a programme. By way of an 
example, a tree of employment  objectives has been constructed to  illustrate 
the  use of  this  tool  in the  context  of  interventions  to  promote  employment 
(see Box 22). 

Box 22 -  Example  of a  thematic 

ctives  tree for  the 

promotion  of  employmc 

abje 
int 

Global  objectives 

Specific  objectives 

Reduce 
disparities  in 
unemployment 
between 
the  regions 

Increased  employment 
potential  of  SMEs 

Increased  employment 
opportunities 

•  Promote  new  business  creation 
•  Promote  investment  by SMEs 
•  Promote  innovation  by SMEs 
•  Promote collaboration  between  SMEs 
and research  institutes 
•  Improve consultancy  services  for  SMEs 

•  Diversify  the  range of products  and  services 
of  regional  firms 
•  Increase  export opportunities  for 
regional  firms 
•  Attract  Inward  investing  enterprises 

Increased  employability 
of workers  already 
In  employment 

•  Adapt  existing  employees  skills  and 
qualifications 
•  Develop  professional training  activities 

Increased  integration  of 
special groups  into 
the  labour  market 

•  Retrain the  long-term  unemployed 
•  Improve  the  qualifications  of  young  people 
•  Improve training  opportunities  for  women 

Evaluating  impacts  on  employment 

In this  example, which  is in fact  a composite  of  different  types  of  interven­
tions not normally found within the context of a single programme, interven­
tions  are targeted  at  a  range  of  specific  objectives  which  themselves  are 
subordinate to four  intermediate objectives: increasing employment  oppor­
tunities,  increasing  the  employment  potential  of  SMEs  (interventions 
targeted  at the productive environment), integrating special groups  into the 
labour  market  (interventions  targeted  at training  for the  unemployed),  and 
increasing the  employability  of  existing  workers. The global  objective  is to 
reduce regional disparities in unemployment. 

•  Estimating net employment impacts of an intervention 

It is important that only those impacts that genuinely arise from an interven­
tion are attributed to it. To do this, the net employment impact of an interven­
tion should be estimated. 

A first step  in estimating  net employment  impacts  is to construct the coun­
terfactual situation  in order to estimate the proportion of gross  employment 
impacts  which  would  have  occurred  in the absence  of  an  intervention (i.e. 
deadweight). 

The  gross  employment  impact  of  an  intervention  is  defined  as the  total 
number  of  jobs  created  or  maintained  or  the  number  of  people  passing 
through training courses, counselling  sessions, etc. who  remain  in employ­
ment  or  find  a  job.  In  practice,  data  on  gross  employment  impacts  is 
collected  by monitoring  systems  and tends to be  used as the basis for  this 
figure. 

When  interventions  focus  on training  either  existing  workers  or the  unem­
ployed, the  deadweight  effect  is defined  as the  number  of  people  passing 
through a scheme who would  have remained in employment  or found a job 
without  receiving  the  training. The  counterfactual  situation  in this  case  is 
usually  estimated  by establishing  a control group. The profile of  addressee 
employees is important  in establishing the extent of this type of  deadweight 
effect.  For example, the training of  management  or other  highly  skilled  staff 
is more likely to occur without the support of interventions. Similarly, the size 
of the firm may be a determining factor, since larger firms tend to  undertake 
more training activities than SMEs. Deadweight  is subtracted from the gross 
impacts of an intervention. 

Displacement  can occur  in the case  of  area-based  interventions, when  the 
creation or maintenance of jobs takes place at the expense of jobs elsewhere 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

in  the  programme  area.  An  example  of  displacement  is  of  companies 
assisted  by an intervention  expanding  their  business and their workforce  by 
taking business from other companies  within the programme area. Displace­
ment should be subtracted from gross impacts when the effect is unintended. 

When interventions are targeted at particular groups of addressees (e.g. long 
term  unemployed, working  mothers, young  people, etc.), the substitution  of 
addressees  for  non-addressees  may  in  fact  explicitly  be  sought,  for 
example, with better trained addressees finding employment  at the expense 
of non-addressees on the labour market. Similarly, interventions which expli­
citly  aim to  create  or  maintain  employment  by  supporting  the  development 
of  a  sector  at  the  expense  of  another  can  be  considered  as  substitution. 
When substitution  is the  explicit  objective  of  interventions,  it should  not  be 
subtracted  from  gross  impacts.  If,  however, the  intervention  was  not  expli­
citly  aimed  at  creating  this  effect,  substitution  should  be  subtracted  from 
gross impacts. 

Example: A training scheme for business creation: from gross to net effects 
A training scheme for  business  creation  might  lead to the creation  of some 400  busi­
nesses of which only 250 businesses were still active two years later, with 50 of  these 
employing the owner  half time only. Assuming that  surviving  businesses created  350 
additional jobs (since the aim of the intervention was to create sustainable jobs, it was 
considered  that  two  years  was  an adequate  time  scale  to  judge  sustainability),  the 
gross number of jobs  at the end of this period would  amount to 300 (employees)  plus 
250 (business owners), a total of 550 gross jobs. However, because  100 jobs are half-
time,  the  number  of  gross jobs  created  amounts  to  525  FTEs  (200  + 50 /  2  +  250  + 
100/2 = 525). 

If,  however,  deadweight  is estimated  to  be 20%, and the displacement  of  businesses 
because  of  the  support  given  to  recipient  businesses  is estimated  at  30%, the  net 
effect  of  the  intervention  will  be  262.5  net  jobs  in terms  of  FTEs  (525  -  [525x0.3]  -
[525x0.2] = 262.5) 

The differences between these concepts as they relate to job creation or mainte­
nance effects or employability effects are summarised in Box 23. 

In addition to these different  effects  employment  may be generated  through 
income multiplier effects (i.e. employment which is created / maintained  indi­
rectly  by the  fact  that  the  intervention  results  in additional  income  for  local 
consumers) or supplier effects (i.e. employment which is created or maintai­
ned as a consequence  of  backward  linkages between addressees  and their 
suppliers) which tend to  be separately  identified. However, these effects  are 
virtually impossible to quantify for individual regional  interventions. 

Evaluating  impacts  on  employment 

Box 23 -  Comparison  of deadweight,  displacement  and 

substitution  effects 

Definition for job creation 
/ maintenance 

Jobs which would have been 
created or maintained in the 
absence of the Intervention 

Jobs created or maintained by a 
company in the area targeted at the 
expense of other companies 
In the programme area 

Deadweight 

Displacement 
(area-based 
interventions) 

Substitution 
(social group-based 
Interventions) 

Definition for 
employability 

Retrained individuals who would have 
remained in employment or found a job 
In the absence of the intervention 

Companies employing or recruiting 
retrained individuals which take 
business and reduce employment In 
other companies in the programme area 
which have not retrained their employees 
or recruited retrained people 

Retrained individuals who remain 
In existing posts instead being 
replaced by previously better qualified 
unemployed persons 
Retrained unemployed individuals 
belonging to specific target groups who 
are substituted for non-addressees 

The impact  of  an intervention, either  positive or  negative will not always  be 
directly  translated  into  an effect  on  unemployment  of  a similar  magnitude. 
This  is  because  any  changes  in  the  number  of  people  actively  seeking 
employment  need to be taken into account. 

One such  effect  is the  impact  on  activity  rates  that  is  induced  by  new  job 
opportunities,  the  so-called  attraction  effect  (or added  worker  effect).  For 
example, new part-time jobs may attract women or retired people who were 
not registered as unemployed before the intervention occurred. 

Another effect on the supply of labour is that of migration: new job opportuni­
ties may have an impact on both immigration (by attracting some  individuals 
to  move into the target area to obtain work) and emigration  (by reducing the 
number who would otherwise leave the target area in search of work). 

In this context, some effects on travel-to-work  patterns may be considered as 
migration effects and these may be even more significant than more permanent 
migration flows since greater numbers of people outside the target area may be 
encouraged  by new job opportunities  to simply  commute  into the area. The 
smaller the target area, the more likely it is that new jobs will go to  individuals 
who were not part of the local supply of labour before the intervention. 

Transversal evaluation of impacts on intervention priorities 

■ Examples of impacts by area of intervention 
In order to illustrate how interventions can lead to employment impacts, 
examples are presented below which correspond with the three broad areas 
of intervention of Structural Funds. 

• Basic Infrastructure 

The aim of interventions in this area is largely to relax constraints on economic 
development resulting from the absence or poor quality of basic infrastruc­
ture. Such interventions do not, therefore, have explicit employment objec­
tives. Job creation and maintenance effects are either of a temporary nature 
during implementation, or indirect due to increased economic activity. 

Example: T he upgrading of analogue telephone e x c h a n g es (Greece) 

The public s w i t c h ed t e l e c om network in Greece w as almost 1 0 0% analogue at the 

beginning of the 1990s. As a result, the network suffered from low quality and reliabi­

lity. One of the interventions designed to improve the network has been the progres­

sive replacement of existing analogue e x c h a n g es w i th new digital e x c h a n g e s. T he 

objective of this p r o g r a m me is to reach a level of 6 8 ­ 7 0% digitisation of e x c h a n g es by 

the year 2 0 0 0. As e x c h a n g es b e c o me d i g i t i s e d, f i r ms a nd h o u s e h o l ds are able to 

benefit g l o b a l ly f r om i m p r o v e m e n ts in t he n e t w o r k. T he m o d e r n i s ed n e t w o rk will 

p r o v i de f i r ms w i th t he o p p o r t u n i ty to t a ke a d v a n t a ge of t he n ew p o s s i b i l i t i es for 

business offered by t e l e m a t i c s. While m o st e m p l o y m e nt effects will be t e m p o r a r y, 

occurring during i m p l e m e n t a t i o n, over the longer term it is possible that permanent 

e m p l o y m e nt opportunities will arise as firms take advantage of the market possibilities 

that the improved infrastructure offers. 

Outputs ~^> 

Effects 

Telephone exchanges 
re­equipped 

Gross temporary jobs 
created or maintained 

Gross jobs created 
or maintained 

Reduction In 
network faults 

Firms potentially 
served by 
modernised 
exchanges 

Firms using telematics to 
expand into new markets 

Increased 
production, 
sales, etc. 

Priority 

Evaluating  impacts  on  employment 

•  Productive  environment 

For  some  regions  the  aim  of  Structural  Fund  interventions  is to  assist  the 

reconversion  or diversification  of the economy  in order to combat  structural 

unemployment.  In  rural  regions,  diversification  of  productive  structures  to 

reduce  dependence  on  agriculture  and  to  encourage  tourism,  a  highly 

labour-intensive  activity,  is an  essential  element  in  a strategy  of  economic 

development. 

Example: Developing tourism in the Republic of Ireland 
In Ireland, an Operational Programme in Tourism was implemented from 1990 with the 
generation  of employment  effects  (the creation  of an additional  25 000  jobs)  as its 
global  objective.  One of the  specific  objectives  of interventions  was to diversify the 
range of tourism  products. One of the  product themes  chosen as a suitable target of 
intervention was the development  of conference  travel to identified tourism  centres. 
During  the 1989 to  1993 period  twelve  new facilities  for conferences  were co-
financed. The construction of conference  facilities  produced  temporary  employment 
effects  and an eventual impact on the growth and seasonality of the activities of local 
businesses such as hotels, restaurants, conference services, etc. 

Outputs 

> 

Effects 

Gross temporary  jobs 
created or maintained 

Construction 
of  conference 
centres 

Organisation of 
conferences and 
events 

Increase in 
non-seasonal and 
bus mess  tourism 

1 

New  activity  generated 
for the  region's  businesses 

t 

Priority 

J\ 
,  > 
1/ 

Gross jobs 
created or 
maintained 

k 

, 

Transversal evaluation of impacts on intervention priorities 

• Human resources 

Interventions in the area of human resources focus on maintaining or 
improving the employability of individuals through enhanced skills and 
abilities, and on supporting Research and Technological Development. 

Example: Support for vocational training in t he North East of England 

The North East of England, o n ce a major industrial region of the UK, has lost over 200 

000 j o bs in traditional sectors (coal mining, shipbuilding, steel making and heavy e n g i­

neering) since 1970. At the s a me t i m e, high t e c h n o l o gy activities in the region have 

d e v e l o p ed in such s e c t o rs as pharmaceuticals, biotechnology, information t e c h n o­

logy and electronics. One of the factors contributing t o w a r ds the emergence of new 

specialisations is the strong links b e t w e en industry and the region's universities and 

o t h er higher e d u c a t i on i n s t i t u t i o n s. S o me of t he m e a s u r es in f a v o ur of industrial 

reconversion of the North East (CSF 1989­91 and 1992­93) are aimed at vocational 

training. These measures include the provision of facilities for training in industrial 

sectors with a high g r o w th potential and especially in technologies having an indige­

nous base in the region. 

Outputs 

> 

Effects 

C> 

Gross temporary jobs 
created or maintained 

Gross jobs created or 
maintained 

/ 

Training projects 
prepared in 
partnership 
with local industry 

Unemployed trainees 
J being offered employment 

during the course 

Trainees successfully 
completing the course 

^ Unemployed trainees 
finding employment 

λ 

* Recipients promoted 

. Employees remaining 

in employment 

Priority 

■ Defining a nd selecting indicators 

Volume 2 of the MEANS Collection is devoted to the subject of indicators for 
monitoring and evaluating Structural Fund interventions. This Volume 
includes an information sheet on indicators for the priority of employment. 

Whilst evaluations are not limited to an analysis of indicators alone, they are 
greatly facilitated by the existence of relevant indicators. The subject of indi­

Evaluating  impacts  on  employment 

cators  and  their  quality  needs  to  be considered  at two  different  levels  in 
relation to employment effects: 
•  Since the promotion of employment is one of the general objectives of the 
Structural Funds, the ongoing development of a system of indicators should 
result in a strengthening of information on the relative employment effects of 
the different components of a programme. 

•  The issue of  indicators  also arises  in relation to more detailed  analyses of 
effects. Any evaluation which focuses on the subject  of employment  effects 
requires  ad-hoc  employment  indicators  to  be  defined  and  selected  in 
relation to the specific evaluation questions being asked. 

In order to evaluate the employment effects of the Structural Funds, it is conse­
quently useful, first, to ensure that some basic and reliable indicators of employ­
ment are included in a programme's system of indicators and secondly, in the 
context of any particular evaluation exercise, to define those central indicators 
which respond to the specific evaluation questions being posed. 

When  measuring  employment  effects,  account  should  be  taken  of  the 
temporary  jobs  needed to  implement  the  interventions  (e.g. jobs  related to 
the  building  of  infrastructure, advisory  services for  businesses  or training). 

Whilst  output  indicators  do  not  focus  directly  on  employment,  they  are 
usually  essential for  an analysis  of the  production  of  employment  effects. 
This  is  particularly  the  case  when  interventions  result  in the  provision  of 
services such as training, information or advice which, due to their intangible 
nature, can be difficult to measure. 

Effort  should be concentrated  on the establishment  of those indicators that 
provide  information  on the  objectives  which  are most  significant  as far  as 
employment  is concerned. These indicators  refer for the most part to  gross 
direct effects. 

The objectives  of  employment-related  structural  interventions  are  diverse 
and the  definition  of  systems  of  indicators  must  incorporate  this  diversity. 
There is no one universal indicator  of employment  effects  which is  relevant 
regardless of the type of evaluation being carried out. For example, if the aim 
of  an intervention  is to  promote the  return to work  of those who are unem­
ployed in an urban area that is being redeveloped, the number employed  in 
this urban area will not in itself  be sufficient as an indicator. Since it does not 
enable the proportion of jobs occupied  by residents of the area to be deter­
mined, the effectiveness of the intervention cannot be judged. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Finally, selected indicators must be precisely defined. For example, the defi­
nition should  state whether a job created or maintained  includes all types of 
job,  or  are  part-time  and  full-time  jobs  distinguished?  How  is  a  job  is 
qualified as sustainable? Is it a job that is expected to remain in existence for 
more  than  two  years,  more  than  five  years,  etc.?  How  are  specific  social 
groups  targeted  by  interventions  defined?  For  example,  the  information 
"30%  of the jobs created by the project  benefited  marginalised people" is of 
little use if the concept of marginalisation  is not clearly defined. 

•  Indicators for a more in-depth analysis of employment  effects 

Apart from the central employment  indicators  in a system  of  indicators,  it is 
often relevant during an evaluation to use ad hoc indicators in order to  make 
a  more  in-depth  analysis,  particularly  of  impacts. Any  such  use of  supple­
mentary  indicators  needs to  be  consistent  with  the  questions  that  the eva­
luation is required to answer, the  particularities  of interventions and the cha­
racteristics  of  the  region  being  evaluated. Supplementary  indicators  might 
therefore focus on: 

•  the proportion  of jobs  created  or maintained  which are unskilled / skilled  / 
highly skilled; 

•  the  remuneration  of jobs  created  or  maintained  (e.g. the effect  will  not  be 
the same if 10 jobs remunerated  at the legal minimum rate or 10 higher  level 
posts are created); 

•  the personal earnings of addressees; 

•  the  profile  of  the  addressees  (even  when  the  objectives  do  not  refer  to 
target  groups,  it  may  be  interesting  to  know  who  is  benefiting  from  job 
creation). An assessment  of compliance with the principle of equal opportu­
nities is relevant at this level; 

•  the career paths of addressees. In addition to knowing that they have been 
re-employed,  it  may  be  interesting  to  know,  for  example,  how  long  it  took 
them to get a job, how appropriate the job is in relation to their qualifications, 
or even what  effect the  intervention  had on their approach to seeking  work; 

•  the type  of the jobs  created  or  maintained. Making  a distinction  between 
part-time  and  full-time  jobs  is  one  way  of  making  inferences  about  the 
quality of jobs created or maintained. However, such inferences have to take 
into  account  the  fact  that  some  addressees  may  prefer  to  work  part-time 
rather than full-time.  Differences  in national  employment  and social  security 
legislation are also key determinants  of the quality of  part-time and full-time 
employment.  It  may  also  be  useful to  look  at the  proportion  of  addressees 

Evaluating  impacts  on  employment 

working  for  temporary  employment  agencies, the  rate  of  unionisation  in 
firms where jobs are created and make comparisons  of wages in respect to 
sectoral / industrial and regional averages. 

Because numerous measures co-financed by the Structural Funds have only 
a long-term effect on employment, apart from the temporary jobs which are 
needed  to  implement  them  in the first  place,  results  and  intermediate  or 
specific impact indicators tend not to reflect employment effects. This is par­
ticularly the case of  interventions  in the areas of  infrastructure and  produc­
tive environment.  For these types  of  intervention  the  more immediate non-
employment  impacts can be converted  into  predicted  employment  effects. 
For example, by dividing the turnover resulting from the intervention, by the 
average additional turnover  needed to create a full-time job, it is possible to 
estimate the maximum level of jobs indirectly created. This method is useful 
in ex ante evaluation to quantify the expected effects of interventions. It may 
also  be  useful  in  intermediate  evaluation  to  assess  indirect  long-term 
employment effects from established intermediate impacts. The conversion 
coefficients  used in such an exercise are determined  in relation to  previous 
experience,  research  or  econometric  models. Whenever  such  coefficients 
are used, the basis of their definition should be clearly indicated. 

When interventions  in the area of  human resources focus on maintaining or 
improving the employability of individuals, employment impacts occur  much 
earlier in the causal chain. In this case the choice of  indicators depends on 
the  definition  of  employability.  An  operational  approach  is to  associate 
employability  with the maintenance of employment  of individuals already in 
work or the recruitment of individuals out of work. 

When  studying  the  maintenance  of  employment  of  existing  workers,  it  is 
necessary  not only to take  into  account  the fact  that  employment  may  be 
maintained  within  a firm  or  sector  (in the case  of  an  industry  level training 
programme), but also that  it may  be maintained  in a wider  inter-firm or inter-
sector perspective (i.e. employment maintained at a comparable level of quali­
fication outside the firm or industry under consideration). The wider the pers­
pective  of  employment  maintenance, the  more problematical  its  evaluation 
since the development  of an individual's situation  is more difficult  to  monitor 
outside  a firm or area of  activity.  Furthermore, the  links between the level of 
qualifications, jobs and salaries can vary greatly, making comparison  difficult. 

The in-depth  analysis  of  employment  impacts  also  requires  an analysis  of 
confounding  factors,  i.e. those factors  over  which  the  programme  has  no 
control  and  which  have  an  influence  -  whether  positive  or  negative  -  on 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

employment  or unemployment. The main effects of confounding factors can 
be identified by means of indicators. 

Indicators  of  adverse  effects  on  employment  may  also  be established (e.g. 
number  of  redundancies  in the  agricultural  sector  due  to  rationalisation  of 
farming  enterprises).  These  serve  as  warnings,  or  to  enhance  knowledge 
about the harmful effects on employment  of the measures for which there is 
no stated employment  objective. 

Indicators  may  also  be  used to  make  a judgement  in terms  of  efficiency,  in 
which  case  they  relate  result  or  impact  indicators  to  the  resources  used, 
producing  a cost /  effectiveness  ratio which  may  be compared  to  reference 
data.  Examples  of  this  type  of  indicator  include  the  average  cost  per  job 
created or maintained. 

•  Precautions to take when using and interpreting indicators of 
employment  effects 

The  accurate  evaluation  of  employment  effects  requires  that  the  time 
needed for the effect to be produced should be taken into account, so as to 
avoid producing invalid information which might be based on the under-esti-
mation  of  expected  effects  (i.e. not  taking  full  account  of  effects  which  will 
subsequently  occur) and / or their over-estimation  (i.e. not taking account of 
the limited duration of temporary  effects). 

In the case of intermediate evaluations, employment  indicators are meaning­
ful above  all for  counting  the  number  of temporary  jobs  created  or maintai­
ned  by  the  implementation.  Impact  indicators  must  take  account  of  the 
delay  that  occurs  before  employment  impacts  are  produced, which  varies 
depending on the type of intervention being studied. 

A single indicator may be used to measure several levels of effects  if used at 
different  points in time. In the case of an intervention aimed at attracting new 
businesses to the area under redevelopment, the  number  of jobs created or 
maintained  might  include: an  indicator  of the jobs  required for the  develop­
ment being undertaken to attract  businesses to the area, an indicator of the 
result of the intervention  if it measures the number of jobs  provided by firms 
when they  set  up, and  an  impact  indicator  if  it  counts  the  number  of  direct 
and indirect jobs created in the area. 

Finally, when comparing  information  provided by an indicator, it is important 
to ensure that the comparison  is relevant. For example, a comparison on the 

Evaluating impacts on employment 

basis of "cost per job created" is only relevant if the context of the reference 
situation is the same. 

By way of an example, a system of indicators that could be used to track the 
production of employment impacts generated by a training programme for 
the unemployed is presented in Box 24. 

Box 24 - Example of a system of indicators for a return-to-work 

programme 

Programme Indicators Context Indicators 

(Related to the Intervention and its effects) (Related to the assisted area) 

Output Indicators 

Number of 
addressees 

Result indicators 

Rate of Immediate 
transition 

Impact indicators 

Gross placement rate 

Net placement rate 

Quality of jobs 

Number of long term unemployed 
participating In the programme 
(of which women) 

Number of addressees who are either 
hired or In a better social situation at the 
end of their participation in the 
programme (of which women) 

Number of addressees hired after 
12 months (of which women) 

Number of long-term 
unemployed in the region 

Gross rate less estimated deadweight 
(trainees who would have been hired 
without assistance) less unacceptable 
substitution (trainees whose hiring has led 
another worker to long-term unemployment) 

% of hired addressees (of which women) 
who's job Is full-time and permanent after 
12 months 

% of jobs In the region that 
are full-time and permanent 

Source: Volume 2 of the MEANS Collection 

■ Sources of useful information 

Numerous series of statistics have been established at the European, 
national and regional levels. At the Community level, a Labour Force Survey 
is conducted annually throughout the Union. The current series of surveys 
was launched in 1983 and a survey is carried out annually. The structure of 
the survey is such that the data presented is disaggregated by gender, 
allowing the equal opportunities dimension of employment effects to be 
integrated into an evaluation. Many other data are also available through this 
survey such as: educational levels, number of hours worked per work, the 
presence of young children, etc. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Category 

Disaggregation by : 

Unemployment 

Jobs 

• Gender 
•  Age group 

• Gender 
•  Professional status 
•  Field of activity 

Groups of job seekers 

• Gender 
•  Duration of unemployment 
•  Mode of job-seeking 

The  Regio  regional  database  covers  unemployment  and  labour  force 
surveys, and the results are disaggregated  by region. Statistics are available 
on  total  employment,  salaried  employment,  active  population,  long-term 
unemployment  and  remuneration  of  employees.  Data  are  available  from 
series which go back to  1970, 1975 or  1983. The reporting  period is annual, 
with  the  exception  of  certain  unemployment  statistics  which  are  monthly3. 
The statistical observatories  MISEP (Mutual Information System on  Employ­
ment  Policies)  and  SYSDEM  (System  of  Documentation  on  Employment) 
provide information on the different  national employment situations. 

Among the drawbacks of using statistical surveys is the problem of  non-cor­
respondence  between  place  of  work  and  place  of  residence  which  may 
create  particular  difficulties  in  the  interpretation  of  statistics  when  an 
analysis is focused on unemployment.  For example, due to commuting, the 
jobs  recorded  in a  region  are  not  all  occupied  by  residents  of  that  region. 
Statistics  may  therefore  indicate  a falling  trend  in  unemployment  levels, 
without  the  majority of the addressees  of that  improvement  being  residents 
of the supported area targeted by the intervention. This phenomenon  is par­
ticularly important when the assisted area is small (e.g. urban areas in indus­
trial  decline  that  are eligible  for  support  to  promote  reconversion)  or  when 
border regions are concerned. 

Permanent  statistical  surveys  on  employment  are  also  available  from 
national statistics institutes, labour and employment  administrations  (statis­
tics on unemployment  databases on human resources, etc.). However, a dif­
ficulty with this sort of data is the time  delay  between the collection  of data, 
its processing and eventual publication. 

Databases  such  as those  compiled  for  the  LEADER  Community  Initiative 
have  proved  to  be  equally  useful. The  LEADER  Observatory  gathers  data 

3. Labour force survey and data from the Regio data base can be obtained from Eurostat. 

Evaluating impacts on employment 

concerning the objectives of projects (including employment) and elements 
concerning counterfactual situations. 

There has been a surge of interest over the last few years in evaluating 
employment effects and evaluations in this field are rapidly evolving. For this 
reason an inventory of existing research reports and previous evaluations 
represents a valuable stock of knowledge on which evaluations can be built. 

The employment data provided by the evaluations of a programming cycle 
provide an extremely useful basis for estimations of effects of the next cycle. 
Estimations of employment effects which feature in programmes and ex ante 
evaluations should also be used in intermediate and ex post evaluations. 

Information produced by a monitoring system includes data that are 
necessary to evaluate employment effects. When the monitoring system 
comprises an entire set of indicators on resources and on physical and 
financial outputs, the extent to which the operational objectives of interven­
tions are being progressively attained and the associated employment of 
human resources during the implementation of interventions can be shown. 
However, it is important to bear in mind that monitoring can only, at the most, 
focus on addressees and would­be addressees (i.e. rejected applicants). 
This means that an analysis which focuses uniquely on monitoring data ­ for 
example, the construction of the counterfactual situation ­ will be biased. 
Furthermore, data gathering which is restricted to addressees and would­be 
addressees will be the subject of respondent bias. Nonetheless, monitoring 
data can be a useful source of information, since monitoring systems of 
most types of intervention include some information on employment. 
Examples of indicators that can be directly informed from the monitoring 
system include the employment generated by businesses attracted into the 
target area and the number of participants in a training programme who have 
obtained a qualification. 

■ Appropriate tools for analysing employment effects 
An assessment of the impact of interventions on employment can be underta­
ken using the "step­by­step" approach presented in Part IV of this Volume. For 
an in­depth evaluation of employment effects, a wide variety of evaluation 
tools can be employed for the analysis of employment impacts. While these 
tools are extensively dealt with in Volume 3 of the MEANS Collection, a 
number of examples, drawn from across the spectrum of evaluation expe­
rience in this theme, are presented in this section. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Tools for  in-depth  analysis of  employment  impacts  can  be divided  into  four 
categories: 

•  descriptive tools that allow for the treatment of both qualitative and quanti­
tative  information  (e.g. full case-studies,  including  longitudinal  studies,  and 
surveys), 

•  comparative  tools  based  on  observation  (either  between  similar  cases  or 
between the subject case and a randomised control case), 

•  analytical tools that  focus  only  on the situation  being  analysed  (e.g. shift-
share analysis), 

•  models  which  are  built  on the  basis  of  economic  analysis  of  other  expe­
riences  and  simulations  of  varying  degrees  of  complexity  (e.g.  multipliers 
and  coefficients,  input-output  and  econometric  models  which  combine  a 
number of  coefficients). 

Tools for quantitative analysis, such as shift-share analysis, do not only need 
to  be  used  with  skill,  their  limitations  also  need  to  be  acknowledged.  The 
measurement  of  their  employment  effects  needs  to  be  interpreted  in  a 
manner which takes account of qualitative factors. 

Both the  level and  scale of the  intervention  being  evaluated  have a  number 
of implications for the type of tool  most  likely to be appropriate for the mea­
surement  of  employment  effects. The  investment  of  resources  needed  to 
produce dependable  simulations  means that full econometric  models, such 
as  the  HERMIN  macroeconomic  model, are  generally  available  for  only 
higher-level, larger-scale  interventions  which operate at the national  level or 
across  very  large  regions. Accordingly,  such  approaches  are more  likely  to 
be  of  most  direct  use  in  the  evaluation  of  the  effects  of  programmes 
designed  to  promote the development  and structural  adjustment  of  lagging 
regions,  or  in the  evaluation  of  the  overall  effect  of  the  variety  of  activities 
typically funded by the Structural Fund on a national level. 

On the  other  hand, simple  coefficients  are rarely  robust  enough  to  transfer 
between  situations  which  include  a  large  number  of  variables.  Ideally  their 
use  should  be  confined  to  very  similar  interventions  with  relatively  direct 
employment  effects  (e.g. those  which  are  targeted  on  similar  groups  in 
similar and smaller regions undergoing industrial reconversion or rural zones 
in the process of development and structural adjustment). 

Evaluating  impacts  on  employment 

The same  limitations  apply  to  comparative  analysis tools, since the  more 
complex  a situation  becomes, the more difficult  it is to find other  situations 
against  which  comparisons  can  be made. They  are, however,  particularly 
useful for establishing the policy-off  scenario of small-scale human resource 
interventions. 

Job creation or maintenance effects lend themselves to analysis by a variety 
of  top-down  approaches  (i.e. relatively sophisticated  statistical  techniques, 
modelling tools, etc.) and bottom-up  approaches  (i.e. surveys, case studies 
etc.). As for  interventions targeted at employability  effects, these are relati­
vely small and should therefore be analysed using the bottom-up approach. 

•  Surveys and case studies: Evaluating Training and Enterprise Council 
(TEC) services (United Kingdom) 

This  evaluation  sought  to  assess  the  overall  cost-effectiveness  of  a 
programme  of  business  counselling,  skills  training  and  information  and 
advice for  existing  small  and  medium  sized  enterprises  (SMEs). Additional 
employment was only one of a number of objectives of the programme and 
the evaluation also took  increased output, improved productivity,  enhanced 
capital  expenditure  and  profitability  into  account.  With  regard  to  employ­
ment  effects, the objectives  of the evaluation  were  primarily  to  assess  dis­
placement and to explore the possibility of any crowding out effects. 

The evaluation used: 
•  a  postal  questionnaire  sent  to  the  management  of  75  English  TECs 
(concerning their activities, targeting, etc.); 
•  personal  interviews with 20 TECs to provide qualitative information on the 
inputs  to  their  activities  (e.g. strategy  developed,  benefits  resulting  from 
different service approaches, etc.); 
•  six  case  studies  mainly  focused  on  customer  benefits  and  on  the 
companies' own views on the impact of the services on jobs and output. 

The  questionnaire  was  designed  to  distinguish  between  gross  and  net 
benefits.  Following  the  conventional  path  to  identify  the  net  impacts  on 
employment, the study attempted to assess (gross) effects, deadweight and 
displacement  effects.  All  of  the  results  of  the  case  studies  on  the  local 
economy were aggregated to the national level and net benefits on the local 
level were carefully distinguished from the net impact at national level. To do 
this, it was necessary to take into account  national displacement  as well as 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

macro-economic  crowding-out effects since gains in one TEC area may lead 
to  displacement  in another  area, and  if all areas  show  increased  economic 
activities, benefits are likely to be crowded out (e.g. by higher inflation rates). 

•  Comparative analysis: Measuring the effects of Swedish  employment 
training on income and employment  (Sweden)  1995 

Two  experimental  groups  consisting  of  4,000 trainees  who  completed  their 
training  in  1992  and  6,000  who  completed  it  in  1994, were  included  in  the 
survey.  A  very  high  proportion  of  these  were  individually  matched  with  a 
control  group  which  received  no training  at the  same time  as the  subjects. 
Some  policy  changes  had  taken  place  between  the  training  of  the  two 
cohorts, so a detailed  study  of these  particular  changes was  also  possible. 

The analysts used a fairly sophisticated control function (including values for 
background  variables  and  error)  to  trace  the  relationship  between  training 
and  income. The findings  obtained  showed  significantly  different  results for 
the  two  years, with  the  1994  trainees  having  significantly  higher  earnings 
than those who had not received training, but the 1992 trainees  experiencing 
no  significant  advantage.  For  those  who  ended  up working  in the  vocation 
for which they trained, earnings were higher than for the control group. 

In order to  measure employment  effects, regression analysis was used. The 
findings reported showed that while the control group had found it increasin­
gly  difficult  to  find  work  during  the  period  when  the  subject  group  was 
training, those who had subsequently  been trained found  it no more  difficult 
to  obtain  work.  Thus,  even  if their  position  had  not  improved  in  absolute 
terms,  it was  relatively  better  than  it would  otherwise  have otherwise  been. 

•  Statistical analysis: An ex post evaluation of the integrated action 
programme for Limburg using shift-share analysis (Belgium) 

The  goal  of  the  programme  being  analysed  was  to  support  job  creation  in 
new  activities  to  compensate  for  the  decline  of  employment  in the  mining 
industry. The evaluators tracked  employment  change during the  1980s  and 
early  1990s  on  the  basis  of  sectoral  and  other  characteristics  for  both 
Limburg  and  Belgium  as  a  whole.  Shift-share  analysis  was  then  used  to 
estimate the  number  of jobs created  by the  programme  in comparison  with 
what  would  have  happened  in its absence, on the assumption  that the  rate 
of decline in the traditional sectors was equal to the Belgian average. 

Evaluating  impacts  on  employment 

The  relatively  positive  experience  of  the  Limburg  region  (high  levels  of 
economic  growth,  new  firm  creation  and  employment)  was  attributed 
primarily to export  growth  in one or two  sectors and exceptional  growth  in 
the  number  of  small  enterprises.  Re-employment  patterns  amongst  ex-
miners  were  put  under  particular  scrutiny  in  light  of  the  objectives  of  the 
intervention, as were the  investment  instruments  chosen. On the  basis  of 
this analysis, specific  proposals  were made for a modification  of these ins­
truments to increase employment effects in subsequent  interventions. 

•  Multipliers and coefficients: Evaluating labour support schemes 
using standardised coefficients (France) 

This approach is used for the ex-post  and on-going  evaluation of effects  on 
employment  and unemployment  of  labour support  schemes financed  by the 
French Government. It takes into account both deadweight and attraction effects. 

For  each  single  measure  a coefficient  of  deadweight  and  a coefficient  of 
attraction  are calculated, based on the evidence of past evaluations and on 
intuitive adjustments. These coefficients are then applied to the variations of 
the stock  of  addressees  of  each  measure,  over  a given  period  of  time,  in 
order  to  obtain  the  effects  on  employment  and  unemployment  of  each 
measure. The effects on employment correspond to the difference  between 
the variation  in the stock  of addressees and the quota of these  addressees 
(obtained  by  applying  the  coefficient  of  substitution)  who  would  have 
obtained a job in any event, or who substitute for another worker. 

Calculations  of effects  on unemployment  also take  into account  the coeffi­
cient of attraction and the absolute value of the reduction  in  unemployment 
resulting from the  difference  between  employment  effect  and the quota of 
new addressees attracted to work by the measure. 

This  macroeconomic  approach  is especially  valuable  since  it  allows  the 
effects  of  cyclical  changes to  be expressed  as changes  in the  coefficients 
over time.  It is globally  useful as it gives a complete view  of existing  labour 
support  schemes  and of  the  net  effects  of  each  one on  employment  and 
unemployment.  However, it has a limited time horizon and does not  provide 
much information about job tenure and, in general, about the quality of jobs. 
The method  is easy to apply once the coefficients  have been obtained, but 
establishing the coefficients  in the first  place demands a high level of  infor­
mation input. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

•  Input-output analysis: Evaluating an objective 1 Operational 
Programme in Italy 

The tool  used  for  this  exercise  is a  Social  Accounting  Matrix  (SAM),  which 
integrates  Keynesian  income  multiplier  mechanisms  into  a standard  input-
output  model. In a SAM the various  interdependent  links of an economy  are 
quantified and simulated. The impact mechanism relies on global multipliers, 
obtained  from  a wider  matrix  system, which  add  additional  rounds  to  the 
input-output  mechanism. The  global  multipliers  are  able to  capture  effects 
arising  not  only  from  the  relations  linking  the  productive  sectors,  but  also 
from other relations linking economic actors. 

The  increase  in factor  revenues  (value-added)  generated  by  input-output 
mechanisms  is translated into an increase in income. The level of consump­
tion  then  rises  (according  to  the  average  consumption  rate),  creating  an 
increase  in  final  demand  which  corresponds  to  a  new  impulse  to  the 
economy and to a new activation of the process. Effects on employment  are 
calculated  on the  basis  of  a  hypothesis  on the  evolution  of  productivity  by 
sector,  which  is  related  to  the  changes  in  value  added  obtained  from  the 
input-output  application. 

The reference model for this exercise is a two-region (Campania and the rest 
of  Italy)  SAM. The  regional  SAM  tables  are  obtained  from  the  1988  input-
output  table  for  Italy, which  is split  using  the  results  of  two  surveys  on  the 
economy  of  Southern  Italy:  one  on  the  purchase  of  intermediate  goods  by 
firms  from  this  area  and  the  other  on  income  and  consumption  of  house­
holds. The SAM  used  has a low  level of disaggregation: two  factors  (labour 
and  capital)  and two  types  of  actor  (firms  and  households).  With  regard  to 
sector  disaggregation,  two  versions  are  provided:  one  with  four  sectors 
(agriculture,  industry,  building  industry  and  services)  and  another  with 
sixteen  (with  industry  and  services  split  into  sub-sectors).  The  model  is 
applied  in two  steps,  each  with  different  procedures,  in order to  separately 
capture demand-side and supply-side  effects. 

•  Macroeconomic  modelling: The HERMIN model in Ireland 

The  macro-economic  model  HERMIN  has  been  developed  by  national 
teams  in four  objective  1 countries:  Ireland, where it was initially developed, 
Portugal,  Greece  and  Spain. This  project  is aimed  at  creating  a  system  of 
harmonised  econometric  models  for  the  evaluation  of  the  medium-term 
economic  performance of the European economies. 

Evaluating impacts on employment 

As econometric models go, the Irish version of HERMIN is relatively small 
with 50 equations, about 150 identities and 60 exogenous variables. The 
standard version of HERMIN has four sectors: tradable (manufacturing 
industry), non­tradable (market services, building and utilities), agriculture 
and public. In the two main productive sectors (tradable and non tradable), 
simple production functions are used, while the agricultural sector has not 
been modelled because of the exogenous factors related to the CAP, which 
influence aggregate farming behaviour. The public sector is policy­driven, 
with expenditure and tax rates set as policy instruments, subject to budget 
constraints and borrowing sustainability. 

The consumption function of the model (based on Keynesian multiplier 
mechanisms) does not play an exclusive role in driving this model as it does 
in demand­side focused models. In this case the supply­side features of the 
industrial sector are also influential. From the production function, factor 
demands are derived in both productive sectors, on the basis of cost minimi­
sation subject to given levels of output. Levels of employment and invest­
ment are then dependent on the level of output, which is demand­driven in 
both sectors. In the tradable sector output is driven by world demand and 
competitiveness. In the non­tradable sector, on the other hand, actual 
output is driven by a weighted measure of final demand. 

The supply of labour is modelled in order to take into account migration 
flows, which are fairly significant in Ireland. The work­force participation rate 
is influenced by migration, unemployment (the discouraged worker effect) 
and the replacement ratio (i.e. the fraction of average earnings replaced by 
social welfare transfers). 

■ Summary 

The creation of employment effects is an objective in all areas of interven­
tion, and the capacity to create or maintain employment is an important 
factor in selecting projects for support. A second type of employment effect 
is sought by interventions in the area of human resources, primarily in 
training. In this case it is linked to the maintenance or improvement of the 
employability of individuals either in work or unemployed. 

In addition to the primary difference of job creation or maintenance effects 
and employability effects, employment effects vary widely. They occur at 
different points of the causal chain, they can be temporary or of a more long­

Transversal evaluation of impacts on intervention priorities 

term nature, they can be sustainable or transitory. For these reasons the 
aggregation of different types of employment effects is often not possible. 

When estimating net employment impacts, nature of deadweight, displace­
ment and substitution effects vary according to whether job creation or 
maintenance effects or employability effects are being evaluated. 

A great deal of experience has been accumulated in this area of evaluation 
and a wide range of bottom­up and top­down tools have been successfully 
used to evaluate employment effects. 

■ For m o re information see... 

Bradley J., Whelan K., Wright J. (1993), Stabilization and Growth in the EC 
Periphery, Aldershot: Avebury. 

Department of the Environment. (1988), An Evaluation of the Urban Develop­
ment Grant Programme, London: HMSO. 

Gautié J. et al. (1994), Les subventions à l'emploi ­ Analyse et expériences 
européennes, Cahier travail et emploi, La Documentation Française. 
Issaev B., Njikamp P., Rietveld P., Snickars F. (1982), Multiregional Economie 
Modelling: Practice and Prospects, North Holland. 

King J. (1990), Regional Selective Assistance 1985­88: An Evaluation by the 
Department of Trade and Industry, the Industry Department for Scotland and 
the Welsh Office Industry Department, London: HMSO. 
Manski CF., Garfinkel I. (eds.) (1992), Evaluating Welfare and Social Pro­
grammes, Harvard University Press. 

Njikamp P., Rietveld P., Snickars F (1986), Handbook of Regional and Urban 
Economics, North Holland. 

OECD. (1991 ), Evaluating Labour Market and Social Programmes ­ The state 
of a complex art, Paris: OECD. 
OECD. (1998), Employment Outlook, Paris: OECD. 

Schmid G., O'Reilly C, Schöman K. (eds) (1996), International Handbook of 
Labour Market Economics, Cheltenham: Edward Elgar. 
Storey D.J. (1990), "Evaluation of Policies and Measures to Create Local 
Employment", Urban Studies, vol.27, n°5. 

Evaluating the impact of structural interventions on equal 
opportunities 

■ Why do impacts on equal opportunities need to be evaluated? 

• The Community context 

The European Council has declared on several occasions (in Essen, Cannes 
and Madrid) that the promotion of the principle of equal opportunities 
between men and women is, along with the struggle against unemployment, 
a fundamentally important task facing the European Union and Member 
States. Glaring inequalities continue to exist between men and women in all 
areas. As far as the participation of women in the job market is concerned, 
their level of activity is lower than that of men in all the Member States of the 
European Union. In 1996, the employment of women over the age of 15 
varied between 34,6% and 58,7% (Eurostat). The wage gap remains signifi­
cant in all countries. Furthermore, women are generally more affected by 
unemployment than men in all age groups, particularly as regards long­term 
unemployment. Finally, most of the actions of the structural policies have dif­
ferentiated effects, reproducing discrimination between men and women. 

Efforts to promote equal opportunities at European level are not intended 
merely to introduce specific measures in favour of women, but to add a 
dimension to all actions and structural policies. This concept is known as 
"mainstreaming" (see Box 2). Mainstreaming is intended to Influence the 
strategies of Structural Funds policies so that they incorporate the 
dimension of equal opportunities" between men and women by systemati­
cally examining their respective situations. Policies must therefore be 
defined by taking into account the diversity of needs, including the needs of 
the various groups of men and women in society. 

The "Fourth Action Programme on Equal Opportunities for Women and Men 
(1996­2000)" is aimed at integrating equal opportunities into the preparation, 
implementation and monitoring of all policies and actions of the European 
Union and Member States, with due respect to their respective competencies. 

With regard to the Structural Funds, the most noteworthy event was the 
revision of the framework­regulation in 1993, in which equal opportunities 
are explicitly referred to, with particular emphasis on the struggle against 
unemployment among women. In 1996 the Council adopted a Resolution 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

reaffirming its political will in this respect.  In decision-making concerning  the 
approval  of  programmes,  a standard  clause  was  specifically  introduced  to 
guarantee compliance with this new obligation. 

Several  examples  of  good  practice  exist, taken  from the  successful  results 
of  the  NOW  1991-94  (New  Opportunities  for  Women)  experience,  and 
covering  a  broad  sample  of  measures.  NOW, the  first  Community  Initiative 
specifically  for women  (focused  on the  promotion  of  equal  opportunities  in 
the job  market),  has opened  new  avenues to  ensure that  women  are  taken 
into account  more fully by all the Structural Funds. 

From  the  beginning  of  the  new  period  of  Structural  Funds  programming 
(1994-1999), the principle of equal opportunities  has been considered as an 
objective  of  structural  policies  in their  entirety. There  has  thus  been  a  shift 
from a passive conception (respect for the principle) to an active  conception 
(objective of all structural policies) of equal opportunities. 

Today, equal opportunities  are no longer considered  only to  be the  problem 
of women. The promotion  of equal opportunities  is recognised for  its  ability 
to contribute  towards  socio-economic  development  in general and  regional 
readjustments in particular. 

In the context of Structural Fund interventions, the evaluation of the applica­
tion of the principle of equal opportunities  involves assessing the potential or 
actual impact on the respective situations of men and women and the extent 
to which equality is taken into account  in processes and mechanisms of pro­
gramming,  implementation,  monitoring  and  evaluation. This  latter  aspect 
involves,  for  example,  analysing  of  the  role  of  women  in  interventions, 
measuring positive developments and highlighting discrimination. 

•  The need to assess the application of the principle of equal 
opportunities 

Evaluation can play an essential role as a mainstreaming tool by ensuring that 
the principle of equal opportunities is taken into account  in structural policies. 
It may, in particular,  help to  enhance the effects  of  synergy for all the actions 
benefiting from Structural  Funds support.  In order to do so, it is necessary to 
apply techniques which take into account the specific nature of equal oppor­
tunities  between  men  and  women.  Through  the  definition  of  evaluation 
criteria  relating  to  this  theme,  and  of  corresponding  indicators,  evaluation 
can  be  used  to  systematically  examine  these  actions  and  policies  and  to 

Evaluating impacts on equal opportunities 

take into consideration their direct or indirect effects in terms of equal oppor­
tunities. 

Methodological difficulties are not as great in the assessment of the impact 
of specific actions in favour of women (who are, in this case, the direct bene­
ficiaries) as in the transversal evaluation of the effects of Structural Fund pro­
grammes in terms of equal opportunities. Interventions likely to produce 
direct effects on women are those focused on the development of human 
resources. Other types of intervention concerning infrastructure or the pro­
ductive environment do not necessarily include equal opportunities among 
their explicit objectives. Their effects are therefore more likely to be indirect. 

In the context of the evaluation of Structural Funds, a few isolated initiatives 
have been taken to evaluate the promotion of principle of equal opportuni­
ties. This experience has, however, been limited to a few cases of ex ante 
and intermediate evaluation. Ex post evaluation experience in the Structural 
Funds context is still embryonic. The NOW Community Initiative which was 
evaluated ex post is too specific to serve as a model in the framework of 
classical operational programmes. 

■ Defining what needs to be measured 

It is not possible to define ex nihilo what an evaluation has to assess, in so far 
as the strategy chosen to promote equal opportunities depends to a large 
extent on national and even regional contexts. Thus, the application of the 
principle of equal opportunities varies considerably from one Member State 
to the next. For certain countries (France, Portugal), it seems to be a general 
objective but without indications as to the strategy to adopt or actions to 
implement. In Spain and Germany equal opportunities are taken into consi­
deration in the context of specific actions, and are promoted by means of 
general measures. In Finland, the principle is visibly and thoroughly integra­
ted into all the components of programmes. 

• Analysing the process of programming interventions 

Because the programming of structural interventions is the result of 
decisions taken jointly (by the EU, state, regional and local authorities), it is 
necessary to examine whether, at all levels, there is a sufficiently strong 
political will to take into account the interests of women and to meet their 
specific needs. Keen awareness of the question is essential if suitable 
solutions are to be conceived and implemented (e.g. the setting up of 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

actions to  inform  and train  the  key  actors). The degree  of  awareness  is  not 
only  related  to  the  level  of  representation  of  women  in  decision-making 
bodies, even if the presence of spokespersons  is a contributing factor. 

Example: Training in occupations traditionally reserved for men. 
Training for women in traditionally  masculine occupations  has frequently been  unsuc­
cessful. Women are not  necessarily  recruited  by firms  in these sectors, or for jobs  for 
which they have been trained. Sometimes they are recruited but give up their jobs  due 
to  pressure  from  an  exclusively  masculine  environment.  An  ex  ante  evaluation  can 
help  to  reduce  this  failure  rate  by  programming,  in addition  to  the  training  measure, 
actions of the following  kind: awareness  in firms, promotion of recruitment  of women, 
advice and assistance to women after  recruitment. 

This example and others presented  in this chapter, are inspired by real situa­
tions. In some cases, however, they have been modified in order to provide a 
concise, but effective, illustration of an idea or a concept. 

that  the  resources  allocated  to  promoting  equal  opportunities  are 

The level of analysis  in the example  is upstream from the implementation  of 
the programme and thus of the production of effects on equal opportunities. 
The evaluator  must  examine the conditions  in which the  programmes  were 
designed and check: 
•  whether  the  announced  objectives  take  into  account  the  interests  and 
specific  needs of women; 
•  whether  groups of women and / or authorities responsible for equal  oppor­
tunities at national and regional level have been involved in the programming; 
•  whether  the  strategy  and  choice  of the  modes  of  intervention  have  relied 
upon a prior assessment of women's needs and potentialities; 
• 
coherent with stated objectives; 
•  the  proportion  of  interventions  programmed  that  are  able  to  facilitate  the 
achievement of priority objectives concerning equal opportunities; 
•  whether the  programming  documents  (CSF, SPD, etc.) provide  qualitative 
and quantitative criteria  and precise  indicators for assessing the considera­
tion given to equal opportunities; 
•  whether  some  measures  are likely to  run counter  to the  principle of  equal 
opportunities, risking the worsening  of the existing  imbalance between  men 
and women; 
•  whether the SPD is consistent  with  national and regional equal opportuni­
ties policies; 
•  the proportion of women in decision-making  bodies. 

Evaluating  impacts  on  equal  opportunities 

•  Analysing the conditions in which interventions are  implemented 

The perception of women's  interests is not always the same during the pro­
gramming of interventions and during their implementation. It is essential to 
analyse the conditions  of  implementation  in order  to  understand  how  they 
can account for the achievement or not of equal opportunity objectives. 

Example: Organisation of training for long-term  unemployed 
Presuming  the  monitoring  of  training  measures  revealed  that  the  drop-out  rate  of 
women was high, an intermediate evaluation would have made it possible to  examine 
the causes, e.g. family reasons (spouse's hostility), lack of infrastructure (lack of child-
care facilities,  means  of  transport,  care  facilities  for  an aged  or  ill  parents,  etc.)  or 
financial reasons (the allowance is too  low to cover child  minding costs, etc.). On the 
basis of such conclusions it becomes  possible for those responsible for the running of 
the scheme to take appropriate steps to remedy the situation. 

To ensure that the  principle  of  equal  opportunities  is taken  into  account  in 
the implementation of structural interventions, the evaluator must check: 
•  whether  the  selection  of  projects  has  included  the  application  of  the 
principle of equal opportunities as a condition in allocating budgets; 
•  whether groups of women and / or authorities responsible for equal oppor­
tunities are represented and active in the monitoring committees; 
•  whether the people responsible for programme management  are aware of 
all  issues  involved  in  promoting  equal  opportunities  (e.g. have  awareness 
campaigns on the subject taken place); 
•  whether  positive discrimination  (sometimes  referred to  by the  term  "affir­
mative  action",  i.e.  the  aiming  of  measures  at  a  particular  group  with  the 
objective of  eliminating  and  preventing  or compensating  for  disadvantages 
which  are a result  of  existing attitudes,  behaviour  and structures)  has  been 
applied to favour the participation of women in project initiation (e.g. whether 
there are women's NGOs among the promoters); 
• whether monitoring systems include the collection of gender differentiated data; 
•  whether  the  interventions  have  been  improved  during  their  implementa­
tion, to better meet women's needs; etc. 

•  Highlighting the effects of interventions on equal opportunities 

The main  methodological  difficulty  encountered  in this type  of  arises  from 
the fact that two concomitant  priorities  have to be evaluated: economic  and 
social cohesion  on the one hand and equal  opportunities  on the other. The 
latter is rarely formulated explicitly as an intermediate objective of  structural 
interventions, except in the case of specific measures or priorities. Moreover, 

Transversal evaluation of impacts on intervention priorities 

structural interventions are rarely, except for those of the ESF, aimed at 
people as direct beneficiaries. The effects of structural interventions are 
usually indirect and long term, which makes their evaluation complex. 

Application of the principle of equal opportunities to all interventions is a 
transversal objective that cannot be analysed immediately in its globality. It 
is therefore advisable to break down the effects into coherent sub­sets. The 
following typology is intended to provide an analytical grid, grouping 
together four types of possible effects in the specific context of Structural 
Funds, in terms of: 
• improving living conditions so as to better meet women's needs; 
• greater accessibility to the job market for women; 
• improving the situation of women at work; 
• promoting of the participation of women in the creation of socio­economic 
activities. 

Box 25 presents an objectives tree that draws on this typology to illustrate how 
this tool can be used to structure the mainly implicit objectives of actions 
promoting equal opportunities. Actions such as support for further or conti­
nuous education, professional training, counselling and orientation can all 
increase the employability of individuals. The introduction of more flexible 
work hours for women with children or aged relatives at home, leave organised 
in conjunction with school holidays, reduction in the physical aspect of jobs, 

Box 25 ­ Example of a thematic objectives tree for the priority 

of equal opportunities 

Global objectives 

Examples of specific objectives 

Balanced 
development 
for men 
and women 

Living conditions which 
meet women's needs 

ι ­ ­

• Improved access to neighbourhood 
services 
• Improved access to specialised medical 
treatment 
• Adaptation of transport systems 

Y 
Access for women to 
the job market 

• Restore employability 
• Adapt available jobs 
• Improving skills 

Women's situation 
at work 

« ­

• Reducing professional segregation 
• Improving working conditions 

Participation of women 

■*■ ­

so cio ­economic activities 

• Favour the creation of business 
or independent activities by women 

Evaluating  impacts  on  equal  opportunities 

can all contribute to  adapting jobs to women's  needs. As a consequence  of 
these actions, access for women to the job market can be improved. 

Furthermore the different intermediate objectives are not unrelated. If the living 
conditions  of women were improved  so that  needs could  be met  more ade­
quately,  the  other  three  types  of  impact  would  more  easily  be  achieved. 
Similarly,  improvements to the situation  of  women  in their jobs  may  improve 
conditions  of  access to  the job  market.  It  is for this  reason that, the  links 
between  intermediate  objectives  in the thematic  objectives  tree  have  been 
shown. 

In order to facilitate the work of evaluation teams, many examples of  effects 
are classified  in terms  of this typology  of  impacts and  presented  in Box 26. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Box 26  -  Example  of  effects  by category  of  impact 

Category 

Lifestyle 

Access for women 
to the job market 

Examples of effects 

•  Relaxing constraints related to responsibility for small children or for other 
persons, either temporarily (e.g. during job seeking) or over a longer period 
(e.g. investing In a middle-term social or professional project) 
•  Improving integration and reinforcing the capacity for initiative of women in rural 
areas, by providing more adequate public transport 
•  Reconciling family and professional constraints more satisfactorily 
•  More assistance in dealing with social problems, particularly through improved 
social services (quality family planning, support for victims of household violence, 
sexual harassment, etc.) 
•  Increased access to good quality care services for children and other depen­
dants 
•  Promotion of gender equality In parental leave 
•  More flexible work conditions 

•  Less sectoral and vertical segregation between men and 
women in education and training 
•  Improvement of job prospects for young women (guidance towards occupa­
tions with a future) 
•  Enhanced employability of women 
•  Provision of jobs which are better suited to meet women's requirements 
•  Adjustment of the content of training actions to meet the particular requirements 
of women (training in fields that improve their career prospects), and adjustment 
of the practical modalities of training to cater for women's specific constraints 
such as their daily schedule 
•  Personalised aid for the unemployed In finding a job. Including more appropriate 
support for women (e.g. for women trying to re-enter the job market) 
•  Setting up professional alternatives for women working in the underground 
economy, etc. 
•  Reconciling family and professional constraints more satisfactorily 

Women's situation 
at work 

•  Reduction of sectoral segregation by promoting a balanced representation of 
women and men in all economic sectors 
•  Jobs better suited to qualifications 
•  Increasing the proportion of women in senior positions 
•  better access to ongoing education, etc 
•  Reconciling family and professional constraints more satisfactorily 

Participation of 
women In the creation 
of new activities 

•  Better recognition and appreciation of the role of women in the creation 
of new businesses 
•  Improvement of women's access to services and infrastructures supporting the 
creation of enterprise, and to financial facilities for business development 
•  Greater participation of women in local development initiatives and in new 
sources of employment, etc. 
•  Reconciling family and professional constraints more satisfactorily 

Evaluating impacts on equal opportunities 

It is also possible that interventions, no matter their effectiveness and 
success, contribute to reproducing existing discriminatory patterns, thus 
having unfavourable or undesirable effects. It is very important that the 
evaluator analyses the over or under­representation of women in particular 
interventions and that the findings are contextualised. Women may for 
example be over­represented in measures not directly or immediately 
leading to employment, e.g. training measures, as opposed to apprentice­
ship schemes or unemployed aids. The importance of providing an explana­
tory socio­economic context to the findings is illustrated by the issue of part­
time work where the occurrence of such work can be considered as positive 
or negative depending, for example, on whether the part­time work was 
chosen or necessitated and taking in account the frequently less favourable 
working conditions, both in terms of work security and wages. When pro­
grammes are orientated towards traditionally male­dominated sectors, the 
evaluator could, in addition to determining to what extent women have parti­
cipated in measures, which types of measure and results, try to identify the 
conditions and necessary changes in these needed to promote increased 
gender balance in the sector in the future. 

■ Examples of impacts by area of intervention 

The approach used to integrate equal opportunities into programming 
documents should be twofold: an approach providing for measures expli­
citly in favour of women, and a transversal approach considering equal 
opportunities as a general guideline applicable to all the measures 
proposed. In the latter case, the promotion of equal opportunities is usually 
simply mentioned as a subsidiary of the general measures. In view of the 
absence of an operational translation of this priority, as well as its marginal 
nature, it is essential to examine how general measures incorporated into 
classical Structural Fund programmes can potentially contribute to equal 
opportunities. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

•  Basic  infrastructures 

Among  the  interventions  in the field  of  infrastructure, those  which  seem  to 
incorporate  an  equal  opportunities  dimension  concern  basic  infrastructure 
and facilities of a social nature. 

Example:  Child  c a re facilities  in  Portugal 

With  s u p p o rt  f r om  the  N OW  Initiative,  a  p l a y s c h o ol  w as  created  In  the  heart  of  the 

industrial  area  of  Evora  (Portugal),  c l o se  to  an  o c c u p a t i o n al  training  c e n t r e.  Child 

m i n d i ng  facilities  were  m a de  available  to  jobless  w o m en  wanting  to  a t t e nd  training 

c o u r s e s,  a nd  to  e m p l o y e es  of  f i r ms  in  t he  Industrial  area.  T he  p l a y s c h o ol  w as 

e q u i p p ed  with  a  playground  and  open-air  s w i m m i ng  pool  in  order  to  make  the  setting 

m o re  homely,  In  w h at  is  an  industrial  a nd  essentially  m a s c u l i ne  e n v i r o n m e n t.  T he 

c e n t re  t h at  h ad  h i t h e r to  o f f e r ed  t r a i n i ng  in  t r a d i t i o n a l ly  m a s c u l i ne  t r a d es  (e.g. 

carpentry),  had  been  frequented  mainly  by  m e n.  Today,  4 0%  of  the  participants  are 

w o m e n.  This  centre  has  enabled  s o me  of  t h em  to  find  j o bs  and  others  to  find  more 

s e c u re  e m p l o y m e nt  t h an  t h ey  f o r m e r ly  e n j o y e d.  T he  c h i ld  m i n d i ng  s e r v i c es  has 

helped  to  ease  the family  constraints  of  w o m en  w ho  consequently  have  the  possibility 

of  involving  themselves  in m e d i u m - t e rm  professional  projects. 

Outputs 

> 

Child  care 
facilities 

Use  of 
facilities 
for  lightening 
family 
constraints 

Increased 
availability 

Effects 

Better 
access 
to  training 

Qualifications 
improved 
and/or 
widened 

Successful 
projects 

O 

Better  access 
to job  market 

Improved 
jobs  situation 

Enhanced 
participation 
in  economic 
activities 

Priority 

I 

I 

Evaluating  impacts  on  equal  opportunities 

•  Human resources 

The  principle  of  equal  opportunities  is  more  visibly  incorporated  into 
measures concerning  human  resources. The  most  significant  effects  are in 
activities  involving  consultancy,  guidance  and monitoring; actions  in favour 
of training and occupational  integration; interventions  in education systems, 
training  and  employment;  and, lastly, on  work  organisation  and  modes  of 
production. 

Example:  Setting  up  measures  for  training, follow-up  and  rehabilitation  tailored 

to  suit  individual  requirements  (Portugal). 

Trainees  with  children  or  adults  under  their  responsibility  were  given  priority  in  training 

actions  and  in the allocation  of  additional  benefits. This enabled  both  w o m en  and  men 

to  reconcile  their family  and  working  lives. The training  also developed  methodologies 

and  specific  strategies  intended  for  w o m e n,  in the  framework  of  measures  providing 

assistance  and  rehabilitation  support  (individual  counselling, j ob  seeking  techniques, 

career  p l a n n i n g,  etc.).  C o n s e q u e n t ly  a  more  b a l a n c ed  p a r t i c i p a t i on  of  m en  a nd 

w o m en  in all sectors  of  activity  was  possible. 

Outputs 

> 

Financial  support 
for  people  with 
children or 
adults  In 
their  care 

System for 
providing support 

Individual 
rehabilitation 
course 

Effects 

Better reconci­
liation of family 
and working  life 

Easy access 
to  training 

Improved and 
diversified 
qualifications 

More  balanced 
participation 
in all  sectors 
of  activity 

Information 
Counselling 
Continuous support 

Quality of  training 
systems 

ΐ> 

Improved 
work 
situation 

Access 
to jobs 

Creation of 
economic 
activities 

Priority 

I 

I 

Transversal evaluation of impacts on intervention priorities 

• Productive environment 

Actions relating to the adaptation of productive activities and to local and 
regional development are likely to have an impact on the promotion of equal 
opportunities. 

Example: Creation of a European Business and Innovation Centre (Italy) 
A centre for the development of business innovation was created in the Pouilles 
region. The Business Information Centre in Puglia Sprind developed an international 
marketing methodology based on employing mainly women in the marketing division. 
Prospecting for business opportunities was based essentially on the ability of the 
women in the team to stimulate their interlocutors' curiosity and interest in their new 
products. Local products of SMEs were introduced into international distribution 
channels and the firms concerned enhanced their competitiveness on a global scale. 
This favoured the establishment of new SMEs in the region, specialising in agricultural 
products. The SMEs created jobs, most of which were occupied by women farmers 
living in and around the villages. 

Outputs 

> 

Creation for 
a development 
centre for 
innovative 
business in 
rural areas 

Establishment 
of other SMEs 

Creation of 
local activities 

Effects > 

Reduction of 
rural women's 
isolation 

Success of 
new enterprises 
created 
by women 

Access to 
employment 

4 

/ 
1 

Creation of 
economic 
activities 

Priority 

■ Defining and selecting indicators 

Volume 2 of the MEANS Collection is devoted to the subject of indicators for 
monitoring and evaluating Structural Fund interventions. The above Volume 
includes an information sheet devoted to the priority of equal opportunities. 

In order to remain as operational as possible, the use of indicators should be 
mainly limited to those which are quantifiable. Yet qualitative information 
must not be overlooked, given the complex and innovative nature of the 
objective of equal opportunities. 

The impacts of interventions on this priority are produced by an accumula­
tion of intermediate results derived from a series of heterogeneous outputs. 
Programme indicators should provide meaningful information on all the 

Evaluating  impacts  on  equal  opportunities 

steps in the process, ranging from the allocation of resources to the produc­
tion of impacts. 

A pragmatic  selection  should  be made of a small number  of ad hoc  indica­
tors relating to equal opportunities.  It should not be necessary to  undertake 
costly  collection  of  information,  nor  should  an  excess  of  information  be 
supplied to decision-makers: 
•  At  European  level, the  priority  need  is to  assess  and  clarify  the  conse­
quences of structural policies in relation to the extent to which equal  oppor­
tunities are being taken into account. 
•  At  programme  level, the  monitoring  committee  needs  to  know  to  what 
extent  resources  have  been  consumed  and  to  monitor  outputs. When  the 
programme  has  not  included  the  principle  of  equal  opportunities  in  its 
explicit  objectives, the  Monitoring  Committee  should  be  kept  informed  on 
the impact of the programme on men and women. 
•  At  project  level,  managers  must  monitor  in  detail  the  consumption  of 
resources and the outputs produced, as well as the reactions of the different 
of beneficiary groups, particularly women. 

Women are not a homogeneous group, but constitute half of the  population 
concerned  by  Structural  Fund  interventions.  Because  of  this, the  range  of 
indicators  should  be  systematically  broken  down  according  to  gender. A 
system  of  indicators  "sensitive  to  the  question  of  equal  opportunities" 
requires  the  close  integration  of  types  of  beneficiaries  and  types  of 
measures. This  makes  it  possible, for  example, to  assess  whether  women 
benefit in the same proportions as men from measures with a direct effect on 
employment  (such as recruitment  or employment  subsidies)  or whether,  on 
the contrary, they benefit only from training measures. 

Context  indicators,  by  providing  a baseline,  show  what  proportion  of  the 
targeted  population,  particularly  of  women, was  reached  or to  what  extent 
the needs of women in the region were met. A programme indicator (e.g. the 
number of women  beneficiaries of the programme who  kept their jobs  after 
having children) can be combined with a context  indicator  (e.g. the  regional 
percentage of women  between the ages of 25 and 35 who  interrupted their 
careers after  having children) to give an indication  of the contribution  of the 
programme  to  solving  one of  the  problems  faced  by  women  in the region. 

No universal rule exists for constructing a system of indicators for the appli­
cation of the  principle  of equal  opportunities. The quality of existing  indica-

Transversal  evaluation  of  impacts  on  intervention 

priorities 

tors has to be checked to determine whether they are acceptable. The most 
frequent shortcomings are: 
•  the absence of relations between the indicators and the targeted; 

•  the difficulty of using the indicators due to the lack of reference data for the 
constitution of the counterfactual situation; 
•  the absence  of  a sub-division  between  men  and women  in the  indicators 
when referring to individuals. 

A  system  of  pragmatic  indicators  taking  into account  the  principle  of  equal 
opportunities  can, however,  limit  the  amount  of  information  collected  and 
transmitted according to its relevance for each managerial level. 

In the  case  of  the  ESF, indicators  are  usually  presented  as  shown  in  the 
upper half of Box 27. When the principle of gender mainstreaming is applied, 
indicators are presented in the form shown in the table. 

Box 27 -  Example  of the  application  of the principle of gender  mainstreaming 

Measure XX : 
Number of beneficiaries : 
• Total 
•  by sex 

·  by age 
·  by level of qualification 

Gender mainstreaming implies the following breakdown of data: 

%Men 

% Women 

1. Total 
2.  by age 
3. by level of qualification 

•  Selecting and interpreting indicators for a given programme 

Programming  documents  reveal  an almost  total absence  of  precise  indica­
tors on how the principle of equal opportunities  is being taken into  account. 
As far  as the  participation  of  women  in the  CSF  and  SPDs  are  concerned, 
indicators  are very general and very  limited. At  best, women  are considered 
as a separate  category  of  beneficiaries,  like young  or  handicapped  people. 

In contrast,  more  specialised  monitoring  bodies  offer  increasingly  sophisti­
cated indicators. These are sometimes too general to be used in the evalua­
tion  of  a structural  programme  that  does  not  have  the  promotion  of  equal 
opportunities as its main objective. An example is the "index of  dissimilarity" 
which  serves  to  analyse  the  degree  of  segregation  between  the  sexes,  by 
sector of activity or by profession. 

Evaluating  impacts  on  equal  opportunities 

Thus, the evaluation team will most  often  have to  propose  ex  nihilo a list of 
indicators  which  incorporate  the  equal  opportunities  dimension  into  the 
global evaluation of a programme. Evaluators, however, cannot be expected 
to define specific  indicators  for the  promotion  of  equal opportunities  within 
this framework. It is therefore necessary to make a selection and to settle for 
relatively  rough  and  approximate  indicators to  assess the equal  opportuni­
ties dimension. Evaluation teams  should  use a solid analysis of the  context 
based  on  a diagnosis  of  the  problems  facing  women, the  identification  of 
prospects for development within the specific national or regional context or 
the  identification  of  the  most  important  aims  within  their  region  for  the 
promotion of women. 

The interpretation of information provided by indicators is not always easy. It 
is  essential  to  use  "control  variables"  on  the  local,  regional  or  national 
context,  particularly to check whether the effect  produced  is truly  beneficial 
for women. For example, an increase in part-time work for women could  be 
explained by the lack of other professional alternatives. It could also reveal a 
lack of infrastructure, child-minding facilities and a persistent lack of sharing 
of family  responsibilities. 

By way of an example, a system of indicators which could be used to track the 
extent to which the equal opportunities dimension has been taken into account 
within a programme of community  centre regeneration is presented in Box 28. 

Box 28 -  Example of a system  of indicators for the  establishment 

or regeneration  of community  centres 

Programme Indicators 
(Related to the intervention and its effects) 

Context Indicators 
(Related to the assisted area) 

Output indicators 

Facilities 

Capacity 

Result indicators 

Satisfaction 

Impact indicators 

Employment 

Purchased, constructed or 
refurbished buildings in m! 
(of which: area devoted to 
meeting women's needs, i.e. nursery, etc.) 

Maximum number of users that can be 
served (of which: maximum number 
of users of women-specific sources) 

% of user that are satisfied or very 
satisfied with the project 
(of which women/men) 

1 

Maximum number of users 
in nurseries in the region 

Number of users holding a permanent 
job (of which women/men) 

% of regional workers 
who use nursery services 

Source: Volume 2 of the MEANS Collection 

Transversal evaluation of impacts on intervention priorities 

■ Suitable sources of information 

Data available from the Statistics Office is usually used to quantify context 
indicators. For example, many indicators can be found at Eurostat in the 
NEW CRONOS database, a collection of about 70 million harmonised items 
of data for the Member States of the European Union. The Regio database, 
devoted to statistics on economic life in each Member State, presents data 
in the form of about 70 standard tables classified by subject domain. The 
data are all broken down by region, according to the nomenclature of the 
NUTS territorial units, and cover a wide range of information: demography, 
unemployment, labour surveys, etc. 

Statistics compiled from labour surveys primarily concern the distribution of 
working hours, the breakdown of jobs by age, gender and socio­professio­
nal category, and the level of unemployment with a breakdown of figures in 
terms of the reason, duration and modalities of job­seeking peculiar to each 
country and sector of activity. A statistical portrait of women living in the EU 
in the early 1990s is presented in the EUROSTAT publication "Women and 
men in the European Union". The document includes opinion polls that 
enrich a highly detailed comparative analysis of women's social and profes­
sional situation compared to that of men. 

Furthermore, the national institutes of certain Member States compile 
national statistics on the subject "women". For example, in Spain the 
Women's Institute (Instituto de la mujer) has created over 100 indicators to 
analyse disparities between men and women in the following fields: demo­
graphy, education, employment, health, policy and social benefits. 

■ S o me e x a m p l es of current practice in the European Union 

A detailed treatment of evaluation tools is provided in Volume 3 of the 
MEANS Collection. Some examples of the use of tools for assessing the 
extent to which the promotion of equal opportunities has been taken into 
account in Structural Fund interventions are presented in this section. 

The methodological singularity of this type of analysis lies in the complex 
and pervasive nature of the phenomena that has to be analysed. The 
construction of the evaluation must take into account the small number of 
former references and the lack of information available to date in this field. 

Evaluating  impacts  on  equal  opportunities 

An  assessment  of  the  impact  of  interventions  on the  priority  of  promoting 
equality  between  men  and women  can  be  undertaken  using the  "step-by-
step"  approach presented  in Part IV of this Volume. For an in-depth  evalua­
tion of impacts, however, other evaluation tools have to be employed. 

With the  exception  of the job  market, for  which  numerous  studies  exist  on 
discrimination  against  women  within  the OECD  area  as well  as at  national 
level, there are  not  sufficient  well-established  foundations  or  prior  informa­
tion available for a "hypothetical-deductive"  type approach, in other  words, 
a procedure which assumes the formulation of precise hypotheses for struc­
turing the tools for data gathering  and processing. The use of  closed ques­
tionnaires, for  example,  must  be  limited  to  certain  well-known  situations 
such as a survey  among trainees who  have attended the same course. It is 
thus appropriate to opt  for an  "inductive"  methodology  suited to  producing 
new  knowledge  about  unfamiliar  situations,  based  on  the  comparison  of 
qualitative observations. In addition to the use of focus groups, other corres­
ponding  techniques  are  unstructured  interviews,  full case studies  or,  more 
rarely, ethnographic observation. 

For  ex  ante  or  intermediate  evaluations, the  techniques  developed  in  the 
field of corporate  management  (e.g. Ishikawa diagram, "I had a dream" and 
the scenario technique) can serve to diagnose problems and find  innovative 
solutions. Descriptions of these tools can be found in most classic  corporate 
management  textbooks.  Other  suitable tools  are SWOT (Strengths /  Weak­
nesses, Opportunities / Threats) and colour vote. 

Ex  post  evaluation  requires  the  use  of  a  methodological  "package"  of 
different techniques  - e.g. questionnaire  surveys, full case studies and com­
parisons  of  several  population  samples  at  regional  and  national  level  -  in 
order to give a credible picture of the real impact. 

•  Ex ante analysis of projects financed by the ERDF in Spain 

In the framework  of the 1994-99 programmes, a case study  was carried out 
to  assess  the  extent  to  which  equal  opportunities  were  being  taken  into 
account.  On the  basis  of  answers  to  questionnaires  sent  to  all the  entities 
promoting  ERDF projects, the evaluator  identified the projects  most  directly 
related to the application of equal opportunities. Several equal opportunities 
criteria defined  by the evaluator  were  used to  perform  an in-depth  analysis 
identifying  projects  with  the  best  practices  in this  domain.  In total,  eight 
projects  with  the  following  potential  effects  were  chosen:  recognition  of 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

equality,  solidarity  between  women  and  self-esteem  building,  entry  by 
women  into  traditionally  male-dominated  sectors,  response  to  women's 
specific  needs,  promotion  of  dynamism  among  women, and  reconciliation 
of their working and family lives. 

The in-depth analysis of these eight cases helped to identify the characteris­
tics  of  projects  which  promote  equal  opportunities,  such  as the  adaptation 
of  the  form  and  content  of  training  actions  for  women, the  articulation  of 
measures to  ensure their  coherence  in terms  of  satisfying  equal  opportuni­
ties,  the  promotion  of  women's  participation  in  project  management,  the 
involvement  of  women's  associations  in  partnerships,  the  guarantee  of 
women's  participation  in project monitoring, and so on (see Box 29). 

Box  29  -  Standard form for  assessing the equal  opportunities 

dimension  in the framework  of ex ante  evaluation 

Name of the entity 
Title of the project 

Territory 
Objective (1,2,..) 

The project 

Objective 
Application of equal opportunities objectives (typology of effects) 
Typology of project promoters (% women) 
Beneficiary public (% women) 
Description of innovative actions and good practice 
Justification of equal opportunities strategies 
System of organisation and facilities developed 

Project programming 
framework 
in the ERDF 

Funds 
Programme 
Objective covered 
Axes, measures et actions 
Public and private co-financing 

Demonstrative value 
of actions 

Diffusion of results at local level 
Diffusion of results at regional level 
Transfer of results or methodology to other projects 
Multiplier effect 

Suggestions for the ap 

plication of "gender Mainstreaming": 

•  Analysis of implementation  of Finnish programmes. 

A thematic evaluation of equal opportunities was based on a matrix to assist 
commissioners and / or evaluators in their strategic  evaluation choice and in 
the  formulation  of  evaluative  questions.  This  approach  emphasises 
structure,  process,  results  and  impacts  on  women  and  men's  lives  and  on 
the  socio-economic  environment,  and  provides  evaluation  criteria  which 
take  into  account  the  principle  of  equal  opportunities. The  criteria  concern 

Evaluating  impacts  on  equal  opportunities 

the  different  roles  of  men  and  women,  women's  access  to  financial  aid 
provided  by the Structural  Funds, and the real needs of women in a specific 
context (see Box 30). 

Box 30 - Matrix for programme  assessment 

Objectives  of the  assessment 

Roles 

Access to  funds 

Needs 

Equal opportunities  criteria 

Structure 

Process 
1.  programming 
2.  Implementation 
3.  monitoring 
4.  evaluation 

Results 

Impact 

The matrix facilitated the selection of evaluative questions for which the eva­
luation had to provide a credible answer. For example, "what are the roles of 
women  and  men  in the  different  phases  of  design  and  implementation  of 
programmes?"  or  "to what  extent  does the  raison  d'être  of the  programme 
and  its  objectives  correspond  to  the  identified  needs  and  priorities  of  men 
and women  in the region?". The evaluation succeeded  in showing that allo­
wances granted  in the framework  of the ERDF programmes  did not  corres­
pond to Finnish women's real needs. 

•  An initiative for the evaluation of the application of equal opportuni­
ties in the framework of objective 2 programmes (Sweden) 

Within  the  framework  of  IQ-NET  (Networking  to  improve  the  quality  of 
objective  2  programmes),  a thematic  study  of  the  incorporation  of  equal 
opportunities  into these programmes  was  performed. The aim was to facili­
tate the exchange of information and experiences between the regions, with 
regard to the design, implementation, monitoring and evaluation of the equal 
opportunities dimension of objective 2 programmes. 

The thirteen objective 2 regions belonging to this network are all characteri­
sed by a particularly high unemployment rate among men. The priority of the 
intervention  is thus to  promote  the creation  of jobs. The  main  object  of  the 
study was, however, to examine the contribution  of objective 2  programmes 
to  the  promotion  of  equal  opportunities.  Within  the objective  2  framework, 
some  regions  incorporate  the  promotion  of  equal  opportunities  into  the 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

implementation  of  projects  and  then  into  their  evaluation.  Several  case 
studies were performed  to  answer  the following  evaluative question:  "What 
are the  employment  effects  of the  objective  2  programme,  bearing  in  mind 
the transversal  priority  given to equal  opportunities  by the Structural  Funds 
policy?". 

The  results of the  case  study  on the  Swedish  MASS  project  are worth  pre­
senting. The  project,  inaugurated  in January  1995  in the  region  of  Ånger­
manlandskusten,  was  aimed  at  providing  administrative  assistance  to 
several  SMEs  and  at  helping  them  improve  their  management.  Jobless 
women were the main beneficiaries of the project. 

A pilot  study was first conducted to define the problems  confronting  SMEs. 
It appeared  that they  encountered  problems  recruiting  skilled workers.  The 
second  step  related  the  needs  of  these  SMEs  to  job  applications.  14 
persons  were  subsequently  recruited,  of  whom  seven  were  women  pre­
viously  employed  in the  public  sector.  Five of  them  were  employed  on  a 
permanent  basis. 

The  project,  above  all,  helped  to  reduce  female  unemployment  and  to 
improve  the  competence  of  the  SMEs  involved.  It  also  contributed  to 
reducing  the  workload  of  managers  in these firms  and thus to  increase  the 
time they spent with their families. 

•  Evaluation of the participation of women in actions co-financed by the ESF 

This  evaluation, conducted  in  1991, was  the  first  thematic  study  on  equal 
opportunities and on the role of women in the Structural Funds. The origina­
lity of the study lay in the incorporation of the equal opportunities  dimension 
in all the  processes  and  methods  of  programming,  implementation,  mana­
gement and evaluation of ESF interventions. 

The construction of the methodological framework was based on  objectives 
defined by the Commission: 
•  qualitative and quantitative  analysis of the  role of women  in  programming 
documents; 
•  analysis of the extent to which of equal opportunities were being taken into 
account  in the  processes  and  methods  of  programming,  implementation, 
monitoring and evaluation; 
•  qualitative  and  quantitative  analysis  of  the  role  of  women  in the  actions 
implemented; 

Evaluating  impacts  on  equal  opportunities 

•  analysis of the coherence and complementarity  between actions financed 
by the  ESF and programmes  and actions set  up nationally and regionally  to 
promote equal opportunities, 
•  analysis  of  the  coherence,  synergy  and  complementarity  between  the 
NOW  programme  and  actions  financed  within the framework  of  objectives 
1,3 and 4. 

For  the  quantitative  analysis,  multiple  and  diverse  sources  of  information 
were used, in addition to the CSFs and OPs, e.g. annual reports  established 
either  by  the  national  and  /  or  regional  authorities,  or  by  the  training 
agencies; or even applications for support. 

For the qualitative analysis, interviews and / or questionnaires addressed  to 
the  actors  involved  in the  preparation  and  implementation  of the  CSFs  and 
OPs were  used, e.g.  ESF  Missions;  national, regional  and  local  authorities 
responsible  for  employment  and  occupational  training; training  agencies 
and private enterprise; and promoters and managers of actions  co-financed 
by the  ESF. National  and /  or  regional authorities  responsible for  policies  in 
favour  of  women, as well  as  correspondents  in each  Member  State  of  the 
Networks  dependent  on the  Unit  for  equal  opportunities  (DGV), were  also 
interviewed. National experts along with national and / or regional  managers 
responsible for the preparation and  implementation  of  NOW  in the  Member 
States were also contacted. 

The indicators used for the quantitative analysis concerned: 
•  resources  allocated  to  women,  through  mixed  and  specific  actions,  as 
compared to all available resources; 
•  number  of  women  beneficiaries of  mixed  and specific  actions,  compared 
to the total number of beneficiaries; 
•  percentage  of women  beneficiaries  by type  of  action, category  of  benefi­
ciary, level of qualification and sector of activity; 
•  proportion  of  actions  and  national  programmes  in favour  of  training  and 
employment of women, benefiting from ESF co-funding. 

The methodology  made provision for the following four typologies: types of 
measures / actions, categories of beneficiaries, sectors of activity, and quali­
fications. It included a series of key questions, grouped together by theme. It 
also provided for the constitution of a sample of mixed actions for each type 
of  measure,  spread  between  actions  with  a high  rate  and those  with  a  low 
rate of female participation, for the purpose of carrying out case studies. 

Transversal evaluation of impacts on intervention priorities 

■ S u m m a ry 

The promotion of equal opportunities between men and women is a priority 
for which very little evaluation experience exists, this is particularly the case 
for ex post evaluation. The incorporation of equal opportunities into Structu­
ral Fund evaluations is therefore an important challenge. 

Actions focusing on the development of human resources are those most 
likely to produce a direct impact on the situation of women. This type of 
impact is easier to evaluate than indirect impacts on the promotion of equal 
opportunities. 

Evaluation can play an essential role in promoting the strategy of gender 
mainstreaming by ensuring that the principle of equal opportunities is syste­
matically taken into account in structural policies. 

■ For m o re information see... 

Àlvaro­Page, M. (1996), Los usos del tiempo como indicadores de la discri­
minación entre géneros, Madrid: Ministerio de Asuntos Sociales; Instituto de 
la Mujer, N°46. 

Bamberger, M. (1996) "Understanding the Impact of Development Projects 
on Women: The Tunisia Institutional Development Fund Project" in Evalua­
tion for the 24st Century: New Topics for Evaluation. 

C.E. (1994), Les femmes et les hommes dans l'Union européenne: portrait 
statistique, Luxembourg: Office des publications officielles des Communau­
tés européennes. 

C.E. (D.G. V) (1998), 100 mots pour l'égalité, Glossaire de termes sur l'égalité 
entre les hommes et les femmes, Luxembourg: Office des publications offi­
cielles des Communautés européennes. 

Rubery, J.; Fagan, C. and Maier, F. "Occupational Segregation, Discrimina­
tion and Equal Opportunity", pp431­461 in Schmid, G.; O'Reilly, J.; and 
Schömann K. (1996) International Handbook of Labour Market Policy and 
Evaluation, Cambridge: EE, University Press. 

Thematic evaluations 
of domains of intervention 

In addition  to interventions  targeted at promoting  political  priorities,  interven 
tions  within  the  framework  of  the  Structural  Funds  take place  directly  in a 
wide range of domains.  These domains are largely situated  upstream  of  prio 
rities and may consequently  have an impact  on them.  For example,  interven 
tions  in  the  domains  of  transport  or  tourism  have  possible  environmental 
impact,  interventions  in  favour  of  RTD are  designed  to  have an  impact  on 
competitiveness  of industry, in particular  SMEs. 

The distinction  between  domains  of  intervention  and  political  priorities  has 
repercussions  on the organisation  of thematic  evaluations.  While the evalua 
tion  of priorities  can  take place  at any level above  that  of measure,  thematic 
evaluations  of  domains  will  be  transnational  in  nature,  taking  place  across 
programmes. 

The selection  of domains  has been made on the basis of recent  transnational 
evaluation experience acquired  within  the framework  of the Structural  Funds. 
The first  chapter  concerns  interventions  in  the  domain  of  RTD.  There is a 
considerable  amount  of evaluation  experience  in this domain,  but  evaluation 
practice  is  undergoing  a period  of  adaptation  to  match  the  evolving  policy 
context.  This structure  of this chapter  is similar  to that of the chapters  on the 
four political  priorities  in Part II. The second  chapter  is, on  the other  hand,  a 
rapid  survey  of  evaluation  practice  and  issues  relative  to  the  domains  of 
transport,  telecommunications  and  tourism. 

Evaluating impacts arising from interventions in the 
domain of Research and Technological Development RTD 

■ W hy evaluate the effects of interventions on RTD? 

For regions depending on agriculture and more traditional labour intensive 
industries, the opening up of national markets has meant increased compe­
tition from lower cost regions both within and outside the European Union. 
This has consequently led to a reduction of low skilled employment opportu­
nities and has exacerbated problems associated with economic develop­
ment and industrial restructuring. 

The decline of traditional sectors has been accompanied by an increasing 
reliance on new, rapidly expanding industries to generate economic growth. 
Because firms in these industries tend to be small or medium­sized, with 
generally limited resources for RTD investment, their research effort often 
has to be undertaken in co­operation with other firms or with research insti­
tutes to obtain optimal results. However, this type of collaborative arrange­
ment tends to occur most often in specific circumstances of which geogra­
phical proximity, the existence of non­pecuniary relations and training­
research­industry linkages are important factors. Particular examples of 
these phenomena that have inspired recent public policy thinking in the 
domain of RTD are the Italian "Districts" and the close links between training 
systems and industry found in Germany. 

As industrial restructuring has progressed, it has become increasingly evident 
that a process of technological polarisation is underway. The more technolo­
gically advantaged regions benefit from prior technological capacity, 
continued high levels of RTD expenditure, a well developed innovation culture 
in general, and the ability to further develop their innovatory capacity, whilst 
the more technologically disadvantaged regions fall further and further 
behind. This divergence of RTD capacity is reflected in the persistence of 
regional disparities in terms of competitiveness, trade, unemployment and 
growth, both within individual countries and across the EU as a whole. 

New insights into the process of innovation have meant that the linear 
"Science­Technology­Production" type model has given way to the concep­
tualisation of innovation as a dynamic, interactive and non­linear process. In 
addition, the understanding of the territorial dimension of RTD, through the 
framework of regional and national systems of innovation, underpins the 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

rationale of interventions aimed at economic and social cohesion. As a result 
of these advances, more emphasis  is being placed on the creation of endo­
genous technological  capacity  and on the diffusion  of  innovations  to  SMEs 
through  increased  co-operation  between  universities,  research  institutes 
and  enterprises,  as well  as  support  and  promotion  of  industrial  RTD. This 
change  of  orientation  reflects  the  fact  that  earlier  interventions  have  had 
generally disappointing  results in regions with low RTD potential. 

Central  to  this  change  of  emphasis  is that,  with  the  demise  of  the  linear 
model  of  innovation,  policies  focusing  on the  "technology  push"  approach 
are inadequate.  It  is now  realised that  technology  supply  and demand  have 
to be matched through an interactive  process  involving both producers  and 
users of new technologies.  Moreover, the development  of a region's innova­
tory  capacity  is intrinsically  linked to the tailoring  of  interventions  to  suit  the 
character  and to  anticipate  the  evolution  of  its  productive  structure.  Where 
SMEs  predominate,  one  of  the  major  requirements  in  RTD  intervention  is 
simply to help them appreciate the importance of innovation for competitive­
ness  in  an  open  economy.  The  present  generation  of  interventions  are 
therefore  largely  composed  of  "mission  oriented",  bottom-up  measures 
aimed at the wider  business and commercial  needs of  SMEs, their  demand 
for outside technology  and  integration of  it  into their  production  processes, 
their own capacity  in product  innovation and marketing, the financing, orga­
nisation  and  management  of  innovatory  activities,  and  the  creation  of  new 
high-tech firms. 

•  The Community  context 

The  importance  of  RTD for the competitiveness  of a region and the  need to 
develop  RTD capacity  is  recognised  within  the  Structural  Fund  framework. 
As a result, Structural  Fund support  for  RTD has risen dramatically  during  a 
period  in which  national  governments  have generally  been cutting  back  on 
science and technology  (S&T) expenditure. 

RTD interventions are financed under Structural Funds, with the ERDF being 
the major  source. RTD support  figures  prominently  in Community  Initiatives 
(INTERREG II, PME, ADAPT...) and is also the subject  of innovatory  actions: 
Regional  Innovation  Strategies  (RIS)  and  Regional  Technology  Transfer 
(RTT). In addition, leading  areas of scientific  and technological  research  are 
the object  of  interventions  associated  with Community  Support  Framework 
programmes. 

Evaluating  impacts  in  the  domain  of RTD 

Within  the  global  aim  of  the  EU's  RTD  policy,  which  is  to  strengthen 
European competitiveness  through the development  of scientific  and  tech­
nological  excellence,  Structural  Fund  interventions  are targeted  at  counte­
racting the concentration  of  RTD capacity  in certain countries, mainly in the 
North  of  the  EU (transnational  cohesion),  and  in certain  regions,  mainly  in 
and  around  countries'  capitals  and  a few  dynamic  regional  poles  (national 
cohesion). 

Structural  Fund  resources  have therefore  been  directed  to  more  peripheral 
regions with the object of developing their S&T base, particularly through the 
establishment  of  sustainable  links  between the  public  research  bodies  and 
regional industry. Over the long term, these interventions should increase the 
innovatory capacity of assisted regions either by reorienting their technologi­
cal  capacity  towards  the  faster  growing,  high  technology  sectors  or  by 
creating  indigenous  technological  capacity  and  therefore  reducing  the 
"technology  gap"  with  developed  regions. The  longer-term  impact  of  this 
process  will  be  increased  regional  specialisation  in  high  value-added 
products  and  services,  increased  growth,  and  enhanced  economic  and 
social cohesion. 

Recent experience of interventions  in the domain of RTD, for the  1989-1993 
programming  period,  has  shown  that  a number  of  problem  areas  in  RTD 
funding exist  in the  regions. Firstly, the relationship  between  increased  RTD 
funding,  increased  innovation  and  economic  development  is  neither 
automatic  nor  direct.  Attention  needs  to  be  paid  not  only  to  technology 
supply,  usually generated  upstream  from  industry  in  public  institutions,  but 
also to  the  demand  for  technology,  fashioned  by  the  needs  of  a  region's 
industrial actors  and their ability to  match  available technologies  with com­
mercial  opportunities.  However,  funding  during  this  period  tended  to  be 
concentrated  in  public  research  and  did  not  consequently  address  the 
practical RTD needs of local firms, particularly those of SMEs. 

In the  least  developed  regions  there  are  major  problems  of  absorption  of 
available  funds  due  to  the  fact  that  suitable  projects  are  often  difficult  to 
identify  or  authorities  have  little  experience  with  or  understanding  of  RTD 
issues. In certain  regions this  problem  has aggravated the  concentration  of 
funds  in  public  RTD  upstream,  with  the  consequent  mismatch  between 
public  technology  supply  and  industrial  technology  demand  (see  Box  31). 

Transversal evaluation of impacts on intervention priorities 

Box 31 ­ Evaluation of the impacts of CSFs for 

Research and Technology in Greece, Ireland and 
Portugal for the 1989 ­ 1993 programming period 

The objective of this evaluation was to assess the impact of interventions, 
make comparisons between the 3 countries studied and make recom­
mendations which would help improve both the efficiency and the effecti­
veness of CSF support for RTD in these countries. The study found that 
the national authorities in the three countries had largely used the oppor­
tunities provided by the funds to considerably enhance public infrastruc­
ture for RTD. It also reported that all three countries had had problems in 
the areas of industrial participation, technology transfer, overemphasis on 
technology supply, concentration of RTD around capital cities and human 
resource shortages. Problems of operational costs, achieving self­
financing targets, and management and evaluation also occurred. Conse­
quently, during the second programming period (1994­1999) more 
emphasis is being placed on better use of existing RTD capacity and less 
on building additional RTD capacity. 

The development of European RTD capacity in general, and in relation to 
cohesion in particular, represents an important strategic objective of the EU. 
This is reflected by the fact that interventions aiming to develop the techno­
logical potential of the regions amount to 7,600 million euros, approximately 
7% of all Community spending/ The evaluation of RTD interventions 
therefore has a central role to play in assessing the extent to which they are 
capable of achieving (or are achieving or have achieved) stated policy objec­
tives. The results and conclusions of these evaluations can then be used to 
inform the decision making process and consequently enhance the effecti­
veness of policies and programmes. A good example of this process is the 
evaluation of the impacts of CSF on research and technology in Greece, 
Ireland and Portugal for the 1989­1993 programming period (see Box 31). 

■ Defining w h at needs to be evaluated 

When evaluating Structural Fund interventions in the domain of RTD, it is 
important to bear in mind that the undertaking of RTD of sufficient quantity 
and quality is not an aim in itself. Nor is it a factor which when introduced into 
policy, leads to convergence towards a desirable goal. It is a prerequisite of 
sustained economic development and socio­economic cohesion. 

4. "Les Fonds structurels en 1996, huitième rapport annuel". This figure includes interven­
tions in the domains of R&D and innovation. 

Evaluating  impacts  in  the  domain  of RTD 

Whilst  it  is relatively straightforward  to  evaluate outputs  and  results of  RTD 
interventions,  it is more difficult  to  estimate the  impacts  produced.  Specific 
impacts  are often of an intangible nature (e.g. qualitative changes  in human 
capital)  and  impacts  at  the  socio-economic  level  only  occur  over  an 
extended time  period, making  impacts  difficult  to  attribute  to  interventions. 

The techno-industrial structure of a region is fashioned over time by a multi­
plicity  of factors. As a result, the  needs of  regions and therefore the  extent 
and  type  of  RTD  intervention  vary  greatly  from  one  region to  another.  For 
instance, in lagging regions and to a lesser degree in those with low popula­
tion densities, the supply  of facilities  and equipment, the number  of  resear­
chers, the  application  of  research  results  and the transfer  of technology  to 
local industry are problems that have to be addressed if endogenous scienti­
fic  and technical  capacity  is to  be  created.  It  is, however,  also  pertinent  to 
question to what extent interventions  in regions with little history of  research 
and  technology  should  be  undertaken  at  all. As  for  declining  industrial 
regions, interventions  are generally oriented towards  support for  investment 
in  specific  high-tech  sectors  (e.g. biotechnology,  microelectronics,  etc.), 
investment  in  key  locations  and  the  creation  of  investment  projects  to 
employ  highly trained personnel. 

•  Some key issues in RTD evaluations 

There are a number  of  key  issues related to  structural  support for  RTD that 
can be used to orient the evaluation of RTD interventions: 
•  Effectiveness  in  promoting  RTD capacity  and  capabilities:  What  consti­
tutes RTD capacity and capabilities?  Do interventions  promote the  intended 
results  and  impacts  in terms  of  RTD capacity  and  capability?  It should  be 
noted that  because the benefits  of  a single  research  project  are not  usually 
captured  by  or  do  not  generally  lead  to  specific  innovations,  particularly 
where collaborative  research is concerned, the results of  cost-effectiveness 
type analyses may be of limited worth unless RTD activities are appraised at 
a portfolio level. 
•  Strength of  linkages generated  by interventions  within the local  economy: 
Whilst  measures  of activity  (e.g. the number of SMEs involved in collabora­
tive RTD projects)  may be high, the  intensity  of  involvement  may differ  from 
firm to  firm.  Focusing  on the financial  participation  of  the  recipient  firms  or 
some  sort  of  qualitative  assessment  of  collaborative  ventures  can  help  to 
give an indication of the strength of these linkages. 
•  Skills  shortages  and  improved  qualification  levels:  In regions  undergoing 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

industrial restructuring, high unemployment  coexists with skills shortages in 
some of the more technologically  advanced  sectors. However, an  important 
question is the extent to which benefits of increased educational and training 
opportunities, especially at post-graduate level, are lost through the outward 
migration of addressees of supported training to other regions? 
•  When  improved  economic  performance  is  primarily  a  consequence  of 
process  innovations  leading  to  reduced  labour  costs,  initial  effects  on 
employment are likely to be a reduction in jobs. This may often be the case in 
regions  where  former  labour  intensive  industries  in  declining  industrial 
regions are attempting to remain competitive through capital investment and 
labour  substitution. The  relationship  between  innovation  and  employment 
creation or maintenance therefore warrants particular attention. 
•  Interactions  between  interventions  and  national  RTD  policy,  and  the 
regional distribution of RTD activities: The geographical distribution of public 
research capacity  is usually  strongly  influenced  by  national  strategic  consi­
derations.  Consequently,  in  more  centralised  states  this  leads  to  research 
facilities  tending  to  be concentrated  around  the  capital  city.  Conversely,  in 
more decentralised states research capacity  is usually found to be less cen­
tralised,  though  it  may  be  concentrated  around  regional  capitals.  Thus, 
national  RTD policies  and Structural  Fund interventions  can display  varying 
degrees of positive or negative synergies. 
•  Synergy  and  integration  with  other  EU  interventions:  Whilst  Support 
Framework  programmes  have,  until  recently,  been  concerned  with  more 
upstream  "pre-competitive"  research  projects,  and  Structural  Funds  with 
enhanced  research  capacity  and  technology  transfer,  the  reality  of  the 
situation  is that the  boundary  between these  approaches  is, in fact,  increa­
singly blurred. This overlap  suggests that there are possibilities  of  synergies 
and /  or  dangers  of  double  employment  between  these  two  policy  instru­
ments.  Furthermore,  research  projects  selected  under  the  Framework 
programme  are  chosen  on  the  basis  of  excellence  and  without  regard  to 
cohesion  and  structural  objectives.  This  generates  negative  synergies 
between these instruments when research capacity, as is generally the case, 
is concentrated  in the more dynamic and wealthier regions. 
•  Future  policy  orientations:  are structural  interventions  capable  of  overco­
ming  the  strong  tendency  towards  geographical  concentration  of  RTD, 
indeed,  is this  objective  relevant?  Or, on the  contrary,  should  interventions 
aim to concentrate RTD capacity in a relatively small number of regions so as 
to  achieve  critical  mass?  Should  Support  Framework  programmes  be 
tailored  to  take  account  of  the  concerns  of  structural  adjustment  and 

Evaluating  impacts  in  the  domain  of RTD 

cohesion?  How  can  overlaps  between  the  two  policy  instruments  be 
reduced  and  how  can  synergies  be  encouraged?  How  can  problems  of 
absorption and additionality  be overcome? 

The choice  of  level  of  study  of  RTD activities  is  largely  determined  by  the 
need  to  take  into  account  the  externalities  created  by  research  projects. 
Because  scientific  and technical  knowledge  are to  a certain  extent  public 
goods, the  estimation  of  effects  at  the  project  level  are  likely  to  seriously 
underestimate the benefits of the project accruing to society as a whole. It is 
for  this  reason  that  RTD  is  being  increasingly  studied  either  at  a  meso-
economic or macro-economic  level. Since Structural  Fund interventions  are 
geographically  delimited, a macro approach  seems to  be the  most  relevant 
level of analysis. Analysis at this level also allows interventions  and  projects 
to  be  examined  in the  larger  context  of  the  regional  or  national  innovation 
system of which they are an integral part. This does not exclude the possibi­
lity of  integrating  a meso-level  analysis  into an evaluation where a sector  or 
filière is of particular importance in a region. 

There  are four  main  channels  through  which  Structural  Funds  intervene  to 
reshape a region's RTD profile: 
•  Support  for  S&T infrastructures:  universities, associated  research  centres 
and public  laboratories, and structures  to  promote the diffusion  of results of 
research  projects. Activities  in this  area are aimed  at  enhancing  scientific 
and technological potential; 
•  Direct  support  for  industry:  subsidies  and  soft  loans to  support  R&D and 
technology  transfer  activities, and the creation  of  high-tech firms. Activities 
in  this  area  aim  at  enhancing  technological  potential,  increasing  links 
between  research and innovation and improving the financial framework  for 
performing RTD; 
•  Indirect  support  for  industry:  services  such  as technology  audits  and 
expert advice, etc., strengthening of innovation and technology centres, and 
efforts  to  increase  participation  in  other  EU  RTD  initiatives  (Support 
Framework,  Community  Initiatives,  etc.). These  activities  are  intended  to 
enhance  technological  potential  and  to  support  innovation  through 
improved links with research; 
•  Training: training for researchers and technology  managers with an overall 
aim of promoting an innovation culture. 

The suggested framework for evaluating RTD interventions  presented in Box 
32 is based on two  broad categories  of activities: the strengthening  of  S&T 

Transversal evaluation of impacts on intervention priorities 

potential and support for innovation. These categories group together a 
number of themes and sub­themes, some of which may warrant a more 
focused analysis depending on the RTD profile of the region under examina­
tion and the type of intervention. 

Box 32 ­ Framework for the evaluation of Structural Fund 

interventions in the domain of RTD 

Structural Fund interventions in the domain of RTD 

Strengthening ( 

1 

1 

■ 

I 

Enhancer 
of 
scientific 
potential 

Construction of 
university 
research 
centres and 

Grants for 
equipment 

Grants for 
projects 

Etc. 

1 

1 

1 

1 1 

1 1 

of an 111 links between I II of financial II 
"innovation II research | || framework for 11 
E ES | 

" ' ■ ­ " ■· * 

I 

ï 1 

Technology 
transfer 

1 1 

Seed and 
venture capital 

Construction of 
technology centres, 
science parks, 
support for SME 
research capacity 

Grants for equipping 
technology centres, 
measurement and 
testing equipment 

Training 

Researcher 
mobility 

Stimulation of 
activity 

Diffusion of 
results 

Grant funding of 
development projects 

Awareness 
raising activities 

Etc 

Etc. 

Etc. 

Promotion of 

Etc. 

Linkages 
between firms, 
universities. 
centres of 
research 

• Clarifying the objectives of RTD interventions 

In the case of RTD interventions, an objectives tree can be used to link up the 
outputs and results of RTD interventions to the specific objectives envisaged 
within the framework of the Structural Funds (see Box 33). 

Evaluating  impacts  in  the  domain  of RTD 

Box 33 -  Example  of a thematic  objectives tree  for 

RTD  interventions 

G l o b al  o b j e c t i v es 

E x a m p l es  of  s p e c i f ic  o b j e c t i v es 

Increased  technology 
transfer  between 
institutions  and 
enterprises 

• Increased number  of  collaborative 

projects  between SMEs and 
research  institutes 

• Increased number  of  post-graduate 

in-firm  placements 

Increased  Innovation 
by  enterprises, 
particularly  SMEs 

• Increased  number of  inter-firm 

RTD collaborations  Involving SMEs 

• Increased  number  of SMEs 

performing RTD 

Improved RTD 
Infrastructure 

•  Increased  number  of S&T parks 
•  Improved  university  research 

facilities 

•  Increased  number  of  new 

university  research  facilities 

Increased 
availability  of 
suitably 
qualified 
researchers  and 
technicians 

•  Increased  number of places on 

science and technology  oriented 
university  courses 

•  Increased  number  of  research 

grants available 

Links  between  intermediate  objectives 

Reduced  inter· 
regional  and 
infra-regional 
disparities  in 
RTD capacity 

Increased 
economic 
and social 
cohesion 

While specific  objectives  are strictly  subordinate to the intermediate  objec­
tives  in the tree, the  latter  are not  independent  of  each other.  For example, 
attaining the objective  of  improved  RTD infrastructure  is likely, by  providing 
facilities  for  firms  wishing  to  perform  RTD, to  contribute  to  innovation  by 
SMEs. 

While giving an overview of the structure of the objectives of an intervention, 
the objectives tree also provides an indication of what an impact typology for 
RTD-oriented  Structural  Fund interventions  might  look  like. Using the  inter­
mediate  objectives  of an intervention  as a starting  point, a typology  of  RTD 
impacts might look like this: 
•  Technology transfer; 
•  Industrial RTD; 
•  RTD infrastructure; 
•  Human resources. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

•  Net effects of RTD  interventions 

The net effects of structural interventions in the domain of RTD are difficult to 
evaluate. Impacts, especially at the socio-economic  level, often occur a long 
time after  RTD investments  have taken  place, which  makes their  attribution 
to  individual  interventions  difficult  to  ascertain.  RTD  impacts  are  heteroge­
neous and must therefore  be estimated using an array of indicators: sales of 
innovatory  products,  processes  and  services;  R&D  employment  created; 
technologies  acquired;  networks  created;  number  of  PhDs  obtained,  etc. 
They cannot therefore be summarised in the form of a single  indicator. 

Generally  speaking, the  estimation  of  net  effects  emphasises  the  assess­
ment  of  deadweight  (i.e. the  change  which  would  have  occurred  to  the 
activity  under examination  in a policy-off  situation).  However,  it is  important 
to  note  that  although  public  funding  dominates  upstream  research,  at  the 
industry  level  the  private  versus  public  funding  balance  varies  between 
sectors  and  the  importance  of  deadweight  will,  as  a consequence,  vary 
accordingly. 

It is for these reasons that the estimation of net effects  of  RTD  interventions 
is generally  undertaken  at the  results  level, by focusing  on changes  in RTD 
expenditure  resulting from an intervention in the domain. 

Example: From gross to net effects in the support for innovation in SMEs 
Assuming  that  support  for  a  venture  capital  fund  to  finance  research  projects 
performed  by  SMEs  has  led to  435  projects  being  launched  at  a cost  of  600  million 
euro.  It is however found that  120 projects  costing  150 million euro would  have  gone 
ahead without support (deadweight). The net effect of the intervention would therefore 
amount to 450 million euro. 

Another factor  which has to  be taken  into account  is the leverage effect (i.e. 
the increase  in privately financed  RTD which  results directly  from  Structural 
Fund expenditure  in RTD projects). Leverage along with substitution  is often 
explicitly  sought  by  policy  makers.  The  latter  results  from  interventions 
aiming  to  persuade  high-tech  enterprises  and  research  institutes  from 
outside the target  region to  relocate within the region in order to  increase its 
RTD capacity.  In the case where diversification of the regional economy  is a 
prime consideration,  substitution  of  innovative  activities for  activities  those 
in declining low-tech sectors will also be an explicit objective. 

Evaluating impacts in the domain of RTD 

■ Examples of i m p a c ts by area of intervention 

A number of different examples of RTD projects are presented below, cor­

responding with the three broad areas of intervention of the Structural 

Funds. 

* Basic infrastructure 

Interventions in the area of basic infrastructure usually have an indirect effect 

on RTD. For example, improvements in telecommunications networks may 

increase the scope for using telematics for long distance RTD collaboration. 

Infrastructure co­financed by Structural Fund interventions, having a more 

direct effect on RTD, includes facilities for university research, science and 

technology parks, etc. 

Example: The construction of a science and technology park in Northern Italy 
The Tecnoparco del Lago Maggiore in Piedmont was conceived to help entrepreneurs 
interested in technology and innovation to turn their ideas into successful businesses. 
The park, co­funded by the ERDF and the Piedmont region, provides rented facilities 
for research projects. Tenants of the Park also benefit from back­up services 
(business counselling, environmental safety and certification, computer­aided design, 
partner and finance search and the maintenance of meeting and training facilities) and 
support for innovative technology projects. The long­term objective of this project is to 
attract research investment and initiatives from outside the region and to stimulate the 
region's economy, allowing traditional industries to be replaced by innovative SMEs. 

Outputs  O 

Construction of 
buildings and 
provision of 
equipment 

Firms setting 
up in the park 

Effects 

New firms becoming 
established 

Jobs directly created by 
the firms located In the park 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

•  Productive  environment 

Interventions  in  the  area  of  the  productive  environment  cover  a  large 

spectrum  of  possible  measures  ranging  from  direct  support  for  RTD  invest­

ments to  support  for  services  aimed at assisting  the technological  activities 

of  firms,  particularly  SMEs  and  NTBFs,  and  at  creating  networks  involving 

both public  and private  institutions. 

Example: Promoting technology transfer in Aquitaine (France) 
The  economy  of Aquitaine  is dominated  by the presence  of large  enterprises  in  the 
armaments,  space  and aeronautics  sectors.  These  sectors  have, for a number of 
years, been  undergoing  large scale  restructuring  which has had dramatic  results not 
only  on their  own activities,  but also  on the activities  of  local  subcontractors  in a 
diverse range of sectors. One of the central priorities established in the  1994-96  SPD 
for  the region  was to adapt  the existing  potential  of the region's  SMEs  through a 
coherent policy for innovation and technology transfer. One of the interventions  within 
the  framework  of this  policy  has been  the setting  up of CREATI Aquitaine,  (Centre 
Régional d'Appui Technique et d'Innovation)  which in 1997 brought  together  10 large 
regional enterprises committed to developing a support  network of technical  services 
for  local  subcontractors  as well  as promoting  technology  transfer. The CREATI has 
also  been  involved  in a number  of projects  launched in partnership  between  partici­
pants and SMEs. These projects  have also benefited from co-operation  with  regional 
universities. 

Outputs 

> 

Effects 

O 

Advice  sessions 
financed  with 
experts in 
technology  Issues 

SMEs  entering  into 
collaborative  arrangements 

Production  of new 
products,  processes and 
services  incorporating 
acquired  technology 

Evaluating impacts in the domain of RTD 

• Human resources 

Interventions in favour of human resources cover such measures as financing 
university research grants in specific areas (e.g. technology transfer, etc.), 
vocational training in high­tech subjects, encouraging links between public 
research and industry (e.g. placements for researchers in industry, etc.). 

Example: Centros Tutelados (Spain) 

This project, set up by the Basque government in 1992, aims to develop a technologi­

cal network, support growth and create j o bs in local industries. The Centres support 

the development of local enterprises by undertaking their o wn research, by monitoring 

of scientific p r o g r e ss a c r o ss the w o r l d, a nd by their t r a i n i ng p r o g r a m m es w h i ch 

represent one of the major aspects of the centres' activities. Highly qualified technical 

and research staff undergo a t wo year training period in the Centres before going to 

work in regional SMEs to ensure that the knowledge they have gained is diffused into 

local industries. The Centres also organise training courses for local enterprises. Up to 

1997 over 1,400 trainees in nearly 900 enterprises had benefited from this service. 

Outputs 

> 

Organisation of 
external training 
courses 

Financing of places 
for technicians and 
researchers within 
the Centres 

Effects 

> 

Increased qualifications and 
skills of employees of 
client firms 

Increased overall skills level 
within participant firms 

Placing of researchers 
in regional companies 

Companies producing 
new or improved 
products, processes 
or services 

■ Defining and selecting indicators 

The evaluation of RTD interventions requires the construction of a system of 
indicators using both standard RTD indicators and ad hoc indicators that 
take into account the particularities of the specific evaluation. The monito­
ring of interventions should generally provide data for resource, output and 
result indicators, whilst the gathering of information on impacts requires the 
use of active data collection techniques. 

Resource indicators such as GERD, BERD and GOVERD5 give information 
on the structure of RTD financing and the consequent distribution of activity 
between public research and industrial RTD. When related to population or 
GDP, comparisons can be made between regions and countries. 

5. GERD ­ Gross Domestic Expenditure on Research and Development; BERD ­ Business 
Expenditure on Research and Development; GOVERD ­ Government Expenditure on 
Research and Development. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Output  indicators  focus  on the activities  of  operators  (contacts  with  addres­
sees, applications  processed, etc.), the number  of  hours of consultancy  and 
training sessions, number of supported research projects, networks and colla­
borative ventures financed, and the creation of infrastructure such as science 
parks, research facilities, etc. (measured in square metres or hectares). 

Result  indicators, in the case of  RDT infrastructure could  provide  information 
on the quality /  cost ratio of, for example, establishing a firm in a science and 
technology  park, or the  attractiveness  of the  park and  its environs to  resear­
chers. As for technology  services, results  indicators  largely focus on  measure 
addressees'  satisfaction.  Indicators  at  this  level  should  also  measure  the 
amount  of  private  money  invested  in the  supported  projects,  the  leverage 
effect  of an intervention. Another  useful  indicator  is the proportion of addres­
sees that undertaking RTD for the first time with the support of the programme. 

Results  indicators  should  also  provide  information  on  networking  between 
firms and institutes when this is the objective of an intervention. 

Impact  indicators  focus  on  wider  and  more  long-term  technological  and 
socio-economic  impacts. Because the developmental  effects  of  RTD occur 
over a long period and some specific impacts are difficult to measure (i.e. the 
knowledge  and  know-how  acquired  by  individuals),  impact  indicators  are 
generally limited to those informing on specific socio-economic  impacts of a 
more  tangible  nature,  for  example,  the  number  of  new  products  and 
processes  marketed,  their  sales  and  the  sales  of  related  patents  and 
licences, increased value added and direct employment effects  (researchers 
and technicians measured in FTEs). Because public and private research are 
increasingly  intertwined  -  indeed,  collaboration  between  public  research 
centres  and  SMEs  and  the  joint  development  of  innovations  is  one  of  the 
objectives  of  interventions  -  many  institutes  are  themselves  starting  to 
behave  like  private  firms:  patenting  innovations,  selling  licences  and 
marketing products, etc. Indicators for university  and research centre activi­
ties  within  the  framework  of  Structural  Fund  interventions  should  therefore 
focus on the more industrial and commercial manifestations  of public  sector 
research. 

By way of an example, a system of indicators that could be used to track the 
production of impacts arising from a project providing financial and informa­
tional  support  for  technology  acquisition  by  SMEs  is  presented  in  Box  34. 

Evaluating  impacts  in  the  domain  of RTD 

Box 34 -  Example  of a system of indicators for an  intervention 

providing financial and  informational  support for 
technology  acquisition  by  SMEs 

Programme  Indicators 
(Related to the intervention  and its effects) 

Context  Indicators 
(Related to the assisted  area) 

Output  indicators 

Activity of the 
operators 

Number  of contacts  between  operators 
andfimns looking for support  for 
technology  acquisition 
Number of advice sessions  organised 
with  experts  in technology  acquisition 

Number of addressees  Number  of SMEs receiving  support 

Volume of technical 
services  provided 

Hours of  consultancy  and 
training  services  dispensed 

Result  indicators 

Satisfaction 

%  of addressees  satisfied or  very 
satisfied with the  service 

Leverage effect 

Investment  in acquisition oriented  RTD 
by firms  benefiting from  financial  support 

Ratio of  BERD to  GERD 

Impact  indicators 

Internal technology 
transfer 

External technology 
transfer 

Technology 
acquisition 

Economic  impact 

Number  of  collaborative  arrangements 
between  research institutions  and 
assisted enterprises  after  one year 

Number  of assisted  firms  purchasing 
patents, licences  or entering  Into R&D 
collaboration  with foreign firms  after 
one year 

Number  of  new  products  or  processes 
incorporating  the acquired  technology 
marketed  by  assisted  firms  after 
one year 

% of  managers  of recipient firms who 
declare  that  the  Intervention  has 
relaxed constraints  in terms  of  access 
to new  technologies 

%  of  regional  managers  who  declare 
that  access to  new  technologies 
is a major  constraint 

Value added 

Employment 

Value added generated  after  one/three 
years in euro /  year 

Value added  generated 
in the  region  by  SMEs 

Net  employment  created after 
one/three years  (of which  women) 

Total number  of jobs  in SMEs 
(FTEs of  which  held by  women) 

Source: Volume 2 of  the MEANS  Collection 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Whilst these  indicators  concentrate  on the quantification  of  outputs,  results 
and impacts, it is also useful to have qualitative information about the effects 
of  interventions.  For example, data on the  number  of collaborative  projects 
between  research  institutes and firms, or the  number  of  innovations  arising 
from them can be enriched by qualitative information about the venture. But, 
because  this  sort  of  collaboration  is  intended  to  improve  the  competitive 
position  of  the  participating  enterprises,  for  reasons  of  commercial  confi­
dentiality  standard peer review or expert  panel procedures used for the eva­
luation  of  public  research  institutes  are  not  particularly  suited  to  this  task. 
However,  partners  could  be  asked  to  assess  the  quality  of  their  venture. A 
possible scoring  system for  measuring the  quality  of collaborative  ventures 
is presented in Box 35. 

Box 35  -  Example  of a score  card for  a joint  research  venture 

Assessment  of  the: 

•  Strategic  relevance  of the joint  research  activity  to  your  commercial 

activities 

0 

12 

3 

4 

•  Quality  of  the  research  performed  within the  project 

0 

12 

3 

4 

•  Quality  of  the  contribution  made  by your  partner(s) 
0 

12 

4 

3 

•  Extent  of  learning  engendered  in your  organisation  by 

the  partnership 

0 

12 

3 

4 

Legend: 
0  -  Very Poor;  1 - Poor; 2  - Low;  3  -  Good;  4  -  Excellent 

Evaluating impacts in the domain of RTD 

■ Suitable sources of information 

When evaluating RTD interventions within the framework of the Structural 
Funds, it is important to remember that the tools used must be suited to 
examining complex phenomena in detail. One must also bear in mind that 
the availability of monitoring data at the regional level is sometimes rather 
patchy, so that in some regions primary data collection will be an expensive 
and time­consuming part of an evaluation. 

Whilst the evaluation of RTD interventions is therefore likely to be heavily 
dependant upon primary data, sources do exist which provide relevant sta­
tistics at the firm level. 

Innovation surveys focus either directly on innovations or on firms and their 
innovatory activities. In the first instance, survey data on innovations 
deemed technologically or economically significant are collected. The 
second type has the advantage of taking into account both innovatory and 
non­innovatory firms as well as the users of innovations, allowing factors 
which restrict innovation and the existence and intensity of user­producer 
linkages to be examined. 

Data on patents is also available at the firm level, giving the technological 
classification of the innovation. Unlike data resulting from innovation 
surveys, patent data is available over a long period. Patents registered at the 
European Patents Office are available on the EPAT database, which has 
been in operation since 1978. 

Data are generally available at the national level (e.g. l'Observatoire des Sciences 
et les Techniques produces an annual report on S&T indicators for France) and 
statistics on national R&D expenditure are provided annually by the OECD. 

Relatively up­to­date European RTD statistics suitable for constructing 
context indicators are available from Eurostat in an annual publication which 
covers data on R&D personnel, R&D expenditure and patent applications 
broken down by region as well as government budget R&D appropriations. 
This source also provides "nearcasts" that provide estimates of current sta­
tistics that are being compiled. 

■ Appropriate tools for evaluating structural interventions 

in the domain of RTD 

While a detailed treatment of evaluation tools is provided in Volume 3 of the 
MEANS Collection, a number of tools particularly suited to evaluating 
impacts on RTD are presented below. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

While a great deal of experience  has been accumulated  in this domain, and 
standard  tools  for  this  type  of  intervention  have  a  long  track  record,  the 
increased  importance  of  interventions  towards  generating  "opportunities" 
for innovation through the creation of better conditions for learning, partner­
ships and networks, has led to new challenges in the area of RTD evaluation. 

First, these  effect  arising  from  RTD interventions  are difficult  to  realistically 
reduce to  simple  monetary variables, which  means that tools such  as cost-
benefit  analysis  and  return on  investment  type  analyses  are of  limited  use. 
This  latter  technique  is  restricted  to  analysing  private  returns  arising  from 
RTD expenditure, it therefore significantly underestimates total returns, since 
a large part of these are social returns generated through RTD spillovers  and 
externalities  (i.e. effects, either positive or negative, that to some extent take 
place  through  non-market  mechanisms  and  are  therefore  not  fully 
accounted for within the price system). 

Secondly,  because of the complexity  of the chain of  causality  between  RTD 
activities  and  their  socio-economic  effects,  increased  emphasis  has  to  be 
placed  on the  use of  tools  allowing  in-depth  qualitative  study. Thirdly,  the 
recognition  of  the  effects  of  RTD  spillovers  on  the  performance  of  firms 
requires  that  evaluations  simultaneously  take  a wider  approach  than  that 
afforded  by  project-by-project  examinations  of  interventions.  Finally,  the 
evolution in the nature of RTD interventions, the increased emphasis on eva­
luating  interventions  in the  area of  industrial  RTD and the  need to  take fully 
into account the spatial objectives of structural interventions, mean that eva­
luation techniques  in this domain are less able to rely on classic  RTD evalua­
tion tools such as expert  panels and bibliometric  indicators  in which a great 
deal of experience has been accumulated. 

In terms  of  the  alternative  approach  required, concrete  evaluation  expe­
rience  remains  rather  limited. The  tools  presented  below  are those  which 
have to a limited extent been used in evaluations in this domain or seem par­
ticularly suited to examining impacts of RTD oriented  interventions. 

By using questionnaire  surveys that  are carefully tailored to the characteris­
tics  of  the  population  concerned,  relatively  simple,  and  clearly  structured 
and designed to cover of the RTD themes depending on their importance  to 
the region or regions studied, evaluators can build up a picture of a region's 
innovation  system  in terms of the type and extent  of interventions, how they 
are implemented, the nature and extent of industrial  RTD, the effect of  inter­
ventions  on  regional  RTD  policies  and  the  organisation  of  RTD  activities. 

Evaluating  impacts  in  the  domain  of RTD 

Whilst informing on resource use, issues of organisational and  management 
quality, the  effectiveness  of  instruments  and  mechanisms,  surveys  should 
largely  focus  on  providing  quantitative  data  on  the  outputs  and  results  of 
interventions.  Questionnaires  may  also  have to  be  designed  to  reflect  the 
cover of programmes (i.e. regional or multi-regional). 

Experience from the EUREKA Initiative has shown that questionnaires  alone 
are  not  able  to  capture  the  richness  and  complexity  of  socio-economic 
effects  arising  from  RTD  interventions.  Furthermore,  experience  from 
Norway has shown that such impacts as extra sales, productivity  gains, net­
working  benefits  and  increased  skill-levels  may  take  over  ten  years  to 
become  evident.  Questionnaires  therefore  need  to  be  supplemented  by 
face-to-face interviews with key figures in the domain of RTD, focusing  both 
on  actual  and  expected  impacts.  The  interviewees  will  be  drawn  from 
programme  actors:  regional  and  local  programme  sponsors,  universities, 
etc.  More  importantly,  they  will  also  include  addressees  such  as  research 
centres,  SMEs  and  larger  companies  involved  in  programmes  along  with 
firms not directly involved (e.g. firms buying process innovations, rather than 
developing them themselves). 

Expert  panels  can  be  used  for  evaluating  interventions  targeted  at  public 
research. However,  because secrecy  is one of the ways firms  maintain their 
innovative  edge,  they  are  of  limited  use  when  private  or  collaborative 
research  activities  need to  be evaluated. On the other  hand, expert  panels 
are useful tools for undertaking  "quick and dirty" evaluations, when these are 
required. 

A number of carefully selected case studies can be used to provide  in-depth 
analysis  and comparison  of  individual  projects  and  measures. These  might 
focus  on projects of outstanding quality  in the area of the promotion of RTD 
capability.  This  technique  may  be  particularly  useful  in examining  internal 
(e.g. the setting  up of a RTD department)  or external (e.g. linkages  between 
firms or between firms and research centres) organisational changes engen­
dered by interventions. Case studies may also cover an entire region in order 
to better appreciate both the functioning of regional innovation systems and 
the  long-term  sustainability  of  a  region's  development.  In this  case  they 
might  look  at  competitiveness,  economic  development  and  employment, 
focusing on the demand side of RTD. Case studies should be designed with 
issues of extrapolation, aggregation and comparison in mind. 

Transversal evaluation of impacts on intervention priorities 

■ S u m m a ry 
The importance of RTD, especially at the industry level, is recognised as an 
essential prerequisite of sustained economic development. This recognition 
has led to a large increase in Structural Funds expenditure in this domain, 
accompanied by the reorientation of interventions towards downstream 
technological development. 

When estimating the net impacts of interventions, in addition to identifying 
deadweight additionality of Structural Fund expenditure is an important 
factor that should be taken into account. The assessment of leverage effects 
also represents an important element in appreciating the impact of interven­
tions on RTD. 

The evaluation of RTD interventions poses particular problems because of 
the intangible nature of many specific impacts and the long­term nature of 
socio­economic impacts. Moreover, the increased emphasis on the role of 
private sector RTD and the need to evaluate at a higher level than individual 
projects, mean that evaluations are less able to rely on assessments of 
expert panels, the established tool of RTD evaluation. 

■ For more information see... 

Archibugi D. and Pianta M. (1996), "Innovation surveys and patents as tech­
nology indicators: the state of the art" in: Innovation, Patents and Technolo­
gical Strategies. Paris: OECD. 
Commission européenne (1998), 2e Rapport européen sur les indicateurs 
scientifiques et technologiques 1998, Luxembourg: Office des publications 
officielles des Communautés européennes. 

European Commission (1999), Impact of Structural Funds (1994­1999) on 
Research Technology Development and Innovation (RTDI) in Objective 1 and 
6 Regions, European Commission, DG XVI. 
Eurostat (1997), R&D Annual Statistics 1997. Luxembourg: Eurostat. 
Georghiou L. (1998), "Issues in the Evaluation of Innovation and Technology 
Policy", Evaluation, Vol 4(1), 37­51. 
Lundvall B.­A. (éd.) (1992), National Systems of Innovation: Towards a 
Theory of Innovation and Interactive Learning. London: Pinter. 
OECD. (1994), The Measurement of Scientific and Technological Activities: 
Proposed Standard Practice for Surveys of Research and Experimental 
Development ­ Frascati Manual 1993. Paris: OECD. 
OECD. (1998), Policy Evaluation in Innovation and Technology: Towards 
Best Practices, OECD Proceedings, OECD: Paris. 
OST. (1996), Science & Technologie: Indicateurs 1996. Paris: Economica. 

Evaluating impacts arising from interventions in the 
domains of transport, telecommunications and tourism 

A small amount of experience in thematic transnational evaluations has been 
gained in the domains of transport, telecommunications and tourism. This 
experience can be characterised as both limited in terms of its transnational 
aspect and predominantly top­down in terms of evaluation design. The 
ambition of this chapter is to draw on this available experience and to com­
plement it with a number of points that should be considered when evalua­
ting interventions in these domains. 

■ Transport and telecommunications 

• Structural interventions in the domain of transport 

Structural Fund interventions in the domain of transport are part of a larger 
Community transport policy which also includes EIB funding and the Trans 
European Network (TEN) initiative. In addition to these instruments focused 
mainly on the funding of transport infrastructure projects, a substantial body 
of legislation, geared towards the harmonisation and liberalisation of market 
conditions within particular transport sectors, now exists. 

Transport infrastructure projects are funded by means of grants and loans 
that are largely allocated on the basis of project­specific appraisal and 
regional development strategy. For the cohesion countries, from both a 
physical and financial point of view, the planning and financing of transport 
infrastructure are among the most important areas of EU policy. 

Transport infrastructure investments are intended to affect socio­economic 
cohesion in two major ways: 
• by eliminating bottlenecks and reducing travelling times, costs and constraints 
on industry (e.g. of transporting raw materials, semi­finished and finished 
products, etc.) are reduced and economic productivity is improved (e.g. by 
allowing new organisational techniques such as just­in­time operations); 
• by increasing accessibility and mobility between regions, therefore 
reducing the effects of peripherality on business and individuals. 

This traditional argument is that improvements in transport and, in particular, 
transport infrastructure, foster economic development and reduce dispari­
ties between regions. This hypothesis is, however, questioned by some who 
argue that improved infrastructure may in some cases worsen disparities, at 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

least  in the  short-term,  by  opening  up  regions  to  imports  originating  from 
lower-cost  sources.  In addition to this  possible  undesirable  effect,  interven­
tions  within  the  framework  of  the  TEN  initiative,  which  account  for  the 
majority of investments  in this domain, are not aimed at achieving  cohesion 
but at increasing the overall competitiveness  of the  EU as a whole. Interven­
tions of this kind may therefore have greater positive effects in non-cohesion 
countries  in which  the  bulk  of  the  investment  in TEN  projects  occurs  and 
which  are  more  favoured  in  terms  of  accessibility  by  their  geographical 
position.  Nevertheless,  the  role  of  the  Community  in  identifying,  planning 
and  co-financing  the  TEN  is seen  as  an  important  factor  in countering  the 
national bias of most transport infrastructure projects. It also seems  possible 
that the tendency  of  improved  infrastructure to attract  individuals  and  busi­
nesses  to  the  more  dynamic  regional  poles  of  activity  at  the  expense  of 
cohesion  may be largely overestimated.  In the medium to  longer term it can 
be expected that there will be a degree of rebalancing of activity as the costs 
and  disadvantages  of  large  agglomerations  (e.g. office  rents,  pollution, 
congestion, etc.) start to outweigh the benefits. 

In addition  to  the  upgrading  and  construction  of  infrastructure,  there  is  a 
growing  emphasis  on  more  efficient  use of  existing  infrastructure  through 
high-technology  solutions  to  problems  of  road  congestion  and  transport 
related  environmental  problems  (e.g. Advanced Transport  Telematics  tech­
nologies). The effect  of this development  on cohesion  may  prove to  be  less 
than  was  bargained  for  since  this  sort  of  solution  seems  more  suited  to 
developed rather than peripheral parts of the Community. 

The  provision  of  transport  infrastructure  may  also  have  negative  effects 
arising  from  such  problems  as  road  congestion,  due  to  the  apparent 
capacity  for  demand  for  road space to  outstrip  supply.  Negative  effects  on 
the quality of life may result from air and noise pollution generated by the use 
of the infrastructure. 

The evaluation of the effects of interventions in this domain will depend  upon 
what  is considered  as the  contribution  of  transport  to  the  economy.  That 
contribution  largely  translates  into  reduced  costs  and  constraints  for  busi­
nesses  and  individuals.  It  also  has  a  strong  qualitative  aspect,  notably  in 
terms of safety and environment. 

•  Structural interventions in the domain of telecommunications 

The transition  from  an  industrial  to  an  information-based  service  economy 
holds  a  number  of  opportunities  for  the  more  peripheral  regions,  since  the 

Evaluating  impacts  of  interventions 

in  the  domains  of 

transport, 

effects  of  geographical  distance  on  business  activity  is  consequently 
reduced.  However,  the  possibility  of  benefiting  from  these  opportunities, 
depends  on these  regions  being  well  equipped  in télécoms  infrastructure 
and services. 

In light  of  advances  in technology,  which  have meant  that  telecommunica­
tions and  information technology  are increasingly  indistinguishable and that 
national telecommunications  monopolies  are  no  longer  tenable, the  role  of 
the EU has been twofold: 
•  First, it has put into place a common framework for a liberalised  European 
télécoms sector within  which the Member States  regulate their telecommu­
nications industries; 

•  Secondly,  it has promoted collaborative télécoms  RTD and supported  the 
modernisation of télécoms infrastructure. 

The effects of liberalisation policies in the domain of telecommunications  will 
lead to  an overall  improvement  in service  quality  and  efficiency.  However, 
their short-term  effects will  probably  accentuate  problems of cohesion. The 
immediate  effects  will  be  a  reduction  in  costs  for  heavy  consumers  of 
télécoms services in the more developed regions as the  cross-subsidisation 
of  services  stops  and  tariffs  are  rebalanced  to  reflect  real  costs.  For 
consumers  in  more  peripheral  regions,  the  same  process  may  lead  to 
increased  costs. Any  sizeable  positive  impact  occurring  in these  regions  is 
likely to be seen over the medium to  long-term as advantages  of  locating  in 
less  central  regions  outstrip  the  agglomeration  benefits  of  a  more  central 
locations. 

The  structural  change  brought  about  by  advances  in  telecommunications 
technology  not only  has an effect on institutions  and organisations, but also 
on the way  individuals  live their  lives. This  has led to  use of the term  "Infor­
mation  Society"  to  encompass  the  broader  aspects  of  this  process.  A 
dimension  of  increasing  importance  in the  "Information  Society"  is that  of 
telework  which  enables  employees  and  independent  workers  to  work  in 
remote  locations  using  the  telecommunication  network  to  receive  and 
transmit data, market their products, search for clients, etc. 

An  increased  focus  on these  broader  aspects  has  led to  the  EU playing  a 
highly  visible  role  in this  area.  However,  infrastructure  and  services  are 
mainly  financed  by the  private  sector  and  Structural  Fund  expenditure  has 
remained  modest,  amounting  to  less than  3%  of total  telecommunications 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

investments. Nevertheless, in lagging regions it has played an important role 
in the development of modern telecom  networks. 

In the case of telecommunications  Trans-European  Networks, the role of the 
Commission  is that of sharing financial  risk  by providing support  for  econo­
mically  beneficial  projects  which  are  of  common  interest  but  uncertain  or 
inadequate  in terms  of  financial  profitability.  Support  is provided  through  a 
number of  instruments: 
•  co-financing of feasibility studies and other technical  measures; 
•  subsidised interest rates; 
•  contributions to loan guarantee premiums. 

•  Some key issues for evaluations in the domains of transport and tele­

communications 

A  large proportion  of  Structural  Fund expenditure  in these two  domains  has 
tended  to  be concentrated  in support  measures  for  infrastructure  projects, 
especially  in the case of transport. Whilst this situation  is likely to continue in 
the foreseeable future, the increased interest in "software" transport  solutions 
and  Information  and  Communication  Technologies  (ICT) applications  deve­
lopment  means  that  less tangible  effects  will  need  to  be  increasingly  taken 
into account  when evaluations are performed. These two domains  have also 
been  the  subject  of  deregulation  policies  that  are  having,  or  will  have 
important  structural  repercussions. The  identification  of  synergies  (either 
negative  or  positive)  between  liberalisation  policies  and  regulatory  frame­
works  and  interventions  with  explicit  cohesion  objectives  represent  an 
important  element  of  evaluation. Another  important  aspect  of  infrastructure 
projects,  relating  to  cohesion,  is their  effect  on  accessibility  to  peripheral 
locations. 

Infrastructure  projects are generally  large and relatively few in number. They 
usually  concern  projects  that  either  take  the  form  of  the  construction  or 
modernisation  of  a  link  between  a  number  of  nodes  (e.g. roads,  railways, 
cables,  pipes,  etc.)  or  are  nodes  themselves  (e.g.  ports,  multimodal 
exchanges, and telephone exchanges). 

One  way  of  establishing  the  relative  importance  of  infrastructure  projects 
prior to  selecting a number  of them for  more detailed  analysis (i.e. for  inter­
mediate or ex-post  evaluation)  is, for example, to  rank them  in terms  of  the 
size of the intervention in budgetary terms. The importance of TEN projects, 

Evaluating  impacts  of  interventions 

in  the  domains  of 

transport, 

for example, is reflected  by a rating system which identifies  priority  projects 
and  other  projects  according  to  their  size. This  approach  is not,  however, 
necessarily  sufficient  to  comprehend  the full  importance  of  a project.  Other 
factors  that  could  be taken  into  account  when  ranking  projects  include 
social  and  financial  benefits  of  the  project,  and  the  participation  of  the 
private  sector.  Since  each  project  is  a component  of  a  system,  it  is  also 
important  to  examine  their  strategic  importance  within  the  system  itself.  It 
may  be  possible to  do  this  by examining the  importance  of the corridor  on 
which  the  project  is  situated,  in  relation  to  other  corridors  in the  region. 
Similarly,  it  may  be  possible  to  classify  nodes  according  to  their  strategic 
importance in the system. 

Interventions  in transport  infrastructure  occur  at the  start  of  a  long  causal 
chain.  Because  of  the  distance  between  an  intervention  and  its  eventual 
socio-economic  impacts (i.e. at the level of employment and GNP growth), is 
difficult  to  establish  clear  causal  links  between  this type  of  project  (i.e. the 
outputs  of  an  intervention)  and  these  global  impacts.  Evaluations  should 
therefore  concentrate  on  tracking  outputs,  results  and  specific  impacts  of 
interventions.  Furthermore,  because  medium  to  long-term  effects  of  infra­
structure  projects  on  cohesion  will  probably  to  some  extent  contradict  the 
short term  effects  (i.e. over time  economies  of agglomeration  may  become 
negative  and  relative  prices  will favour  locations  outside  central  poles  of 
activity), the timescale  of  evaluation  is of  the  utmost  importance  if the  full 
impact of an intervention is to be observed. 

In  addition  to  the  direct  effects  that  supported  infrastructure  projects 
produce,  they  also  give  rise  to  indirect  or  induced  effects  by, for  example, 
reducing  constraints  on the economic  activity  of indirect addressees. In this 
way  they  can  give  rise to  supply-side  effects  by  making  productive  struc­
tures more efficient  (e.g. by reducing delivery times) and reducing costs (e.g. 
by  reducing  distances,  travelling  times,  etc.). They  also  stimulate  demand 
(demand-side  effects)  by  mobilising  human  and  material  resources  during 
the construction period. 

While  Volume  2  of  the  MEANS  Collection  presents  an  information  sheet 
devoted to indicators for the domain of transport infrastructure, by way of an 
example, a system  of  indicators that  could  be used to track the  production 
of impacts arising from an upgrading project for a railway line is presented in 
Box 36. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Box 36  -  Example  of a  system  of indicators for  upgrading 
of a  railway  line for freight  operations 
Programme Indicators 
(Related to the Intervention and its effects) 

Context Indicators 
(Related to the assisted area) 

Output indicators 

Progress 

Quantity 

Quality 

Result indicators 

Speed 

Impact indicators 

Traffic flow 

Transport system 

Indirect economic 
effect 

Compliance with project duration 
Rate of completion 

Number of km upgraded 

% of line allowing speeds of at least 
a desired threshold value 
(i.e. 100 km/h) 
% of line length allowing trains of 
heaviest standard load 

Typical commercial speed of 
freight trains in that line 

Average speed on regional lines 

Freight traffic flows on the line 
(Tonnes / year) after one year 

Traffic flow of freight using the new 
infrastructure and withdrawn from road 
(Tonnes / year) 

Traffic flow to and from region 
(Tonnes / year) 

Modal distribution of traffic flows 
within the region (Tonnes / year) 

% of regional managers who declare 
that rail freight accessibility ¡s a major 
constraint 

Source: Volume 2 of the MEANS Collection 

Interventions  in telecommunications  infrastructure are also difficult to  link up 
to  socio-economic  impacts.  In this  case, evaluation  at the  impact  level  will 
normally  focus  on the  increased  use (e.g. penetration  rate, changes  in tele­
communications  traffic,  etc.)  of  teleservices  and  equipment  by  enterprises 
and individuals, dueto  improvements in the network. Where telecommunica­
tion  services  supported  by  Structural  Fund  interventions  are  concerned, 
these tend to be of an experimental nature (e.g. support for the setting up of a 
telework centre, support for on-line education services, etc.). This means that 
as well  as  looking  at  any  specific  socio-economic  impacts  that  they  may 
produce, the survival of projects is an important evaluation issue. 

While  Volume  2  of  the  MEANS  Collection  presents  an  information  sheet 
devoted to  indicators for the domain of telecommunications, an example of 
indicators that could  be used to track  impacts  produced  by the  deployment 
of a broadband  Hybrid Fibre Coax (HFC) network is presented in Box 37. 

Evaluating impacts of interventions in the domains of transport, 

Box 37 ­ Example of a system of indicators for the deployment 

of a broadband Hybrid Fibre Coax (HFC) network 

Programme Indicators 
(Related to the intervention and its effects) 

Context Indicators 
(Related to the assisted area) 

Output Indicators 

Progress 

Quantity 

Result Indicators 

Network performance 

Services supplied 

Impact indicators 

Connections 

% of HFC kms in the total network 
(In the considered region) 
% of buildings / apartments passed 

Compliance with project duration 
Rate of completion 

Number of HFC kms laid 
Number of buildings/apartments 
passed 

Bit rate Improvement 
Number of network failures 

Number of services 
Number of TV channels 
Hours of broadcasting supplied 
(through the network) 

Number of households that have 
connected their equipment (TV, PC) 
to the HFC network (after 6 months) 

Growth of PC and / or CATV 
penetration in households 
in the considered region 

ICT usage 

Number of subscriptions to the services 
Total hours of connections / month 

Growth in teleservices adoption rate 
In households in the considered region 

Source: Volume 2 of the MEANS Collection 

Given that the main actors in the domain of telecommunications are in the 
private sector and that some potentially profitable projects, which might 
have been financed exclusively by private actors, may be assisted by Struc­
tural Funds, the major preoccupation in estimating net effects is the need to 
identify deadweight (i.e. those investments that would have taken place in 
the absence of the intervention and their associated effects). 

■ Tourism 

• Structural interventions in favour of tourism 

The objective of interventions in this domain is to support the diversification 
of regional economies away from excessive dependence on agriculture and, 
to a lesser extent, on declining industries. However, while tourism has the 
capacity to play an important role in supporting socio­economic cohesion, it 
is generally characterised by a number of structural problems: 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

•  tourism demand  is highly seasonal, leading to infrastructures  being  under­
utilised for substantial  periods of the year, and to seasonal / casual  employ­
ment patterns; 

•  value  added  generated  by  tourist  activities  is often  weak,  leading  to  low 
profitability; 
•  product quality is highly variable; 
•  levels of training are relatively low; 
•  transport access is a limiting factor to tourist  development. 

Furthermore, the  development  of  tourism  needs  to  be  managed  in such  a 
way  as to ensure its sustainability  (i.e. by concentrating  on activities that  do 
not produce negative environmental  impacts and by mitigating any  negative 
impacts that do occur). 

Interventions  in the  domain  of  tourism  are therefore  essentially  aimed  at 
resolving these  problems  by, for  example, supporting  the  provision of  infra­
structure, widening  the  range of tourist  products, and  so  on.  Interventions, 
such as support  for the construction  of conference facilities, can  contribute 
to  reducing  seasonality  by  promoting  all  year  round  conference  tourism. 
Similarly, the development  of  new attractions  can  lead to  reduced  pressure 
on established tourist sites during the peak season. This type of  intervention 
also  has  an  impact  on  transport  access  (e.g. by  smoothing  out  seasonal 
travel  needs)  and  on  product  quality  and  profitability  (e.g. by  encouraging 
clients who demand higher quality products and services). 

Since tourism is a labour intensive activity relying heavily on locally  supplied 
products  and locally  based services, it is a domain  in which development  is 
largely endogenously  generated and which  has potentially  important  effects 
on employment and living standards, and thereby on the promotion of socio­
economic cohesion. 

•  Some key issues for evaluations in the domain of tourism 

Although certain interventions support relatively large and visible infrastructure 
projects, a large number of interventions in this domain are fragmented, hete­
rogeneous and  relatively small (e.g. support  for  improving  tourist  accommo­
dation, support for museums, heritage centres, training courses, etc.). 

The  development  of  tourism  is  heavily  dependent  on  qualitative  factors. 
Many  tourists  are  discretionary  visitors  (i.e. they  are  not  constrained  by 

Evaluating  impacts  of  interventions 

in  the  domains  of 

transport, 

factors such as business and family needs to visit a particular region), which 
means that they are highly sensitive to the quality and price of the product or 
service they  buy. The fact  that  tourism  has traditionally  had the status  of a 
commodity  is reflected in its lack of variety and its variable quality (as well as 
low  profitability  and  job  quality,  if  measured  in terms  of  low  training  levels 
and the casual nature of employment). Evaluation of interventions that aim to 
support  the  development  of  tourism  must  therefore  take  these  qualitative 
factors into account. 

In this  domain  it  is  highly  probable  that  a  number  of  projects  would  have 
been undertaken  In the absence of Structural  Fund interventions  (e.g. hotel 
modernisation, golfcourses, campsites, etc.). The calculation of deadweight 
is therefore  important  when  estimating  net  effects.  Since one  of  the  objec­
tives  of  interventions  in tourism  is to  develop  new  attractions  to  reduce 
pressure on established tourist sites during the peak season, substitution  is 
often an aim of interventions. On the other hand, visitors are only able to visit 
a limited number of sites during a stay. This implies that assisted sights  may 
benefit  at the  expense  of  unassisted  sites.  It  is therefore  necessary  to  be 
aware of possible displacement  effects. 

By  way  of  an example,  a system  of  indicators  for  tracking  the  creation  of 
impacts  by  interventions  supporting  the  creation  of  tourist  facilities  is 
presented in Box 38. 

Transversal evaluation of impacts on intervention priorities 

Box 38 ­ Example of support for the creation of tourist facilities 

(museums, theme parks, etc.) 

Programme Indicators Context Indicators 

(Related to the intervention and its effects) (Related to the assisted area) 

Output Indicators 

Activity of the 
operator 

Number of contacts with potential 
addressees 

Number of addressees 

Number of assisted economic units 

Total number of tourist facilities 

Capacity 

Maximum number of visitors per day 

Result Indicators 

Duration of visit 

Time normally required to visit 
the facility (hours) 

Cost of visit 

Average cost per visit of the facilities 
(euro per person) 

Impact indicators 

Fréquentation 

Attractiveness 
to Tourists 

Value added 

Jobs created 

Number of visits per year for supported 
facilities 

% of visits made by tourists from 
outside the region 

Value added generated in euro 
per year 

Value added generated In the 
tourist industry In euro / an 

Net jobs created (% of which are 
occupied women) 

Number of jobs in the tourist industry 

Source: Volume 2 of the MEANS Collection 

A more detailed treatment of indicators and other examples relating to 
tourism are provided in Volume 2 of the MEANS Collection devoted to indi­
cators for monitoring and evaluation. 

■ Examples of tools for evaluating structural interventions 
in t he d o m a i ns of transport, t e l e c o m m u n i c a t i o ns and 
tourism 

Transnational evaluations in the domains of transport and telecommunica­
tions have tended to be approached in a top­down fashion, relying heavily 
on documentary research and secondary data complemented by interviews 
with key actors, expert reports and, in some instances, case studies and the 
scores and weights technique. 

Evaluating 

impacts  of  interventions 

in  the  domains  of 

transport, 

The  objective  of  this  section  is to  present  a small  number  of  examples  of 
tools  used in these domains  within the framework  of transnational  thematic 
evaluations.  A  more  detailed  treatment  of  evaluation  tools  is  provided  in 
Volume 3 of the MEANS Collection. 

•  SWOT: Example of the strategic overview of transport issues for the 
Republic of Ireland 

SWOT analysis (Strengths, Weaknesses, Opportunities and Threats) is parti­
cularly  useful for  performing  a strategic  analysis  at the time  of the  concep­
tion or evaluation  of an infrastructrure  project.  Its implementation  involves a 
number of steps: 
1. Scanning the key issues and problems faced by a region; 
2. Ordering priorities; 
3. Identifying opportunities and threats (factors outside the project, region, etc.); 
4.  Identifying  strengths  and weaknesses  (factors  related to the  characteris­
tics of the project, region, etc.); 
5. Synthesising results and analysis of the SWOT matrix; 
6. Formulating strategies. 

In this  particular  example  a SWOT analysis was  undertaken to  examine the 
position  of  the  Republic  of  Ireland  with  regard  to  the  national  transport 
sector  and  in relation to  the overall  context  in which  it exists. The analysis, 
based on the factors summarised  in the matrix  presented  in Box 39, comes 
to  the  conclusion  that  the  overall  position  of  transport  for  the  Republic  of 
Ireland is one of weakness and inadequacy. This analysis allows the elabora-

Box 39  -  Example  of a SWOT analysis  matrix 

Strengths 

Weaknesses 

Industrial base relatively modern 
Tradition of open economy 
Educated workforce 
Unspoiled environment 
Proximity to sea 
Political stability 

Small size, peripheral location 
Poor indigenous resources 
Quality of domestic infrastructure 
Unsatisfactory access to transport 
Limited finance 
Unemployment, social problems 

Opportunities 

Threats 

SEM - access to mainland 
Role as entrepot for the USA? 
Shannon as European air hub? 
Suited to tourism and leisure 
Telecoms based services 
Mariculture, natural energy 

European centre moving eastwards 
Uncertainty of future EC funding 
Scotland and Wales, post Channel Tunnel 
Large industrial alliances 
Unstable Europe- social / economic 
Emigration of workforce 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

tion  of  strategy  options  aimed  at  eliminating  identified  weaknesses  or  at 
compensating for them through the development  of identified strengths. 

•  Scores and weights: Example of the  "Corridor Ranking  Framework" 

(Greece, Ireland and Portugal) 

This tool  is often  used  in the ex ante appraisal  of  projects, using a standard 
set of selection and ranking criteria. It usually involves two stages: 
1. Screening out projects that do not satisfy the relevant criteria; 
2. Ranking projects that have been retained according to the criteria. 

In  the  case  of  the  "Corridor  Ranking  Framework",  presented  here,  the 
relative  importance  of  corridors  depends  on  the  specific  objectives  that 
underpin  transport  infrastructure  interventions  and  that  contribute  to 
attaining the global objective of socio-economic  cohesion. 

The ranking  of corridors  in this three-country  evaluation was  undertaken  on 
the  basis  of  seven  criteria  that  were  themselves  ranked  according  to  their 
relative importance. The criteria used were the following: 
1. Does the corridor provide linkages to the rest of the European Community? 
2.  Does  the  corridor  connect  important  population  centres  (more  than 
50,000  inhabitants)? 
3. Does the corridor  provide links to external access  points (i.e. airports  and 
seaports)? 
4. Are traffic  volumes on the corridor  high and, as a result, does  congestion 
occur regularly, etc.? 
5. Does the corridor complete land and / or sea routes? 
6. Does the corridor facilitate international traffic? 
7.  Does the  corridor  integrate  remote  or  isolated  regions  with  development 
potential? 

In the case of evaluation of the transport sector in Greece, corridors  between 
economic  centres  were  ranked  according  to  these  criteria  reflecting  a 
number of transport-related  objectives  (e.g. improving  connections to  other 
European  countries,  linking  up  centres  of  high  population  concentration, 
improving links to ports, etc.) The method of scoring allows the relevance of 
the  project to  be taken  into account  by indicating whether the criteria apply 
to  the  project  fully,  partially  or  not  at  all.  The  criteria  are  also  ranked 
according  to  their  overall  importance. This  information  is  presented  in  the 
form of a matrix, an extract of which figures in Box 40. 

Evaluating impacts of interventions in the domains of transport, 

Box 40 

­ Extract from a 

'corridor ranking framework" applied 

to Greece 

1 
Trans­Euro­
pean Road 

• 

O 

• 

O 

o 

Potential 
candidate 
corridors 
for 
develop­
ment 

1) 
Athens ­
Patras 
2) 
Athens ­
Thessaloniki 
3) 
Thessaloniki­
Igoumenitsa 
4) 
Corinthois ­
Kalamata 
5) 
Igoumenitsa 
Patras 

2 
Connects 
population 
concent­
rations 
(cities > 
50k) 

• 

• 

loannina 

3 

O 

Decreaslr 

.apriority. 

3 
Links exter­
nal access 
points 

4 
Has traffic 
volumes 

5 
Completes 
land / sea 
routes 

6 
Facilitates 
International 
traffic 

• 

• 

O 

O 

• 

• 

• 

O 

O 

o 

O 

O 

• 

O 

o 

O 

O 

o 

o 

o 

■w. 

7> 
Inte­
grates 
remote / 
isolated 
areas 

O 

O 

• 

• 

• 

• Criteria fully applies 

φ Criteria applies partially 

O Criteria does not apply 

It is important to note that this example Is only one way of ranking projects. 
Another set of criteria would reflect another set of objectives or a different 
type of intervention (e.g. total project cost, financial return, socio­economic 
value, relationship to objectives, leverage effects, timing considerations, 
local geo­political factors, environmental safety and other related objectives, 
network effects, etc.). 

• Case studies: Example of transport sector projects (Ireland) 

Case studies were employed in Ireland to obtain both quantitative and quali­
tative data on the economic impacts of a number of specific transport 
projects. Surveys of users (local companies, road hauliers) and cost­benefit 
analysis were used to inform the case studies. 

Whilst the case studies selected (a by­pass, a motorway, regional airport 
development and a seaport) provided some information to complement 
macro­level data resulting mainly from secondary sources, their contribution 
to the evaluation could be considered as less than satisfactory for a number 
of reasons: the benefits arising from the projects were considered indivi­

Transversal  evaluation  of  impacts  on  intervention 

priorities 

dually  and  therefore  could  not  be  fully  appreciated  because  the  projects 
form  an integral  component  of  a system; and, secondly, competing  factors 
overwhelmed the specific effects arising from the individual  projects. 

Although case studies in this example provide useful illustrations of interven­
tions  and  their  effects, their  contribution  to the  evaluation  could  have  been 
increased  if a number of methodological  considerations had been taken  into 
account.  Firstly, the  selection  procedure  for  cases  should  be  transparent 
and the importance of the case in relation to overall  policy objectives  needs 
to  be clarified. Secondly, as part  of a transnational  evaluation, the  selection 
of cases should allow for the greatest possible degree of comparison  across 
countries and regions to be undertaken. Finally, the measuring of impacts at 
the  project  level  neglects  the  synergies  and  externalities  generated  by  an 
intervention. In order to take these factors into account, a case study  should 
focus on a number of key projects within a circumscribed geographical area. 

•  DELPHI: Example of forecasting the development of transport tele­
matics technologies 

The advantages of telematics  in the domain of transport  is that they  provide 
a way of optimising the use of existing networks through better traffic  mana­
gement  and  information. They  therefore  represent  a  partial  alternative  to 
road  infrastructure  investment.  In  this  particular  example,  the  DELPHI 
technique  was  used  to  forecast  the  likely  level  of  use  of  a  selection  of 
transport  telematics  technologies  in  the  year  2015  in  medium-sized 
European  cities.  The  DELPHI  technique  allows  experts  to  collectively 
perform quantitative estimations of potential impacts of  interventions. 

The study  itself  involved  a total  of 224 experts  in Advanced Transport Tele­
matics technology  (ATT) from the vast majority of Member  States. It focused 
on 18 different ATT technologies  and their possible adoption over the  period 
up  to  2015.  By  participating  in  a  consultation  process,  this  technique 
allowed the experts to provide information on the technologies they thought 
were  likely  to  be  adopted,  the factors  that  might  lie  behind  their  adoption, 
and those that might restrict or retard this process. 

A  technique  related  to  DELPHI, the  matrix  of  cross  impacts  (SMIC),  was 
used  in parallel with this study. This technique,  unlike  DELPHI, allows  inter­
actions  among events to  be considered  by refining the  probabilities  relating 
to the occurrence of individual future developments. The resulting probabili-

Evaluating  impacts  of  interventions 

in  the  domains  of 

transport, 

ties can be used directly  in the  planning  process or as the basis for  develo­
ping scenarios that can subsequently  be used for planning. 

In total,  126 experts were  involved from across the  EU. Six  possible  events 
were identified which could be broken down into two main groups: 
•  events involving the use of technologies to reduce transport  problems 
•  events involving traffic and transport problems or consequences). 

A first stage of analysis assessed the single probability of the occurrence of 
the six events  considered  separately. A second  stage examined  the condi­
tional  probability  of  the  events  forecast.  The  result  of  the  study  was  a 
selection  of  scenarios  chosen  on  the  basis  of  the  estimated  likelihood  of 
their occurrence. 

IV 
Framework for performing 
thematic evaluations 

Thematic  evaluations  can  be  performed  within  the  framework  of  a  single 
programme  or across a number  of programmes.  For regional or national level 
conclusions  to be aggregated  and compared  at European level, evaluations 
have to be similar in the way they are structured  so that they produce  results 
that are relatively homogenous.  The first chapter  of Part IV presents  a stan 
dardised procedure  which, although  flexible enough  to be adapted  to  indivi 
dual evaluation situations,  is sufficiently  robust  to ensure that this objective is 
attained. 

Firstly,  the  second  chapter  examines  the  basic  principles  of  transnational 
thematic  evaluation,  a number  of  organisational  issues associated  with  this 
type of evaluation and how  they are used to highlight  best practice.  There is, 
however, another approach  to undertaking  transnational evaluations which is 
to move away from  transnational  comparisons,  based  on synthesising  what 
are essentially a number  of individual  country  or regional  level  evaluations, 
towards a multi-country  or pan-European  impact  assessment approach.  The 
future  development  of  transnational  evaluations,  with  an  emphasis  on 
moving  towards  this new perspective,  is discussed.  The chapter  concludes 
with  a  brief  overview  of  three  evaluation 
tools  that  are  suited  to  this 
approach. 

A step­by­step approach for thematic analysis 

The methodological approach proposed below is intended to provide a 
practical model for assessing, in a way that is relatively standardised, the 
impacts produced in a particular priority or domain. The need for a standar­
dised approach is particularly important in the case of structural policies 
because programmes have to be monitored and evaluated throughout the 
European Union and, as far as possible, the results aggregated. This method 
can be used to visualise and specify the results of a programme and to 
produce sufficiently coherent information to enable the Commission 
services to synthesise data from a number of programmes. 

■ A "step­by­step" approach for a thematic evaluation 

On the basis of the different elements described in this volume, it is possible 
to establish the different steps required in a thematic evaluation at 
programme level. This six­phase procedure, comprised of eleven different 
steps, is set out in Box 41. 

In the case of an ex ante evaluation, the evaluation team is required to use 
the first six steps of the procedure. These correspond with the following 
three phases: preparation of the evaluation, strategic assessment and 
screening of the programme (screening involves the examination of projects 
and programmes to determine the degree of impact assessment which will 
be required). 

In cases of intermediate or ex post evaluation, all the steps are appropriate. 
In this instance, the thematic assessment matrix will be initially used to focus 
the evaluation and secondly to synthesise the results of the analysis. 

The greater part of the work recommended in this approach should in any 
case be carried out in the framework of the evaluation of a programme as a 
whole. Thus, it does not necessarily concern additional work, in so far as the 
steps described are not intended only for the assessment of the theme. They 
also serve as a reminder, while the evaluation is underway, to produce 
results on this theme. 

Transversal evaluation of impacts on intervention priorities 

Box 41  Transversal evaluation of a theme : 

"Step by step" approach for a programme 

Steps 

Outputs 

Preparation 

• Examine the programming documents. 

<1 > Identification of the peculiarities of the 

• Interview the programme managers and 
stakeholders. 

programme in relation to the theme: 
specific objectives, % of budget allocated 
to specific measures. 

• Collect available data (time series, 
Information drawn from the monitoring system, 
etc.) and record existing documentation (literature). 
• Analyse available statistics. 

<2> Assessment of the most Important Issues 

in the region in relation to the theme. 
Reference situation or context analysis. 
Information concerning programme 
outputs and results. 

Strategic assessment 

• Detail the transversal effects of the 
programme in relation to the theme according 
to a typology of impacts. 

<3> Detailed list of potential effects organised 
according to the generic 
criteria selected. 

• Complete the "thematic assessment matrix" 
(see Box 42). 

■:4> 

Classification of measures by category: 
favourable, unfavourable or neutral effect. 

Screening 

• Prepare a score card for important impacts 
assessing the effects of measures (see Box 43). 

<5> 10 to 20 score cards provisionally 

completed for the most interesting 
measures. 

• Consultation with programme managers and 
stakeholders. 

<6> 

Selection of measures for further 
analysis. 

• Collect and analyse information by means 
of the minimal case study technique, 
Interviews and / or focus groups. 

Scoping 

<7> 

Score cards completed on the basis 
of the analysis of selected measures. 

Programme level synthesis 

• Complete "summary table" (see Box 45) 
for each type of impact. 

<8> Summary tables providing a transversal 

view of impacts. 

• Detail the thematic assessment matrix. 

^9> 

Overall view of the value of programmes on 
the basis of types of impacts defined. 

Conclusions 

• Examine the progress made: success and <10> 
weaknesses of the programme. 

Useful conclusions for reviewing for the 
promotion of a priority or interventions in a 
domain. 

• Forward conclusions to the Commission <11 > 
services in a standardised format. 

Comparable view of the progress made 
in the EU in respect to the theme. 

A step­by­step approach for thematic analysis 

This above procedure is neither exhaustive nor exclusive. Nor is it intended 
to be a rigid framework. Evaluators have to determine just how far this 
approach is applicable to the programme in question and, if necessary, add 
descriptive comments and appropriate indicators. 

It should be noted that this procedure is not sufficient for the evaluation of 
the effects of specific measures in a domain: the screening (the identification 
and selection of measures for more detailed analysis) and scoping (the 
detailed analysis of impacts) of measures, in particular, needs to be further 
developed. 

This procedure is detailed below, using the priority of equal opportunities 
between men and women as an illustration. 

■ Initiating the procedure (steps 1 & 2) 

Step <1 > is intended to ensure that the evaluators have in­depth knowledge 
of the content of the programme concerned, including its objectives and 
information considered to be relevant. A thematic objectives tree can be 
drawn up, highlighting the objectives of the priority. This phase implies 
obtaining and consulting all the programming documents and interviewing 
programme managers. Based on the typology of potential effects described 
in the chapter on the promotion of equal opportunities, it consists of: 

• assessing the extent to which the chosen objectives include this priority 
and, where relevant, are consistent with EU policy, and checking whether the 
strategy is relevant from this point of view; 

• analysing the content of the programme in order to assess the extent to 
which the promotion of equal opportunities has been taken into account 
during the programming stage; comparing financial grants measure by 
measure, bearing in mind their specific (targeted) or general nature; making 
an inventory of the targeted publics of the different interventions in order to 
identify those interventions which explicitly aim at the priority. 

It is then essential to define the context indicators which measure the extent 
of the needs to be met or problems to be solved. Within the framework of ex 
ante evaluation, they help to show the raison d'être of interventions and to 
clarify the strategy to be chosen. They serve as a medium for the quantifica­
tion of clear and realistic objectives. Within the framework of intermediate or 
ex post evaluation, context indicators furnish information about the extent to 

Transversal evaluation of impacts on intervention priorities 

which the target has been reached, or the extent to which the needs in the 
region have been met. 

In step <2>, the problems encountered by the public (or a geographical 
zone) concerned are diagnosed, and the potentialities to be developed in the 
specific national or regional context are identified. A list can thus be drawn 
up of the main challenges to the effective promotion of equal opportunities. 
For this purpose it will be necessary to collect literature and have access to 
databases in order to analyse the national and regional socio­economic 
context and, more particularly, identify the expectations of potential benefi­
ciaries and political stakeholders. 

■ M a k i ng a strategic a s s e s s m e nt (steps 3 & 4) 

Step <3> consists of a thematic assessment of the programme measures. 
This implies the classification of each measure in terms of its potential 
impact on the promotion of equal opportunities. It is necessary first to draw 
up a typology of impacts. This can be done by drawing up a list of impact 
descriptors, based on the preceding contextual analysis, and grouping them 
according to the similarity of impacts (see Volume 4 of the MEANS Collec­
tion for a description of the use of the "impact mapping" technique in this 
respect). In the case of the promotion of equal opportunities between men 
and women, the types of impacts selected are: "Living conditions which 
meet women's needs", "Access for women to the job market", "Women's 
situation at work" and "Participation of women in the creation of economic 
activities". 

The thematic assessment matrix (see Box 42) of the contribution of structu­
ral programme interventions to objectives relating to this priority can be pro­
visionally completed in step <4>. In this way the most promising measures 
can be identified in terms of potential impacts. 

The use of this matrix allows the classification of interventions in terms of 
whether they have a favourable, unfavourable or neutral effect. Thus, it is 
possible to identify interventions that warrant further analysis during the 
phases of screening and scoping. Finally, the matrix enables the identifica­
tion of those objectives mentioned in the programme but which few 
measures help to achieve. 

A step­by­step approach for thematic analysis 

Box 42 ­ Thematic assessment matrix. Example relating to equal 

opportunities 

Type of impact expected / achieved on the promotion of equal opportunities 

Area of intervention 
(Measures) 

More appropriate 
living conditions 

Access to the job 
market 

Situation at work  Participation in the 

creation of socio­
economic activities 

Infrastructures 
Measure N°: 
Measure N°: 

Human resources 
Measure N°: 
Measure N°: 

Productive 
environment 
Measure N°: 
Measure N°: 

Added value of 
the programme 
In terms of equal 
opportunities 

Provisional 
assessment of 
impact 

/ /? ) X 

Beneficial effect Effect probably No significant Negative effect 

beneficial effect 

■ Screening and scoping (steps 5, 6 & 7) 

Step <5> concerns the identification and selection of measures for further 
analysis. It requires the creation of a score card for each measure identified 
as having a potential impact on the promotion of equal opportunities. An 
example is provided in Box 43. In the preparation of the score cards the 
following types of impact should be covered: 

• positive or negative; 

• direct or indirect; 

• temporary or permanent; 

• short or long term. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Box 43  -  Score  card. Example: assessing the  extent to  which 
a  measure takes  into account  of the  priority of  equal 
opportunities 

Sub-programme / priority: 
Measure (title): 
Objective: 
Budget (% of funding by the EU): 
Output: 

Potential positive impact of  Significant bene-  Hardly signifi- 
cant beneficial 
effect 

projects/actions on: 

ficial effect 

No significant 
effect 

Negative effect 

Living conditions better suited 
to women's needs 
Women's access to the job 
market 
Situation of women at work 
Participation by women in the 
creation of socio-economic 
activities 

1 

1 

1 

0.5 

0.5 

0.5 

0.5 

0 

X 

X 

X 

X 

Total number of points | 

| 

If the measure has a direct or explicit impact In terms of equal 

opportunities, multiply the score by 1.5 

Give details on the two most significant positive or negative effects: 

Secondly, the evaluators  are advised  to  consult  with  programme  managers 
and  stakeholders  about  the  results  obtained  during  the first  two  phases  of 
the  evaluation  (step  <6>). The objective  of this consultation  is to  select  key 
indicators  and  the  measures  that  are to  be the  subject  of  further  analysis. 

Until now there has been no attempt to quantitatively  measure the  changes 
that  have  occurred  in  relation  to  the  priority. At  this  stage, the  information 
summarised  in the  thematic  assessment  matrix  and  on  the  score  cards 
should  be  complemented  by  data  gathered  by  the  relevant  statistical 
agencies on the indicators selected. 

After the score  cards  have been the subject  of consultations, the  measures 
most  likely to have a significant  impact on the promotion of equal opportuni­
ties  are selected  for analysis  (step  <7>). This analysis  involves  an examina­
tion of impacts using minimal case studies, interviews and / or focus  groups 
(see  Box  44). This  technique,  based  on  the  limited  analysis  of  impacts  in 
order  to  consolidate  the  assessment  of  impacts  carried  out  on  a  strategic 
basis, is known as scoping. 

Cross programme 

thematic  evaluation 

Box 44 -  Interviews, focus groups and  case  studies 

Individual  interviews  can  be  used  for  all types  of  evaluation  where  the 
information needed is more complex than that which can be conveyed by 
questionnaire. Interviews range from being relatively informal and unstruc­
tured to  highly formalised,  based on  pre-determined  questions. They are 
particularly  useful  for  describing  procedures  and  processes  (e.g.  the 
implementation  of a programme)  and the  likely  impacts of an  intervention 
or  impacts  that  are difficult  to  quantify,  such  as  behavioural  changes  or 
intangibles.  Group  interviews  can  largely  be considered  as  a number  of 
simultaneous individual interviews. 

Focus  groups  actively  seek to  harness  the  dynamics  of the  interactions 
between the  individual  participants  and to  integrate  it into the analysis  of 
complex  issues. The  interaction  among  participants  is  particularly  useful 
when  opinions  about  a  certain  issue  are  not  fully  formulated.  The 
technique  can therefore  be used to  identify  the  needs  of a region, clarify 
the  objectives  of  interventions,  examine  the  impacts  of  innovative 
measures,  focus  or  refocus  the  implementation  of  a  programme,  or 
interpret the impacts of an intervention. 

Case studies  are used for the analysis of  specific  cases, ranging from  an 
individual  to  a geographical  region. They  can  also  be  used  to  analyse 
decision  making  processes.  Both  interviews  and  focus  groups  can  be 
used  as data  gathering  tools  within  the  framework  of  case  study.  Case 
studies are particularly useful for making detailed analyses of the complex 
process  of  programme  implementation  and  of  their  impacts. They  are 
often  undertaken  to  verify  causation  or to  qualitatively  test  and  validate 
measurements  of  effects  obtained  by  other tools. This tool  can  also  be 
used when there are very few units of analysis by matched  pairs of similar 
units. This  involves finding  two  units  of  observation  -  one of  which  is an 
addressee of a programme and one of which is not - that closely  resemble 
each  other  (e.g.  SMEs  in  the  same  sector,  with  the  same  number  of 
employees, etc.) and that are in a similar context (e.g. in a declining indus­
trial region where unemployment  is high, etc.). The relative evolution of the 
units can then be studied. 

Transversal evaluation of impacts on intervention priorities 

■ Synthesising the results (steps S & 9) 

The analysis by measure is essential for the construction of an evaluation of 
the programme as a whole. Evaluation results must also be produced at 
programme level, then at the higher levels. To globally assess the effects of 
measures at the level of each programme, a summary table (see Box 45) can 
be prepared for each of the four main types of impact, identified as being 
relevant In the case of structural interventions (step <8>). The summary table 
will be completed using: 
• the quantitative assessment provided by the score cards (far right-hand 
column of the matrix); 
• the analysis, providing both quantitative and qualitative data, performed 
during the scoping phase (cells of the matrix). 

Box 45 - Summary table of measures and their effects. Example 
of improving the suitability of living conditions to women's needs 

Summary table: synthesis of impacts in terms of living conditions better suited to women's needs 

Measures with a 
significant impact 

Budget 
(% of funding by 
the EU) 

Outputs 

Results which were 
expected and / or 
produced 

Impacts which 
were expected 
and / or produced 

N°: 
(title) 

N°: 
(title) 

N°: 
(title) 

N°: 
(titre) 

Total points or X ( in the case of a negative impact (source: score cards): 

General comments 
this type of impact: 

for 

Since the Commission will need to have an overall view of the impact of 
interventions on the priority of promoting equal opportunity between man 
and women, it may be useful to prepare a brief, very general summary on the 

Cross programme thematic evaluation 

(positive and negative) impact of each programme (<9>). The thematic 
assessment matrix, previously completed on a provisional basis can be used 
for this purpose (see Box 46). The matrix approach is sufficiently simple (without 
being simplistic) to meet practical needs. However, it remains necessary to 
analyse available indicators, in particular as far as context is concerned. 

The matrix approach makes it possible to achieve the standardisation 
needed for an overall view at European Union level. It serves to simplify an 
extremely complex reality and to present it in a form which is synthetic and 
comprehensible at policy level. 

The key steps and evaluation tools employed in this approach are summari­
sed in a highly simplified example in Box 46. Four measures initially enter the 
strategic assessment phase, of which three are considered sufficiently 
important to be retained for further study. At the screening phase Measure 4 
is eliminated which leaves Measures 1 and 3 to enter the scoping phase. The 
conclusions of the analysis made at the scoping phase are presented in a 
summary table and this is then used to complete the thematic assessment 
matrix. 

Box 46 ­ Summary of the key steps and evaluation tools 

Thematic assessment matrix Screening 

Step <4> and Step <9> Step <5> and <6> 

Scoping 
Step <7> 

Measure 

1 

2 

3 

4 

k 

f , 
) 

, I «* 

• 

1 

3 

> 4 

­ * ·! 1 ' 

­ * ¡ 3¡ 

_l 

S u m m a ry t a b le (one per t y pe of i m p a c t) 

S t ep < 8> 

Measure 

Resources 

Outputs 

Results 

Impacts 

1 

τ 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

I  Concluding  on  the  merits  of  t he  p r o g r a m me  as  regards 

the theme evaluated (steps 10 & 11) 

The  conclusion  of  the  evaluation  report  (step  <10>)  will  obviously  include 
recommendations  on  the  best  way  of  taking  the  evaluated  theme  into 
account when formulating objectives, programming and selecting  projects. 
So that a priority can be better taken into account  by decision-makers, the 
evaluation team must make significant efforts to: 
•  propose new strategies of intervention; 
•  strengthen the most promising  measures; 
•  formulate  recommendations  for  enhancing  the  potential  of  the  targeted 
social groups and areas. 

On  a  more  technical  level, recommendations  are  needed  so  that  in  the 
future  the  systems  of  monitoring  indicators  and  of  statistics  can  include 
information that will provide a better understanding of the situation. 

Finally, in step <11 > the conclusions  provided by the procedure will be suf­
ficiently  standardised  to  nourish  debates  and  comparisons  of  the 
promotion of equal opportunities between men and women (or other priori­
ties or domains) at the European level. 

Cross programme thematic evaluation 

■ Basic principles of transnational t h e m a t ic evaluations 

The preceding chapters have examined the evaluation of a programme from 
a thematic perspective. Evaluations at the programme level focus principally 
on the examination of programming undertaken by Member States. Whilst 
these evaluations allow the appreciation of interventions on a national / 
regional scale, they do not provide information that allows a Europe­wide 
analysis of impacts, the appreciation of Community value­added or a pan­
European strategic analysis of Structural Fund interventions. 

In order to achieve these more ambitious evaluation objectives, it is 
necessary to undertake thematic evaluations at transregional or transnatio­
nal level. It is important to note, however, that for this type of evaluation to be 
fully effective the methodology used should go beyond the simple supra­
national synthesis of a number of single evaluations (see Box 47, upper 
figure). The approach followed should be based on the collection, analysis 
and synthesis of data within a common and co­ordinated single evaluation 
framework across all the countries involved (see Box 47, lower figure). 

Transversal evaluation of impacts on intervention priorities 

Box 47 ­ Comparison of supra­national synthesis of programme 

evaluations and transnational evaluation 

Supranational synthesis of evaluations 
(a), (b) and (c) 

J_ 
Evaluation (a) 

Evaluation (b) 

Evaluation (c) 

Programme 
(a) 

Programme 
(b) 

I 

Programme 
(c) 

Single transnational evaluation 

Programme 
(a) 

J 

1 Programme ! 
; (b) : 
1 J 

1 Programme ! 
; (c) : 

ι I 

■ Organisational issues 

A major problem in commissioning thematic transnational evaluations is the 
question of how to take into account differing national or regional contexts, 
thematic specificities and interrelations between different themes and 
domains. The consequence of these complexities is that the drawing up of 
the terms of reference for an evaluation of this type is extremely problemati­
cal. It is for this reason that an essential first step is to set up a technical 
committee composed of experts and academics with relevant sectoral 
knowledge, and national experts who understand the national and regional 
socio­economic contexts in which the evaluation will take place. 

The role of the technical committee is to steer a feasibility study by evalua­
tors of two or three countries (carefully chosen for their representativeness 

Cross  programme  thematic  evaluation 

and relevance in terms of the theme of the evaluation). This study will involve 
the  elaboration  of  a  formal  pre-evaluation  document  in  which  priority 
questions, programmes to  be evaluated, timescale, budget  and the evalua­
bility  of  interventions  are  addressed  (e.g. Is the  evaluation  politically  and 
technically  opportune?  Is the  budget  and  timescale  adequate,  considering 
the  characteristics  of theme  and the  stock  of  existing  knowledge  that  may 
be mobilised?). 

When designing the evaluation the evaluation team should: 
•  draw  up evaluatory  questions that  disregard  country  and  regional  factors 
and concentrate solely on thematic aspects of the evaluation, and which are 
specifically centred around a European perspective; 
•  tightly define the criteria for the selection of case studies and sampling so 
as to  improve  comparability  and take  into  account  of  country-  and  region-
specific contextual factors; 
•  elaborate  an analytical  grid  which  takes  competing  factors  into  account. 

This work should be then validated by the technical committee. 

Case  studies  should  be selected  by the technical  committee  and  validated 
by the relevant national experts who would judge the pertinence of the cases 
chosen. The number  of case studies  performed  should  be relatively  limited 
and they should be likely to capture  strong structuring effects. These struc­
turing effects should  be analysed against a contextual  backdrop which may 
be regional, national, European or even international, depending on the par­
ticular cases selected. 

The output  of this process will be a document  which  is the result of  interac­
tions between actors with sectoral and national socio-economic  experience. 

When the final evaluation  report is produced  it is important that the  Member 
States  involved  in transnational  evaluations  should  have the opportunity  to 
verify the interpretation of the results of the evaluation. 

Transversal evaluation of impacts on intervention priorities 

■ Highlighting e x a m p l es of best practice 

Carefully designed and implemented transnational evaluations can comple­
ment existing programme level evaluation by highlighting examples of best 
practice and allowing experiences to be transferred between regions and 
countries. The objective of this section is to suggest a number of practical 
recommendations that are aimed at ensuring that evaluations are fully 
adapted to the goal of highlighting best practice. 

Since Community programmes are both complex and open ended, the way 
in which they are implemented is an important evaluation issue. The search 
for examples of best practice should therefore examine not only the impacts 
of programmes but also the process of implementation, and should make 
comparisons of delivery mechanisms in different countries. 

In order to find examples of best practice, it is important that populations of 
beneficiaries, geographical units (regions, sub­regions, towns, etc.), 
projects, interviewees, etc. selected for case studies should be similar in 
terms of the context in which they are situated. The contextual situation in 
which the selected case studies are embedded can be initially established 
by a SWOT analysis. Points of comparison also have to be established by 
which performances can be judged. This can be done using the benchmar­
king technique. 

The SWOT framework (Strengths­Weaknesses­Opportunity­Threats) can be 
used to undertake a strategic analysis of a programme and the region in 
which it is to be implemented (internal factors) as well as the overall policy 
environment (external factors). This tool can be used to outline an economic 
development strategy and is therefore particularly useful during the design 
and ex ante assessment stage of a programme. It can also be used for mid­
term and ex post evaluations to take into account contextual and identify 
problems of comparability. Box 48 illustrates this principle. 

Cross programme thematic evaluation 

Box 48 ­ Sampling with the objective of highlighting best practice 

Country 1 

Country 2 

Country 3 

SMEs 
medium­tech 
20­50 
employees 

  _ Λ_ 
e
.

t
x
e
t
n
o
C

d

.

c

.

b

.

a

SMEs 
medlum­tech 
20­50 
employees 

L 

Λ 1 

I 

1 

1 

1 
1 

1 

SMEs 
medium­tech 
20­50 
employees 

m ι 
• I 

I 

1 
1 

1 

■ 1 

1 

1 

1 

_. 

( · \ 
V · J 

Cross­country 
sample of various 
practices embedded 
in similar contexts 

If the theme which is the object of the evaluation is SMEs, the constitution of 
a cross­regional sample requires that the firms have similar characteristics 
(e.g. between 20 and 50 employees in a medium­tech sector of the 
economy). In order to assure comparability, the firms are drawn from similar 
contextual situations, minimising the difference in regional context. A direct 
comparison is then feasible across countries. 

Benchmarking consists of making a qualitative comparison between one 
intervention and another, comparable in terms of being embedded in the 
same contextual situation, which is reputed to be an example of best 
practice. By doing this, the strengths and weaknesses of the intervention 
being evaluated can be highlighted. This technique can also be used at a 
more strategic level to compare the organisation of programmes. 

■ Future directions for transnational evaluations 

Evaluation practice and experience in the field of transnational thematic eva­
luations have grown rapidly and a second generation of this sort of evalua­
tion, with an emphasis on highlighting best practice, is already developing 
(see Box 49). 

 
Transversal evaluation of impacts on intervention priorities 

While the type of transnational evaluation discussed above is capable of 
highlighting examples of best practice, it is not capable of providing a picture 
of pan­European achievements. This sort of study, which needs to take into 
account different national or regional contexts, could be used to inform 
European­wide strategic planning, for example, permitting the drawing up of 
a masterplan for Trans­European Networks or for the production of recom­
mendations and guidelines adapted to Community­wide conditions. 

Box 49 ­ The development of transnational evaluations 

1 ■ Experience of transnational 
evaluations 
• mainly based on documentary 
research, secondary data, and 
interviews with key actors 
• scores and weights 
• SWOT analysis 
• project case studies 

Examples: 
• Transportation 
• Telecommunications 
• Tourism 

$ 

2° Experience of transnational 
evaluations 
1° + additional tools including: 
• country­by­country reviews of 
existing literature 
• secondary data analysis 
• surveys 
• regional and project case 
studies 

Examples: 
• Business information 
Centres 
• Environment 
• RTD 
• SMEs 

^ 

3° Experience of transnational 
evaluations 
1° el 2° + additional tools such 
as: 
• research synthesis 
• expert panels 
• pan­European surveys 
• embedded case studies 
•others... 

In addition to the range of tools suited to thematic evaluations presented 
throughout this volume, there are a number of tools are suitable for underta­
king pan­European analysis. These tools are presented below. 

• Multi­country / multi­regional econometric analysis 

When surveys can be undertaken, the data collected can be used to 
construct ad hoc multi­country or pan­European econometric analysis of 
impacts. 

The results furnished by this type of analysis can transformed into coeffi­
cients which can subsequently be used to make a projection of the results 

Cross programme  thematic  evaluation 

for  either  particular  aspects  of  an  intervention  or  for  its  overall  projected 
effects.  The  use of  coefficients  also  requires  monitoring  data  to  provide 
information on the addressees of interventions and the distribution of expen­
diture,  etc.  Impacts  can  be analysed  both  in a cross-sectional  perspective 
(i.e. a variety of  interventions  at a given  moment  in time)  and over time (i.e. 
using time-series data). 

The  use of this tool  requires that  sampling  be correctly  undertaken.  Unlike 
the sampling technique used when benchmarking is the objective of an eva­
luation,  in this  case  samples  should  be  constituted  from  across  a  diverse 
range of country  and  regional  contexts.  In order  to  do this, a common, co­
ordinated  methodological  approach has to  be used which involves carefully 
controlled  data  collection  to  ensure  that  the  selection  of  units  of  analysis 
features the level of representativeness required. This approach is illustrated 
in  Box  50 where  careful  sampling  allows  a cross-regional  sample,  drawn 
from a variety of different contexts, to be constituted. 

Box 50 - Sampling with the objective  of  making  a pan-European  analysis 

Country  1 
SMEs 
medium-tech 
20  -  50  employees 

Country  2 
SMEs 
medium-tech 
20  -  50  employees 

Country  3 
SMEs 
medium-tech 
20  - 50  employees 

I 

1  1 

> 

| 

1 
1 

1 

t
x
e
t
n
o
C

e
.

d
.
c
.
b
.
a

1 

' 

1 
t 

1 

1 

1 

1 

i 
1 

f 

» 

•  \ 
· 
y 

f 
/ 
• 
V 

· 

· 

· 

 
 
 
Transversal  evaluation  of  impacts  on  intervention 

priorities 

Whenever  feasible,  comparative  analyses  should  be  performed  with  non-
European countries (e.g. the United States, Canada, Australia, etc.). 

•  Multi-site case studies with replication design 

Case  study  analysis  is  a  useful  tool  for  analysing  socio-economic 
phenomena  that  are  difficult  to  separate  from  their  contextual  conditions. 
The unit of analysis of the case study may be an individual, an organisation, a 
process,  an  industry,  a  region, etc. The  main  unit  of  analysis  (e.g. a  region) 
may  furthermore  contain  sub-units  of  analysis  (e.g. firms,  infrastructure 
projects, etc.). Case studies can either be single or multiple. 

The  use  of  multiple  case  studies  with  replication  design  is  a  particularly 
important  technique  for  strengthening  or  broadening  analytical  generalisa­
tions. This type of  case study  is based on a series of waves  (usually two  or 
three)  of  case  studies  carefully  chosen  to  support  a  given  proposition. 
Depending  on  the  design  of  the  evaluation  the  case  studies  will  produce 
either: 
•  similar results providing corroborative evidence; 
•  contrasting  results  but  with  similar  theoretical  underpinnings  (i.e.  the 
theory is corroborated); 
•  both replicated and contrasting results within the same study. 

If, however, the results produced in some of the cases are contradictory in some 
way, the initial proposition cannot be accepted. This may cause it to be retested, 
by means of a second and, when necessary, a third wave (see Box 51 ). 

Example: Providing advice to city administrations 
In a study designed to examine attempts by universities and research groups to colla­
borate  with  local  government  officials,  four  different  sets  of  replicated  case  studies 
were undertaken (see Box 51). The articulation of the propositions supported by each 
of  the  different  sets  allowed  an  overall  conclusion  to  be  reached  which  was  not 
evident at the outset  of the study. An initial group  of case studies  concluded that  uni­
versities and research centres  had failed to  help city administrations. A second  group 
concluded that this phenomenon  was not restricted to  universities  but included  other 
organisations. A third group showed that in some cases  universities  had been able to 
help  other  types  of  organisations.  Finally,  a fourth  group  showed  that  some  groups 
had  been  able  to  help  city  administrations  by  taking  implementation  issues  into 
account.  The  overall  conclusion  was  that  city  administrations  had  particular  needs 
when  receiving  advice  from  outside  bodies,  a conclusion  which  differs  significantly 
from the initial one. 

Cross  programme 

thematic  evaluation 

Box 51 - Example of multiple-case studies with replication design 

(Wave  1 ) 
Case studies  focusing 
on universities  and 
their  relations 
with  city 
acministrations 

(Wave 2) 
Case studies  on 
relations  between 
non-university  groups 
and  city 
administrations 

(Wave 3) 
Case studies  on 
relations  betwwen 
universities  and 
organisations  other 
than  city 
administrations 

(Wave 4) 
Case studies  on 
groups  concerned 
not  only  with  new 
ideas, 
but  also  their 
implementation 

Source: from Yin R.K. (1994), Case study Research: Design and Methods, London: Sage. 

•  Research synthesis 

Research or evaluation synthesis  is the analysis of existing research using a 
set  of  systematic  procedures  - or  meta-analysis  techniques  - for  summari­
sing quantitative results from  multiple studies. This technique  is based on a 
systematic  and  rigorous  approach  to  synthesising  past  research. Unlike  a 
literature review, research synthesis takes into account contradictory  results 
when analysing the existing stock of knowledge about a subject. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

This  technique  has  two  benefits:  first,  by  drawing  on  a  number  of  well 
designed  and  executed  studies,  the  resulting  analysis  is  highly  robust 
because  based  on  solid  and  reliable  knowledge;  secondly,  if  research 
synthesis shows that the existing base of knowledge  is not well-founded, it 
enables decision makers to avoid basing their policy making on  incomplete 
or  erroneous  information,  and  provides  guidance for  future  research. The 
procedure for performing research synthesis is summarised in Box 52. 

Box 52  - The seven steps  of  research  synthesis 

Siepi 
Selection of the questions  to be  addressed 

Step  2 
Collection  of the  all  relèvent documents  and identification  of  key  issues 

Step  3 
Development  of screening criteria and screening  out of irrelevent  material 

Step 4 
Development  and implementation  of critical  asessments  of  studies 

Step  5 
Redetermination of the appropriateness  of this technique  in the  light of the material  reviewed 

Step 6 
Undertaking  of the evaluation  synthesis 

Step  7 
Reporting the findings  of the  synthesis 

Conclusion 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

The development of thematic  evaluations within the framework of the Struc­
tural  Funds  has  been  shaped  by  the  evolving  policy  context  within  the 
European  Union. As  a consequence,  the  state  of  the  art  in this  area  was 
initially dominated  by experience in the evaluation of employment effects. As 
the decade  has  progressed, however,  environmental  concerns  and  equality 
between  men  and  women  have  climbed  the  political  agenda,  and  this  has 
been reflected in the accumulation of evaluation experience in these themes. 

At  the  same  time  as  this  process  of  accumulation  of  thematic  evaluation 
experience  has  been  occurring, the  need  for  comparisons  between  pro­
grammes  has  become  increasingly  important,  hence the  need  for  transna­
tional  thematic  evaluation.  Experience  in this  still  emerging  area of  evalua­
tion remains relatively limited for the moment. 

While  an  in-depth  evaluation  of  impacts  within  a  particular  theme  requires 
corresponding expert  knowledge which can only be provided  by specialists, 
the  central  premise  of  this  book  is that,  with  a  basic  understanding  of  a 
theme's characteristics, an impact assessment  can be designed or underta­
ken  by  evaluation  units  and  evaluators  without  in-depth  knowledge  of  the 
theme.  It  is on this  basis that  existing  thematic  evaluation  experience  and 
general  evaluation  expertise  have  been  combined  to  produce  this  book 
focusing on how to assess the impacts of Structural Fund interventions on the 
four political priorities of the EU and on number of domains of intervention. 

The  approach  taken  in  this  book  has  been  to  examine  thematic  impact 
assessment  in such  a way  that  methodological  considerations,  rather  than 
evaluation  themes,  are the  dominant  perspective. This  approach  has  been 
particularly  fruitful since  it has given  rise to a the construction  of an evalua­
tion  method  -  the  step-by-step  procedure  -  which  draws  upon  evaluation 
practices  and  tools  that  have  been  developed  within  the  various  thematic 
frameworks  An  example  of  this  is  the  screening  and  scoping  technique 
which  has  been  mainly  used for ex ante environmental  impact  assessment. 
Another  example  of  "cross-fertilisation"  between  themes  is that  of  mains­
treaming, which is finds its way into the step-by-step approach by way of the 
score card.  Moreover, this  is a strategy  which  is particular  to the  evaluation 
of  equality  between  men  and  women,  but  which  seems  to  be  a  promising 
approach for evaluating  impacts on SMEs within the framework  of  interven­
tions  aimed  at  improving  the  competitiveness  of  the  productive  sector.  At 
the conceptual level, the notion of sustainability which can be applied across 
a wide range of themes. Similarly, the concept  of leverage can be applied  to 

Conclusion 

all types  of  interventions  where the objective  is to  encourage  private  sector 
co-financing. Substitution, displacement and deadweight, concepts origina­
ting from the evaluation of employment  effects, are also applicable to  other 
themes. 

Finally,  although  a  large  part  of  this  Volume  has  focused  on  drawing  on 
existing  evaluation  experience,  the  future  of  the  evaluation  of  Structural 
Fund interventions  has not been forgotten. It is for this reason that a number 
of  tools  are discussed  that  seem  particularly  adapted  to  complementing 
transnational thematic  evaluation aimed at highlighting best practice with in-
depth evaluations of impacts at the pan-European level. 

Appendix 

Appendix  1 contains an index of tools classified by their thematic suitability in 
the form of a matrix. 

Appendix  2 presents  the  terms  of  reference  for  a programme  evaluation 
incorporating  explicit  instructions  relating  to  the  inclusion  of  Community 
political  priorities:  "Intermediate  Assessment  of  the  Highlands  and  Islands 
objective  1 Programme". 

Appendix  3 presents  the terms of reference  of a transnational  thematic  eva 
luation:  "Evaluation of  RTD actions  in  the case of Structural  Funds" (1998). 

Appendix  4 presents a list of boxes. 

The documents  presented  in Appendices  2 and 3 are not  intended  to serve 
as strict  models  upon  which  terms of reference  should  be based.  They can, 
nonetheless,  be  used  to guide  the  drawing  up  of  future  terms  of  reference 
which  the authors  should  tailor to  the specific  characteristics  of  the evalua 
tion to be performed  (e.g. in terms of particularities  of the theme,  evaluation 
tools to be chosen, organisational factors, etc.). 

Interviews 

Case 

Focus 

studies 

group 

Question­

naire 

SWOT 

surveys 

CBA 

and 

MCA 

Scores 

Screening 

and 

and 

DELPHI 

weights 

scoping 

Expert 

panels 

coefficients 

Input­

Macro­

and 

output 

ecowmethc 

statistical 

analysis 

models 

analysis 

Multipliers, 

Environ­

ment 

Page 54 

Page 55 

Page 52 

SMEs 

Page 73 

Page 75 

Page 73 

Page 73 

Page 73 

T3 
CD 

< 

ω 
5' 
3 

Employ­

ment 

Equal 

opportunities 

Page 101 

Page 101 

Page 103 

Page 104 

Page 104 

Page 126 

Page 125 

Page 123 

Page 126 

Page 123 

RTD 

Page 147 

Page 147 

Page 147 

Page 147 

·­*■ 

O 
o_ 
C/5 
03 
3 
O. 
O 

Π) 

O ^ -\ 
(Λ •a 
o 
=i o. 
Ξ" 
en 

Transport 

Page 163 

Page 161 

Page 162 

Page 164 

re 
3 
re 
C/5 

Appendix 2: Extracts from a Terms of Reference for a 
programme evaluation  incorporating thematic  elements 

Intermediate  Assessment  of the  Highlands  and  Islands  Objective 1 
Programme 

This document  explicitly presents the requirement to evaluate  Community 
priorities in the paragraph on "Specific Objective" of the evaluation. 

Examine  each  priority  and  measure  and  consider  their  appropriateness  in 
addressing the aim of the programme. Consideration should  be given to the 
number of measures, their suitability and scope and the continued  relevance 
of  targets,  indicators  and  outputs.  Recommendations  should  be  brought 
forward on amendments. Annex A sets out  in summary form the key  issues 
which  the  partnership  has  identified  in terms  of  implementing  the  SPD  by 
priority  and  measure. The final  report  should  address  each  of these  issues 
albeit in varying detail, and where appropriate, provide advice or recommen­
dations for the future implementation of the programme. 

Assess  performance  at  programme  priority  and  measure  level,  identifying 
cause of variance in terms of commitment and expenditure, having regard to 
external factors  and  match funding  (comment  on the  extent to  which  such 
factors can be accommodated within the existing implementation). 

Examine the synergy between the different Structural Funds, between Priori­
ties  and  between  the  SPD  and  Community  Initiatives.  To comment  and 
advise on linkages within the programme and the extent to which  objectives 
are  being  achieved  in  particular  job  creation,  environmental  sustainability 
and community  development. 

Details are given of key issues concerning various priorities 

PRIORITY 1 : SUPPORT FOR  BUSINESS 

•  Is there adequate balance across different  measures? 
•  Can different measures and different funds be better  integrated? 
• Are the targets appropriate in relation to the strategy? 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Measure  1: Financial support for the development of new ventures  and 
SMEs 

• Scheme coverage - does this require to be focused? - does it require better 
co-ordination. 
•  Number of schemes - Scope for more? Capacity for more? 
•  Leverage - Is there a need to establish  thresholds? 
•  Does there need to be greater of sectoral focus 

Measure  2:  Financial  support  for  research  and  development  appro­
priate to SMEs 

• Why are SMEs not more interested in R&D? 
•  Does product development require more emphasis? 
•  How can the measure be made more attractive? 

[etc.] 

PRIORITY 3: PRESERVATION AND  ENHANCEMENT  OF THE ENVIRON­
MENT 

• Are the measures too or not sufficiently  constrained? 
•  Is the scope of measures too  limited? 
•  How can a better integration between-funds  be achieved? 

Measure 1 : enhancement and protection of the environment and mana­
gement  schemes 

•  Is the scope of measure sufficient? 
• Are targets suitable? 
•  How can support  be adjusted in relation to the targeted  problem? 

Measure  2: pollution  control, waste  disposal  and  the  development  of 
recycling  systems 

•  How can statutory and non-statutory  responsibilities be better reflected in 
the system? 
•  How can re-cycling be encouraged? 
•  Can the innovatory character of projects be increased? 

[etc.] 

Appendix 3: An example of terms of Reference for a 
transnational thematic evaluation 

Evaluation of action in favour of RTD within the framework of the 
Structural Funds 

■ Background 

Presentation of the theme, related policy interventions, the context in 
which it is situated and the raison d'être of the evaluation. 

The importance of research and technological development (RTD) for 
improving the competitiveness of the productive sector through product and 
process innovation, especially by SMEs, is recognised in the context of 
Structural Funds. 

The Commission's First Action Plan for Innovation in Europe also stressed 
the need for all Community instruments to be mobilised to support RTD and 
innovation with an increased support from the Structural Funds through 
other forms of intervention such as Article 10 of the ERDF and some 
Community Initiatives (SME initiatives, ADAPT, LEADER II). 

The build up of a RTD capacity in assisted regions is a necessary but not a 
sufficient condition to enhance their competitiveness and economic perfor­
mance. The RTD investment effort should be related to the build up of the 
wider innovation system, in particular through the creation of durable links 
between the RTD system and the regional industrial structures. 

Within assisted regions, Community interventions for RTD and innovation 
under the Structural funds provide an opportunity to strengthen their capabi­
lities and to reduce the technology gaps with the most developed regions. 

Most objective 1 regions (and to a lesser extent objective 6 regions) face a 
number of problems relating in particular to the concentration of RTD related 
activities in the most developed areas, weak industrial participation as well 
as human resource shortages. In these regions, the provision of facilities and 
equipment and the availability of researchers are priorities as well as the 
application of research results and the transfer of innovation and technology 
to local business and industry. Most objective 2 regions, which often are 

Transversal evaluation of impacts on intervention priorities 

endowed with better RTD capacity, focus their actions towards the develop­
ment of specific segments in line with the needs of regional SMEs, and espe­
cially to technology and know­how transfer to local business and industry 
through regional and inter­regional co­operation networks. 

Given the complexities of developing RTD policies for both types of areas, it 
is imperative to have a detailed understanding of the current regional 
situation in order to determine what policies might be most appropriate. A 
region­specific analysis, according to characteristics of the objectives, 
enables current innovation and technology levels to be assessed and gaps 
to be identified in the policy framework. 

In view of the preparation of the post­1999 Structural interventions, the 
Commission is currently preparing a communication on RTD and innovation. 
Simultaneously, it is envisaged to launch a comprehensive thematic study to 
feed the reflection on these policy issues 

■ Aims and objectives of the study 

This section of the document should clearly list the central questions to be 
addressed by the evaluation. While these questions are obviously specific 
to the theme to be evaluated in terms of the programmes implemented 
and the impacts likely to be generated, they will be focused on key factors 
that are generally applicable to the evaluation of Structural Fund interven­
tions: 
• Impacts to be analysed 
• Effectiveness, efficiency, relevance and utility of interventions 
• Coherence and synergies between different levels of intervention (i.e. 
regional, national and European) 
• Coherence and synergies between different Community level interven­
tions (i.e. Community Framework programmes, Structural Fund pro­
grammes, Community initiatives, innovative or specific measures) 
• Future directions for policy 

The importance of research and technological development (RTD) for 
improving the competitiveness of the productive sector through product and 
process innovation, especially by SMEs, is recognised in the context of 
Structural Funds. 

The Commission's First Action Plan for Innovation in Europe also stressed 
the need for all Community instruments to be mobilised to support RTD and 

Appendix 

innovation  with  an  increased  support  from  the  Structural  Funds  through 
other  forms  of  intervention  such  as  Article  10  of  the  ERDF  and  some 
Community  Initiatives (SME initiatives, ADAPT, LEADER II). 

The build  up of a RTD capacity  in assisted  regions  is a necessary  but  not a 
sufficient  condition to enhance their competitiveness  and economic  perfor­
mance. The  RTD investment  effort  should  be  related to the  build  up of  the 
wider  innovation  system, in  particular  through the  creation  of  durable  links 
between the RTD system and the regional industrial structures. 

Within  assisted  regions,  Community  interventions  for  RTD and  innovation 
under the Structural funds provide an opportunity to strengthen their capabi­
lities and  to  reduce the technology  gaps with  the  most  developed  regions. 

Most  objective  1 regions  (and to  a lesser extent  objective  6 regions) face a 
number of problems relating in particular to the concentration of RTD related 
activities  in the  most  developed  areas, weak  industrial  participation  as well 
as human resource shortages. In these regions, the provision of facilities and 
equipment  and  the  availability  of  researchers  are  priorities  as well  as  the 
application of research results and the transfer of innovation and technology 
to  local  business  and  industry.  Most  objective  2 regions,  which  often  are 
endowed with better RTD capacity, focus their actions towards the develop­
ment of specific segments in line with the needs of regional SMEs, and espe­
cially to  technology  and  know-how  transfer  to  local  business  and  industry 
through regional and inter-regional co-operation  networks. 

Given the complexities  of developing  RTD policies for  both types of areas, it 
is  imperative  to  have  a  detailed  understanding  of  the  current  regional 
situation  in order  to  determine  what  policies  might  be  most  appropriate. A 
region-specific  analysis,  according  to  characteristics  of  the  objectives, 
enables current  innovation  and technology  levels to  be assessed  and  gaps 
to be identified in the policy framework. 

In view  of  the  preparation  of  the  post-1999  Structural  interventions,  the 
Commission is currently preparing a communication on RTD and innovation. 
Simultaneously, it is envisaged to launch a comprehensive thematic study to 
feed the reflection on these policy issues 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Scope 

The field  of  application  specified  should  delimit  the evaluation  according 
to  the  relevant  dimensions  such  as time,  geographic  zone, and  type  of 
intervention: 
•  On what  period should the evaluation focus? Will the evaluation  involve 
a comparison between programming  periods? 
•  Which  countries /  regions will  be  involved  in or excluded  from the  eva­
luation? 
•  Which policy  instruments would be included in the study? 

·-  Suggest  how  interventions  and  their  objectives  of  might  be categori­
sed. This will help give evaluators to better orient their proposals. 

Considering the content and policy relevance of RTD actions and the need to 
cover all relevant objectives  under Structural Funds, the study will be divided 
into two  lots. It will focus on the one hand on objective  1 and 6 regions (lot 1) 
and on the other hand on objective 2 regions (lot 2), but due consideration will 
be given where relevant to other forms of intervention such as Community  ini­
tiatives  and  innovative  actions  under  the  ERDF art.  10.  It  will  cover  both 
Structural Funds programming periods but particular attention will be given to 
the 1994-99 period, for which most information is available. 

The  study  will  also  consider  the  EU  Framework  Programmes  and  relevant 
national RTD policies including significant regional state aids for RTD. 

Regarding the actions or activities to be evaluated, there needs to be a clear 
classification of what makes up RTD activities. The coverage will be  broader 
than  RTD  in the  strict  sense  covering  largely  the  innovation  system.  The 
analysis  should  refer  to  a common  framework,  as  developed  under  the 
Action Plan for Innovation, in order to allow for a more coherent evaluation of 
RTD and innovation content of Structural Funds. For example, in objective 1, 
it should  be possible to  assess the catch-up  process  in terms  of  RTD infra­
structure,  and  to  distinguish  between  scientific  investments  (research 
centres,  research  projects...)  and  technological  investments  (technological 
parks, technology centres...). 

The suggested framework  encompasses the following  categories: 

(1) Enhancement of scientific and technological  potential 

•  Scientific potential (infrastructure, equipment, scientific  actions,) 
•  Technological  potential  (infrastructure,  equipment,  technological  actions,) 

Appendix 

(2) Support to Innovation 

• Promotion of an innovation culture, which includes human resources deve­
lopment through education and training measures, research mobility, stimula­
tion actions, dissemination of RTD results, promotion of public awareness, etc. 
• Regulatory and financial framework (legislative and regulatory environ­
ment, innovation financing through seed capital, venture capital or other) 
• Link between research and innovation , including technology transfer, 
promotion of networking, strengthening links between enterprises, universi­
ties and research centres 

In reviewing the different types of RTD innovation actions, it would also be 
interesting to consider relevant sectors of the Framework Programme such as: 
­ information and communication technologies, 
­ industrial technologies, 
­ life sciences, including bio­technologies and agriculture, 

in order to assess complementarity and synergy between Structural Funds 
interventions and the Framework Programme itself. 

As regards the development of RTD and innovation indicators, the Eurostat 
"Innovation manual" should constitute an essential source of information for 
this study. 

■ Content of the study 

This section outlines how the evaluation will be structured. It should 
present the methodological framework of the evaluation including: 
• evaluation tools to be used 
• which methods of data collection will be used 
• details of the types of analysis to be performed 

·■ The methodological framework should be drawn up in such a way as to 
ensure that the evaluators, while respecting its overall structure, feel able 
to use their experience and know­how to make practical suggestions on 
how the evaluation could be performed. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

It should comprise the following aspects: 

*  Methodological  framework 

This part will contain the development  of a methodology to allow for compa­
rative  analysis  and  measurement  of  performance  of  RTD  related  actions. 
This should include: 

(a) the adoption of a thematic approach requiring the comparative and simul­
taneous assessment of common themes and issues 

(b)  the  proposal  of  appropriate  indicators,  relating  to  inputs,  outputs, 
impacts, management and economic  context 

(c) the design of selected instruments to assess performance  of a sample of 
RTD related actions 

(d) the  design  of  questionnaire  forms  and  identification  of  key  informants 
(basis for field work) 

(e) the selection and implementation of case studies 

*  A  comparative  analysis  of  RTD  policies  and  programmes  and  the 
impact of Structural Funds  interventions 

In the  context  of  Structural  Funds, the  Commission  wishes  to  analyse  in  a 
comparative fashion: 

- t he  RTD  funding,  i.e.  the  amount  and  proportion  of  Structural  Funds 
budget  allocated  to  RTD,  and  a  comparison  with  national  or  other 
Community  funds  allocated  to  RTD for  each  relevant  objective,  member 
state and region; 

-  the importance assigned to different  RTD related actions and their diversity 
throughout the regions; 

- t he  quality  of  RTD  organisation  and  structures,  relating  in  particular  to 
management structures and evaluation procedures; 

-  the  effectiveness  of the  instruments  or  mechanisms  to  promote  RTD and 
innovation actions; 

-  the impact on human resources and training for RTD. 

*  Assessment  of  a  sample  of  RTD  related  actions  at  country/regional 

level (field work) 

This  part will consist  of  an in-depth  analysis  of  a significant  sample  of  RTD 
actions funded by the Structural Funds for each of the countries concerned. 

Appendix 

The selection of RTD actions will be based on criteria such as financial 
dimension and strategic importance. Fieldwork will be conducted in relevant 
countries or regions in accordance with a standardised methodology. 

• Assessment of Community value-added and best practices 

This part of the study will evaluate the RTD related actions funded by Structural 
Funds, and draw lessons for regional policy. Particular attention is to be paid to: 
- t he quality of RTD actions, and the changes observed between the 1989-
93 and 1994-99 programming periods; 
- the value added of Structural Funds actions in the field of RTD 

• the identification of best practices in terms of RTD industry linkages, of 
technology transfer mechanisms and institutional designs in the assisted 
regions and analysis of transferability conditions to other regions 
• the complementarity and synergy within Structural Fund programmes and 
with the EU RTD Framework Programme as well as with national RTD policies. 

• Synthesis of main findings and policy issues 

This part will draw conclusions and recommendations, with special regard to 
the future of structural policies. Some of the issues at stake have already 
been addressed in the first Cohesion Report and the current Communication 
on "RTD, Innovation and Cohesion". 

Nonetheless, further thought should be carried out on the following 
questions: 
• Should there be more RTD activities funded under the Structural Funds? 
• How should RTD actions be focused? 
• Have the Structural Funds been sufficiently considered in the Fourth 
Framework Programme and vice versa? 
• Should more be done in the future to achieve better synergy of both EU 
policies? 

■ Sources and methods 

The objective of this section is to outline the data collection stage of the 
evaluation. 

·■ Information should be provided on the balance between primary data 
generation and the use of secondary sources. This will allow evaluators to 
better plan the work programme and calculate its cost. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

The  development  of  the  evaluation  methodology  will  be  undertaken  in  co­
operation  with  the  Commission  services  and the  relevant  experts  selected 
for the study. The evaluation methodology  should comprise  both  qualitative 
and quantitative assessments. 

The study would involve two main elements: desk research and field work  in 
the regions concerned. 

•  Desk  research 

The consultant  should undertake a detailed  comparative  assessment  of the 
areas  for  which  he  will  be  appointed. The  review  will  concern  past  and 
current  programmes  (CSFs/SPDs,  OPs  and  other)  and their  main  compo­
nents  relating  to  RTD  activities  such  as  regional  analyses,  strategic 
approaches,  hierarchy of objectives, priorities and measures, indicators  and 
profiles, synergy between RTD measures and other components. 

The study  will draw  extensively  upon the  contents  of the  programmes,  and 
the  available  evaluation  exercises  for  those  programmes.  In  particular,  it 
should take into consideration the methodology and lessons drawn from the 
RTD  Thematic  evaluation  for  Ireland,  Portugal,  Greece  for  the  1989-93 
period. It should also take into account the results of recent evaluations such 
as the  mid-term  evaluations  (for  objective  1 regions)  and  the  evaluation  of 
RTPs (Regional Technology  Plans) under the ERDF art. 10. 

The consultant  is invited to collect data and statistics and to develop appro­
priate  indicators  which  may  be  relevant  for  evaluation  purposes  in order  to 
provide  key  elements  on  RTD  expenditure,  as  well  as  the  nature  and  the 
impacts of the actions funded. 

•  Field work 

In order to derive first hand information on the effects of RTD related actions, 
the consultant will interview key informants in the regions concerned. A short 
list  of  key  issues  (to  be  carefully  selected)  should  be  used to  structure  the 
questionnaire  or  checklists  for  use  in the  interviews.  A  number  of  case 
studies  will  be  selected  to  provide  in-depth  and  contextual  evidence  and 
information  on  best  practices  on  RTD  related  actions  under  the  Structural 
Funds. 

Appendix 

Deliverables 

The deliverables (inception report, interim and final reports, case study 
reports, etc.) that the evaluation will produce, both intermediate and final, 
should be listed and their contents detailed when necessary. 
• What format should be used for the production of documents? 
• What type of computer file should be used? 
• Which language(s) should be used? 

*■ The timetable for an evaluation is usually determined by the decision­
making calendar and not by methodological requirements. Rather than 
impose a timescale that may reduce the depth, scope or even quality of 
the evaluation, the production of an interim report can be an acceptable 
compromise. 

The proposed work programme and calendar should include: 
­ a detailed work programme covering essential questions and a methodo­
logical framework for the thematic study 
­ national reports covering the desk research and field work stages 
­ a final synthesis report, including a summary focusing on policies to be 
followed, which must be understandable and usable independently of the 
main report. 
­ All the texts and the corresponding database will be established or trans­
mitted (hard copy and diskette) in a format determined by DG XVI of the 
European Commission. 
­ Reports and documents will be presented in English, French or German. A 
supplementary budget is planned for translation into at least one of the two 
languages. 

Calendar 

The duration of the evaluation should be indicated along with any inter­
mediate deadlines. 

The study, divided into two separate lots will be completed in one year. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Appendix 4: List of boxes 

Introduction 

Box 1 - Priorities, domains and themes 
Box 2 - Mainstreaming: a definition 

8 
11 

Technical considerations relating to the assessment of thematic 
impacts 

Box 3 - Using objectives trees for clarifying implicit and explicit objectives 
related to a given theme 
Box 4 - In-depth analysis of impacts: appropriate tools and constructing an 
evaluation method 
Box 5 - Typical strengths and weaknesses of sources of existing 
information 
Box 6 - Estimating impacts net of deadweight 
Box 7 - Additionality, deadweight  and leverage effects 

21 
23 
25 

16 

20 

Evaluating the environmental impacts of structural interventions 

Box 8 - An adaptation of the Driving Force-State-Response  (DF-S-R) 
model of sustainable development 
Box 9 - Detailed typology of environmental impacts of structural 
interventions 
Box 10 - Environmental objectives of Structural  Fund Interventions 
presented in the form of a thematic objectives tree. 
Box 11 - Example of a system of indicators for the construction of 
waste-water treatment plants 
Box 12 - Example of an environmental screening form 

33 

36 

37 

47 
53 

Evaluating the impact of structural interventions on the 
competitiveness of Small and Medium-Sized  Enterprises 

Box 13 - Definition of SMEs 
Box 14- Example of a thematic objectives tree for SME 
interventions 
Box 15 - Example of a system of indicators for an intervention  providing 
informational support for SMEs to promote exports 
Box 16 - Example of a factor analysis of the Catalan co-operative 
sector 

59 

65 

71 

74 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Box  17 - Example of the use of the case study technique in the 
evaluation of the NordTek programme, Denmark 

75 

Evaluating the impact of structural interventions on employment 

80 
80 

Box  18 - Different types of skill obsolescence 
Box  19 - Breakdown of different types of employment effects 
Box 20 - Production of employment  effects through structural 
interventions 
Box 21 - Structural  Fund interventions and employment  effect 
mechanisms 
Box 22 - Example of a thematic objectives tree for the promotion of employ­
ment 
Box 23 - Comparison of deadweight, displacement and substitution 
effects 
Box 24 - Example of a system of indicators for a return-to-work 
programme 

86 

89 

97 

82 

82 

Evaluating the impact of structural interventions on equal opportunities 

Box 25 - Example of a thematic objectives tree for the priority of equal 
opportunities 
Box 26 - Example of effects  by category of impact 
Box 27 - Example of the application of the principle of 
mainstreaming 
Box 28 - Example of a system of indicators for the establishment  and 
re-generation of community centres 
Box 29 - Standard form for assessing the equal opportunities dimension in 
the framework of ex ante evaluation 
Box 30 - Matrix for programme assessment 

121 

124 
125 

112 
114 

120 

Evaluating impacts arising from interventions in the domain of 
Research and Technological  Development 

Box 31 - Thematic evaluation of the impacts of CSFs for Research and 
Technology  in Greece, Ireland and Portugal for the 1989  -1993 
programming period 
Box 32 - Framework for the evaluation of Structural Fund interventions in 
the domain of RTD 
Box 33 - Example of a thematic  objectives tree for RTD 
interventions 

134 

138 

139 

Appendix 

Box 34 - Example of a system of indicators for an intervention  providing 
financial and informational support for technology acquisition by 
SMEs 
Box 35 - Example of a score card for a joint research venture 

145 
146 

Evaluating impacts arising from interventions in the domains of 
transport, telecommunications and tourism 

Box 36 - Example of a system of indicators for upgrading of a railway line for 
freight operations 
156 
Box 37 - Example of a system of indicators for the deployment of a 
broadband Hybrid Fibre Coax (HFC) network 
Box 38 - Example of support for the creation of tourist facilities (museums, 
theme parks, etc.) 
Box 39 - Example of a SWOT analysis matrix 
Box 40 - Extract from a "corridor ranking framework" applied to 
Greece 

160 
161 

157 

163 

A step-by-step approach for thematic analysis 

Box 41 - Transversal evaluation of a theme : "Step by step" approach for a 
programme 
Box 42 - Thematic assessment matrix. Example relating to equal 
opportunities 
Box 43 - Score card. Example: assessing the extent to which a measure 
takes into account of the priority of equal opportunities 
Box 44 - Interviews, focus groups and case studies 
Box 45 - Summary table of measures and their effects. Example of 
improving the suitability of living conditions to women's needs 
Box 46 - Summary of the key steps and evaluation tools 

170 

173 

174 
175 

176 
177 

Cross-programme thematic  evaluation 

Box 47 - Comparison of supra-national synthesis of programme evaluations 
180 
and transnational evaluation 
183 
Box 48 - Sampling with the objective of highlighting best practice 
Box 49 - The development of transnational evaluations 
184 
Box 50 - Sampling with the objective of making a pan-European 
analysis 
Box 51 - Example of multiple-case studies with replication design 
Box 52 - The seven steps of research synthesis 

185 
187 
188 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Index 

ethnographic observation, 123 
Europe-wide analysis of impact, 179 
evaluation research See research synthesis, 

187 

evaluation team, 16,120,169,178,181 
evaluatory question, 17 
expert panel, 148 
external coherence, 15 
externality, 137,148,164 

-F-

factor analysis, 74 
feasibility study, 180 
focus group, 73,123,175 
framework conditions, 62 
full-time employment, 94 
full-time equivalent (FTE), 85 

-G-

gross employment impact, 23, 87 
gross impact, 22 

-I-

impact descriptor, 172 
implicit objective, 15 
income multiplier effect, 24,25,82,88 
indicator, 18 
indicator (examples of), 47, 71,97,121,145, 

156,157 

indirect effect See induced effect, 155 
induced effects, 155 
information system, 20 
input-output model, 104 
internal coherence^ 5,38 
interview technique, 73,123,127,149,160, 
175 
Ishikawa diagram, 123 

job creation or maintenance effect, 77,80, 

88,101 

absorption, 86,133,137 
activity rate, 89 
ad hoc indicator, 44, 94,118 
additional effects, 25 
additionality, 25,137 
attraction effect, 89,103 

-B-

baseline, 23,43,119 
benchmarking, 182 
best practice, 63,123,182 
bibliometric indicator, 148 

-C-

case study technique, 54, 75101,123,125, 
149,160,163,175,181 
coefficients, 22,55,95,103,184 
colour vote, 123 
Community value-added, 179 
comparative analysis, 102,123,186 
confounding factor, 18,79,95 
context indicator,  17,72,119,122,147,171 
contingent valuation method, 55 
control group, 87,102 
cost-benefit analysis, 55,148,163 
cost-effectiveness analysis, 101,135 
counterfactual situation, 22,65,73, 87,99, 

119 

deadweight, 22, 38, 65, 87,101,103,140, 
159 
DELPHI technique, 164 
demand-side effect, 155 
displacement, 22,38,65,88,101,159 
domain of intervention, 129 
Driving Force-State-Response model, 33,43 

-E-

employability, 78, 79,95 
employability effect, 80,88,101 
employment effect, 25, 74,77,80 
environmental impact assessment (EIA),29, 

36,52 

leverage effect, 24,66, 70,140,144 

Transversal evaluation of impacts  on intervention  priorities 

-M-

strategic environmental assessment (SEA), 

32,50 

substitution, 22, 65, 88,103,140,159 
summary table, 176 
supplier effect, 25,84,88 
supply-side effect, 155 
sustainable development, 29 
sustainable employment, 84,94 
SWOT analysis, 123,161,182 
synergy effect, 65, 79,136,154,164 

-T-

technical committee, 180 
temporary employment, 84, 93 
terms of reference, 180 
thematic assessment matrix, 172,177 
transitory employment, 84 
typology of impact, 35,113,139,172 

-U-

unemployment, 77,98,125 
unemployment (effect on), 89 

macro-economic model, 104 
mainstreaming, 11,  70,107,120 
matched pairs, 175 
matrix of cross impacts (SMIC), 164 
mitigation, 42 
monitoring data, 73, 99,147,185 
multi-country econometric analysis of 
impact, 184 
multi-criteria analysis (MCA), 55 
multiple case studies with replication design, 

186 

multiple case study technique, 55 

-N-

net employment impact, 79, 87 
net impact, 22,38,66,101 

-O-

objectives tree,  16,37,65,86,112,139,171 

-P-

part-time employment, 94 
policy-off situation, 22 
political priority, 16,27 
positive discrimination (affirmative action), 

111 

primary data, 20, 71,147 

-Q-

questionnaire survey, 73,101,123,127,148 

reference situation, 17,96 
regression analysis, 102 

-s-

sampling, 181,185 
scenario technique, 123 
scoping, 50, 52,171,174 
score card, 146,173 
scores and weights, 160 
scoring See scores and weights 
screening, 50, 52,169 
shift-share analysis, 102 
spillover, 148 
statistical surveys, 21,98 
step-by-step approach, 169 

Acknowledgements 

The preparation of this Volume took place over several years from 1996 to 1998. The pre­
paratory work included the following steps: 

Several workshops and meetings which allowed the points of view of experts in different 
themes and the transversal (or multi-thematic)  perspective  of those responsible for the 
Structural Funds to come together. 

This process firstly gave rise to the production of two MEANS Handbooks on employment 
(Handbooks  3 and 6) as well as a methodological  note  on evaluating  human resource-
related interventions. These documents were used to provide the basis of the chapter on 
employment. 

Similarly the  chapters  on the environment  and equal opportunities  are both based on 
recent  methodological  notes  resulting from a confrontation  of the ideas of  experts and 
Commission officials. The consultation procedure employed during the preparation of the 
note on the environment  is particularly  noteworthy  for  its  use of the  Impact  Mapping 
technique (see Volume 3 of the MEANS Collection). 

The production of this Volume was coordinated by Anastassios Bougas and David Sweet 
(D.G. XVI.G.2  : Coordination of Evaluation) and involved several  Directorate-Generals  of 
the European Commission, notably D.G. II, V, VI, XI, XIV and XVI and in particular Philippe 
Goybet, Anne Bucher, Nelly  Bandarra, Miguel  Benito Alonso, R. Bradley, Ciaran Dearie, 
Sylvie Finne, Martin Hallet, Richard Harding, Mikael Kirosingh, Frédérique Lorenzi, Andrea 
Mairate, René Marty, Valéry Morard, Irmengard Nelles, Anastasios  Nychas, Julius Op de 
Beke, Sergio Piccolo, Brigitta Renner-Loquenz, Jacqueline Soulier Oliveira, Martin Tyrrell, 
Martine Voets and Dominique Van Der Wee. 

This Volume has been written by the Centre for  European  Evaluation Expertise, notably 
with the participation, Cristina Diaz Gomez, Andreas Esche, Connor  Kearney and Kevin 
Williams. It was translated into English by Liz Libbrecht. 

The authors wish to thank all those who contributed, either directly or indirectly, to the pro­
duction of  this  Volume, and  in particular  the following  experts:  Christine  Abrossimov 
(DARES), Mariano d'Antonio  (Universita da Roma), Peter Auer  (Wissenschaftszentrum, 
Berlin), Jean-Claude Barbier (Centre d'Etudes pour l'Emploi), Martin Bekker (B&A Groep), 
Peter  Bosch  fThe  European  Environment  Agency),  Marco  Causi (CLES  Sri), Antonio 
Estevan Estevan (Gabinete de Economia Aplicada), Marta Ferrés (Interés General), Chris­
topher  Hasluck  (University of Warwick), Andy  Hirst  (PA Cambridge  Economics Consul­
tants), Liisa Horelli (ITSU OY), Japp de Koning (Netherlands Economic Institute), Kierän A. 
Kennedy  (The Economic  and Social  Research  Institute), Ulla Koski  (Finnish  Ministry of 
Environment)Marie-Christine  Lefebvre  (Lconsult  SPRLHedwig  Lutz  (WIFO),  James 
Medhurst (Ecotec), Dr Heino Von Meyer.Chris Newton, (The Environmental Agency), John 
Rhodes (PA Cambridge Economics Consultants), Barry Sadler (Institute of Environmental 
Assessment), Claude  Seibel (DARES), Thanos  Skouras (Reseach Centre of the Athens 
University  of  Economics  & Business), Antoni  Soy  (Centro  d'Estudis  de  Planiflcacio), 
Joachim Spangenberg (Wuppertal Institute for Climate, Environment & Energy), Peter Tyler 
(PA Cambridge Economics Consultants), Pieter Van Run (CBEA), Claudia Villante (ISFOL). 

European  Commission 

Evaluating socio-economic  programmes 
Transversal evaluation  of  impacts on the  environment,  employment 
and other  intervention  priorities 

Luxembourg: Office for Official  Publications  of the  European Communities,  1999 

1999 —  215  p p .—  14.8x21  cm 

ISBN  92-828-6626-2 

Venta  ·  Salg  ·  Verkauf  ·  Πωλήσεις  ·  Sales  ·  Vente  ·  Vendita  ·  Verkoop  ·  Venda  ·  Myynti  · Försäljning 

BELGIQUE/BELGlE 

ÖSTERREICH 

Jean De Lannoy 
Avenue du Roi _02/Koninn__an 202 
B­1190 Bruxe«es_russo[ 
Tel. (32-2)538 43 08 
Fax (32-2) 538 06 41 
E­mail, jean de.LannoyOintoooard.be 
URL: http www jean­ae­iannoy De 

La librairie européenne/ 
De Europese Boekhandel 
Rue de la Loi 244/Wotsttaal 244 
B­1040 Pruid'es­Brussel 
Tel. (32-2)295 26 39 
Fax (32-2) 735 08 60 
E­mail: mailOlibeurop be 
URL: hrtpJ/www hbeurop.be 

Moniteur bel ge/Belg lac h S t u t s b l ad 
Rue de Louvam 40­42/leuvenseweg 40­42 
B­1000 Bruxelles/­russel 
Tôt. (32­2)552 22 11 
Fax (32­2) 511 01 84 

DANMARK 

J. H. Schultz Information A/S 
Heretedvang 10­12 
DK­2620 Albertslund 
Tit (45)43 63 23 00 
Fax (45) 43 63 19 69 
E­mail. schuttzOschuüz dk 
URL: htlp:/Avww Schultz dk 

DEUTSCHLAND 

Bundesanzeiger Vertag GmbH 
Veni iL'bsaDtei lung 
Amsterdamer Straße 192 
D­50735 Köln 
Tel (49-221)97 66 80 
Fax (49-221) 97 66 82 78 
E­Mail: vertnebObundesanieiger.de 
URL: rittp^/www.bundesanzeiger.de 

ΕΛΛΑΔΑ/GREECE 

G. C. Eleftheroudakl· SA 
International Bookslore 
Panepisiimrøu 17 
GR­10564 Athina 
Tel (30­1 ) 331 41 80/1/2/3/4/5 
Fax (30­1)323 9821 
E­mail: elebooksOnelof gr 

ESPAÑA 

Boletín Oficial del Estado 
Trafalgar. 27 
E 28071 Madrid 
Tel (34)915 38 21 11 (Libros). 
913 B4 17 15(Suscnp.) 
Fax (34) 915 38 21 21 (Libros). 
913 B4 17 14 (Suscflp.) 

E­mail: clientes O com boe es 
URL httpi/www boe es 
Mundi Prensa U b r o s, SA 
Castello. 37 
E­28001 Madrid 
Tel (34)914 36 37 00 
Fax ¡34)915 75 39 98 
E ­mail 11 br e na O mundiprensa. es 
URL: httpy/www.murtdipiensacom 

FRANCE 

Journal ottici«! 
Service des pubi­cali on s des CE 
26. rue Desaix 
F­75727 Paris Cedex 15 
Tel (33) 140 58 77 31 
Fax (33) 140 58 77 00 
URL: http //www journal'Officiel gouv Ir 

IRELAND 

Government Supplies Agency 
Publications Section 
4­5 Harcourt Road 
Dublin 2 
Tel (353­1)681 31 11 
Fax (353­1)47527 60 

ITALIA 

Licosa SpA 
Via Duca di Calabria. 1/1 
Casella postale 552 
1­50125 Firenze 
Tel (39) 055 64 83 1 
Fax (39) 055 64 12 57 
E­mail: hcosaOftbcc it 
URL: http //www.ftbcc.it/licosa 

LUXEMBOURG 

Messageries du livre SARL 
5. me Raiffeisen 
L ­2411 Luxembourg 
Toi (352)4010 20 
Fax (352) 49 06 61 
E mail mailOmdl lu 
URL­ http //www.mdl lu 

NEDERLAND 

SOU Servicecentrum Uitgever* 
CiriMorfei Planton straat 2 
Postbus 20014 
2500 EA Den Haag 
Tel (31­70) 378 90 80 
Fax (31­70) 378 97 83 
E­mail sduOsdu n 
URL: utip . www sdu nl 

Manz'sche Verlags­ und 
Universitátsbuchhandlung GmbH 
Kohlmarkt 16 
A­1014 Wien 
Tel. (43­1) 53 16 11 00 
Fax (43­1)53 1611 67 
E­Mail: bestellen β manz co at 
URL r, tip ://www. manz. at/index, htm 

PORTUGAL 

Distribuidora de Livros Bertrand L d .' 
Grupo Benrand. SA 
Rua das Tertas dos Vales. ■'■ ­ A 
Apañado 60037 
Ρ­2700 Amadora 
Tel. (351-1)495 90 50 
Fax (351­1) 496 02 55 
Imprensa NaclortaI­Casa da Moeda, EP 
Rua Marqués Sá da Bandeira, 16­A 
Ρ­1050 Lisboa Codex 
Tel.(351­1)353 03 99 
Fax (351­1)353 02 94 
E­mail del irtcmOmail lolepac pt 
URL: http^/www.incm.pl 

SUOMI/FINLAND 

Akateeminen Kirjakauppa/ 
Akademiska Bokhandeln 
Keskuskatu i /Cent raiga tan 1 
PL/PB 128 
FIN­00101 Helsinki/Helsingfors 
PJtfn (358­9) 121 44 18 
V : . i. (358­9)121 44 35 
Sähköposti. akatilausOakaloeminen.com 
URL. htlp^/inrww.akaieeminen com 

SVERIGE 
B T J AB 
Traktorvager 11 
S­221 82 Lund 
TIn (46­46) 18 00 00 
Fax (46­46) 30 79 47 
E­post bljfìu­pubQblj.se 
URL: hnp://www b1j.se 

UNITED KINGDOM 

The Stationery Office Lid 
International Sales Agency 
51 Nine Elms Lane 
London SW8 5DR 
Tel. (44­171)373 90 90 
Fax (44­171)873 64 63 
E­mail: ipa.enqumesOlheso.co uk 
URL http7/www the­stabonery­oftice co.uk 

ISLAND 

Boka bud La rusar Blondel 
Skólavordustig. 2 
15­101 Reykjavik 
Tel (354) 551 56 50 
Fax (354) 552 55 60 

NORGE 

S wet s Norge AS 
O s le n j ovo ie n 18 
Boks6512Et1erstad 
N - 0 6 0 6 O s_ 
Tel. (47­22) 97 45 00 
Fax (47­22) 97 45 45 

SCHWEIZ/SU ISSE/SVìZZERA 

Euro Info Center Schweiz 
_o OSEC 
Slamptenbachstrafle 85 
PF 492 
CH­8035Z_nch 
Tel. (41­1) 365 53 15 
Fax (41­1) 365 54 II 
E­mail eicsOosecch 
URL: htlp://www osee ch/eics 

BAL GARDA 

Europrees Euromedia Lid 
59. brvd Vilosha 
BG­1000 Solia 
Tel (359-2) 960 37 66 
Fax (359-2) 980 42 30 
E­mail MilenaCmboi cri bg 

CESKA REPUBLIKA 

ÚSIS 
NlS­prodejna 
H a velkova 22 
CZ­l30 00Praha3 
Tel (420­2)24 2314 86 
Fax (420­2) 24 23 11 14 
E­mail nkpospOdoc nis cz 
URL: http//ustscr cz 

CYPRUS 

Cyprus Chamber of Commerce and Industry 
PO Box 1455 
CY­1509 Nicosia 
Tel (357-2)66 95 00 
Fox (357­2) 66 10 44 
Email demelrapOccci org cy 

HHVATSKA 

Medialrade Lid 
Pavia Hatza 1 
HR­10000 Zagreb 
Tel. (385­1)481 94 1 
Fax (385­1) 481 94 1 

MAQYAflORSZAG 
Euro into Service 
Europa Hâz 
Margiiszigel 
PO Box 4 75 
H­1396 Budapest 62 
Tel- (36-1)350 80 25 
Fax(36­t)350 90 32 
E ­ m a 11 e υ r ο ι π ίο « î mail mata ν hu 
URL: http//www.eurotnlo.hiilnoex him 

MALTA 

Miller Distributors Ltd 
Malta International Airporl 
PO Box 25 
Luqa LG A 05 
Tel (356) 66 44 88 
F a x ( 3 5 6 ) 67 67 99 
E­mail: gwirthOusa.net 

POLSKA 

Ars Polona 
Krakowskie Przedmiescie 7 
Skr. pocztowa 1001 
PL­00­950 Warsiawa 
Tel (48­22)826 12 01 
Fax (4S­22) 826 62 40 
E­mail: ars pol ­ b e vy hsn com pi 

ROMANIA 

Euromedia 
Str. G­ral Berthelot Nr 41 
RO­70749 Bucuresti 
Tel (40­1)3154403 
Fax (40­11314 22 86 

R05SIYA 

CCEC 
60­letiya Oktyabrya Av 9 
117312 Moscow 
Tel. (7-095) 135 52 27 
Fax (7­095) 135 52 27 

SLOVAKIA 

Centrum VTI SR 
Nam Slobody, 19 
5K­61223 Bratislava 
Tel (421­7)54 41 83 64 
Fax (421­7) 54 41 83 64 
E­mail: europOlbb1.5tlk.stuba.sk 
URL: http .'.'www 5ltk.stuba sk 

5 LOVEHIJA 

Gospodarski Vestnlk 
Dunajska cesia 5 
SLO­10OO Ljublana 
Tel (386)613 09 16 40 
Fax (386)61309 1645 
E­mail: europe ovest m k si 
URL: httpJ/www.gveslnik.si 

TÙRKIYE 

Dünya Intotel AS 
100. Yil Mahallessi 34440 
TR­80O5O Bagolar­Istanbul 
Tel (90­212)629 46 89 
Fax (90­212) 629 46 27 
E­mail, intolel θ dunya­gazeie com.tr 

AUSTRALIA 

Hunter Publications 
PO Box 404 
3067 Abbolsford. Victoria 
Tel (61­3)94 17 53 61 
Fax (61­3) 94 1971 54 
E­mail: jpda «es Ooze mail com au 

CANADA 

Les «dillons La Liberté Inc. 
3020. chemin Samte­Foy 
G1X 3V Samle­Foy, Quebec 
Tel (1­418)658 37 63 
Fax (1­800) 567 54 49 
E­mail libertoOmodtom.oc ca 

Rr­nouf Publishing Co. Ltd 
5369 Chemin Canotek Road Unii 1 
K1J 9J3 Ottawa. Ontano 
Tel (1­613)745 26 65 
Fax(1 ­613)745 76 60 
E­mail: order deptOrenoutbooks.com 
URL: http //www renoufbooks com 

EGYPT 

The Middle East Observer 
41 Shenf Street 
Cairo 
Tel (20-2)392 69 19 
Fax (20­2) 393 97 32 
E­mail matoudaOmeobserver com eg 
URL: htlp7/www.meobserver com.eg 

EESTI 

INDIA 

Eesti Ka u b i nd us­Toost vi skods (Estonian 
Chamber of Commerce and Industry) 
Toom­Kooli 17 
E E­0001 Tallinn 
Tel (372)646 02 44 
Fax(372)646 02 45 
Email einloOkodaee 
URL: htlp .'www koda ee 

EBIC India 
3rd Floor. Υ Β Chavan Centre 
Gen J. Bhosale Marg 
400 021 Mumbai 
Tel (91­22)282 6064 
Fax ¡91­22)285 45 64 
E­mail: BbicOglasbmOl vsnl.net.in 
URL: htlpy/www ebondia com 

ROY luteiiintiun:il 

41. Mishmar Hayarden Street 
PO Box 13056 
61130 Tel Aviv 
Tel. (972-3)649 94 69 
Fax (972-3) 648 60 39 
E­mail: royilOnetvision.net il 
URL: htlp^/www foyinl.co il 

Sub­agent for the Palestinian Authority: 

Index Information Services 

PO Box 19502 
Jerusalem 
Tel. (972­2) 627 16 34 
Fax (972­2) 627 12 19 

Ρ Si­J apan 

Asahi Sanbancho Plaza «206 
7­1 Sanbancho. Chi yoda­ku 
Tokyo 102 
Tel (81­3)32 34 69 21 
F_x(81­3Í32 34 6915 
E­mail: books ü ps ι­japan co jp 
URL: httpiWwww psi ­japan com 

EBIC Malaysia 

Level 7. Wisma Hong Leong 
1B Jalan Perak 
50450 Kuala Lumpur 
Tel. (60­3) 262 62 98 
Fax (60­3) 262 61 98 
E­mail: ebic­klOmol.net.my 

Mundi Prensa Mexico. SA do CV 

Rio Panuco No 141 
Colonia Cuauhtemoc 
MX­06500 Mexico. DF 
Tel (52­5) 533 56 58 
Fax (52-5) 514 67 99 
E­mail: 1015452361 Ocompuserve com 

EBIC Philippines 

19th Floor, PS Bank Tower 
Sen Gil J. Puyal Avo. cor. Tindalo Si. 
Makati City 
Metro Manilla 
Tel (63-2)759 66 80 
Fax (63-2) 759 66 90 
E­mail: occpcomOglobe com ph 
URL: hltpJAwww eccp.com 

EBIC Sri Lanka 

Trans Asia Hotel 
115 Sir chittampalam 
A Gardiner Mawatha 
Colombo 2 
Tel. (94­1)074 71 50 78 
Fax (94­1)44 87 79 
E­mail: ebicsieitmin.com 

EBIC Thailand 

29 Vanisse Building. Btrt Floor 
Soi Chidlom 
Ploenchl 
10330 Bangkok 
Tel (66-2)655 06 27 
Fax (66-2) 655 06 28 
E­mail: ebicbkkOkscl5.thcom 
URL: httpJVvww obicbkk.org 

UNITED STATES OF AMERICA 

Berna n Associates 

4611­F Assembly Dnve 
Lanham MD207Ó6 
Tel (1­800) 274 44 47 (toll Iroe telephone) 
Fax (1­800) 865 34 50 (ton Iroe lax) 
E­mail: queryObeman.com 
URL http //www beman com 

Bille wenden Sla sich an ein Büro Ihrer 
Wahl/ Please contaci the sales office 
of your c h o i e ·/ Veuillez vous adresser 
BU bureau de venie de votre choix 

Office tor Official Publications 
ot lha European Communities 
2. rue Mercier 
L­2985 Luxembourg 
Tel (352)29 29­42455 
Fax (352) 29 29­42758 
E­mail: into info Oopoce cec be 
URL: httpJ/eur­op.eu int 

The  MEANS  collection: 
"Evaluating  socio-economic  programmes" 

This  six-volume  set  gives  wider  public  access  to  the  lessons  of  the  MEANS 
programme which the  European Commission  introduced with a view to  impro­
ving  and  promoting  evaluation  methods.  Its  publication  meets  a  growing 
need for  reliable  benchmarks  to  deal with  the  problems  of evaluation  encoun­
tered when implementing  public  measures in general and Structural  Fund ope­
rations  in  particular. 

In  this  context,  the  set  constitutes  a  unique  methodological  guide  that  can 
solve  the  technical  and  organisational  problems  arising  in  the  course  of 
evaluation  work  and for  which  the existing  literature  provides  little  help. 

The  collection  is  aimed  not  only  at  professional  evaluators,  for  whom  it  will 
provide a valuable  handbook,  but also at non-specialists,  in particular  manag­
ing  authorities  and  structural  programme  Monitoring  Committees. 

Evaluation  design  and  management 

Each  volume  is  designed  to  be  independent  of  the  others,  yet  they  are  also 
complementary. Together, they  deal with all aspects of the evaluation of  socio­
economic  programmes: 
I. 
II.  Selection  and  use of  indicators  for  monitoring  and  evaluation 
III.  Principal  evaluation  techniques  and  tools 
IV.  Technical  solutions  for  evaluation  within  a partnership  framework 
V.  Transversal  evaluation  of  impacts  on  the  environment,  employment  and 

other  intervention  priorities 

VI.  Glossary  of  300  concepts  and technical  terms 

OFFICE FOR OFFICIAL  PUBLICATIONS 

OF THE EUROPEAN  COMMUNITIES 

_-2985  Luxembourg