Evaluating socio-economic programmes : volumes 1-6.
Date: 2009-07-04
Institutions: Directorate-General for Regional and Urban Policy
Subjects: EU programme, Structural Funds, economic and social cohesion, economic development, project evaluation, social development, structural policy

Evaluating socio-economic programmes 

Evaluation design and management 

EC Structural Funds 

Evaluating socio-economic programmes 

Evaluation  design  and 
management 

Volume  1 

European Commission 

MEANS is a programme  of  the European  Commission. 
It is aimed  at improving  Methods  of Evaluating Structural  Policies. 
The MEANS  Programme  and  Collection  was supervised  by 
Phillipe  Goybet and  Miguel  Angel  Benito  Alonso, 
Unit for  the Coordination  of Evaluation in  the Directorate  for  Coordination 
and  Evaluation of  Operations  of DG XVI 
Coordination-evaluation@fmb.dg16.cec.be 

The realisation  of  the Programme  was entrusted  to the 
Centre  for European  Evaluation Expertise (C3E) 
under  the direction  of Eric Monnier  and Jacques  Toulemonde. 
Responsibility  for  the opinions  expressed in this 
boon lies solely with  the authors  of  the text: 
C3E  -  13 bis Place Jules Ferry  -  F  -  69006 Lyon  -  France  -  c3e@c3e.fr 

A great deal of additional information on the European Union is available 
on the Internet. 
It can be accessed through the Europa server (http://europa.eu.int) and 
on the Inforegio Website (http://inforegio.cec.eu.int). 

Cataloguing data can be found at the end of this  publication. 

Luxembourg: Office for Official Publications of the European 
Communities,  1999 

ISBN  92-828-6626-2 

©  European Communities,  1999 
Reproduction is authorized, except for commercial purposes,  provided 
the source is  acknowledged. 

PRINTED  ON WHITE CHLORINE-FREE  PAPER 

Printed in Italy 

Foreword 

is  directly 

Developing  good  evaluation  practices 
has  become  a  priority  area  for 
the 
relevant 
Commission  and 
to 
the  effective  use  of  Community 
resources. 
Indeed, the sound  financial  management 
principles  which  the  Commission  seeks 
to  promote  via  its  SEM2000  (Sound  and 
Efficient  Management)  Programme  rely, 
specifically,  on  the  regular  evaluation  of 
the  Community  activities. 
The  Directorates  General  responsible  for 
structural policy  have considerable expe­
rience in this area.  Since  1988 they  have 
in  the  process  of 
made  evaluation  an  integral  part  of  each  stage 
Community  Structural  intervention.  Thus,  the  programme  development 
stage  includes  a  prospective  evaluation;  the  implementation  stage  a 
strengthening  of the monitoring systems and, at the end of the intervention, 
evaluation  establishes  what  has  been  achieved  in  terms  of  programme 
impacts. 
However, the  capacity  of  evaluation  to  contribute  to the  effectiveness  and 
transparency  of  structural  interventions  rests  ultimately  on  the  integrity  of 
the  methods  adopted  to  verify  the  credibility  of  results. 
This  is why, four  years  ago, on  the  initiative for  the  Directorate  General  for 
Regional  Policy  and  Cohesion  (DG  XVI),  the  MEANS  programme  was 
initiated  as a response to the  demand  for  developing  evaluation  methodo­
include  practices  combining  both  general  scientific 
logy.  This  also 
requirements  and  the  specific  requirements  of  Community  interventions 
whilst,  at the  same time,  remaining  accessible  to  a large  audience. 
One  of  the  more tangible  results  of  the  MEANS  programme  has  been  the 
publication  of this six volume set  by a group  of  independent  experts,  focu­
sing  on  the  principal  elements  of  socio-economic  programme  evaluation. 
A short  summary  of this work  can be found  at the  beginning  of the  present 
volume. This collection covers the various  methodological  approaches  and 
innovations  that  have  taken  place  ¡n the  context  of  structural  funds  eva­
luation.  It  will  enable  those  responsible  for  evaluation  to  develop  their 
approach  in  a  flexible  and  independent  manner  and  for  the  various 

programme  partners  to  exercise  greater  control  over  the  evaluation  pro­
cess.  Attending  the  conferences  of  evaluation  specialists  and  "clients" 
convened  in  Brussels  (1995),  Berlin  (1996)  and  Seville  (1998),  I  gained 
some impression  of the interest that  new evaluation techniques  can gene­
rate.  I would  stress that  the development  of  evaluation  needs to  progress 
in  tandem  with  efforts  to  incorporate  evaluation  findings  in  the  decision 
making  process. This  implies  evaluation  as a  means of  establishing  whe­
ther  policies are equitable, effective and relevant to the identified  needs. 
In satisfying  these  requirements,  evaluation  should  be seen as a valuable 
instrument  affording  both  Member  State  and  Community  authorities  the 
necessary  guarantees  that  Community  funds  are  being  deployed  in  the 
best interests of the  EU citizen. This is a cost worth paying when the goals 
of  Economic  and  Social  Cohesion  and  the  European  Union  enlargement 
are at stake. 

Finally,  I wish  to  extend  my thanks  to  the  members  of  the  Committee  of 
Independent  Experts  established  at  the  launch  of  the  programme.  Their 
expertise and standing  have contributed greatly to the quality and relevan­
ce  of  the  MEANS  programme.  I therefore  thank:  E. Chelimsky  (USA), ex-
President  of  the  "American  Evaluation  Association";  K.  Kennedy  (IRL), 
Director  of  the  "Economic  and  Social  Research  Institute"  (ESRI);  J.  R. 
Cuadrado  Roura  (SP),  President  of  the  "European  Regional  Science 
Association"; C. Seibel (F), Director of the "Statistiques démographiques et 
sociales de l'Institut  National de la Statistique et des Etudes  Economiques" 
(INSEE);  L.  Senn  (I),  Director  of  the  Department  of  Economie  Regional  of 
the  University  Bocconi; E. Stern (U.K.), Director of the  unit of evaluation of 
the  Tavistock  Institute;  A.  Sorber  (t)  (NL),  Head  of  the  "Policy  Analysis 
Department of the Ministry of  Finance"; L. I. Strom (SV), Statens  Institut för 
Regionalforsking  (SIR);  L.  Tsoukalis  (GR.),  professor  at  the  University  of 
Athens;  H. Wollmann  (D), professor  at the  University  of  Humboldt  -  Berlin 
and founder  member  of the "European  Evaluation  Society". 

E.  Landaburu 

: 

Contents 

The  MEANS Collection 

Presentation  of Volume  1 

5 

11 

I  Evaluation issues within the framework  of the Structural  Funds  15 

The value added  of  evaluation 
The specificity  of  programmes  and their  evaluations 
Development  of  practice within the framework  of  European 
Structural  Funds 

II  Preparing an evaluation  and defining  its content 

Defining the general framework  of an evaluation 
Deciding  to  launch  an evaluation 
Ordering  an evaluation 

III  Performing  an evaluation 

Analysing the  logic  and  content  of the  programme 
Evaluating the  impacts  of  an entire  programme 
Analysing  macro-economic  impacts 
Carrying  out  more in-depth  analyses 

IV  Using an evaluation 

Optimising the  use of an evaluation 
Assessing the  quality  of  an evaluation 

V  Guaranteeing the  success  of an evaluation 

Better  preparation  of  an evaluation 
Opening  evaluation to  more partners 
Consolidating  evaluation techniques 

Annexes 

17 
27 

35 

49 

51 
65 
75 

85 

87 
109 
135 
147 

159 

161 
169 

181 

183 
189 
197 

203 

Annex  1 -  Process  of ex ante, mid-term  and ex post  evaluation  205 
211 
Annex 2 -  Guide for drawing  up the terms of  reference 
217 
Annex  3 -  Bibliographic  references 

Index 

219 

The  MEANS Collection 

in 

For  several  years  the  European  Commission  has  been 
training  and 
research,  development, 
investing 
communication, with a view to  improving the quality  and 
utility  of  evaluations  performed  within  the  framework  of 
economic  and  social  cohesion  policy.  This  work  has 
taken  place within  a global  programme  entitled  MEANS: 
Means  for  Evaluating  Actions  of  a  Structural  Nature, 
initiated by the Directorate General XVI (Regional Policies 
and  Cohesion).  The  MEANS  programme  is  justified  by 
the importance of evaluation in a domain which accounts 
for about one third of the Union's  budget. The aim of the 
programme  is  to  devise  a  coherent  set  of  methods  for 
addressing  a  wide  range  of  evaluation  problems 
encountered  in the field of Structural  Fund interventions. 

The 

also 

training  seminars  and 

The outputs  of the  MEANS programme comprise a body 
of  methodological  experiences,  ideas  and  innovations 
which,  until  now,  were  presented  in  seven  50-60-page 
handbooks. 
published 
programme 
methodological  reports  and  quarterly  newsletters,  and 
organised 
three  European 
conferences (the most recent, in Seville, was attended by 
just  under  600  practitioners  involved  in the evaluation  of 
the  Structural  Funds).  An  effort  was  made  to  take  full 
advantage of existing experience and recent innovations, 
particularly  in  the  following  areas:  measurement  of 
employment  effects;  development  of  systems  of 
indicators;  and  evaluation  of  macro-economic  effects. 
Initiation  and  training 
in  evaluation  practices  was 
organised  throughout  the  Commission  in  the  Structural 
Funds sector. 

Through  the  evaluations  initiated  at  the  European  level 
and  the  work  that  has  gone  into  building  on  these 
experiences,  sufficient  knowledge  has  now  been 
to  produce  a  comprehensive  publication 
acquired 

Evaluation design and management 

accessible  to  a  wide  public.  The  result  is  a  collection 
covering  all  aspects  of  the  evaluation  of  structural 
policies,  and  which  takes  full  advantage  of  existing 
MEANS  documents,  revised  and  comprehensive 
to 
guarantee their coherence. 

It  was  decided  to  design the  Collection  without  referring 
too  directly  to  the  regulations  governing  European 
cohesion  policy  for  the  period  1994-1999.  Experience 
had  shown  that  the  regulatory  framework  changes 
frequently  whereas  technical  problems  and  professional 
standards  change very  little. With  a view to  ensuring  the 
relevance over time of the  Collection, it was  intentionally 
decided  to  focus  on  "socio-economic  development 
programmes".  This  term,  considered  in  its  broadest 
sense,  applies  to  interventions  financed  within  the 
framework  of  European  cohesion  policy  and to  all  other 
similar  interventions  which  share  the  same  essential 
characteristics, 
i.e.:  a  development  aim,  multiple 
objectives,  a partnership  dimension  and a  multi-sectoral 
content.  In  fact,  the  evaluation  of  such  programmes 
requires special methods that are not readily found in the 
existing  literature.  This  is  the  main  justification  for  the 
MEANS Collection. 

An  aid  for  those  who  manage 
and  perform  evaluations 

The Collection is intended for two types of readers: those 
involved  in  managing  evaluations  and  those  involved  in 
performing them. Readers in the first category  participate 
in  the  entire  evaluation  process,  including  decisions 
taken  before  and  after  the  actual  evaluation  work.  They 
are primarily  local, national and  European  officials  (in the 
latter  case,  desk  officers  of  the  geographic  units  at  the 
Commission)  responsible  for  managing  programmes 
financed  by  the  European  Union.  The  Collection  must 
offer  them  a complete  set  of  information  on the  state of 
the  art,  particularly  when  they  lack  know-how  in  this 
domain,  so  that  they  are  able  fulfil  their  missions.  The 

The MEANS Collection 

work  of  the  MEANS programme  has  nevertheless  led to 
more  general  methodological  progress  as  well,  which  is 
likely  to  interest  those  responsible  for  commissioning 
evaluations  in other  domains. 

The second  category  of  readers for whom this  collection 
is intended consists of teams responsible for carrying out 
the  actual  evaluation  work:  public  research  centres, 
university 
firms, 
independent  consultants  and  evaluation  units  within 
administrations. 

laboratories,  private  consultancy 

This  Collection  is,  moreover,  likely  to  interest  teachers 
and  students  of  academic  programmes  on  regional 
economies,  economic  policy,  employment  policy,  public 
management,  etc. 

However,  it  does  not  have  an  academic  vocation.  In 
particular,  the  bibliography,  limited  to  a  few  reference 
texts, is not  intended to  be exhaustive. 

General  survey  of  evaluation  in  six  volumes 

The  complete  Collection  consists  of  six  volumes 
designed  to  complement  each  other  and  to  provide 
essential  elements  for  professionals  to  perfect  their 
evaluation  practice  autonomously.  Moreover,  each  book 
is  organised  in  such  a  way  that  it  can  be  read  and 
understood  independently  from  the other volumes in the 
Collection. Apart from the present volume, the Collection 
is organised on the following  lines: 

The  second  Volume  covers  the  selection  and  use  of 
indicators for  monitoring  and evaluation.  It shows  how 
indicators  can  be  used  to  back  political  decisions,  to 
reinforce management and to measure global performance 
of  programmes.  Volume  2  proposes  a  methodological 
framework, numerous examples of indicators, and a guide 
for assessing the quality of the indicators. Its purpose is to 
improve  practices  in terms  of  quantification  of  indicators 
and comparisons  between programmes. 

Evaluation design and management 

The  third  Volume  presents  the  principal  evaluation 
techniques  and  tools  adapted 
to  socio-economic 
programmes.  These  tools  are  taken  from  six  or  seven 
different  scientific  disciplines:  economics;  sociology; 
political  science; management;  geography;  etc.  It is, in a 
sense, an encyclopaedia  of tools presented in the form of 
information  sheets.  Each  sheet  consists  of  a  general 
description  of  the  technique,  the  principles  of  its  use,  a 
list  of  strengths  and  weaknesses,  an  example  of  its 
application, and  a bibliography  of  references. Apart  from 
giving an  idea of the possible  uses of each tool taken  on 
its  own,  the  document  includes  a  guide  for  choosing 
tools  and  for  their  combination,  in  relation  to 
the 
questions  asked  and  the  context.  To  our  knowledge, 
there  is no  other  multidisciplinary  guide  of  this  nature  in 
the  existing  literature.  It  is  certainly  one  of  the  most 
original contributions  of this  Collection. 

Technical  solutions 
for  evaluation  within  a 
partnership  framework  are  the  subject  of  the  fourth 
Volume. The  need for this  book  stems  from  the fact  that 
socio-economic  development  programmes  are  unusual 
objects  of  evaluation.  First,  they  always  involve  several 
public  authorities  within  a  partnership  framework  which 
recognises  the  existence  of  legitimately  different  points 
of  view. They  generally  have  numerous  objectives  which 
are not always clear and coherent enough to form a basis 
for  evaluation.  Secondly,  the  programmes  concern 
multiple  sectors,  which  increases  the  observation  and 
analysis  work  considerably  and  complicates  the  writing 
of  a 
the  multiple 
components of the programme. Finally, since each public 
authority 
its  citizens,  each 
administrative  level  introduces  into  evaluation  its  own 
criteria  and  priorities.  No  complete  solution  to  these 
problems  can  be  found  in  existing  evaluation  manuals. 
Thanks  to  a  number  of  pilot  evaluations,  the  MEANS 
programme  has  tested  and  perfected  four  techniques 
which,  used  together,  offer  a  complete  methodological 
solution.  The  volume  provides  instructions  for  using 

transversal  synthesis,  covering 

is  accountable 

to 

The MEANS Collection 

these  four  techniques:  impact  mapping,  the  cross-
impacts  matrix,  the  scoring  card  and  multicriteria 
analysis. 

The fifth Volume deals with the transversal evaluation of 
impacts  on  the  environment,  employment  and  other 
intervention  priorities.  The  evaluation  of  programmes 
financed by the Structural Funds cannot be limited to the 
examination  of  impacts  aimed  at  directly;  it  has  to 
analyse  the  extent  to  which  European  priorities  as 
diverse  as  employment  impacts,  the  maintenance  of 
jobs,  equal opportunities  between  men and women, the 
environment,  the  competitiveness  of  SMEs,  etc.,  have 
been  taken  into  account.  These  priorities,  or  priority 
themes,  are  often  referred  to  in  operational  documents 
without, however, being the object of mainstreaming. It is 
acknowledged  that  the  evaluation  of  such  transversal 
impacts,  known  as  thematic  evaluation,  is  difficult  to 
achieve  and that  the fact  of there  being  indirect  impacts 
compounds 
the  problem.  This  volume  provides  a 
methodological  framework  for  identifying  the  potential 
into 
impacts  of  a  programme, 
accountóincluding  for  measures  which  are  not  referred 
toóand  identifying and defining appropriate indicators for 
empirically  monitoring  and  analysing their  effects  on  the 
publics concerned. It offers the elements required for the 
analysis and choice of the best methodological  solutions 
for thematic  evaluations. 

taking 

them 

Finally,  the  last  Volume  closes  the  Collection  with  a 
glossary  of  300  concepts  and  technical  terms.The 
writing  of  the  Collection  required  an  extremely  rigorous 
effort  to  homogenise  and  define  the terms  used, and  to 
write  a  specific  index  for  each  volume.  The  glossary 
draws  together  several  hundred  terms  taken  from  the 
different 
indexes,  with  definitions,  translations  and 
descriptions.  It  meets  a  genuine  need  in  so  far  as  the 
quality  of  an evaluation  partly  depends  on the quality  of 
communication  between  the  partners  who  commission 
it,  and  between  the  commissioners  and  the  evaluation 

Evaluation design and management 

team.  In  the  case  of  European  economic  and  social 
cohesion,  interaction  is  made  more  difficult  because  the 
evaluation  is  performed  in  an  international  and  inter-
sectoral  context  involving  people whose  mother  tongues 
and  professional  references  differ.  Moreover,  the  recent 
development  of  evaluation  means  that  many  actors  are 
unfamiliar  with  standard  evaluation  vocabulary  which, in 
turn,  has  not  always  been  firmly  established.  The 
resulting  difficulties  of  communication  can  seriously 
compromise  the  quality  and  utility  of  evaluation  work. 
This Volume  broadly  covers all the subjects  addressed in 
the  rest  of  the  Collection.  In the  numerous  cases  where 
several  definitions  exist  at the  Commission  and/or  in the 
scientific  community,  a single, clear  choice  is  made  and 
the  main  alternative  definitions  prevailing 
in  other 
sectoral  or  national  contexts  are given. 

Presentation  of Volume 1 

This first  Volume  in the  MEANS collection  serves  as a "entry"  both  into the 
Collection  and  into  the  evaluation  of  socio-economic  programmes.  In 
addition  to  the  issues  traditionally  considered  in  currently  available 
evaluation  guides 
(General  Accounting  Office,  1991;  European 
Commission,  1997;  Conseil  Scientifique  de  l'Evaluation,  1994;  see 
bibliography  in Annex  3),  it  presents  and  treats  in  an operational  manner, 
questions  specific  to  socio-economic  programmes,  such  as  interaction 
between  the  different  management  levels  (project,  programme,  policy), 
partnership  between  several  government 
levels,  overall  evaluation  of 
complex  programmes and in-depth analyses of actions  or specific themes. 

This  book  helps  readers  to  discover  and  understand  the  implications  and 
value  of  evaluation.  At  the  same  time,  it  shows  what  can  and  cannot  be 
expected  from  an evaluation  at a particular  time  and  in a specific  context. 
It  provides  the  elements  required  for  steering  an  evaluation  from  A  to  Z: 
defining  responsibilities,  specifying  the  scope  and  objectives  of  the 
evaluation,  outlining  the  method,  schedule  and  budget;  assessing  the 
quality  of  the  work  and,  finally,  supervising  the  use  of  evaluation 
conclusions.  It  is  applicable,  generally,  to  evaluations  performed  before, 
during and after a programme, that  is to say, for the prospective analysis of 
needs,  strategies  and  expected  effects,  and  for  the  retrospective  analysis 
of  impacts  observed  in the  field. The  modes  of  organisation  proposed  are 
general  in  scope  and  their  aim  is  not  limited  to  current  organisational 
schemas  which  are  bound  to  evolve  during  the  new  generation  of  post-
1999  programmes. 

This  first  Volume,  which  is  to  serve  as  a  general  introduction  to  the 
evaluation of socio-economic  development  programmes and as a synthesis 
of  the  entire  Collection,  is  intended  for  a  less  specialised  public  than  the 
following  volumes  which  are  more  technical  and  intended  for  more 
informed  readers. 

This  Volume  has  been  set  out  so  that  readers  may  have  easy  access  to 
information  corresponding  to  their  own  specific  interests. Box  1 and  Box 2 
propose  several approaches  which  can  be used to  go  directly  to  the  most 
relevant  sections. 

Evaluation design and management 

Box 1 - For those who wish to know more about evaluation 

by going directly to the essential 

For a rapid overview: 

> Scan the messages in italics heading each chapter, which briefly 

summarise the key points in the book - start on page 27 

For the reader who is relatively unfamiliar with European structural 
interventions: 

> Carefully read Part I, "Evaluation issues within the framework of the 

Structural Funds" 

For the reader who has to commission an evaluation: 

>■ Move directly on to Part II, "Preparing an evaluation and defining its 

content" 

For readers who design, perform or supervise the collection and analysis 
of data: 

>· Read, in particular, Part III, "Performing an evaluation" 

For getting the most out of your evaluation: 

>· Advice and recommendations are presented in Parts II and IV 

For ideas on perfecting evaluation practice: 

>■ See the suggestions and recommendations made in the conclusion 

to the Volume, in Part V 

Presentation of Volume 1 

Box 2 - For those who have to design and perform an evaluation 

Ex-ante evaluation: 

> Consult annex 1 - The evaluation process - column "ex ante" 

> Follow the symbols -^ heading certain sections or paragraphs 

Intermediate evaluation: 

>■ Consult annex 1 - The evaluation process - column "mid-term" 

> Follow the symbols M heading certain sections or paragraphs 

Ex-post evaluation: 

> Consult annex 1 - The evaluation process - column "ex post" 

> Follow the symbols  ► heading certain  sections  or paragraphs 

I 
Evaluation  issues within the  framework 
of the Structural  Funds 

The value added of evaluation 

The evaluation of public intervention consists of judging its value in relation 
to explicit criteria, and on the basis of information that has been specially 
gathered and analysed. Beyond this general, and necessarily, abstract 
definition, the growth of evaluation practice within the countries of the 
European Union has taken a number of forms: managerial evaluation aimed 
at improving management, democratic evaluation used for accounting to 
citizens and pluralistic evaluation that tries to bring about agreement 
between the partners involved in public action. These three concepts are to 
be found within the experience of evaluating the European Union's policies, 
with one of them predominating according to whether the evaluation 
concerned is ex ante, ex post or intermediate. 

■ A recent approach, specific to the public sector 

The purpose of evaluation is to check the raison d'être of a public 
intervention, to confirm both reproducible success stories and failures not 
to be repeated, and to report back to citizens. Whereas private sector 
organisations have little need to justify or account for their actions, since 
they are judged by the market, public sector organisations are obliged to 
evaluate. 

Origins of evaluation 

Evaluation is related to the analysis of public actions and, more generally, 
to the social sciences. It systematically uses the methods of economics and 
sociology, but also those of social psychology, economic geography, 
anthropology, etc. As a tool for enhancing knowledge it developed along 
with the regular collection of socio-demographic and economic data (e.g. 
public records and census). By contrast, the evaluation of programmes as 
a decision-making aid is more recent. It appeared in the 1950s, when in the 
USA the effectiveness and neutrality of the administration were being called 
into question. At the time, this was part of the effort to rationalise resource 
planning and allocation. The PPBS (Planning, Programming, Budgeting 
System) was born and spread to most European countries (see Rist, 1990, 
for an historical analysis of evaluation). 

In the USA, evaluations started to be performed on a wide scale in the 
1960s. They focused primarily on innovations in the fields of education, 

Evaluation design and management 

social  services, public  hygiene and crime  prevention. For the  U.S.  scientific 
community  many  social  programmes  became  real  laboratories  which  were 
expected  to  produce  knowledge  and  to  help  improve  the  actions  of  public 
authorities. 
to 
become  aware  of  the  restrictive  and  artificial  nature  of  the  "social 
experiment"  type  of  approach  and  of the  PPBS norm. 

In  the  1970s,  however,  administrative  officials  started 

From  then  on  evaluation  was  seen  more  as  a way  of  managing  necessary 
budgetary  restrictions. This was the case in the UK, where the  development 
of  evaluation  was  linked  to  the  concern  to  limit  public  spending  following 
the  public  service  reforms  in the  1980s. 

Evaluation  became  part  of  a  new,  more  decentralised  and  responsible 
concept  of  public  management.  It  is  the  reflection  of  several  concerns 
which  can  be  defined  in terms  of  three  main  objectives.  First,  there  is  the 
need  to  estimate  as  objectively  as  possible  the  effects  of  policies  or 
programmes  on society, and to understand the logic that  governs  how  they 
function  (cognitive aim). Secondly, there  is the desire to  help  policy-makers 
and  stakeholders  to judge  the value of  these  programmes  (normative  aim). 
Thirdly, there is the need to contribute  in a practical way to the  improvement 
of  programmes  by  informing  different  protagonists  on  the  meaning, 
conditions  and  consequences  of  their  actions  and  decisions  (instrumental 
aim). 

Evaluation  brings  together  existing  information  in a coherent  whole.  Where 
relevant,  it  produces  new  information  for  enhancing  programme  managers' 
knowledge  about  the  social  and  economic  context  in  which  their  public 
interventions  are  situated.  Moreover,  evaluation  is  an  instrument  at  the 
service  of  strategic  and  operational  decision-making. 
Its  practical 
usefulness can be defined as the linking up of objectives, resources,  results 
and  effects  of  interventions,  with  a  view  to  improving  the  efficiency  and 
effectiveness  of  a  programme.  Finally,  evaluation  introduces  the  notion  of 
value and, a fortiori,  of  value judgement,  into  administrative  practices. This 
dimension,  which  can  be  qualified  as  "political",  is what  makes  evaluation 
so  original.  It  implies  the  emergence  of  a  new  profession  in  its  own  right, 
distinct  from  others  such  as research. 

A  necessity  for  the  public  sector 

The  need  for  evaluation  in the  public  sector  can  be  understood,  especially 
at the ex post phase when efforts are made to measure the impacts of  public 

Evaluation issues within the framework of the Structural Funds 

activity,  by  comparing  it  to  the  role  of  the  market  for  a  private  enterprise 
selling  goods  and  services.  By deciding  either  to  buy these  products  or to 
go  to  a  competitor,  each  customer  calculates  all  the  benefits,  costs  and 
disadvantages  to  be  derived  from  the  purchase  in  the  short  or  long  term. 
Customers'  buying  behaviour  therefore  indicates  what  the  firm  should  or 
should  not  do.  Finally,  the  firm  does  not  need  to  justify  its  existence.  It 
survives  if it is financially  viable, that  is to say, if it is useful to  its customers. 

By analogy with the private firm, a public organisation also produces  goods 
or services.  It offers  (and sometimes  imposes) them, not  to  customers  but 
to  addressees,  users, etc.  If the  people  participating  in a public  action  are 
satisfied  with  it,  that  is  generally  not  enough  to  say  that  the  action  is 
successful. On the  contrary,  public  authorities  often  try  to  produce  distant 
and  global  effects,  such  as  sustainable  socio-economic  development, 
which are not synonymous with addressee satisfaction. Evaluation makes it 
possible to  know and to judge all the effects of a given  public action: direct 
and  indirect,  positive  and  negative.  The  sanction  of  the  electorate  is 
certainly an important  signal for any government, but its main disadvantage 
is  the  non-discriminatory  nature  of  the  judgement  thus  expressed.  Only 
direct democracy  as practised  in Switzerland  allows, to a certain extent, for 
the  expression  of  a judgement  on  each  intervention  (evaluation  is  no  less 
advanced  in  Switzerland!).  Because  the  democracies  represented  in  the 
European  Union are essentially  indirect; evaluation  plays a crucial role, both 
as  a  signal  to  show  what  should  and  should  not  be  done,  and  as  an 
instrument  capable of justifying the existence  of public action in the eyes of 
citizens. 

The  development  of  evaluation  in  Europe 

Different  conceptions  in the  European  countries 

The  development  of  evaluation  in Europe  started  in the  northern  countries 
(UK,  Sweden,  Denmark,  and  the  Netherlands),  where  it  has  now  become 
standard  practice  in  the  administrations.  All  the  European  countries  are 
concerned, and have adapted  evaluation to suit their own cultural  attitudes 
to the role of government and the relationships between their  legislative and 
executive  powers. 

The  major  policy  decisions  taken  within  European  countries  can  be 
classified  in three  main  categories.  To the two  traditional  lines that  can  be 

Evaluation design and management 

qualified as "managerial" and "democratic", has been added a third that 
can be referred to as "pluralistic". These three directions or conceptions are 
different without being mutually exclusive. Box 3 describes them briefly by 
showing the principles and recommendations characterising each one 
(Duran, 1992). 

Box 3­

Three conceptions of < 

¡valuation 

Managerial 

Democratic 

Pluralistic 

Optimisation of 
public resources 

Involve citizens and 
report to them 

Collective problem 
solving 

Global 
project 

Valuing 

Independent 
« Evaluators » 
(experts) 

Standards Objectiveness 

Judgement by 
elected 
representatives 

Publicising 
the approach 

Judgement by 
those involved 
in the action 

Impartiality 

The general idea of managerial evaluation is to ensure that limited public 
revenue is efficiently managed. The underlying question can be formulated 
as follows: how can trade­offs between the different sectoral policies be 
justified? The approach preferred here is that of "optimisation" of existing 
budgets, within administrations. 

The political dimension of the evaluation act is disregarded. The 
"evaluator", in official terminology, is above all a methodologist whose 
legitimacy is grounded in his or her scientific knowledge and approach. The 
deontology is based on two principles: the independence of the evaluator 
and the objectiveness of the method. The reference to scientific rigour 
constitutes, in a sense, the guarantee of the evaluation's independence vis­
à­vis policy. 

Example: The British " Value for Money " approach 

During the 1980s, many functions of the British administration were privatised or 
delegated to semi­independent agencies. At the same time, demands for the 
efficiency of these services grew, both at the central administrative level and among 
citizens. This tendency seemed even more obvious during this period within the 
framework of ε managerial culture in the administration. This culture was based on 
"Value For Money" evaluations and on the publication, in the "Green Book" by the 
Treasury, of technical guidelines for this type of evaluation. 

Evaluation issues  within the framework of the Structural Funds 

The second, "democratic",  conception shows the growing awareness of the 
crisis  in  traditional  systems  of  representation  (turning  away  from  political 
parties,  trade  unions,  etc.).  In  this  conception,  public  authorities  must  be 
fully  accountable  for  their  actions, to  the  citizens. The emphasis  is thus  on 
the  need for  evaluation to  strengthen  counter-powers  to the  administration 
and,  more generally, the executive, accused of  monopolising  expertise. 

Evaluation, defined as a "judgement of the value of public  policies", must feed 
into  the  democratic  debate.  It  is  recommended  that  these  evaluations  be 
managed  by  evaluation  authorities  consisting  of  democratically  elected 
persons. Evaluation professionals are at the service of such committees but for 
technical issues only. The process must be publicised to ensure its democratic 
nature.  From  a  general  point  of  view,  this  model  seems  to  be  developing, 
particularly  in a context  of  modernisation  of  legislative and political  practices, 
where  keen  interest  is currently  shown  in  the  notion  of  "guideline  laws"  and 
"sunset  legislation". These  are  laws  which  need to  be confirmed,  by vote, in 
view of the results obtained after the first few years of application. 

Example: Consensus  conferences  in  Denmark 

Among various types of democracy, a Danish practice has become a reference. "Public 
Consensus Conferences" have been part of the political scene in Denmark for over ten 
years.  The  Danish  Board  of  Technology  calls  on  ordinary  citizens  to  express 
themselves  and  to  debate  with  experts  and  representatives  of  various  interests,  on 
particularly  controversial subjects  (for example, genetic  manipulation  of animals, tests 
likely to be carried out during recruitment, or the signing of insurance contracts). These 
conferences, spread over three weekends, enable citizens to express themselves  and 
to  undertake  a  dialogue  with  experts.  The  reports  produced  in  this  way  aim  at 
informing public opinion and facilitating democratic  debate. Without  replacing  political 
decision-making,  they  lay  the  foundations  of  debate  between  the  executive  and  the 
legislative  powers and become  part  of the decision-making  process. 

The  last  type  of  evaluation  is  called  "pluralistic  evaluation",  "participative 
evaluation"  or,  in  the  American  literature,  "stakeholder  evaluation".  This 
approach  tries  to  find  a  compromise  between  the  technical  requirements 
justifying  the  first  conception  and  the  political  requirements  inspiring  the 
second. 

In this conception, evaluation is part of a collective problem-solving  process. 
On the  basis of  reliable  information  acceptable to  all, value judgements  are 
formulated  by  seeking  consensus  within  a  pluralistic  evaluation  authority 

Evaluation design and  management 

composed  of  political  and  administrative  officials,  as  well  as 
spokesperson  for  the  publics  concerned  . 

the 

Example: Pluralistic  authorities for  evaluation 
In France, ad hoc evaluation authorities are often formed to conduct evaluation work. 
Thus,  within  the  framework  of  national  evaluation  of  the  prevention  of  natural 
hazards,  the  evaluation  authority  was  composed  of  actors  of  diverse  origins: 
ministries,  local  authorities,  public  agencies,  community  groups,  etc.  Within  the 
framework of  policies co-financed  by the state and the regions, certain evaluations 
are steered jointly  by political  and administrative  officials  at  both levels, along with 
representatives of the publics concerned. 

the  1950s.  So-called 

A  comparison  can  be  drawn  between  the  different  conceptions  stated 
above,  and  the  "four  generations"  identified  by  publications  in  the  U.S. 
These  refer  to  the  different  stages  of  development  of  practices  in  this 
country  since 
first  generation  evaluations 
concentrated  on  the  description  of  the  effects  on  society  of  public 
interventions. The second  generation focused  on an analysis  of  relations  of 
causality  and  on  the  attribution  of  effects  to  interventions.  The  third 
generation  started  systematically  to  verify  the  achievement  of  these  aims, 
which  makes  it  very  similar  to  managerial  evaluation  described  above. 
Finally,  fourth  generation  evaluations  have developed  in a  pluralistic  mode, 
that  is to  say,  by  associating  the  different  protagonists  with  the  evaluative 
approach  (Guba,  Lincoln  1989). 

Evaluation  procedures  at  the  Commission 

Evaluation  at  the  European  Commission  dates  back  to  the  1980s,  in  the 
fields  of  research  and  aid  to  developing  countries.  From  the  early  1980s 
European  science  and  technology  policy  was  based  on  a  system  of  pluri-
annual  programming,  and  evaluation  was  rapidly 
the 
management  of  these  programmes.  A  specific  programme  was,  moreover, 
set  up  (Monitor,  1989-1992)  to  improve  methods  and  indicators  used  in 
evaluating  research.  Evaluation  in  the  field  of  aid  to  developing  countries 
also  increased  significantly  during  the  1980s.  The  most  commonly  used 
approach  was  that  of  Project  Cycle  Management,  partly  discussed  in  this 
book  (see  page  148).  Evaluation  practice  later  extended  to  other  fields, 
particularly  that  of  the  Structural  Funds,  with  the  establishment  in  1988  of 
specific  regulations  on the  monitoring  and  evaluation  of  programmes. 

included 

in 

Evaluation issues within the framework of the Structural Funds 

In  1991, a  little  over  half  the  Commission's  Directorates  had  an  evaluation 
unit. This  unequal  development  can  be explained  by the  birth  of  European 
policies. Some policies were a priori  legitimate, that  is to say, recognised by 
the  Treaties  as  the  exclusive  competence  of  European  institutions  (e.g. 
foreign trade, competition, and agricultural  market  policies). These  policies 
were  scarcely  evaluated  until  very  recently.  The  few  existing  evaluations 
have all been carried out  internally. Thus, in the case of  competition  policy, 
for example, evaluation was for a long time carried out by European officials 
who relied on consultation with the industrialists concerned and on informal 
meetings  with  national  administrations.  The  Commission,  in  a  strong 
position  because of  the  legal texts  defining  its competence  in these areas, 
is able to  conduct  its own  evaluations. 

interventions  has  been 
In  other  areas,  the  legitimacy  of  European 
constructed  progressively  over the years. These concern  policies that  have 
traditionally  been  the  province  of  national  authorities  (e.g.  scientific 
research,  consumer  rights,  transport  networks,  etc.).  Despite  the  growing 
recognition  of  European  initiatives,  these  policies  still  have to  be justified. 
Interventions  are  recognised  as  being  necessary,  provided  that  the 
Commission can prove "the advantages  gained from implementation  of the 
programme, throughout  the Community".  In order to do so, it  commissions 
numerous  evaluations.  Thus,  the  evaluation  of  research  programmes  is 
aimed  at  legitimising  European  policy,  particularly  vis-à-vis  the  Member 
States.  Use  is  made  of  expert  committees  that  are  known  to  be 
independent  and are recruited from the  different  countries. 

In the field of political cohesion, evaluation was born in 1988 and has since 
developed  steadily.  Evaluations  are  sometimes  similar  to  the  pluralistic 
conception  described  above,  in so far  as some  of  them  involve  European, 
national  and  regional  officials,  as  well  as  socio-economic  partners.  This 
without  being  too  removed  from  the  other  conceptions  of  evaluation. 
Moreover, on its own initiative (unilateral order), the Commission carries out 
thematic  evaluations  or  evaluations  of  transversal  programmes  across 
several  countries.  Part  I of  this  Volume  features  a  detailed  analysis  of  the 
evaluation  of  Structural  Funds. 

Finally, in 1995 the Commission  launched a programme aimed at  improving 
its financial management: "Sound and  Efficient  Management -  SEM 2000". 
This  programme  generalises  evaluation, to  include  all  policies  which  have 
an effect  on  its  budget.  It  institutionalises  the  use of the  managerial  model 
in  improving  management  at  the 
of  evaluation  as  a  key  element 

Evaluation design and management 

Commission. The Directorates are requested to establish on-going 
evaluation programmes and to create evaluation units. Support 
mechanisms have also been set up (network of evaluation services, 
manuals, training, data base of past evaluations and pilot projects). Any 
renewal of a pluri-annual programme is now subject to evaluation of its 
previous results. 

■ Three good reasons to evaluate 

Verifying that public action replies to uncovered or insufficiently 
satisfied needs 

One of the aims of evaluation is to verify the raison d'être of public action. A 
public intervention can be justified only if it is intended to satisfy needs or 
solve socio-economic problems. Evaluation serves to clarify these needs or 
to verify the real existence of these problems, especially if it is ex ante. When 
it is ex post it serves to examine whether the needs or problems still exist. 

Furthermore, it is essential to rule on the appropriateness of the public 
intervention as such. A public programme can be justified only if the needs 
cannot be met by private initiative or by any other existing public initiatives. 
Thus, the inadequate nature of private initiative constitutes the first motive 
for public intervention, by virtue of the principle of subsidiarity, in the 
original sense of the term, meaning that public authorities act only when the 
private sector fails to do so. 

More particularly, for European institutions this concept of subsidiarity 
takes on additional meaning in so far as the legitimacy of the European 
Union must also be justified in relation to national sovereignty. The 
Maastricht Treaty reaffirms the importance of respecting subsidiarity in 
relation to the national level: "the Community acts when an objective can 
be met more satisfactorily at the European level than at the level of Member 
States taken in isolation". 

Improving interventions 

By observing the reality of results and impacts, the evaluation exercise 
provides an opportunity for feedback arising from actions underway. 
Certain lessons drawn from the experience lead to proposed improvements 
in the implementation of evaluation, without necessarily changing the basic 

Evaluation issues within  the framework of the Structural Funds 

assumptions.  For example, the  budget  for the  least  successful  actions  will 
be  reduced. This  is  a case  of  direct  feedback  or  simple  learning  (called  a 
"single loop learning"). Evaluation may also propose strategic  reorientation, 
for  example  by  re-organising  the  implementation  of  actions  so  that  they 
succeed  better.  In  that  case  evaluation  produces  indirect  feedback  or 
"double  loop"  learning,  in  so  far  as  it  challenges  the  logic  of  the 
intervention. 

is  commonly  called  the 

This  evaluation  aim  is  what 
instrumental 
contribution, that  is to  say, concrete  assistance  in terms of solutions to  the 
problems  encountered  or  adaptation  to  an  unstable  environment.  Its 
function  is  qualified  as  formative,  in  the  sense  that  it  has  the  merit  of 
providing  all the elements  required  by the operational  officials to  be able to 
improve  the  programme:  more  rational  distribution  of  human  and  financial 
resources, improved management  of services responsible for  implementing 
them,  etc. The goal  is to  analyse the factors  of  success  or failure,  in order 
to help managers adjust their  modalities of intervention  and  implementation 
of the  programme. 

Accountability 

The third  aim of  evaluation  consists  of  reporting  to  political  authorities  and 
citizens  on  results  obtained  and  on  the  sound  use of  allocated  resources. 
This aim  is called  responsibility,  liability  or  accountability. 

The duties of the public  authorities  are the reverse side of their  right to levy 
taxes  and their  power  to  compel.  Evaluation  then  takes  on  its full  political 
dimension  in the functioning  of  democracy. The Commission, although  not 
politically  accountable  to  the  European  electorate,  has  progressively 
asserted  a demand  for  transparency,  vis-à-vis taxpayers,  in the  use of  the 
funds  it manages. 

Evaluation  must  help  to  explain  in  simple  terms  where  public  money  was 
spent,  what  effects  it  produced  and  how  the  spending  was  justified.  The 
addressees  of this  type  of  evaluation  are obviously  political  officials  and in 
fine,  most  often  via the  media, the  citizens.  An  essential  notion  within  the 
framework  of  European evaluations  is that  of  "Community  value added"; in 
other  words,  is  the  support  given  by  the  intervention  of  European 
institutions justified  in terms of the  progress  made? 

The specificity of programmes  and their  evaluations 

Socio-economic  development  programmes must be evaluated by means of 
methods  adapted 
these 
characteristics  is  that  they  are  co-financed  and  involve  joint  decision­
making.  This 
implies  that  evaluation  must  be  carried  out  with  the 
participation of several partners and must  meet their different  expectations. 

their  specific  characteristics.  One  of 

to 

These  programmes  are  also  characterised  by  multiple  objectives  and 
domains of intervention. The result is greater complexity  of evaluation work. 

Finally,  unlike  those  contexts  in  which  evaluation  is  considered  to  be  a 
purely  retrospective  exercise,  the  system  chosen  for  European  cohesion 
policy  consists  of  programmes  at  three  successive  stages:  ex  ante, 
intermediate  and ex  post. 

Evaluation of a set of combined  objectives 

From the  outset,  the  purpose  of  European  economic  and  social  cohesion 
policy  was  to  reduce  economic  and  social  disparities  between  regions  in 
Europe  and  between  their  inhabitants.  In  the  early  1990s,  differences  in 
levels of  development  and  standards  of  living  throughout  Europe  were  still 
marked. The per capita Gross  Domestic  Product of the ten poorest  regions 
in  the  European  Union  was  still  3.5  times  lower  than  that  of  the  richest 
regions.  The  unemployment  rate  of  the  ten  regions  with  the  lowest 
employment  levels was as much  as seven times  higher than that of the ten 
regions with the  highest  employment  levels. 

The reduction  of these disparities, which  cannot  be achieved solely  by free 
market  forces,  is a challenge for  public  interventions  in the very  long term. 
The  interventions  of  the  European  Union  reinforce  those  of  each  Member 
State  by helping to  enhance  European  solidarity. 

The objective of economic and social cohesion was introduced in 1986 in the 
Single European Act, triggering the  1988 reform which included a substantial 
increase  in  resources  allocated  to  its  financial  instrument,  the  Structural 
Funds:  ERDF,  ESF,  EAGGF  and,  more  recently,  FIFO  The  European  Union 
Treaty,  ratified  in  1993,  reinforced  this  orientation  placing  economic  and 
social cohesion among the Union's  main objectives with, as a backdrop, the 
objective  of  "economic  convergence", so  essential  for the  achievement  of a 
Single Market  but also of the economic  and monetary Union. 

Evaluation design and management 

The  long-term  global  objective  of  cohesion  in  the  European  Union  can  be 
broken  down  into  several  general  objectives  such  as  solidarity  and  mutual 
support.  These,  in  turn,  are  expressed 
in  concrete  and  measurable 
economic terms. Economic  and social cohesion  is officially  defined in Article 
130a of  the  European  Union Treaty  in terms  of  "harmonious  development", 
with  a  specific  geographical  dimension:  the  lesser-developed  regions, 
including  rural  areas,  should  be  helped  to  catch  up  with  the  others. 
Moreover,  this  aim  includes  support  to  the  most  underprivileged  social 
groups,  particularly  the  unemployed  and  people whose jobs are threatened. 

Acting  together,  the  Structural  Funds  have,  until  1999,  been  concentrated 
on  seven  objectives.  Objectives  1,  2,  5b  and  6,  all  specifically  regional, 
concern:  the  structural  adjustment  of 
lagging  regions  (Objective  1); 
economic  conversion  of  declining  industrial  areas  (Objective  2);  economic 
diversification  of  vulnerable  rural  areas  (Objective  5b)  and  structural 
adjustment  of  regions  with  very  low  population  density  (Objective  6). 
Objectives  3, 4 and 5a, covering the entire Community, aim at fighting  long-
term  unemployment,  the  integration  of  young  people  into  the  professional 
world  and  people  threatened  with  exclusion 
the  job  market 
(Objective 3), the adaptation  of workers to changes  in industry  and  systems 
of  production  (Objective  4),  and  the  adaptation  of  agricultural  structures 
and fishing  (Objective  5a). 

from 

In order  to  enhance  the  visibility  and  effectiveness  of  the  Structural  Funds, 
the  Commission  has  proposed  (in  Agenda  2000)  to  reduce  these  seven 
objectives  to  three:  two  regional  objectives  (lagging  regions  and  regions 
undergoing  economic  and  social  reconversion)  and  a  third,  horizontal 
objective  devoted  to  human  resources. 

The  global  objective  of  cohesion  and  reduced  disparities  corresponds  to 
several  priorities  defined  by  the  Member  States  and  the  Council.  These 
priorities  must  be respected  within the framework  of the objectives  of  each 
of  the  programmes  and,  in  fine,  in  the  selection  criteria  of  the  financed 
projects.  These  priorities  correspond  to  the  four  transversal  challenges 
which are: protection of the environment,  competitiveness, job creation  and 
equal  opportunities  between  the  sexes.  Volume  5  of  this  Collection  is 
devoted  to taking  these  priorities  into  account  within  an evaluation. 

Consequently,  the  evaluation  of  programmes  must  simultaneously  analyse 
several  long-term  objectives  and  an  even  larger  number  of  medium-term 
objectives,  without  forgetting  to  supply  information  on  the  merits  of  the 

Evaluation issues within  the framework of the Structural Funds 

programme  in relation to the above-mentioned  transversal  policy  priorities. 
Administrative  officials  and,  to  an  even  greater  extent,  evaluation  teams, 
have to deal with considerable technical  difficulties  because of these  inter­
linking  objectives,  resulting  in  multiple  objectives  that  are  difficult  to 
observe and to  analyse  in detail. 

Evaluation  of  interventions  in  multiple  domains 

With the strengthening  of the  process of  European  integration, the areas in 
which  the  Member  States  have decided  to  pursue  a common  policy  have 
increased.  Some  specifically  target  enhanced  cohesion  while  others  are 
supposed  to contribute to  it, with their own  objectives. 

Interventions can be put into various categories: interventions in agriculture 
and  fishing;  measures  aimed  at  increasing  competitiveness  (research  and 
development, industrial and trade policies), policies on networks  (transport, 
telecommunications,  energy);  measures  aimed  at  improving  the  quality  of 
life (social policy, education and training, protection and improvement of the 
environment); etc. 

These  are  covered  by  the  four  Structural  Funds  (ERDF,  ESF,  EAGGF-
Orientation, and FIFG): 

•  The European Regional Development  Fund (ERDF), created in 1975, aims 
at  enhancing  the  economic  potential  of  the  assisted  regions,  supporting 
structural  adjustments  and  promoting 
the  growth  of  sustainable 
employment.  It  supports  productive  investment,  infrastructure  projects 
and actions to develop the economic  potential of regions by  co-financing 
operational  programmes,  assisting  large  projects,  and  providing  global 
subsidies, technical  assistance  and preliminary  studies. 

reformed 

in  1988, 

•  The European Social  Fund (ESF), created  in 1958 with the Treaty  of Rome, 
long-term 
but 
unemployment  and  improving  the  capacity  of  young  people  to  find 
employment. Since  1993, incentives for adapting to industrial change have 
been added. The Fund helps to finance occupational training and measures 
for supporting  employment  and improvements  in the educational system. 

is  primarily  aimed  at 

fighting 

•  The  "guidance"  part  of  the  European  Agricultural  Guidance  and 
Guarantee  Fund  (EAGGF-G),  created 
in  1962  with  the  Common 
Agricultural  Policy,  is  aimed  at  encouraging  structural  adjustment  in 
agriculture through  measures to  modernise  production  and  development 
in rural areas. 

Evaluation design and management 

•  Finally, the  Financial  Instrument  for  Fisheries  Guidance  (FIFG), created  in 
1994  to  replace  various  separate  financial  instruments  in  force  since 
1976, supports  the structural  adjustment  in this  sector. 

The  Cohesion  Funds  and  the  European  Investment  Bank  also  contribute 
towards  the  cohesion  policy  through  subsidies  and  loans  for  projects. The 
Cohesion  Fund, instituted  by  the  Maastricht  Treaty,  is  intended  to  facilitate 
the  preparation  of  Greece,  Portugal,  Ireland  and  Spain  for  entry  into  the 
economic  and monetary  Union. This Fund can be considered as  supporting 
the  weakest  Member  States,  in  exchange  for  their  undertaking  to  satisfy 
convergence  objectives  and to  control  their  budget  deficits. 

i.e. 

the 

This  Collection,  and  particularly 
this  Volume,  uses  a  typology  of 
interventions  based  on  three  main  domains  of  intervention  covered  by  the 
policies  implemented  within  the  framework  of  the  Structural  and  Cohesion 
Funds, 
(transport, 
communication, and  public  facilities  in general), the development  of  human 
resources  (through training  actions and consultancy  services to  businesses 
and  individuals)  and  support  for  the  productive  environment,  including 
assistance  to  economic  actors  for  expanding  their  capacity  to  produce, 
export,  innovate,  etc.  (see page  137 and  onwards). 

improvement  of  basic 

infrastructure 

This  wide  range  of  heterogeneous  domains  of  intervention  is  frequent  in 
socio-economic  programmes.  Yet,  for  programmes  financed  by  the 
Structural  Funds,  this  diversity  reaches  levels  rarely  equalled.  This  makes 
evaluation all the more  difficult,  which  is why two volumes  of the  Collection 
provide  possible  methodological  solutions  from  the  point  of  view  of 
indicators  (Volume 2) and  evaluation  methods  (Volume 4). 

Several  levels  and  several  stages  of  evaluation 

In  the  public  domain,  evaluation  is  a  wide-ranging  operation  which  may 
concern  activities,  services,  organisations  and  practices.  It  is performed  at 
three  main  levels: the project,  programme  or  policy  level. In North America, 
Northern  Europe  and  at  the  European  Commission,  the  notion  of 
programme  evaluation  dominates.  In  the  Southern  European  countries, 
evaluation  literature  more often  relates to  public  policy. 

The  term  "programme"  signifies  (1)  a  coordinated  set  of  different  types  of 
action  (e.g. training  courses;  personalised  advice;  subsidies  to  employers, 
etc.),  of  which  (2)  the  operational  objectives  are  defined  precisely  (e.g.  a 

Evaluation issues within the framework of the Structural Funds 

programme  for  the  conversion  of  the  long-term  unemployed)  without 
implementation  being defined  entirely  in advance  and, finally  (3) which  has 
a limited time-scale  and  budget. 

The notion of "policy" refers to a set of programmes and/or measures which 
have  the  same  general  objective  but  not  necessarily  the  same  specific 
objectives,  schedule  and  modes  of  management  (e.g.  an  employment 
policy will refer to a set of measures which have accumulated over the years 
but  are not always  coordinated). 

The definition of the preceding two terms relates to that of a "project"  which 
is  a  non-divisible  action  with  a  mode  of  management,  a  schedule  and  a 
budget  that  are all defined  precisely from the  outset. 

These  three  levels  of  evaluation  are  relevant  for  European  cohesion  policy 
and  it  is  worth  while  linking  them  (see  page  51).  However,  it  is  at  the 
programme  evaluation  level  that  the  most  activity  is  concentrated.  The 
MEANS Collection  primarily  concerns work  at this level. 

In  the  evaluation  literature,  contradictory  opinions  are  expressed  on  the 
timing  of  evaluation.  Some  authors  consider  that  evaluation  is  only  a 
retrospective  exercise  while  others  also  attribute  a  prospective  dimension 
to  it.  In  the  context  of  European  cohesion  policy,  evaluation  is  clearly 
defined  as  an activity  which  takes  place  before,  during  and  after  a  public 
action.  The  three  steps  or  stages  in  the  evaluation  of  a  programme  are 
related  to  its  life  cycle  and  consist  of  ex  ante  evaluation,  mid-term 
evaluation  and  ex  post  evaluation.  Given  the  uninterrupted  sequence  of 
programme  cycles,  there  seems  to  be  an  overlap,  in  time,  between 
evaluations  performed  at  different  stages  and  for  different  cycles.  This 
necessitates  a  careful  articulation  so  that  the  exercises  reinforce  one 
another  without  being  redundant  (see page 51). 

Partnership  and  evaluation 

Partnership 

Programmes  financed  by  the  Structural  Funds  are  systematically  co-
financed by the Member States at the national and infra-national levels (the 
Regions or other  local authorities).  European  financing  is accompanied  by 
a demand for additionality. The principle  of additionally  is that  Community 
support  for  economic  and social development  must  not  be substituted  for 

Evaluation design and management 

efforts by national governments; financing by the beneficiary State must 
remain at least equal to what it was before the Structural Fund 
contribution. In concrete terms, this means that any Community spending 
will be in addition to national spending. The application of this principle is 
verified nationally within the framework of financial control and not during 
evaluation as such. This verification still involves methodological problems 
(incomplete data, modes of calculation, etc.). 

The combination of European and national financing has a decisive 
consequence for evaluation, i.e. the existence of joint decision­making and 
partnerships. The design and implementation of co­financed programmes 
requires the establishment of common objectives, a contractual framework 
of action and mutual trust. In the case of the Structural Funds, joint 
management is often taken to mean vertical partnership, that is to say, the 
fact that close collaboration must be established between the Commission 
and all competent authorities appointed by each Member State at the 
national, regional and local levels. There also exist several forms of 
horizontal partnership: geographical, sectoral, administrative, etc. Since 
1994, partnership has spread to all competent organisations, including (as 
a rule) economic and social partners appointed by the relevant Member 
State. 

Box 4 ­ Partners in the European programmes 

/ \ 

/ \ 
■Representatives^ 

/ o f t h e E . U. \ 

ι ' N ν 
ι / \ \ — — *" 
; , national x ι ^ 
ι ι representatives \ < 
'ι „ ­ ­ » ' .'' 
ι / s " * ·. \ ι .' 

• * 

I, \ \ 1 

I
1 ' ' regional \ ' ' 
' ' representatives \ ¡(' 

. J IJ . . . _ ­ . .. }. 11. . 

« Vertical » 
partners 

,­r Decision 

• 

• 

>. Monitoring 

"*■ Evaluation 

Monitoring 
Committee 

A 

Other actors 

« Horizontal >· 
partners 

Evaluation issues within  the framework of the Structural Funds 

Evaluation  in  partnership 

represents  an  opportunity 

In  terms  of  evaluation,  partnership  and  more  particularly  "vertical" 
partnership  between  the  different  administrative  levels  will  generate 
opportunities  that  need  to  be  exploited  and  constraints  that  must  be 
the 
overcome.  Vertical  partnership 
development  of  pluralistic  evaluations,  known  as  "fourth  generation" 
evaluations  (see  page  19),  but  also  introduce  a  whole  series  of  new 
constraints.  Partnership  policies  produce  situations  in  which  evaluations 
are  difficult  to  perform  and  in  which  their  scope  is  reduced.  Common 
objectives  often  lack  precision  and  coherence  due to the  negotiations  and 
compromises  which  governed 
joint 
their 
responsibility  complicates  resource  management.  As  a result,  it  is  difficult 
to  define  the  object  of  evaluation  and  partners  tend  to  fear  that  the 
negotiated  modus  operandi  might  be called  into question. 

formulation.  Moreover, 

for 

Because  evaluation  in  partnership  has  to  take  into  account  the  partners' 
diverse  expectations  and  seek  convergence  between  differing  points  of 
view,  it  is confronted  with  the  fact  that  the  stakes  are  not  the  same for  all 
the  partners. The existence  of  specific  stakes  is often  overlooked, with  the 
idea that  what  is good for the group  is necessarily  good for  everyone. This 
implicit  assumption  results  in  the  absence  of  differentiation  between  the 
questions  asked  and  judgement  criteria.  Finally,  an  evaluation  that  is 
constructed  without  taking  into  account 
initial  differences  between 
partners, supposedly  because of the partnership, is very  likely to become a 
procedural  construct  without  substance. 

From  one  situation  of  partnership  to  the  next,  there  are  profound  de  facto 
differences  in  the  application  of  mandatory  evaluation.  In  many  countries 
and  regions  this  obligation  is reduced  to  a formal  and  relatively  superficial 
exercise.  In  other  cases,  it  has  led  not  only  to  more  effective  use  of  the 
Structural  Funds  but  also  to  a  general  evolution  in  administrative  models. 
Thus,  for  example,  in  several  Member  States  and  regions  which  were 
recently  declared  eligible,  Community  requirements  concerning  evaluation 
framework  of 
the 
have  helped 
programming  and the  implementation  of  regional  policies. 

to  promote  new  practices  within 

Moreover, the application of these regulations varies as much between the 
different  Member States as between regions within the States themselves. 
It  has  been  observed  that  evaluations  are  more  likely  to  be  of  a  high 
quality  when  relations  between the  partners are balanced, in other  words, 

Evaluation design and  management 

when  one  funding  organisation  is  not  too  dominant.  The  richest  regions 
are  not  necessarily  those  that  comply  with  Structural  Fund  regulations 
most  satisfactorily,  probably  because they  do  not  contribute  substantially 
to  the  budgets  of  these  communities.  The  least  privileged  countries  or 
regions  receive  a large amount  of  assistance  but  are  not  always  the  most 
active  as regards  evaluation, either  because they  already find  it difficult  to 
implement  the  available  resources,  or  because  they  see  evaluation  as  a 
threat  due to their  being  indebted to the other  partners. It can  be  assumed 
that  each  partner's  desire to  evaluate depends  on their  assessment  of  the 
gains  as  well  as  the  costs  and  risks  involved  in  the  evaluation  of  their 
programmes. 

In  the  future,  the  European  Commission  will  focus  more  on  the  choice  of 
strategic  priorities,  leaving  the  national,  regional  and  local  authorities  to 
steer  programming  and  the  management  of  programmes.  This  more 
decentralised  and  simplified  management  will  require  in  return  more 
extensive  monitoring  and  evaluation. 

Development  of  practice within the  framework 
of European Structural  Funds 

The  diffusion  of  evaluation  within  the  framework  of  European  cohesion 
policy  has  occurred  within the  last  ten  years. As  it  spreads  throughout  the 
Member  States,  it  encounters  widely  diverse  references  and  ways  of 
thinking  (macro-economic  expertise, regional economics, political analysis, 
strategic  analysis, etc.)  and  is  based  on  concepts  that  may  vary  from  one 
country  to the  next. 

Whatever  the  vocabulary  and  structures  adopted, the  evaluation  systems 
set  up  in  the  different  Member  States  have  helped  to  enhance  the 
transparency  and  effectiveness  of  interventions  financed  by  the  Structural 
Funds  and,  in  many  cases,  even  of  other  public  interventions  in  similar 
domains. 

From a situation of virtual  non-existence  of practice, considerable  progress 
has been  made  in the quality  of  evaluation  reports. A number  of  difficulties 
still  exist,  however,  particularly  in  the  analysis  of  impacts  and  the  use  of 
reports. 

Changes  in  Structural  Fund  regulations 

At the  Commission, a demand  for transparency  vis-à-vis taxpayers, on  the 
sound  use of funds from the  Member  States, has progressively  taken  hold. 
This aspiration is reflected in a simple expression: "striving for  effectiveness 
in European spending". 

The Single  European Act, adopted at the beginning of  1986, introduced  into 
the  EEC  Treaty  a  new  section  V  containing  article  130D.  This  article 
announced  the  reform  of  the  Structural  Funds,  intended  to  improve  their 
effectiveness  with a view to enhancing economic  and social cohesion  in the 
Community.  Similarly,  in February  1992, the  Maastricht  Treaty  included  this 
imperative need for cohesion, and Article  130B stipulates that the Council is 
to define measures required for ensuring that the Funds are effectively used. 

In accordance with these requirements, the regulations relating to Structural 
Funds (those of  1988 for the first  generation  of  Structural  Funds and those 
of  1993 for the  current  generation)  included  specific  articles  on evaluation, 
particularly  Article  6  of  the  framework  regulation  and  Article  26  of  the 
coordination  regulation. 

Evaluation design and management 

Regulations  applicable  between  1988-93 

Article  6,  Paragraph  2,  of  the  1988  regulations  stipulated  that  "in  order  to 
assess  the  effectiveness  of  structural 
impacts  of 
Community  actions  must  be  evaluated  ex  ante  and  ex  post.  For  this 
purpose, app ¡cations for support  by the competent  authorities  must  include 
the  necessary  information  so that  it can  be evaluated  by the  Commission". 

interventions,  the 

These  first  regulations,  adopted  by  the  Council  on  19  December  1988, 
concerned  regional  development  plans  which  had  to  be  submitted  by  1 
January  1989.  For  many  national  and  regional  administrations,  the  need  to 
establish pluri-annual  programming was in itself a kind of revolution. Moreover, 
most  competent  authorities  and  the  Commission  services  did  not  have  the 
required  evaluation  skills,  at  least  as far  as  evaluating  Structural  Funds  was 
concerned.  Consequently,  evaluations  carried  out  during  the  1989-93  period 
were not of a very high quality, despite efforts by many actors, especially at the 
Commission. It was observed, in particular, that the framework  defined by this 
first  version  of the  regulations  failed to  create  adequate  conditions  for  the ex 
ante evaluation  phase to take  place in the normal way. 

Regulations  applicable  between  1994-99 

Acknowledging  this  partial  failure, at  the  instigation  of the  Commission  but 
also  through  the  strong  impetus  given  by  certain  Member  States  (the 
Netherlands  and  the  U.K.,  in  particular),  the  new  July  1993  regulations 
considerably  strengthened  requirements  concerning  evaluation. 

This  new  regulation  (Article  6  of  the  framework  regulation)  specifies  the 
notions  of  prior  assessment,  monitoring  and  ex  post  evaluation.  The 
Articles  detailing  the content  of  programming  documents  submitted  by  the 
Member  States,  introduce  the  notion  of  "specific  quantified  objectives, 
where  possible".  Similarly,  Article  26  of  the  coordination  regulations 
presents  a  precondition  clause  specifying  that  "support  will  be  granted 
when a prior  assessment  has shown the socio-economic  advantages  to  be 
gained  in the  medium-term,  in relation to the  resources  mobilised". 

for  ex  ante  evaluation,  a 
In  practice,  this  very  strong  requirement 
Community  variant  of  the  British  notion  of  Value-For-Money,  was  not 
applied  very  strictly.  By  contrast,  the  introduction  of  mid-term  evaluation 
which  had  not  existed  before  was  particularly  successful,  as  our  survey 
below  shows.  This  strengthening  of  European 
legislation  as  regards 
evaluation  was  clearly  a  strong  incentive  for  spreading  the  idea  that 
evaluation  was  unavoidable  if Community  Funds were to  be obtained. 

Evaluation issues within  the framework of the Structural Funds 

Regulations  of the third generation  of  programmes  (2000-2006) 

With  the  third  generation  of  Structural  Funds  the  assessment  of  their 
effectiveness  will  be  reinforced. Community  actions  will  henceforth  be  the 
object  of  ex  ante  evaluation  by  the  Member  States  themselves,  of 
intermediate  evaluation  by  programme  managers,  and  of  ex  post 
evaluation, at the initiative of the Commission, for assessing their  effects in 
relation  to  the  objectives  (their  particular  objectives  and  cohesion 
objectives)  and for analysing their  impact  on specific  structural  problems. 

The effectiveness  of the Funds  is to  be measured at three levels: the overall 
effect  on  the  objectives  of  Article  130  A  of  the  Treaty  (particularly  the 
strengthening  of social and economic  cohesion), the effect  on the  priorities 
proposed  in the  Plans and  provided  for  in each CSF, and the  effect  on  the 
specific  priorities  selected 
interventions.  Complementary 
the 
evaluations,  where  relevant  of  a thematic  nature,  may  be  launched  with  a 
view  to  identifying  experiences  that  are transferable  from  one  programme 
to  another. 

for 

In  order  to  improve  transparency,  evaluation  reports  are  to  be  made 
available to the  public  more extensively  than  in the  past. 

Recent  developments  in  evaluation  practice1 

Prior  appraisal: mixed  findings 

Evaluation  of  the  second  generation  of  programmes  was  stimulated  by 
the  revised  and  reinforced  regulations  of  1993  and  given  a  rigorous 
framework.  In  particular,  the  prior  appraisal  phase  has  to  specify  the 
objectives  of  regional strategy  and ensure that  Community  priorities  have 
been taken  into  account.  It thus  provides  the  responsible  services  at  the 
Commission  with  elements  for  judging  the  relevance  of  actions  during 
negotiations. 

As  a  rule,  prior  appraisals  are  performed  by  national  administrations.  For 
the  first  time,  they  are  now  also  systematically  being  performed  by  the 
Commission. However,  prior  appraisals  have not always  been as useful as 

'  This section is based on a survey carried out in 1997 within the framework of the 
MEANS  programme.  The  survey  consisted  of  telephone  interviews  in  all  the 
Member States and visits in five countries. 

Evaluation design and management 

expected. Due to a lack of time, analyses have often  remained too  superficial 
to  produce  value  added,  in  respect  of  both  the  formulation  of  general  and 
specific  programme  objectives,  and the  modalities  of their  implementation. 

Efforts  have  nevertheless  been  made  by the  Member  States, particularly  in 
the  quantification  of  objectives.  Programming  Documents  of  the  1994-99 
period  are  unquestionably  of  a  better  quality  than  those  of  the  preceding 
period. They  were  enriched  by  debate  around  prospective  analyses,  which 
helped  to  clarify  objectives,  structure  programme  components  and  grade 
priorities. 

The  spectacular  development  of  mid-term  evaluation 

Significant  progress  has  been  made  in the  area of  the  mid-term  evaluation 
of  the  second  Structural  Funds  programming  cycle.  Because  this  period 
was  able  to  build  on  the  experience  gained  during  the  preceding  period, 
evaluations  were  systematically  performed  in most  cases. 

From  a  quantitative  point  of  view,  the  number  of  reports  produced  has 
increased five or six fold compared  to the  preceding  period, and  evaluation 
practice  has developed  spectacularly. 

Thus,  in  the  fifteen  Member  States,  for  the  period  from  1996  to  the 
beginning  of  1998,  over  380  mid-term  evaluations  were  organised  by 
Monitoring  Committees  for  the  six  Objectives.  Among  them, there were  no 
less  than  250  evaluations  for  Objectives  1 and  2  alone, for  the  single  year 
1997.  By comparison, for the  period  1989-1993, an average of  less than 40 
evaluations  per  year  were  performed  on  all  the  programmes  and  for  all 
forms  of  evaluation  (ex  ante,  ex  post  and  thematic).  The  number  of 
programme  evaluations  within  the  framework  of  Structural  Funds  was 
therefore  mu tiplied  by  5 or  6 from  one generation to  the  next. 

Systematic  overall  evaluations  and  sometimes  in-depth  thematic  and 
geographic  analyses  have  been  performed  for  Objective  1  and  5b 
programmes.  These  mid-term  evaluations  aim  at  providing  the  first 
programme  results and indicating adjustments to  be made during the 1994-
1999  programme. 

For  Objective  2  programmes,  evaluations  were  performed  in  1996-97,  at 
the  end  of  the  first  programming  phase  (1994-96),  in  the  majority  of 
countries. The new  Member States also supplied their evaluation  reports. In 
view  of  the  delayed  launching  of  programmes,  the  implementation  period 

Evaluation issues within  the framework of the Structural Funds 

taken  into  account  in  reports  was  shorter  but  the  work  was  nevertheless 
useful.  In Sweden, for  example, the  evaluation  of  Objective  2  programmes 
was recognised  for drawing  initial conclusions  on implementation  methods 
and the appropriateness  of the  measures. 

In general, the usefulness  of the exercise from the  managerial  point of view 
is  enhanced  with  experience.  This  is  true  to  such  an  extent  that  many 
managers  have  done  more  than  demanded  by  the  regulations.  Over  and 
above the thematic  and transnational evaluations  performed at the initiative 
of  the  Commission  services,  numerous  sectoral  evaluations  have  been 
launched  by  national  administrations  or  programme  managers. 

Noteworthy  improvements  in the  quality  of evaluation  reports 

The  average  quality  of  evaluations  carried  out  during  the  1994-99  period 
has  unquestionably 
to  the  preceding  period, 
particularly  as  far  as  mid-term  evaluations  are  concerned.  There  is  of 
course still  some  mediocre  work,  but this  can  be explained  by the  delay  in 
launching certain programmes in some Objective  1 or Objective 5b regions. 

increased  compared 

Most  reports  can  be  criticised  for  not  being  analytical  enough  and  for 
providing  conclusions that  are too  descriptive  to  be truly  useful  in decision 
making.  There  are  nevertheless  some  practices  which  meet  the  best 
professional  standards.  Thus,  evaluations  (particularly  in  Belgium  and 
Ireland)  have  provided  credible  conclusions  on  the  tricky  question  of 
reallocating budgets: identifying measures to be reinforced or reconsidered. 

With  more  experience  in the  domain  and  better  knowledge  of  the  specific 
nature  of  evaluation, evaluators  have  been  able to  produce  better  reports. 
Many  of  them  are  based  essentially  on  the  analysis  of  secondary  data 
(collected  from  data  bases  or  existing  documents)  and  monitoring  data 
which  have  improved  in  recent  years.  On  the  basis  of  these  data,  the 
Member  States  have  been  able  to  comply  with  the  regulations  which 
require  the  examination  of  physical  and  financial  data  for  evaluation. 
Significant  work  has been carried  out  in Italy, for  example, where  a system 
of  standard  indicators  was  established  in  the  area  of  training,  and  in 
Portugal where  proposed  indicators  were constructed  for  all sectors. 

It  is  in  the  collection  and  processing  of  socio-economic  data  that  most 
improvements  need  to  be  made.  Primary  data,  collected  directly  by 
surveying  the  publics  concerned, remain  rare and  information  provided  by 

Evaluation design and  management 

context  indicators  fails  to  take  full  advantage  of  national  and  regional 
statistical  systems.  Some  of  these  statistical  systems  have  not  greatly 
developed  (e.g.  in  Greece  and  Portugal)  or,  as  in  the  case  of  the  new 
German  Länder,  have  not  been  harmonised  in  line with  European  systems 
(EUROSTAT).  It  is  within  the  perspective  of  optimising  existing  data  that 
Volume 2 of this  Collection  has  been written. 

Multiplication  of  initiatives  and  increase  in  resources 

Remarkable  efforts  have  been  made  by the  Commission  services,  both  on 
their  own  and  in close  cooperation  with  national  administrations. 

The  Community  services  can  be  credited  with 
the  creation  and 
strengthening,  as  from  1994,  of  evaluation  units  and  the  diffusion  of 
specialised  know-how  to  all  the  operational  services.  Between  1994  and 
1998,  close  to  260  officials  in charge  of  Structural  Funds  attended  two-  to 
three-day  evaluation  training  courses  as  part  of  the  MEANS  programme 
and  SEM 2000  (i.e.: almost  half the officials  concerned). 

Depending  on  the  Member  State,  the  national  administration  has  an 
essential  role at  the  central  level  (e.g. the  Irish  Department  of  Finance;  the 
German  Treasury  and  Ministries  of  Agriculture  or  Transport)  or  the 
decentralised  level (e.g. the  Prefectures  in France). In some countries,  such 
as  Spain,  responsibilities  are assumed  by  both  types  of  authority  (national 
and  regional).  In  the  Netherlands,  for  example,  the  national  authorities 
formulated  a  methodological 
to 
guarantee the  relevance and quality  of  evaluation  reports  carried out  under 
the  responsibility  of  the  different  committees. 

framework  (see  the  example  below) 

Example: The  Dutch  methodological  framework 

In  1994  this  country  developed  a  common  methodological  framework  for  all  the 
stages  of  evaluation  and  all  Structural  Funds.  As  a  result,  each  evaluation  report  is 
set  out  in  the  same  way:  socio-economic  situation  of  the  region,  strategy  and 
coherence,  organisation  and  monitoring,  implementation  and  synergy,  and  effects. 
This  framework,  combined  with  the  use  of  indicators  common  to  the  different 
regions,  and  with  an  efficient  monitoring  system,  allows  for  an  exchange  of 
information  and  experiences  owing to  the  comparability  of  tools  used  in the  regions 
concerned  by the  Structural  Fund  programmes. 

Participation  by  regional  authorities  in  evaluation  exercises  has  increased 
substantially  since  1994:  programming,  definition,  selection  of  evaluators, 

Evaluation issues within  the framework of the Structural Funds 

taking into account  of recommendations, etc. This trend can be considered 
as a factor in the development of evaluation, especially when these regional 
authorities  have  a  high  degree  of  autonomy  and  are  therefore  dominant 
partners. 

In  Belgium,  the  federal  level  does  not  intervene  in  the  management  of 
structural  evaluations.  Evaluation  is consequently  managed  at the  regional 
level.  In  Germany,  the  administrations  of  the  Länder  are  primarily 
responsible  for  the  implementation  of  Structural  Funds  and  therefore  for 
their  evaluation. Significant  differences  nevertheless  exist  between  the  old 
and  the  new  German  Länder.  The  former,  familiar  with  an  evaluative 
approach,  involve the national level only when this is essential, whereas the 
latter  are systematically  supported  by the  national  administrations. 

In countries such as Portugal or France, ministries concerned with technical 
issues are not  necessarily  very  active in the evaluation  of  Structural  Funds. 
By contrast, the administrations  which coordinate at the  national level have 
a  key  role  in  constantly  managing  and  monitoring  evaluation  work 
performed  under the  responsibility  of  each of the  Monitoring  Committees. 

The  dynamics  created  by  the  Structural  Funds 

The  value  added  of  partnerships 

Partnerships,  by  creating  relationships  of  trust,  have  made  evaluation 
credible  and  have  thereby  facilitated  its  implementation.  The  reservations 
and even hostility vis-à-vis evaluation which persisted in the early 1990s has 
gradually  disappeared  in  most  Member  States.  Naturally,  this  has  greatly 
facilitated  the implementation  of evaluation. 

The management  of  Structural  Fund  programmes  involves a large  number 
of actors and coordination  between  European, national and regional levels. 
Partnerships,  involving  socio-economic  partners,  were  built  up  around 
Monitoring  Committees.  In  almost  all  countries  Monitoring  Committees 
have entrusted the responsibility  for  evaluation to Steering Committees  (or 
technical  or  ad  hoc  political  supervisory  committees,  etc.)  which  are 
smaller  and  therefore  more  apt  to  address  technical  issues.  These 
committees  are  usually  composed  of  European,  national  and  regional 
authorities  and sometimes  of experts  (academics, statisticians, etc.). They 
play  a  key  role  in the  methodological  conception  and  examination  of  the 

Evaluation  design  and  management 

quality  of  evaluation  work.  Evaluation  reports  are  discussed  within  these 
restricted  groups  before  being  presented  to  the  Monitoring  Committees  in 
plenary  sessions.  The  application  of  this  organisation 
in  Portugal  is 
presented  below. 

Example: Partnership, a key factor  of success  in  Portugal 
Each  Monitoring  Committee  constitutes  an  evaluation  group,  consisting  of 
Commission  desk  officers  and representatives  of  Portuguese  national  and regional 
administrations,  responsible  for  relations  with  the  evaluation  team.  This  technical 
group  meets  at  least  three times  a year  (to validate  the  terms  of  reference,  select 
offers, and discuss reports before submitting them to the Monitoring Committee). 
Good  relations  between  partners  within  the  technical  groups  and  constructive 
interactions  between  commissioners  and  evaluation  teams  are  two  strong  points 
which, according to national officials, have had a highly positive impact on the quality 
of evaluations. 

Partnerships  with  Commission  services  have  also  played  a  positive  role, 
e.g.: contributions  in terms  of  technical  documents  to the  different  phases, 
participation  in the selection  of  evaluation teams and analyses of the  quality 
of  reports. 

Many  initiatives  have  been  launched  by  the  Commission  to  disseminate 
knowledge  and  promote  exchanges  between  Brussels and  its  partners.  For 
example,  the  Commission  has  set  up  Technical  Evaluation  Groups 
(consisting  of  representatives  of  national  authorities  and  Commission 
services)  which  meet  several  times  a  year.  For  the  commissioning  of 
evaluations  it has, in partnership with several States, drawn  up model  terms 
of  reference.  Meetings  are held  regularly  in the  Member  States  (Greece  and 
Finland, for  example)  to  share and  exchange  information. 

The subject  of  partnership  cannot  be addressed  in the context  of  Structural 
Funds  without  referring  to  one  of the  conclusions  produced  by the  MEANS 
programme 
totally  relevant  today:  although 
partnership  is  a  driving  force  for  the  development  of  evaluation  practice  it 
can  also undermine the quality  of evaluations  when too  many  compromises 
are  made  between  co-decision-makers. 

in  1996,  which  remains 

The  professionalisation  of  evaluation  teams 

During the  past five years the evaluation  market  has developed  remarkably, 
as  have  the  number  of  agents  performing  evaluation  work.  The  supply  of 

Evaluation issues within the framework of the Structural Funds 

evaluation services often remains external to administrations, except for the 
central  administration  in  Ireland, France and  Italy  which  have created  their 
own specialised  evaluation teams. When they  exist, these evaluation  units, 
whose independence  is guaranteed  by a special legal status and a position 
"outside  the  hierarchy",  benefit  from  experience  that  outside  practitioners 
do  not  necessarily  have  since  evaluation  accounts  for  only  a  part  of  their 
professional  activities.  The  case  of  the  Irish  administration  is  presented 
below. 

Example: Permanent evaluation teams in the Irish administration 
In 1992 the  Irish government  and the Commission  established  specific  Evaluation 
Units for Industry and Agriculture, as well as a specific Unit for the ESF and, among 
others, for the entire CSF, to act as a "horizontal" coordinator. These units, thanks to 
their independence and professionalism, contribute towards the smooth functioning 
and quality of the work produced. 
The units carry out thematic evaluations and in-depth analyses. Their work is planned 
by  pluralistic  Steering  Committees.  Most  evaluations  are  conducted  by  the  units 
themselves, although the Agriculture unit sometimes sub-contracts work. Reports on 
all these evaluations are published. 

The  number  of  consultancy  firms  active  in the  field  of  evaluation  has  risen 
sharply in all countries. It is considered that entrusting evaluation to  outside 
evaluators, whether  private consultants or universities, is a guarantee of the 
independence  of  the  team  in  relation  to  the  authorities  concerned.  This 
sector  has  developed  in different  ways: the  emergence  of  teams  linked  to 
the  academic  world  and the  creation  of departments  or  subsidiaries  of  the 
major  international  groups.  Thus,  in  Greece  and  Portugal  a  significant 
number  of  small firms  (over thirty  consultancy  firms)  are currently  involved 
in the evaluation of structural  programmes.  In some cases, such as France, 
a  phenomenon  of  concentration  can  be  observed,  i.e.  a  few  firms  share 
most  of the  market. 

The  conditions  for  performing  quality  work  are  not  always  met.  For 
example,  the  time  allocated  is  often  too  short  (from  one  to  six  months) 
because of scheduling reasons. In some countries  such as Finland, Ireland, 
France  or  Germany,  or  for  in-depth  evaluations  or  certain  specific  cases, 
schedules  are more reasonable  (up to twelve months). The corollary  of this 
is that the budgets allocated are too small (30 Keuros for certain evaluations 
in the  Netherlands  and Spain)  to  carry  out  proper  field  surveys  among  the 
individuals  or businesses  concerned. 

Evaluation  design and  management 

In  general,  the  cost  of  evaluation  work  remains  fairly  low  compared  to  the 
overall  budget  of the  programme:  0.02%  on average  and  probably  0.01 %  in 
the Objective  1 regions.  However, these  ratios will have to  be brought  closer 
to  international  standards  (see  page  78)  when  the  budgets  for  in-depth 
evaluations,  either  underway  or  ex  post,  in the  future, are  made  known. 

The  quantitative  development  of  supply  is also  accompanied  by  qualitative 
developments  generated  by  greater  professionalism.  Skills  have evolved  in 
recent  years,  owing  more  to  accumulated  experience 
the 
specialisation  of  evaluation  service  suppliers.  Evaluation  teams  often  have 
multidisciplinary  competencies,  although  economists  play  a dominant  part 
in the  evaluation  of  regionalised  programmes. 

than 

to 

It  is  increasingly  frequent  that  consultants  request  the  collaboration  of 
academics  who  have  more  specialised  and  precise  expertise  in  specific 
areas  of  public  policy.  In fact,  many  consultancy  firms  have  been  set  up  by 
academics,  especially  in the  southern  European  countries  (Portugal,  Spain 
and  Italy).  Know-how  in  Germany,  Finland  and  Belgium  is  based  primarily 
on  universities,  institutes  and  research  centres. 

Certain  methodological  progress  made  in  recent  years  warrants  attention. 
Know-how 
in 
in  the  modelling  of  regional  economies  has 
Germany,  Italy,  Ireland  and,  more  recently,  Belgium.  Input-Output  analyses 
have  been  skilfully  implemented  for  conducting  evaluations  in  Portugal, 
Germany, 
Italy  and  Spain.  The  analysis  of  relations  of  causality  has 
developed  and  specialists  are  using  comparative  statistical  analyses  more 
often.  The use of  comparison  groups, limited to  Denmark  and the  U.K.  until 
recently,  has  spread  in  Sweden,  Germany,  France  and  Spain,  primarily  in 
the  domain  of  employment. 

increased 

The  build-up  of  evaluation  capacities  within  countries 

It  has  been  noted  that  institutional  peculiarities  of  each  State  result  in 
significant  differences 
in  the  implementation  of  evaluation.  The  same 
applies  to  the  appropriation  and  use,  by  commissioners,  of  results  in 
reports.  Unsurprisingly,  the  European  landscape,  with  regard  to  the  use  of 
evaluation,  varies  widely.  Nevertheless,  three  phases  in  the  level  of  utility 
attributed  to  evaluation, and  hence  in the  quality  of  its organisation, can  be 
distinguished  schematically. 

7s' phase:  Evaluation  is  seen  as  an  answer  to  regulatory  obligations.  It  is 
therefore  a  constraint  and  an  additional  workload  weighing  on  managers 

Evaluation issues within  the framework of the Structural Funds 

who  consider  it,  above  all,  as  a  demand  from  Brussels-even  though 
regulations  are decisions  unanimously  adopted  by the Council. 

2nd phase: Evaluation becomes a system to aid the design and  management 
of interventions. A dialogue is established with the evaluators and the quality 
of  the  information  gathered  improves.  In this  highly  operational  phase, the 
evaluation approach  is refined and progress  is made rapidly. 

3rd  phase:  Evaluation  becomes  a  political  act,  the  results  of  which  are 
publicly  debated. The  aim  is to  inform  public  opinion  on the  effectiveness 
of  the  use of  Public  Funds, and  on the  demonstration  of  their  efficiency  in 
terms  of  obtaining  the  expected  impacts  (Value  for  Money).  In  a  sense, 
evaluation  becomes  a  tool  of  democracy  by  informing  citizens,  if  not 
seeking their  consent  on State spending. 

These  three  phases  are  represented  simply  in  Box  5  which  shows  the 
different  Member  States'  use  of  evaluation  work,  in  relation  to  each  other. 
The  use  of  evaluation  is  represented  by  the  vertical  axis,  while  evaluation 
practice  within  countries  is  represented  by  the  horizontal  axis.  For  each 
case, the diagram shows whether  practice is limited to the Structural  Funds 
or  whether  evaluation  is  also  practised  within  the  framework  of  other 
national  or  regional  policies. 

Box 5 -  Practice  and use of  evaluation 

Netherlands) 

(  Sweden ] 

Limited to the Structural Funds 

Also exists outside 
the Structural Funds 

Evaluation design and management 

This graph  is only  a transitory  image (from a survey carried out  in 1998). The 
dynamics  of  each  country  could  also  be shown. For example,  Portugal  has 
become  largely  familiar  with  evaluation  practice  and,  in  recent  years, 
significant  progress  has  been  made  in  terms  of  quality  and  the  use  of 
evaluation  work. 

The  influence  of  Community  regulations  on the  development  of  practices  is 
significant  in most  of the  Member  States. Thus, in Objective  1 countries,  for 
example,  evaluation  work  is  essentially  focused  on  the  Structural  Funds. 
This influence, although  perceptible, seems less marked  in certain  countries 
such  as  Ireland where  recent  reforms to  improve the  management  of  public 
actions  stem  mainly 
the  performance  management  movement 
developed  in  the  English-speaking  countries.  In  certain  States,  European 
requirements  governing  the  implementation  of  structural  policies  have 
prompted  remarkable  changes  in  attitudes  and  have  led  to  systematic 
evaluation.  Thus,  in  Germany  and  France,  for  example,  evaluation  was 
being  institutionalised  but  Community  regulations  accelerated  the  existing 
process. 

from 

As  far  as the  most  recent  members  of  the  European  Union  are  concerned, 
Community  regulations  have  had  particularly  noteworthy  effects  because 
evaluation  practice,  skills  and  culture  are  still  limited.  The  obligation  to 
evaluate,  imposed  by  the  Community,  has  provoked  the  emergence  of  a 
new  market  accompanied  by  greater 
institutionalisation  and  a  more 
assertive  evaluation  culture.  This  phenomenon  can  be  found  in  Finland 
the  relative 
is  developing 
where  evaluation  practice 
integration  of  European  structural  policies  into  Finnish  national  policies. 

fast,  owing 

to 

It  is  interesting  to  note  that  the  democratic  use  of  evaluation  is  promoted 
when  a  national  evaluation  culture  exists  independently  of  Community 
requirements.  In other  words,  regulatory  obligations  can  help to  establish  a 
managerial  use but  are probably  not  enough to  induce a contribution  to  the 
public  debate.  By  contrast,  the  number  of  countries  in the  bottom-left  part 
of  Box  5  reflect 
from  an 
institutional  point  of  view,  as  well  as  its  influence  on  the  development  of 
practice  in countries  where  the  evaluation  culture  was  previously  weak. 

importance  of  Community  regulations 

the 

International  reflections  on  the  building  of  evaluation  capacity  emphasise 
the  importance  of  incentives  and  actions  to  promote  awareness,  as 
significant  factors  in  the  development  of  evaluation.  Various  initiatives  can 
the  creation  of  national  evaluation  societies  or 
contribute, 

including 

Evaluation issues within the framework of the Structural Funds 

associations  such  as the  recently  founded  German  Evaluation  Society  and 
the  Italian  National  Evaluation  Society  (Boyle,  Lemaire,  to  be  published). 
The particularly  active  British evaluation  society  (UKES) is briefly  presented 
below. 

Example: The British professional evaluation community 
British  teams  in the  private  sector  and  universities  have developed  internationally 
recognised  competencies  and expertise  in evaluation. The U.K.  is one of  the rare 
Member  States  to  have  evaluation  research  teams  which  work  in  this  domain 
"transversally", without limiting themselves to a particular methodological aspect nor 
to  a  given  political  sector.  The  British  evaluation  society  (UKES),  with  over  200 
members,  organises  an  annual  conference  and  workshops  for  developing  the 
exchange of methodology. 

Training  is  an  essential  tool  in  promoting  awareness.  Those  countries  in 
which  administrative  personnel  have  received  evaluation-specific  training 
remain rare. However, wherever these activities  have been set up  (seminars 
on  evaluation  organised  in  France  by  private  firms  or  administrative 
services,  or  in  Greece  by  Commission  services)  the  results  are  positive. 
Moreover,  a  few  courses  on  the  evaluation  of  public  policies  exist  or  are 
being  set  up  in  universities,  most  often  as  part  of  training  focused  on 
specific  sectors  of the  economy. 

Based  on  the  above  observations  concerning  recent  developments  in 
evaluation  practice, it is possible to  identify two  major challenges to  be met 
if  the  current  dynamic  of  the  evaluation  of  the  Structural  Funds  is  to  be 
maintained: 

•  the  professionalism  of  practitioners  must  be  enhanced  to  complete 
inadequate  know-how  among  administrative  officials  and  evaluation 
teams, and  improve the quality  of  evaluation  work; 

•  effective  action  to  increase  awareness  and  provide  incentives  must  be 
implemented;  in other  words,  evaluation  must  move  on from  a  response 
to  an obligation  to  a voluntary  approach. 

Preparing an evaluation 
and defining its content 

Defining the general framework of an evaluation 

Evaluation  can  have  a  number  of  forms,  depending  on  the  stage  of  the 
programming  cycle  at  which  it  is  performed  (ex  ante,  intermediate  or  ex 
post  evaluation),  the  level  of  decision-making  concerned  (evaluation  of  a 
policy,  programme  or  project)  and  the  scope  of  the  evaluation  (overall, 
thematic  or in-depth  evaluation). These different  forms  of evaluation, rather 
than  being  independent  from  one  another,  have to  be  linked  together.  It  is 
therefore  preferable  to  view  evaluation  initiatives  in  a  general  framework 
covering  several  programming  periods  and  levels  of  responsibility. 

In  particular,  the  succession  of  European  Structural  Fund  programming 
periods  imposes  a  schedule  that  creates  overlaps  between  ex  ante, 
intermediate  and  ex  post  evaluations.  It  is therefore  necessary  to take  into 
account  the  complementarity  between  evaluation  work,  so  that  each 
evaluation can draw on preceding ones, contribute to future ones and avoid 
redundancy. 

the  overlaps  between 

Similarly, 
(political 
/programme  /  project)  means  that  evaluations  performed  at  one  level 
produce  conclusions  relevant  to  other  levels.  In  particular,  programme 
evaluations  take  into  account  the  objectives  and  priorities  of  the  policy 
context  in which they  are situated. 

responsibility 

levels  of 

Linking  the  different  stages  of  evaluation 

In order  to  anticipate  the  usefulness  of  an evaluation,  it  has to  be  situated 
in  the  life-cycle  of  the  programme  considered  as  it  is  represented  in  the 
following  diagram. 

Box 6 -  Evaluation  in the  life cycle  of  a  programme 

Design of the  programme 
Ex ante  evaluation 
Feasibility  study 

Ex post  evaluation 
Report 
Conclusion 

Implementation 
Monitoring  outputs 
Mid-term  evaluation 

Evaluation  design  and  management 

Ex  ante  evaluation 

-4  Ex  ante  evaluation  takes  place  at  the  beginning  of  the  cycle  before  a 
programme  has  been  adopted. 

This  form  of  evaluation  helps  to  ensure  that  the  final  programme  is  as 
relevant  and  coherent  as  possible.  Its  conclusions  are  intended  to  be 
integrated  into the  programme  when  decisions  are taken. 

Ex  ante  evaluation  focuses  primarily  on  an  analysis  of  the  strengths, 
weaknesses  and potential  of the  Member State, region or sector concerned. 
It  provides  the  relevant  authorities  with  a  prior  judgement  on  whether 
development  issues  have  been  diagnosed  correctly,  whether  the  strategy 
and  objectives  proposed  are  relevant,  whether  there  is  incoherence  in 
relation  to  Community  policies  and  guidelines,  whether  the  expected 
impacts  are realistic, and so on. It also  provides the required foundations  for 
monitoring  and for future evaluations, by ensuring that there are explicit and, 
where  possible, quantified  objectives.  It helps to specify  selection criteria  for 
the  selection  of  projects  and  to  ensure  that  Community  priorities  are 
respected.  Finally,  it  helps  to  ensure  the  transparency  of  decisions  by 
allowing for  a clear  explanation  of  choices  made  and their  expected  effects. 

Ex  ante  evaluations  are  performed  at  the  time  when  public  authorities  are 
involved  in discussions  and negotiations on the future programme. They are 
therefore  subjected  to  strong  constraints:  pressure  of  deadlines,  vague 
formalisation  of the  proposed  programme  to  be evaluated, amendments  to 
this  proposal  while  the  work  is  underway,  demands  for  confidentiality,  etc. 
The evaluation team must  therefore be able to  intervene flexibly and  rapidly, 
and  be  able to  apply  techniques  for  analysing  needs  and  simulating  socio­
economic  effects. 

Mid-term  evaluation 

M  Mid-term  evaluation  is  performed  during  the  second  stage  of  the 
programming  cycle,  during  the  implementation  of  the  interventions. 

Depending  on the conclusions  of  mid-term  evaluation, adjustments  may  be 
made  during  the  cycle.  This  evaluation  critically  analyses  the  first  outputs 
and  results  of  interventions.  It  also  assesses  the  financial  management  of 
the  programme  and the quality  of  the  monitoring  and of  its  implementation. 
It  shows  how  and  whether  original  intentions  have  been  carried  out  and, 

Preparing an evaluation and defining  its  content 

where  relevant,  checks  whether  de  facto  changes  have  been  made  to  the 
initial  objectives.  By  comparison  with  the  initial  situation,  it  highlights 
changes  in  the  general  economic  and  social  context  and  judges  whether 
the objectives  remain  relevant.  Mid-term  evaluation  also  examines  whether 
the  evolution  of  Community  priorities  and  policies  poses  a  problem  of 
coherence,  and  helps  to  prepare  adjustments  and  reprogramming,  and  to 
argue them  in a transparent  manner. 

Mid-term evaluation  relies heavily on information  drawn from the  monitoring 
system,  but  also  on  ex  ante  evaluation  and  on  information  on  the  context 
and  its  evolution.  It  generally  consists  of  short  and  exhaustive  exercises 
focusing  primarily  on  the  results  of  the  programme  evaluated,  without 
attempting  an  in-depth  analysis  of  impacts  that  have  not  yet  had  the  time 
to  emerge.  It  is,  however,  possible  and  advisable  to  refer  to  in-depth  or 
thematic  evaluations  of  former  programmes  when  such  analyses  do  exist. 
Mid-term  evaluation  has  a  "formative"  nature,  that  is  to  say,  it  produces 
direct  feedback  into the programme  which  it is helping to  improve  as far as 
its  management  is concerned. 

Ex  post  evaluation 

►  Ex  post  evaluation  recapitulates  and  judges  the  entire  programme, 
particularly  its impacts.  Its aim  is to account  for the use of  resources and to 
report  on the effectiveness  and efficiency  of  interventions  and the extent  to 
which  expected  effects  were  achieved.  It focuses  on factors  of  success  or 
failure,  and  on  the  sustainability  of  results  and  impacts.  It  tries  to  draw 
conclusions  that  can  be  generalised  and  applied  to  other  programmes  or 
regions. 

Ideally,  the  results  of  this  evaluation  should  be  available  when  the  next 
programme  is  planned,  that  is,  at  least  a  year  before  the  end  of  the 
programme.  However,  for  the  impacts  to  have  been  produced,  ex  post 
evaluation  would  have to  be  performed  two  to  three  years  after  the  end  of 
the programming  period. While waiting for this period to  pass, a provisional 
review  is often  requested  shortly  before the end  of  the  programming  cycle, 
in liaison with  the ex  ante  evaluation  of  the following  cycle. 

Impact analysis is always a large-scale exercise if performed  systematically. 
Ex post evaluations therefore tend to involve surveys  in the field and to take 
place over  long  periods  lasting  from  twelve to  eighteen  months. 

Evaluation design and management 

Successive programming cycles 

The sequence of three evaluation phases in successive cycles creates 
overlaps that have to be organised as efficiently as possible to avoid any 
duplication of work. The basic principle is that of combining evaluation 
work, during a programme, with the use of conclusions of evaluation 
performed on the preceding programme. The relative continuity of actions 
programmed from one period to the next makes it possible to use 
conclusions from the recent past to judge the relevance of the new 
measures proposed. The following diagram shows that interactions are 
possible between evaluation work performed at the different phases of 
several successive programmes. 

Box 7 - Connections between evaluation and programming cycles 

Programmes 

(1) 

(2) 

(3) 

Results, 
impacts 

Evaluation 

mid-term (1) 
ex ante (2) 

ex post (1) 
mid-term (2) 
ex ante (3) 

Observation 

Feedback 

■^ Thus, an ex ante evaluation that prepares the adoption of a future 
programme has to take advantage of the results of earlier work, i.e.: 

• the intermediate evaluation of the period drawing to a close. This 

evaluation will have produced conclusions on the first years of activity 
and on the ensuing programmes. It may have been completed by a final 
review of the outputs and results of the current programme, based on 
information from the monitoring system. 

Preparing an evaluation and defining its content 

•  ex  post  evaluation  of  the  period  preceding  the  current  period,  possibly 
completed  by  thematic  evaluations  and  in-depth  analyses.  These 
evaluations  will  have  made  it  possible  to  observe  and  analyse  the 
impacts  of  former 
interventions  which  are  similar  to  the  planned 
interventions  and which  took  place  in a partly  similar  context. 

H  Since  the  role  of  intermediate  evaluation  is  to  check  whether  the 
objectives  are still  relevant  and  in the  process  of  being  achieved, it  will  be 
necessary  to  refer  primarily  to the  monitoring  system  data  but  also: 

•  to the ex ante evaluation and particularly to the diagnosis made in relation 
to  the  prevailing  socio-economic  context  before  the  start  of  the 
programme, and which  needs to  be updated; 

•  to  the  ex  post  evaluation  of  the  preceding  programme,  of  which  the 
conclusions  concerning  the  same  areas  of  intervention  could  serve  as 
references. 

►  Ex post evaluation  is based on management  and monitoring data and on 
surveys  in  the  field  which  will  help  to  observe  and  analyse  the  real  and 
sustainable impacts of the interventions. It refers to ex ante evaluation  in so 
far  as  it  has  to  report  on  the  attainment  of  objectives.  It  refers  to 
intermediate evaluation, particularly to identify the success or failures which 
were identified  at that  stage. 

In  so  far  as  evaluation  must  draw  conclusions  from  the  experience  of 
preceding  programmes  to  improve  future  programmes,  an  interesting 
solution  is to establish a pluri-annual  evaluation  plan. The idea is to  identify 
the different  possible  evaluations  and to establish their date and content  in 
relation to the  political  schedules  and deadlines  for  decision-making  of  the 
evaluation  "customers"  at  various  levels. 

■ Performing evaluation at different levels 

Since the Structural Funds reform in 1988, the Commission has favoured the 
integrated programming of Community interventions. A programme is 
composed of numerous projects within the framework of specific objectives, 
which are themselves directed towards the global objective of the programme. 

The evaluation of a programme is neither the sum nor the synthesis of the 
evaluations of the projects comprising it. Some questions are not raised at 
the individual project level but concern the programme evaluation. That is, 
for example, the case when the evaluation of the effects of synergy is 

Evaluation  design  and  management 

required.  This  point  is  considered  below  (see  page  128)  as  well  as  in 
Volume  4  of the  MEANS  Collection. 

Evaluating  projects  at their  selection  stage 

The  national  and  regional  authorities  are  responsible  for  selecting  projects. 
This  is frequently  accompanied  by  an  ex  ante  evaluation,  also  called  prior 
appraisal.  The  example  described  below  is that  of  a  regional  development 
programme  in  the  U.K.,  a  country  where  particular  care  is taken  in  project 
appraisal. 

Example: Selection  of the  projects  within the  Highlands and  Islands  programme. 

Within  the framework  of the  programme  under  consideration, there are two  cycles  of 
calls  for  projects  each  year.  All  the  applications  for  funding  are  first  examined  in 
relation to their  eligibility.  Projects  considered  to  be eligible  are given  scores  in terms 
of eight  general  criteria, common  to  the  entire  programme  (relation to  the  objectives, 
probable  effectiveness,  respect 
innovation, 
relations  with  other  projects,  intervention  of the private sector  and support  from  local 
partners),  and  in  terms  of  several  specific  criteria  which  vary  depending  on  the 
measure.  Since the criteria are weighted, it  is  possible  to  give a global  score to  each 
project. After  comparing  applications  for funding with available  resources, a standard 
for  admission  is  established  and  a  list  of  projects  to  be  financed  is  recommended. 
The final  decision  is taken  by  the  programme  managers. 

the  environment,  additionality, 

for 

This  example  shows  that  project  evaluation  must  be  performed  in  relation 
to  the  programme  objectives.  Close  ties  with  the  objectives  are one  of  the 
main  selection  criteria  in  the  scoring  system.  Inversely,  during  the  overall 
evaluation  of  a  programme,  the  approach  is  often  from  the  project  to  the 
programme  level,  i.e.  bottom-up.  In  many  cases  analyses  are  initially 
focused  on  specific  projects  or  measures,  which  are  therefore  micro,  and 
progressively  extended  to  the  entire  programme  to  evaluate  hypotheses  of 
causality  which  are, by  nature, always  macro.  In this way  project  evaluation 
feeds  into  programme  evaluation. 

The  limits  of  the  bottom-up  approach  lie  in  the  fact  that  it  causes  the 
evaluation  team  to  make  increasingly  fragile  assumptions  because  the 
effects  to  be  assessed  are  more  and  more  distant  and  complex.  The 
advantage  of this articulation  of  project  and  programme  evaluation  is that  it 
provides  a detailed  explanation  of the  sequence  of  short  and  medium  term 
impacts. 

Preparing an evaluation and defining its content 

Example: evaluation  of  a  project  in  its regional  context. 

An  evaluation  team  performed  an ex  ante  evaluation  of  public  support  attributed  to 
the  establishment  of  a car  manufacturing  plant  in southern  Italy. The  main  question 
concerned  the  economic  and  social  impact  of  this  plant  on  local  business  and,  in 
particular,  on  the  reduction  of  unemployment.  By  examining  the  project  in  isolation, 
the  evaluation  team  showed  that  the  impact  was  weak.  In fact,  the  plant  remained 
largely  dependent  on  resources  from  outside  the  region  in  terms  of  labour  and 
suppliers.  Thus,  from  a  purely  micro-economic  point  of  view,  the  impact  of  the 
project  was limited. 

On  the  other  hand,  by  examining  the  project  in  its  regional  context,  the  evaluation 
team showed that  it was likely to generate new activities, to develop local  production 
of goods  and services and, finally, to  produce significant  long-term  effects,  provided 
that  macro-economic  mechanisms  were taken  into  account. 

Evaluation  of  large  projects 

An  exception  is made  of the  evaluation  of  large  infrastructure  projects  which 

are  important  instruments  in the  implementation  of  European  Union  regional 

policy.  The  major  projects  have  been  defined  as  "those  of  which  the  total 

cost  [...]  generally  exceeds  25  million  euros  for  infrastructure  investments 
and  15  million  euros  for  productive  investments"2.  The  amounts  at  stake 
justify  the  projects'  in-depth  and  independent  prior  appraisal,  by  comparing 

their  costs  and  benefits  according  to  the  technique  described  in  Volume  3 

of  this  Collection.  These  evaluations  are  the  joint  responsibility  of 
Commission  and  Member  States3. 

the 

For 

investments 

in 

infrastructure,  an  analysis  of 

the  project's  socio­

economic  advantages  includes  the  expected  rate  of  use,  the  potential 

impact  on  the  development  or  reconversion  of  the  region  concerned,  and 

the  estimation  of  the  consequences  of  the  Community  intervention  on  the 

realisation  of  the  project.  For  productive  investments,  the  analysis  will  focus 

on  market  prospects  in  the  sector  concerned,  employment  effects  and 

expected  profitability  of  the  project. 

In  the  processing  of  individual  applications  for  funding,  the  Commission 

takes  into  account  the  results  of  prior  appraisals  and  evaluations.  If  the 

appraisal  presented  by 

the  applicant 

is  considered 

insufficient  and 

2 Article  16 of the  Council  regulation  2082/93 
3  Regulation  2082/93. 

Evaluation  design  and  management 

unconvincing, the Commission may require the applicant to expand the analysis 
or, alternately,  may  itself  appraise  the  project  by  requesting, where  necessary, 
the  opinions  of  independent  technical  experts.  In any  case, the  Commission's 
decision  will  be  the  result  of  dialogue  and  collaboration  with  the  applicant,  to 
ensure that the  investments  made  produce the most  satisfactory  results. 

Evaluation  of  policy 

The results  of  programme  evaluations  are  systematically  used  in  syntheses 
produced  at  a  national  or  European  level. At  these  levels,  they  serve  as  a 
basis for  reflection and support  for decision-making  in the area of  economic 
and  social  cohesion  policy.  Box  8  lists  a  series  of  direct  and  indirect 
"clients"  of  programme  evaluations  at the  European  level. 

Box  8 -  The  "customers"  of  evaluations  at the  European  level 

Programme  evaluations  are directly  or  indirectly  useful: 

•  to  European  governments,  as  a basis  for  strategic  planning  of  cohesion  policy 

and  for  preparing  their  proposals  to  the  Council; 

•  to the Commission's  operational  units, in the preparation of  decision-making  on 

the  financing  of  programmes  and their  content; 

•  to  the  representatives  of  the  Commission,  when  they  express  the  Community 
the 

point  of  view  during  decision-making  on  adjustments  during 
implementation  of the  programme; 

•  and,  finally,  to  the  services  responsible  for  preparing  annual  reports  on  the 
implementation  of  Structural  Funds  and  the  tri-annual  report  on  social  and 
economic  cohesion. 

A particularly  interesting form of evaluation at the European level is that  which 
analyses  certain  impacts  considered  as  priorities  at  that  level.  Thus,  an 
evaluation  may  address transversal  questions  such  as employment  impacts, 
the  competitiveness  of  SMEs,  equal  opportunities,  the  environment,  etc. 
These  Community  priorities  are  often  recalled  in  programming  documents 
without,  however,  being  fully  integrated  or  mainstreamed. Trying  to  evaluate 
impacts  in  relation  to  priorities  amounts  to  designing  what  is  called  a 
thematic  evaluation  (see Volume  5 of  the  Collection). 

A  thematic  evaluation  as  such  covers  several  programmes  and  may  be 
carried  out  at the  national  or  European  level.  Its  main  characteristic  is  that 
it  treats  a  specific  theme,  generally  a  priority  for  the  commissioner.  For 
example, the  European  Commission  performed  an evaluation  on the  theme 

Preparing an evaluation and defining its content 

of  equal  opportunities,  in  about  ten  programmes 

in  different  States. 

Thematic  evaluations  serve to compare  different  programmes  and to gain a 

better  understanding  of  processes  of  impact  production  in  a  particular 

domain.  Major  transnational  evaluations  have  thus  been  launched  on  the 

themes  of  the  Environment,  R&D and  support  for  SMEs. 

Selecting  t he  different  levels  of  analysis 

Evaluation experience  of Structural  Fund programmes  shows a wide  variety 

of  ways  of  addressing  evaluative  questions.  Fairly  often,  the  evaluation 

team  is  expected  to  treat  ten  or  more  questions  in  a  few  months.  Other 

evaluations,  by contrast,  devote  a whole  year  to  the  in-depth  analysis  of  a 

single  question. The  choice  of  studying  a question  in greater  depth  will  be 

guided  by the  need  to  better  understand  a difficulty  encountered,  to  show 

the  exemplary  character  of  a  success  story,  to  better  understand  the 

effectiveness  of  a measure, to  identify  the  conditions  in which  an  objective 

was  met,  etc.  In  the  following  example,  the  steering  group  considered 

analysing  several  priority  questions  in  more depth. 

Example: Seven  questions to consider  in depth when  evaluating a  programme 

In  1997, within  the  framework  of  the  evaluation  of  an  industrial  reconversion 
programme  in  Franche-Comté  region  in the  North-East  of  France,  the  evaluation 
steering  group  devoted  its first  two meetings  to the identification  and grading  of a 
series of questions. 

Seven  fields  of  questioning  were  thus  identified.  These  were  then  discussed  in 
workshops  in order  to formulate,  for each  of  them,  a  realistic  and useful  evaluative 
question.  In the order  of  priorities  established  by the steering  group,  these  questions 
were  as  follows:  (1)  Which  categories  of  projects  have  the  best  impact  on the 
attractiveness of the territory? (2) Has support to enterprise helped to create or maintain 
jobs  that  are firmly  rooted  in the area?  (3) Have  services  for enterprises  contributed 
sufficiently  to the diversification  of economic  activity?  (4) Does  training  serve  to raise 
qualifications  or  rather  to  make  them  more  relevant?  Which  of  these  two  expected 
effects  is the  most  necessary?  (5) Is the  probable  impact  of  higher  education and 
research  projects  sufficient?  (6) What  is the effectiveness  of the different  categories of 
projects in terms of creation and maintenance of employment?  (7) Which categories of 
projects guarantee or improve equal opportunities  between  men and  women? 
An in-depth study of all seven questions would have been too costly. It was therefore 
decided to limit the number of questions to study in depth to three. Certain questions 
were  eliminated  so  that  there  would  not  be  duplication  with  other  evaluations 
underway  in the region. Finally, questions  (1), (4) and (5) were judged  priorities. 

Evaluation  design  and  management 

It  is  the  combination  of  several  criteria  that  leads  to  the  selection  of 
questions  to  be  evaluated  in depth. These  may  be: 

•  the  most  costly  measures  (assuming  the  stakes  involved  in  decision­

making  increase  in  proportion  to  the  budget); 

•  the  most  innovative  measures  (considering  that  these  parts  of  the 
to  be  validated  by  an 

programme  are  experiments  which  need 
evaluation); 

•  the  most  strategic  objectives  (assuming 

that 

these  parts  of 

the 

programme  are the  most  likely  to  gain  from  an  informed  debate); 

•  the  most  controversial  objectives  (assuming  that  these  parts  of  the 
programme  are the  most  likely to  gain from  the clarification  of  judgement 
criteria),  etc. 

•  priorities  which  interest  a specific  decision-making  level  in particular,  etc. 

Box  9  shows  that 
depending  on the  object  analysed. 

in-depth  analyses  may  assume  different 

forms, 

Box  9 -  Three  situations  of  in-depth  ana 

ysis 

Measures 

Concerned 
groups 

S *^ 

1 

^ ~ ~ ~\ 

^ 

2 

SMEs 

Impacts 

Exports 

New  business 
creators 

JT 

j " * "^ 

^ "~ 

3 

~~ 

Innovation 

Advice 

Investment 
incentives 

Training 

The  first  form  of  in-depth  analysis  (see  circle  1  in  the  Box)  concerns  the 
impacts  of  a  measure,  an  action  or  a  major  project.  Depending  on  the 
nature  of  the  public  concerned,  the  impacts  sought  will  be,  for  example, 
jobs  created  or  maintained,  investments  induced,  the  consolidation  of 
activities,  the  diversification  of  activities,  etc.  The  addressees  of 
the 
intervention  are  known  and  can  be  questioned.  Primary  data  obtained  in 
this  way  provide  indications  relative  to  gross  effects.  The  analysis  of 

Preparing an evaluation and defining its content 

causality  may also  be carried out  by means of the methods and techniques 
described  below  (see  page  147): the  search  for  confounding  factors,  the 
estimation of deadweight, eviction or displacement  effects, the  constitution 
of  a comparison  group, and  the  construction  of  a counterfactual  situation. 
In  this  situation,  the  evaluation  produces  conclusions  on  the  success  or 
failure  of  a  given  intervention.  Where  possible,  these  conclusions  will  be 
generalisable to  other  similar  interventions  (external  validity). 

The second form  of  in-depth  analysis (see circle 2 in the  Box) focuses  on a 
particular  public,  such  as  the  long-term  unemployed,  creators  of  new 
enterprise,  small  farms,  etc.  The  evaluation  is  performed  from  the  point  of 
view  of  the  public  concerned.  It  analyses  this  public's  needs  and  attitude 
towards  the  programme  and  its  judgement  criteria.  In  this  situation,  the 
evaluation  produces conclusions on the comparative  impact of the different 
measures directly  affecting  this  public.  It can also show  how these  different 
measures  reinforce  or contradict  one another  (synergy  effects). 

The  third  form  of  in-depth  analysis  (see  circle  3  in  the  Box)  concerns  an 
impact  or  a  category  of  impacts.  The  question  then  applies  to  all  the 
programme  measures  and  focuses  on  their  respective  contribution  to  the 
production  of  this  impact.  The  particularity  of  this  type  of  situation  is  that 
the  public  concerned  is  homogeneous,  whereas  the  actions  which  impact 
on this public are heterogeneous and numerous. In this situation, evaluation 
leads  to  conclusions  on  the  comparative  impact  of  the  different  measures 
contributing  directly  or  indirectly  towards  the  impact. 

The notion  of thematic  evaluation  calls to  mind that  of  in-depth  evaluation. 
Some  thematic  evaluations  examine  how  the  same  impact  (e.g.  the 
competitiveness  of  SMEs)  is obtained  by  several  sorts  of  intervention  (see 
Box 9 - situation 3) or how  interventions financed  in a specific  domain  such 
as  Research  & Development  produce  several  kinds  of  impact  (see  Box  9 -
situation  1). 

Thematic  evaluations  can  be  carried  out  within  a  single  programme  (the 
term  is  then  equivalent  to  in-depth  analysis)  or  transversally,  on  several 
programmes. These  evaluations  are  described  in Volume  5 of  the  MEANS 
Collection. 

Thematic  evaluations  are  particularly  useful  for  evaluating  programmes 
from  the  point  of  view  of  Community  priorities.  They  are  grounded  in  the 
principle  of  "mainstreaming"  whereby  a  priority  or  specific  needs  are 
systematically  taken  into account  in all Community  policies  and  particularly 

Evaluation  design  and  management 

in  the  evaluated  programme.  This  concept  is  used  mainly  for  taking  into 
account  women's  and  men's  needs  in  all  policies,  in  order  to  promote  the 
priority  equal  opportunities.  This  objective  can  be  achieved  if  the  effect  of 
the  intervention  on  the  specific  situation  of  women  and  men  is  taken  into 
account  from  the  design  stage  of the  programme,  through  implementation, 
monitoring  and  evaluation,  to  the  systems  of  indicators.  Mainstreaming 
necessitates 
the 
organisation  of  society  to  guarantee  a  more  equitable  distribution  of  roles 
between  men  and  women. 

the  development  of  measures  aimed  at  adapting 

Differentiating  audit,  monitoring  a nd  e v a l u a t i on 

The expectations  placed  on evaluation  should  not  exceed what  it  can  really 
contribute, and should not try to give it roles which  it is not designed to play, 
such  as audit  and  monitoring. 

Within  the  framework  of  structural  policies,  the  Commission  wishes  to 
satisfy  the  requirements  of  both  the  Council  and  the  Parliament  as  regards 
audit,  monitoring  and evaluation. The three exercises  are similar  in so far as 
they  are judgements  on public  activity  and can  mutually  enrich one  another. 
However, 
any 
misunderstandings.  These  practices  require different  professional  qualities, 
time  frames  and  modes  of  organisation. 

distinguished 

they  must 

to  avoid 

clearly 

be 

An  initial clarification  concerns  the  point  of view  from which  public  action  is 
judged.  Audit  verifies  the  legality  and  regularity  of  the  implementation  of 
resources.  Monitoring  verifies  the  sound  management  of  the  interventions 
and  produces  a  regular  analysis  of  the  progress  of  the  outputs.  Evaluation 
juges  programme  implementation  on  the  basis  of  the  outputs,  results  and 
impacts  it  has  produced  in  society. 

A  second  distinction  concerns  judgement  criteria. Audit  judges  in terms  of 
criteria  that  are  known  and  clarified  in  advance  (budgets,  regulations, 
professional  standards).  Monitoring 
in  terms  of  operational 
objectives  to  be  achieved.  By  contrast,  evaluation  often  has  to  start  by 
choosing  its  judgement  criteria.  These  are  formulated  on  the  basis  of  the 
objectives  of the  evaluated  public  action.  However,  experience  shows,  and 
the  evaluation  literature  confirms,  that  the  objectives  of  public  action  are 
rarely  clear,  quantified,  verifiable  and  immediately  usable  as  evaluation 
criteria. 

judges 

Preparing an evaluation and defining its content 

These  basic  differences  distinguishing  audit,  monitoring  and  evaluation  are 
presented  in  Box  10. 

Box  10 -  Articulating  audit,  monitoring  and  evaluation 

Inputs 

Output 

Outcome / Outreach 

Control 
(of  legality) 

Monitoring  (of  management) 

Evaluation  of  quality,  management  and  effects 

References 

Compliance with 
administrative texts 

Good management 
standards 

> 

Meeting citizen's 
needs and solving 
problems 

Thus,  the  professional  qualities  and  know-how  required  by  the  different 
exercises  are  not  the  same.  Auditors  have  in-depth  knowledge  of  pre-
established  criteria  (legal  or  accounting  knowledge,  for  example).  They 
have authority  and their  behaviour  inspires respect. They  do  not  hesitate  to 
use constraints  and to  publicly  stigmatise  errors and dysfunctioning  for the 
purpose  of  dissuasion.  Monitoring  officials  need  to  have  recognised 
competence  in terms  of  organisation,  management  and  the  establishment 
of  information  systems.  They  supervise  and  assist  operators  in  so  far  as 
they  provide  tools  and  procedures  that  help  them  in  their  professional 
practices  and  also  serve  in  a  critical  analysis  of  the  conditions  of 
implementation.  Finally,  because  evaluation  practitioners  cannot  rely  on 
unanimously  recognised  criteria,  they  need  the  collaboration  of  public 
managers to collect  relevant information, interpret  it correctly and formulate 
a legitimate judgement. They cannot  be effective if their interlocutors  do not 
act  willingly  and 
implies  a  degree  of 
confidentiality. 

faith,  which  often 

in  good 

If  the  three  exercises  have  intrinsic  differences,  they  all  make  use  of  one 
another, owing to their complementarity.  For example, audit and  monitoring 
can identify  apparent  successes  and failures, which  evaluation  will then  try 

Evaluation  design  and  management 

to confirm  by means  of an  in-depth  analysis.  By contrast,  an evaluation  can 
show  that  the  application  of  a  legal  or  professional  standard  constitutes  a 
particular  factor  of  effectiveness  or  ineffectiveness.  Interdependence  exists 
between  exercises  of  monitoring  and  evaluation, 
that 
monitoring  indicators  are  useful  for  evaluation. 

in  the  sense 

Risks of conflict  between  the three  approaches  are  nevertheless  important. 
In  particular,  an  evaluation  must  not  be  performed  with  audit  methods  or  it 
could  create  contradictions  which  hinder  the  collection  of  information  and 
the  use  of  conclusions.  From  another  point  of  view,  no  evaluation  can 
replace  audit  or  monitoring.  For  example,  in  case  of  doubt  as  to  the 
regularity  or  professionalism  of  the  implementation,  it  is  necessary  to  carry 
out  an  inspection  or  an  audit  in  order  to  take  necessary  decisions,  and  to 
wait  for  the  situation  to  become  less  troubled  before  investing  in  an 
evaluation. 

Deciding to launch an evaluation 

Most  evaluations  which  fit  into  the  framework  of  European  economic  and 
social cohesion  policy  are of a mandatory  nature. This should not,  however, 
be  cause  for  underestimating  the  role  of  the  decision  to  launch  an 
evaluation.  This  decision  is  an  opportunity  to  frame  the  work  to  be 
undertaken,  by  means  of  a  precise  definition  of  its  scope  (i.e. what  has  to 
be evaluated), its schedule and, above all, its real motivations apart from the 
need to  comply  with  regulations. 

formulating  a  clear  and 

By 
the  public  authorities 
commissioning  the  evaluation  will  give  it  full  political  legitimacy.  They 
appoint  the  people  who  are  to  participate  in  the  evaluation  process,  by 
distributing  responsibilities  and setting  the  rules of the  game. 

firm  decision, 

When  evaluation  is  mandatory,  there  is  a  risk  of  the  commissioning 
authorities  being tempted  by the easiest  solution, which  consists  of  basing 
their decisions on established models, without enough thought to what they 
can really expect  from the evaluation  in the context  of their  programme  and 
in the  limits  of the  political  schedule.  As a  result, evaluation  could  become 
a  uniform, bureaucratic  procedure.  Several  recognised  authors  in this  field 
affirm  that,  on  the  contrary,  each  evaluation  must  be  considered  unique 
(Patton, 1986) because  it has to  meet expectations  which  change with time 
and  the  context,  but  also  because  experience  has  shown  that  there  is  no 
universal  evaluation  method. 

Defining  the  scope 

The  decision  to  evaluate  is  an  opportunity  to  define  the  limits  of  the 
programme  in  terms  of  institutional,  temporal,  sectoral  and  geographical 
dimensions.  This  is  what  is  known  as  the  scope  of  the  evaluation  or  the 
"evaluanda".  Defining  the  scope  of  an  evaluation  amounts  to  asking  the 
question:  "What  is going to  be evaluated?". The field  of  investigation  must 
be  delimited  in  at  least  four  respects:  institutional  (European,  national  or 
local 
level),  temporal  (period  under  consideration),  sectoral  (social, 
industrial,  environmental,  rural,  etc.)  and  geographical  (which  part  of  the 
European territory, which  region, town, nature  reserve, etc.). 

Within  the  framework  of  the  Volume  scope  is,  theoretically,  delimited  by 
financing of the programme to be evaluated, by the territory  concerned  and 

Evaluation design and management 

by the programming period. It is, however, useful to have first considered 
the following points: 

• Is the intention to limit evaluation to the funding of the programme or to 
include other national, regional or local funding which is, to a greater or 
lesser degree, directly related to the programme? 

• Is the intention to limit the evaluation to interventions in the eligible area 

or to extend observations to certain neighbouring areas which encounter 
similar development problems? 

• Is the intention to limit the evaluation to funding allocated to the 

programming cycle under consideration or to a certain extent to include 
funding of preceding cycles? 

It is necessary to stress the importance of a relatively strict definition of the 
scope of the evaluation. Experience has shown that during the evaluation 
process participants tend to want to examine, to a greater or lesser degree, 
everything concerning the programme. To reach conclusions, the 
evaluation must be reduced to an examination of the programme and its 
most essential interdependences with other public policies. 

■^ The risk is particularly great in the case of ex ante evaluations which can 
turn into an exercise in forecasting or speculation which is far from the 
object of the evaluation. In this case, limiting the scope of the evaluation 
strictly to the programme proposal is recommended. 

A n n o u n c i ng t he m o t i v es a nd p l a n n ed use 

When an evaluation is launched, it is strongly recommended to specify the 
motives underlying the decision to evaluate, and to draw up a list of 
potential users. Even if the evaluation is compulsory, this exercise is a way 
of ensuring that no work is carried out unnecessarily. It is also important to 
specify how far the evaluation team is expected to involve itself. Are 
expectations limited to mere conclusions, or is there a hope of obtaining 
recommendations of an operational or strategic nature? The expectations 
are obviously different, depending on the stage at which the evaluation 
takes place (ex ante, intermediate or ex post). 

It is very important to note that, in order to obtain a useful evaluation, the 
decisions likely to be taken and which can be informed by the evaluation, 
must be stated as precisely as possible. Often commissioners, not wanting 
to influence the evaluation team too much, are reluctant to express in 

Preparing an evaluation and defining its content 

advance the changes they think  should  be made to the programme or their 
doubts  concerning  the effectiveness  of  a particular  action. The intention  is 
commendable:  reveal  nothing  in  advance  to  see  whether  the  evaluation 
team  reaches  the  same  conclusions!  Experience  shows,  however,  that 
evaluation  has little chance of documenting  intended decisions  if these are 
not  known  in advance  by those  who  are going  to collect  and  analyse  data 
in  the  field.  Socio-economic  reality  is  in  fact  highly  complex  and  the 
evaluation  team  is  confronted  with  a  large  number  of  observations  and 
possibilities  for  making  improvements.  It  may  therefore  happen  that 
attempts  are  made  to  verify  hypotheses  which  may  be  interesting  in  the 
absolute,  but  are  hardly  realistic  in  the  eyes  of  political  officials  or 
managers, given  the existing  margins for  manoeuvre  or  emergencies. 

Distributing  roles  within  the  setting  of  an  evaluation 

The  evaluation  setting  refers  to  the  roles  and  responsibilities  within  the 
evaluation  process,  and  relations  between  evaluation,  monitoring  and 
management  of the  programme.  Clear  and fair  rules of  conduct  contribute 
towards  quality  work  and  avoid  blockages. The  main  choice  relates  to  the 
steering  of  the  evaluation,  a role which  includes,  in  particular,  the  drawing 
up  of  the  terms  of  reference, the  selection  of  the  evaluation  team  and  the 
validation  of  its  report. 

Box  11 -  The  evaluation  setting 

Commissioning 
public  authorities 

Steering group 

6. Decisions, directions 
communication 

Concerned 
groups 

Operators 

Evaluation team 

Evaluation  design  and  management 

In the  past, the  steering  of  evaluations  was  often  either  entrusted  to  a single 
official  or  exercised  collectively  by  the  co-funders  of  the  programme  within 
the  framework  of  formal  meetings  held  infrequently  and  attended  by 
In  practice,  the  system 
numerous  partners  (Monitoring  Committees). 
functions  well  when  the  responsibility 
is 
for  steering 
informally,  to  a  small  group  of  officials  from  the 
entrusted,  relatively 
administrations  most  directly  concerned  by  the  programme,  sometimes 
supported  by a few experts. Meetings  must be held frequently (once a month 
at certain  times)  and the working  procedures  must  be flexible  and  informal. 

the  programme 

Schematically,  (see  Box  11), the  setting  of  the  evaluation  can  be reduced  to 
the  distribution  of  roles  between  the  following  five  key  players:  the 
commissioners,  the  steering  group, the  evaluation  team, the  operators  and 
the  publics  concerned.  In some  situations  other  entities  need  to  be  added. 
For  example,  in  some  cases  the  steering  of  an  evaluation  has  successfully 
been  organised  on  the  basis  of  a  global  committee  coordinating  several 
sectoral  committees  (or  committees  created  on  a  smaller  geographical 
scale). The responsibilities  of these  entities are described  by their  respective 
roles  during  the  six  main  steps  comprising  the  evaluation  process: 

•  Step  1:  The  public  authorities,  commissioners  and  future  users  of  the 
evaluation  formulate  their  instructions  to  a  steering  group.  This  step  is 
materialised  at  least  by  a  paragraph  in  a  programming  document.  It 
could, 
in  due  form,  as 
recommended  in the  concluding  chapter  of  this  Volume. 

in  the  future,  be  set  out 

in  a  mandate 

•  Step 2: Tne terms  of  reference are drawn  up  by the steering  group on  the 
basis  of  the  mandate  given to  it. This document  usually  serves  in the  call 
for  tenders  and  in  the  selection  of  the  evaluation  team.  The  steering 
group  will  interact  with  the  evaluation  team  as  much  as  is  necessary  to 
ensure that the team  does  not  diverge  from  the guidelines  in the terms  of 
reference  and that  it  works  towards  answering  the  questions  asked. 

•  Step  3: The  evaluation  team  collects  secondary  and  primary  information 
from  the  operators  and  publics  concerned, and then  analyses them. This 
is the  longest  step. 

•  Step  4:  A  report  which  can  be  qualified  as  "technical"  is  written  by  the 
evaluation  team.  It  includes  conclusions  and, where  relevant,  preliminary 
recommendations  for  the  steering  group. 

•  Step  5:  The  conclusions  are  partly  or  entirely  validated  by  the  steering 
group. This group furthermore  formulates  recommendations  to the  public 

Preparing an evaluation and defining its content 

authorities  in  the  evaluation  report  which  either  includes  the  technical 
report  or  presents  it  as an annex. 

•  Step  6: The authorities  use the evaluation  by disseminating  it  among  the 
public  concerned  and/or  by  deciding  on  new  guidelines  that  the 
administrations  will  have to  implement. This crucial  step  is often  isolated 
from the evaluation  process  as such, because the steering  group  can  be 
dissolved  after  submitting  its  report  and  is therefore  unable  to  follow  up 
on  subsequent  decisions.  Nevertheless,  it  is  possible  to  ensure  that  at 
least  some  follow-up  occurs,  by  including  a  commitment  in  the  initial 
mandate by which the authorities undertake to request, if not the  steering 
group then at least its President, to present the conclusions and to  attend 
the  meeting  at which  decisions  are taken. 

towards 

•^  It  is  important  to  emphasise  that,  in  the  past,  evaluation  settings 
gradually  converged 
intermediate 
evaluations. This was not, however, the case with ex ante evaluations. Time 
constraints  and  the  difficulty  of  mobilising  protagonists  with  regard  to  a 
programme  that  did  not  yet  exist,  may  explain  the  absence  of  a  steering 
group, without  necessarily justifying  it. 

the  above  schema 

for 

►  The  same  shortcomings  are  found  in  ex  post  evaluation  although  the 
same difficulties  do  not  exist. This type  of  evaluation  is often  requested  by 
an  administration  without  the  necessary  care  being  taken  to  clarify 
expectations  at  the  most  important  decision-making 
levels  or  in  the 
steering  of the  process. 

Formulating  the  evaluative  questions 

List  of  possible  questions 

Probably the most fundamental aspect  in the design phase of an evaluation 
is  the  formulation  of  evaluative  questions.  It  is  these  questions  which 
constitute the core of the evaluation  exercise. 

Possible  questions  include: descriptive  questions  intended to  observe  and 
measure  changes  (what  happened?),  causal  questions  which  strive  to 
understand  and  assess  relations  of  cause  and  effect  (how  and  to  what 
extent  is that  which  occurred  attributable  to  the  programme?)  and, finally, 
normative  questions  which  apply  evaluation  criteria  (are  the  results  and 
impacts  satisfactory?).  Ideally,  any  evaluative  question  must  encompass 
these three  dimensions  and  have the following  qualities: 

Evaluation design and management 

The question must correspond to a real need for information, understanding 
and/or identification of new solutions, otherwise it will merely be an "alibi 
question", aimed, for example, at endorsing a decision already taken. If a 
question is of interest in terms of new knowledge only, without an 
immediate effect on decision-making or public debate, it is more a matter 
of scientific research and should not be included in an evaluation. 

The question concerns an impact, a group of impacts, a result or a need. 
That is to say, it concerns, at least partly, elements outside the programme, 
notably its addressees or its economic and social environment. If a question 
concerns only the internal management of resources and outputs, it can 
probably be treated more efficiently in the course of monitoring or audit. 

The question concerns only one judgement criterion. This criterion may 
sometimes be difficult to meet, but experience has shown that it is a key 
factor in the usefulness of the evaluation. Without judgement criteria clearly 
stated from the outset, the final report rarely provides conclusions. 

Example: Some evaluative questions relating to the European Structural Funds 
Has support to scientific and technical centres initiated the catching up of regional 
SMEs in terms of innovation? 
Has the programme created or maintained an adequate number of jobs compared to 
its cost? 
Do the outputs and results of the programme correspond to the specific needs and 
expectations of women? 
Which measures have the most positive (or negative) impacts on the environment? 

Evaluation criteria 

When they include a judgement criterion, evaluative questions fall primarily 
into one of the following four categories: those related to the relevance of 
the programme, those related to its effectiveness, those related to its 
efficiency and those related to its utility. These four main categories are 
represented in the following diagram. 

■^ M The term "relevance", in the context of an evaluation, refers to the 
appropriateness of the explicit objectives of the programme in relation to the 
socio-economic problems it is supposed to address. In ex ante evaluation, 
questions of relevance are the most important because the focus is on the 
choosing the best strategy or on justifying the one proposed. In intermediate 

Preparing an evaluation and defining its content 

evaluation,  the  aim  is  to  check  whether  the  socio-economic  context  has 
evolved  as  expected  and  whether  this  evolution  calls  into  question  a 
particular  objective. 

Box  12 -  Main evaluation  criteria 

Needs 
Problems 
Issues 

II 

|| 

Society 
Economy 
Environ­
ment 

Programme 

Objectives 

Outreach 

,—t-
Ζ 

Outputs 

Evaluation 

Relevance 

Efficiency 

Effectiveness 

Utility 

Η  ►  The  basic  questions  of  intermediate  evaluations  and,  more 
particularly, of ex post  evaluations, concern the effectiveness and  efficiency 
of  the  interventions  implemented  and  of  the  entire  programme.  As  far  as 
effectiveness  is concerned, the question  is knowing  whether the  objectives 
formulated  in the  programme  are  being  achieved, what  the  successes  and 
difficulties  have  been,  and  how  appropriate  the  solutions  chosen  or  the 
influence of  confounding  factors  outside the  programme are. 

In the context  of evaluation, efficiency  is assessed by comparing the  results 
obtained  or, preferably, the impacts  produced, and the resources mobilised. 
In  other  words,  are  the  effects  obtained  equal  to  the  inputs?  The  terms 
"efficiency"  and "effectiveness"  are, moreover, commonly  used in a different 
sense  by  managers  who  will  seek,  within  the  framework  of  monitoring,  to 
analyse outputs  rather than  results or  impacts.  In this  respect  it  is  important 
to  emphasise  that  questions  concerning  the  performance  of  a  programme 
are  increasingly  frequent.  This  notion  encompasses  those  of  effectiveness 
and  efficiency  as  defined  both  for  monitoring  and  evaluation.  Thus,  it  is  a 
fairly  large set  of  questions  that  will  be grouped  under this  heading. 

The  criterion  of  utility  judges  the  impacts  obtained  by  the  programme  in 
relation  to  society's  needs  and  economic  issues.  Utility  is  a very  particular 

Evaluation design and management 

evaluation  criterion  in  so  far  as  it  makes  no  reference  to  the  official 
objectives  of the  programme.  It may  be judicious to  formulate  a question  of 
utility  when  the  objectives  are  badly  defined  or  when  there  are  many 
unexpected  impacts. This  criterion  must  nevertheless  be  used  with  caution 
to  ensure  that  the  evaluation  team's  selection  of  important  needs  or  issues 
is  not  too  personal.  Some  North  American  authors  have  advocated  this 
type  of  goal  free  evaluation. 

The four categories  of questions can  be characterised  simply  and  concisely 
as  follows: 

•  Relevance:  To  what  extent  are  the  programme  objectives  justified  in 
relation  to  needs?  Can  their  raison  d'être  still  be  proved?  Do  they 
correspond  to  local, national  and  European  priorities? 

•  Effectiveness:  To  what  extent  have  the  expected  objectives  been 
achieved?  Have  the  instruments  used  produced  the  expected  effects? 
Could  more  effects  be  obtained  by  using  different  instruments? 

•  Efficiency:  Have the  objectives  been  achieved  at  the  lowest  cost?  Could 

better  effects  be  obtained  at  the  same  cost? 

•  Utility: Are  the  expected  or  unexpected  effects  globally  satisfactory  from 

the  poini  of  view  of  direct  or  indirect  addressees? 

These  four  criteria  are  not  exclusive.  For  example,  a typical  question  might 
be:  Will  the  effects  obtained  last  after  the  end  of  the  programme,  even  if 
there  is  no  more  public  funding?  The  criterion  here  will  be  that  of  the 
sustainability  of  the  impacts  produced.  Other  criteria  such  as  equity, 
coherence,  synergy,  reproducibility,  etc.  may  also  be  used. 

Probability  of  an  answer 

Once  the  evaluative  questions  have  been  identified, their  evaluability  has  to 
be considered. A prior assessment  has to  be made of whether the  evaluative 
questions  are  likely  to  be  answered,  given  the  available  resources.  In 
concrete  terms  it  is  a  matter  of  ensuring  that  an  evaluation  team,  with  the 
available  time  and  resources  and  using  appropriate  evaluation  tools, will  be 
able to find  all the  elements  needed for the construction  of  credible  answers 
to  the  questions  asked.  In  order  to  know  whether  it  is  possible  to  attain  an 
acceptable  degree  of  credibility,  an  evaluability  study  can  be  carried  out. 
This  point  is  considered  in the  last  part  of  the  book  (page  199)  devoted  to 
possible  improvements  to  existing  practices. 

Preparing an evaluation and defining its content 

For each evaluative question it is recommended  to check,  even very  briefly: 

•  whether  the concepts  are stable, 

•  whether  explanatory  hypotheses  can  be formulated, 

•  whether  available  data  can  be  used to  answer  the  question, without  any 

further  investigation, 

•  whether  access to the field will  pose  major  problems. 

A number of factors can make a question very difficult  to answer, e.g. if the 
programme  is very new, if it has not yet  produced significant  results or if the 
available  data  are  totally  inappropriate.  These  reasons  may  lead  to  the 
decision  not  to  undertake  the  evaluation,  to  postpone  it,  or  to  ask  more 
realistic  questions. 

Selecting  priority  questions 

Because trying  to  answer  all the  questions  in depth  would  be too  costly  in 
time  and effort,  it  is always  recommended  that the evaluative  questions  be 
graded. 

Box  13 -  Selecting  priority  questions 

Stakeholders 
Types  of  decisions 
to  be  taken 
Political  context 

^ 

^ 

J 

Initial  list  of  evaluative 
questions: 

Probability  of  answer 

Probability  of  use 

v^ f 

> 

Priority 
questions 

Evaluation design and management 

In order  to  do  so, it  is  necessary  to  assess the  issues  associated  with  each 
of these  questions  (knowledge,  negotiation, decision, and  communication). 
The  probabilities  of  finding  answers  to  the  questions  and  of  using  these 
answers  must  guide the  choice  of  priority  questions. This  choice  will  be  far 
more  "successful"  when  potential  users  have  been  consulted  and  have 
appropriated  the  selected  questions. 

It  is  obviously  better  to  obtain  slightly  inaccurate  answers  to  important 
questions  than  accurate  answers  to  questions  that  nobody  asks  and  which 
a  fortiori  will  never  be taken  into  account. 

As  Box  13  shows,  a  question  is  considered  a  priority  if  the  probability  of 
obtaining  an  answer  and  of  this  answer  being  used  is  high.  For  this 
purpose,  the  following  questions  must  be  asked: 

•  Will  the  conclusions  be  used?  By  whom?  For  what  purpose  (deciding, 

debating,  informing)?  When? 

•  Is it  politically  appropriate to  perform  such  an evaluation  at this  particular 
time  or  in  this  particular  context?  Is  there  a  conflictual  situation  which 
could  compromise  the  success  of  the  exercise? 

•  Has  a  recent  study  not  already  answered  most  of  the  questions? 

•  Are  evaluation  priorities  not  too  unstable? 

If  necessary,  these  points  can  be  studied  prior  to  the  evaluation  (see 
page  185). 

Ordering an evaluation 

The  terms  of  reference  reflects  the  decision  to  launch  an  evaluation,  in 
terms  which  are  sufficiently  precise  so  that  several  evaluation  teams  can 
compete  and  so  that  the  work  may  be  performed  in  good  conditions.  The 
drawing  up  of the terms  of  reference  therefore  requires  particular  care. 

The  terms  of  reference  recapitulates  the  origin,  scope  and  aim  of  the 
evaluation, as well as the distribution of  roles. It describes the content  of  the 
programme  to  be  evaluated  and  specifies  information  which  is  already 
available and will be made available to the evaluation team. It states  precisely 
a limited number  of questions to which the evaluation  must  provide answers. 
It also outlines the evaluation method and sets the steps in the schedule. The 
document  may  indicate  the  budget  earmarked  for  this  purpose. 

The  evaluation  team  is  chosen  after  a  call  for  tenders.  Independence  and 
competence  are two  essential  criteria  in the  selection  of  a team. 

Carefully  writing  the  t e r ms  of  reference 

The  terms  of  reference  is  the  document  which  serves  as  the  basis  of  a 
contractual  relationship  between the commissioner  of an evaluation and the 
team  responsible  for  carrying  out  the  work.  Constructing  the  terms  of 
reference  clearly  is a vital  step  when  an evaluation  has to  be  performed  by 
outside  consultants. This work  is nevertheless  equally  important  when  part 
of  the  evaluation  is performed  in-house,  although  it  is often  believed  to  be 
unnecessary  in such  cases. The terms  of  reference  may  concern  either  the 
evaluation  operation  as a whole  (when  it  is entrusted  to  a single team)  or a 
part  of  the  research  work  programmed  in  the  evaluation  project  (in-depth 
analysis  of  an evaluative  question). 

Example: an  evaluation  consisting  of several terms  of  reference 

Within  the  framework  of  their  development  programme,  the  authorities  of  a  region 
chose  to  divide  up  the  work  into  separate  batches  according  to  the  technical 
capacities  required  to  carry 
it  out:  (1)  questionnaire  survey  on  a  sample  of 
addressees, (2) interviews  with  managers  and operators  and  (3) economic  expertise. 
Other  evaluations  are  broken  down  by  territory  (e.g.  rural  development  /  urban 
development)  or  by  sector  of  intervention  (e.g.:  technology  /  training  /  tourism). 
Note  that  this  type  of  division  means  that  the  commissioner  is  fully  responsible  for 
methodological  choices  and for  the  synthesis  of  results. 

Evaluation design and management 

The terms  of reference is a 5 to 20 page document,  generally  complemented 
by  administrative  annexes.  Its  contents  are  described  in  Box  14.  Only  the 
points  which  have  not  already  been  considered  are  explained  below. 

Box  14 -  Standard  layout  of the  terms  of  reference 

1  -  Regulatory framework 
2  - Scope of the evaluation 
3  - Who is going to use the evaluation and what for? 
4  -  Evaluative questions and criteria 
5  -  Knowledge and available data 
6  -  Method (guidelines) 
7  -  Evaluating  setting 
8  -  Schedule 
9  -  Suggested  budget 
10  -  Required qualifications 
11  -  Structure of the proposal 

Regulatory  framework  of  an  evaluation 

For the  benefit  of the  team  responsible  for  the evaluation, it  is necessary  to 
recall the  context  in which  it  is situated. Although  the  legal  and  contractual 
foundations  are set out  in the regulations  of the  European  Union and  in each 
programming  document,  it  is advisable  to  refer  to  them. 

The  terms  of  reference  must  specify  who  initiated  the  evaluation  project 
and,  where  relevant,  who  was  involved  in  formulating  the  order  for  the 
evaluation, as well as the  motives for the decision to evaluate, if they are not 
purely  regulatory. 

Moreover,  the  aim  of  an  evaluation, that  is to  say, the  reasons  for  which  an 
evaluation  is requested, must  be clearly  stated. Is the  intention  a change  of 
direction?  Why?  Is  the  intention  to  modify  the  monitoring  or  arbitration 
procedures?  Is the  intention  to  reallocate  funds? 

Available  knowledge 

The  terms  of  reference  contain  a  review  of  the  current  state  of  knowledge 
on  the  programme  and  its  effects.  The  elements  in  this  inventory  include 
extracts  or  references  of  programming  documents, 
lists  of  previous 
analyses  and  evaluations  with  important  extracts,  a  description  of  the 

Preparing an evaluation and defining its content 

monitoring  system, quantified  indicators,  and the  various  reports  and  data 
bases  available from  the  services  managing  the  programme. 

is  essential  for  the  candidate  teams  to  adjust 

This  inventory 
their 
methodological  proposals  in  relation  to  the  size  and  nature  of  the  data 
collection  and  interpretation  work  which  will  have  to  be  organised  during 
the evaluation  exercise. 

Principal  outlines  of the  method 

Each  evaluation  uses  an  ad  hoc  method  constructed  with 
the 
commissioner's  agreement,  on the  basis  of  the  selected  evaluation  team's 
proposal. At the terms  of  reference stage, the choice  is generally  made  not 
to  describe  the  method  completely,  but  to  outline  it,  as  shown  in  the 
diagram  in  Box  15. The  aim  is not  to  confine  the  candidate  teams  to  strict 
methodological  constraints,  but  to  state  a  small  number  of  requirements 
that  the  commissioner  considers  essential  for  obtaining  answers  to  the 
evaluative  questions.  Should  an  inventory  be  made  of  evaluations  and 
research  reports  on  similar  subjects?  Will  there  be  a  survey  on  the 
addressees  or not? Will there be a comparative  analysis or not?  Should the 
evaluation  team  look  for  judgement  references  or  will  these  references  be 
provided  by the steering  group? 

The  choice  is  generally  made  to  maintain  sufficient  flexibility  to  allow 
candidate  teams  to  differentiate  themselves  in  terms  of  the  relevance  and 
clarity  of  their  methodological  proposals. 

Box 15 -  Planning an evaluation method at the terms of reference stage 

First  evaluative  questions 

-—  r u ai  c 

^ ^ — ^ - E x p l a n a t i on  of  questions 

Evaluability  study,  methodological  proposals 

Priority  questions 

Verification  of  compatibility  with  budget 

Evaluation design and management 

Schedule 

The  evaluation  schedule  is  established  by  taking  into  account  various 
constraints,  especially  those  concerning  the  decision-making  schedule.  As 
Box  16  shows,  it  is  also  necessary  to  integrate  the  main  deadlines, 
generated  by  the  procedures  of  calls  for  tenders  and  by  the  phases  of 
primary  data  collection, and  which  cannot  be  reduced. 

Box  16 -  Constraints  bearing  on the  evaluation  schedule 

D e c i s i o n - m a k i ng 
s c h e d u le 

P r e p a r a t i on 

Decision 

Evaluation 

L a u n ch 
Terms  of 
reference 

Call  for  t e n d e rs 
Selection 

O u t p u ts 

Validation 

Dissemination 

Utilisation 

Indicative  budget 

The  financial  resources  available  for  a  study  partly  determine  the  limits  of 
an  evaluation,  especially  the  number  of  issues  that  will  be  addressed  and 
the  extent  to  which  they  will  be  analysed.  When  possible,  the  evaluation 
budget  should  be  established  by  the  commissioner  by  considering  the 
interest  and  difficulty  of  the  questions  asked.  To  establish  a  suggested 

Preparing an evaluation and defining  its  content 

budget,  a "top-down" or a  "bottom-up"  approach, or a combination  of  the 
two,  can  be adopted. 

to  1.4%  (Review  of  ongoing  evaluation  Function 

A  "top-down"  approach  consists  of  determining  the  evaluation  budget  in 
relation to the  budget  of the  programme  to  be evaluated.  Some  references 
can  be  used  in  this  approach.  A  review  of  evaluations  conducted  in  the 
framework  of  the  Irish  CSF  for  the  1994-1999  period  found  that  the 
annualised  cost  of  evaluation  in  relation  to  annual  OP  expenditure  ranged 
from  0.6% 
in  the 
Community  Support  Framework  (CSF)  for  Ireland,  1994-1999,  published 
1998).  These  figures  are  far  higher  than  current  spending  devoted  to  the 
evaluation of  European Structural  Funds, which  is globally  around 0.02%  of 
the  programmed  funds.  In certain  contexts, the  cost  of  the  evaluation  was 
as  high  as  10%  or  even  50%  of  the  cost  of  the  evaluated  action.  Such 
percentages  can  be  justified  for  highly  innovative  interventions  or  for 
experimental  operations,  but  it  is  not  relevant  to  refer  to  the  cost  of  the 
evaluated  intervention.  A  better  reference  is  the  cost  of  the  programme 
which  will  be  decided  on  if  the  evaluation  concludes  that  the  experiment 
was successful. 

The main factors which  can justify  a small  or large evaluation  budget  are as 
follows: 

•  a major, expensive  policy  which  is  both  simple  and  routine (reduced %); 

•  the  programme  is  frequently  evaluated  (high  %  for  all  the  evaluations 

together,  but  reduced  %  for  an  individual  evaluation); 

•  a regulatory  policy  with  significant  effects  and  minimum  costs  (high %); 

•  an innovative  or  experimental  programme  (high %). 

The "bottom-up" approach consists of estimating the probable cost  of each 
evaluated  question.  The  cost  of  an  evaluation  consists  of  two  main 
categories  of  spending:  general  costs  (structuring  the  evaluation  and 
producing  a  report)  and  costs  concerning  the  collection  and  analysis  of 
data.  General  costs 
include  expenditure  related  to  the  analysis  of 
programme  documents  and  monitoring  data,  an  overview  of  all 
documentation,  interviews  and  meetings  with  managers  and  the  main 
stakeholders,  and  the  establishment  and  presentation  of  intermediate  and 
final  reports  and  discussions  on  them.  The  second  category  of  spending 
concerns case studies, focus  groups,  individual  interviews,  questionnaires, 
statistical  analyses,  etc.  The  cost  and  duration  of  the  implementation  of 
some  evaluation  techniques  can  be estimated  as follows: 

Evaluation design and management 

Box 17 -  Indications of the cost of some evaluation techniques 

Questionnaire  surveys 

Surveys  based  on  individual  interviews 

Surveys  based  on  focus  groups 

Surveys  based  on  case  studies 

Statistical  analysis 

Expert  Panels 

Use  of  an  econometric  model 

Development  of  an  econometric  model 

Cost 
(person  /  month) 

Duration 

Medium 

Medium 

Low 

Medium 

Low 

Medium 

Low 

High 

XXX 

XX 

XX 

XXX 

XX 

xxxx 

XX 

XXXXXX 

Χ = approximately  1 month 

In reality, the  cost  of  evaluating  a  programme  varies  considerably  from  one 
case to another. The suggested  budget  is often  fixed  by combining  bottom-
up  and  top-down  approaches  and  taking  into  account  significant  factors 
such  as: 

•  the  number  of  questions  to  consider  in  depth.  In  general,  the  cost  of 
collecting  and  analysing  data  is  proportional  to  the  number  of  questions 
studied  in depth,  but  it  is sometimes  possible  to  group  together  some  of 
these  analyses; 

•  expectations  in  terms  of  credibility.  The  soundest  evaluation  methods 
tend  to  be the  most  expensive  ones,  although  this  is  not  necessarily  the 
case; 

•  the  quality  of  the  monitoring  system  and  the  existence  of  recordings  of 

outputs  and  results; 

•  the  cost  of  time  devoted 

the 
production  of  data  within  the  administration.  These  costs  are  often 
hidden  and  should  not  be  underestimated. 

to  steering  group  meetings  and  to 

S e l e c t i ng  t he  e v a l u a t i on  t e am 

The  evaluation  team  is  selected  by  taking  into  account  its  independence 
and  profile  and  the  methodological  qualities  of  its  proposal. 

Preparing an evaluation and defining  its  content 

Independence  of  the  evaluation  team 

in  a  context  of  joint  decision-making.  In  particular, 

The  independence  of  an  evaluation  or  of  the  evaluation  team  is  a  strict 
requirement 
the 
evaluation  team  must  not  find  itself  in  a  relationship  of  subordination  with 
the  managers  of  the  evaluated  programme.  This  is  especially  important  in 
intermediate  evaluations  or  when  the  evaluation  is  supposed  to  judge  the 
success  of  a programme.  In the  last chapter  of  this Volume  it  is shown  that 
the  concern  for  independence  must  not,  however,  prevent  the  evaluation 
team  from  establishing  close  relations  with  the  managers. The  principle  of 
independence  must  also  be  applied  with  discernment  when  it  is more  cost 
effective  to  entrust  part  of  the  work  to  the  administrations  responsible  for 
monitoring  the  programme.  This  is  the  case,  for  example,  when  certain 
management  data  have  to  be  found  and  processed  for  the  first  time.  In 
some circumstances, skilful mixes of  internal and external contributions  are 
probably  more  productive  than  entrusting  the  entire  exercise  to  outside 
professionals.  The evaluation  team  should  nevertheless  be  in  a position  to 
control  the  impartiality  and  quality  of  the entire  evaluation. 

The independence of the evaluation  is not entirely guaranteed  by the fact of 
it  being  entrusted  to  an outside  team. The  members  of  an evaluation  team 
may  be  subjected  to  multiple  pressures  during  the  programming  of  the 
work, the collection of information and the validation  of the report. The team 
does  not always  have enough weight to resist  pressure, especially  if it  lacks 
familiarity  with  evaluation,  cannot  rely  on  sound  data  or  does  not  have  the 
time to  make complete  analyses.  In all cases, the steering  group can  play a 
very  useful  complementary  role  in  improving  the  independence  of  the 
evaluation,  particularly  if  it  includes  one or  more outside  experts. 

In some cases Structural  Fund programmes  undergo a series of  evaluations 
by  the  same  team,  for  example  several  in-depth  analyses  and  an 
is  usually  entrusted  to  a  firm  of 
intermediate  evaluation.  The  work 
consultants  or  a  research  centre,  but  in  some  cases  evaluation  units  are 
formed for this purpose, alongside the national  administration  (e.g. Ireland). 
In these  situations,  the  evaluation  team  works  for  several  years  in  contact 
with  the  programme  managers.  Whether  the team  is  internal  or  external, it 
manages  far  more successfully  to  maintain  its independence  when there  is 
a well-structured  steering  group  comprising  outside  experts. A  particularly 
interesting  solution  consists  of  calling  on  evaluators  from  outside  the 
country  under  consideration,  even  if  linguistic  differences  often  limit  this 
type  of  practice. 

Evaluation design and management 

Profile  of  evaluation  teams 

In the field  of  European  Structural  Funds,  more  and  more  organisations  are 
present  in  the  evaluation  market.  The  service  providers  who  are  selected 
belong to  local, national  or  international  consultancy  firms. The  commercial 
sector  accounts  for  most  of the  interventions  but the  presence  of  university 
research  centres  is  also  significant.  Each  evaluation  team  has  its  own 
profile,  depending  on  its  basic  profession.  An  important  distinction  relates 
to  the  difference  between  evaluation  and  research. 

The  purpose  of  research,  particularly  fundamental  research,  is  mainly  to 
"advance"  knowledge;  its  aim  is  therefore  cognitive.  Theoretically,  it  is  not 
concerned  with  changing  behaviour  or, at  least, its approach  is not  aimed  at 
producing  immediately  useful  results.  Although  evaluation  uses  the  same 
methodological  rules  as  research  (in  sociology,  economics,  etc.),  it  is 
primarily  intended  to  directly  serve  actors  and  to  make judgements.  Its  aim 
is therefore  normative  and  instrumental. Its main vocation  is to  be useful  and 
to  be  used. This  basic  distinction  between  the  "research"  and  "evaluation" 
approaches  explains  why,  even with  precise terms  of  reference, opting  for  a 
consultancy  firm  or a  university  laboratory  can  have  implications  in terms  of 
the  approach  and therefore  of the  results  of the  evaluation. Academics  have 
the  advantage  of  being  perceived  as  independent  and  highly  credible  owing 
to  their  academic  requirements.  On  the  other  hand,  private  firms  are  often 
more  readily  available  as far  as  time  is  concerned  and  are  more  concerned 
with  meeting  the  commissioner's  expectations. 

When  the  steering  group  puts  out  a  call  for  tenders,  it  must  specify  the 
required  competencies.  While determining  the  main  lines of the  method, an 
evaluability  study  may  define  these  competencies  and  specify  whether  the 
evaluation  needs  experts  specialised  in  the  fields  of  intervention  of  the 
labour  market,  rural 
programme  (regional  sciences,  macro-economics, 
development,  etc.)  or  whether  the  need  is  rather  for  general  evaluation 
know-how  or  specific  methodological  skills  (survey,  audit, 
information 
systems,  causal  analysis,  etc.). 

In  general,  each  consultant  or  academic  has  his  or  her  strengths  in  certain 
technical  domains  of  evaluation  and  his  or  her  weaknesses  in  others. 
Cooperation  within  a  mixed  team  is  often  a  solution,  provided  that  the 
coordination  is  effective.  If  the  candidate  teams  are  versatile  enough,  it  is 
possible  to  maintain  a  competitive  situation  and  to  avoid  using  the  same 
experts  repeatedly.  This  situation  is the  best  for  ensuring  independence. 

Preparing an evaluation and defining its  content 

Methodological  qualities  of  the  proposal 

The selection of a team  is never  independent  of the criterion of  price. In this 
case,  the  evaluation  method  must  be  determined  in  great  detail  and  the 
candidate  teams  pre-selected  according  to  their  technical  skills.  This 
method  forces  the  commissioner  and  the  steering  group  to  carry  out  a 
considerable  amount  of  methodological  work  upstream. 

When  possible  from  an administrative  point  of view,  it  is preferable to  work 
the  other  way  round,  that  is  to  say,  to  determine  a  budget  or  volume  of 
work,  to  describe  only  the  main  lines  of  the  method  in  the  terms  of 
reference, and then to select the team which  proposes the most  interesting 
method. The  selection  of  the  team  will  then  require the  ability  to judge  the 
methodological  quality  of  a tender. This quality  is not  necessarily  related  to 
the  use  of  obscure  terminology  and  complicated  models.  The  proposed 
method  must,  above all, be suited  to  answering  the  questions  asked  in the 
terms  of  reference. The  steering  group  can  lead this  work  by  checking  the 
points  presented  in Box  18 for  each  of the tenders. 

Box  18  -  Examining  the  methodological  quality  of  a  proposal 

For  each  evaluative 

question 

Question  1 

Question  2 

The proposition  of  the  candidate 
team: 

Does  it  include  the  collection  of 
information 
sufficiently  relevant 

Is it  based  on  rigorous  analytical 
techniques? 

Is it  able  to  clarify  the  evaluation 
criteria  in an  impartial  manner? 

1st it  likely  to produce  credible 
findings? 

++ 

-

+ 

+ 

+ 

+ 

+/-

+ 

Was the  respective 
of  the  questions  well  understood? 

importance 

++ 

Ill 
Performing an evaluation 

Analysing the logic and content of the programme 

As  a  first  step,  the  evaluation  team  reconstitutes  the 
logic  of  the 
programme,  i.e.  the  needs  to  be  fulfilled,  the  strategy,  the  objectives,  the 
expected  effects,  the  inputs,  and  the  measures  taken  to  implement  and 
monitor  the  programme.  An  analysis  of  the  programme  logic  provides  an 
indispensable framework  for the study of  results and impacts. It is therefore 
a key  step  in any  evaluation. 

In  an  ex  ante  evaluation,  this  analysis  constitutes  the  bulk  of  the  work 
because  it  produces  a  judgement  on  the  validity  and  coherence  of  the 
programme  and  its  strategy. 
to 
recommendations  to  improve  the  monitoring  system  and  management 
methods. 

In  a  mid-term  evaluation,  it  leads 

In  an  ex  post  evaluation,  this  analysis  is  used  to  clarify  objectives  when 
these were not clearly expressed. This clarification  is necessary to  structure 
the  collection  of  information  and  analyse  impacts  more  effectively. 

In general, this step  is used to verify  the coherence  of  objectives  within  the 
same  intervention, and the  coherence  of  the  same  objectives  in  relation  to 
those of other interventions. The programme content  is examined  in light  of 
the  budget  allocated, the  modes  of  intervention  chosen  (projects  selected, 
participants'  responsibilities),  the  impacts  identified  and  the  monitoring 
system  put  into  place, in which the system of  indicators  plays an  important 
part. 

The  logic  of  a  programme  is  defined  as  the  connections  between  all  its 
elements,  i.e.  the  needs to  be met, the  strategy  chosen, the  objectives  set 
(global, specific,  operational), the  inputs, the  measures  taken to  implement 
it,  the  outputs  and  the  expected  effects  (results  and  impacts).  The 
articulation  of  all  these  elements  is, by  custom,  presented  in the  form  of  a 
logical series of steps called the "programming cycle", as shown in Box  19. 

Evaluation design and management 

Box 19 ­ T he  orograr 

nmi ng cycle 

r 

Needs 
issues 
Problems 

Objectives 

■ *­

' r 

Specific objectives 

­*­

J 

Operational 
objectives 

■ ♦­

V 

Resources. 
Implementation 
system 

*­

*­

»­

Λ 
ì 

Expected impacts 

41 

Expected results 

1 1 

Expected 
outputs 

;> J 

­^ Ex ante evaluation examines the logic of the proposed programme and 
judges whether it is clear, complete and coherent. This is its main purpose. 
The following pages therefore present a key step in the realisation of an ex 
ante evaluation. In the second stage, ex ante evaluation forecasts impacts. 

M  ►  In mid-term  and ex  post  evaluation the first  step  may  also  be  devoted 
to  the  programme  logic,  as the  diagram  in  Box  19  shows. This  is  obviously 
necessary  if  points  in the  initial  logic  are  obscure  or  poorly justified, that  is, 
if  the  ex  ante  evaluation  was  not  performed  correctly  or  if  its  conclusions 
were  not  taken  into  account.  It  may  happen  that  the  objectives  were 
amended  midway.  In that  case  it  is  necessary  to  present  a new  programme 
logic  at  the  beginning  of  the  evaluation  report  and  to  check  whether  the 
overall  coherence  of  the  programme  has  been  maintained  despite  these 
changes.  More  often,  the  needs  to  be  met  or  the  problems  to  be  solved 
have  changed  during  the  implementation  of  the  programme.  If  necessary, 
the  intermediate  or  ex  post  evaluation  re-examines  the  programme  logic  in 
light  of  this  new  context. 

When  the  evaluation  team  examines  the  logic  of  a  programme,  it  starts  by 
analysing  the  official  texts  (the  proposed  programme  in  the  case  of  an  ex 

Performing an evaluation 

ante  evaluation,  otherwise  the  approved  programme).  The  "official  logic" 
has to be clarified and its coherence evaluated. If the programme  managers 
agree, the evaluation may  be carried out in the form of a series of  interviews 
and/or  meetings in which the participants explain their understanding  of the 
strategy,  objectives  and expected  effects. The evaluation  team  then  has  to 
produce,  in a participative  manner,  a "reconstructed  logic". The first  option 
is quicker  and  has the  advantage  of adhering to  official texts, which  makes 
it  possible  to  report  on  a  legitimate  basis.  The  disadvantage  is  that  the 
evaluation  may  reveal  inconsistencies  or  inaccuracies  which  are  already 
known  to  the  decision-makers.  The second  option  requires  more  work  but 
produces  more  useful  conclusions  for  improving  the  programme. 

Assessing  the  validity  of  the  p r o g r a m me 

The  rationale  of  a  programme  is  to  satisfy  needs,  solve  problems,  avoid 
threats  and  grasp  opportunities.  The  needs  or  problems  considered  are  of 
a social, economic  or environmental  nature. They concern  a territory  as well 
as one or more groups  of  people  in that territory.  If public funding  has to  be 
justified, the  needs concerned  must  be those which  cannot  be satisfied  by 
the  market  and  the  problems  must  be  those  which  cannot  be  solved  by 
private  initiative. 

Prior  appraisal  of  socio-economic  problems 

-4 The evaluation team's initial and essential task  is to re-examine the social 
the 
needs  or  development  problems 
programme. The logic  that  supports  this justification  has to  be  investigated 
thoroughly  and  systematically,  especially 
in  the  case  of  an  ex  ante 
evaluation. 

the  existence  of 

justify 

that 

The  evaluation  details  the  socio-economic  handicaps  and  development 
problems  of  the  region  or  Member  State  under  consideration  that  warrant 
the attribution  of  support  within the framework  of the Structural  Funds. The 
evaluation team  must  also establish  a position on whether  a region's  needs 
actually  correspond  with  the  general  objectives  of  European  cohesion 
policy. 

H 
In  the  case  of  a  mid-term  evaluation,  the  evaluation  team  must  also 
examine  the  extent  to  which  socio-economic  needs  and  development 
problems are evolving, whether  they are worsening  or  improving, and try  to 

Evaluation design and management 

explain  why.  The  object  of  this  questioning  is  above  all  to  know  whether 
private  initiative  could  satisfy  these  needs  or  solve  these  problems,  in  the 
place  of,  and  as  effectively  as, the  considered  programme. 

the  example  below  shows,  depending  on 

As 
the  period  under 
consideration, certain  problems  can  be  requalified  as  being, to  a greater  or 
lesser  degree,  a matter  of  public  intervention. 

Example:  A  new  examination  of  the  rationale  of  interventions 

initiative;  those  which  offset  shortcomings 

In thè  mid-term  evaluation  of  a set  of  programmes  on  a national  level, the  evaluation 
team  sortee  the  interventions  into  four  main  categories,  depending  on  the  way  they 
could  be justified: those  which  produced  public  goods  which  could  not  be  provided 
by  private 
in  the  market;  specific 
subsidies;  and  distribution  measures.  Within  each  category,  the  evaluation  team 
noted  the validity  of each intervention  by  means of the same assessment  grid. On the 
basis of this evaluation, it appeared that  certain  measures  could  be considered  as no 
longer  having  any  justification.  Among  them  were  certain  incentives  for  the  tourist 
sector,  which  were  no  longer  of  use  due  to  the  fast  growth  of  the  industry.  By 
contrast,  other  measures  had  become  more  relevant,  for  example  the  improvement 
of  inter-urban  roads  and  the fight  against  dropping  out  of  school. 

Relevance  of  the  strategy  and  objectives 

-4  Establisning  the  strategy  of  a programme  amounts  to  prioritising  actions 
on  the  basis  of  the  most  important  needs  to  be  fulfilled, the  importance  of 
problems  to  be  solved,  and  the  probability  that  envisaged  actions  will  be 
successful.  The  formulation  of  a  strategy  amounts  to  selecting  and 
establishing  a  hierarchy  of  objectives  and  fixing  the  extent  of 
their 
ambitions.  Not  all  territories  have  the  same  development  strategy.  An  ex 
ante  evaluation  examines  whether  the  strategy  is  suited  to  the  current 
context  and  to  its  probable  evolution.  It  is  therefore  advisable  to  verify 
whether  the  programme  has  been  constructed  on  the  basis  of  objectives 
that  have  been  classified  according  to  their  degree  of  importance. 

The evolution  of  the  context  is characterised  by the  issues that  the  strategy 
has to  deal  with. The  strategy  could  be either  active  or  reactive,  depending 
on the  nature  of the  issue. There  could, for  example,  be a threat  that  had  to 
be  avoided,  a  possible  economic  obstacle  that  had  to  be  removed,  a 
weakness  that  had to  be  taken  into  account,  an  opportunity  that  had to  be 
grasped,  a  development  potential  that  needed  to  be  stimulated,  a  strength 

Performing an evaluation 

which  could  be  relied  on, etc. The  strategy  is composed  of  defensive  and 
offensive  objectives.  In  the  first  case,  the  programme  is  aimed  mainly  at 
combating  a weakness  or avoiding  a threat.  In the  latter case, it  is aimed  at 
building  on  a strength  or taking  advantage  of  an  opportunity. 

An evaluation considers  the  strategy  of  a programme  by trying to  ascertain 
whether  it  is  primarily  offensive  or  defensive,  or  possibly  mixed. A  purely 
defensive  strategy  comprises  the  risk  of  having  to  confront  new  problems 
in the long run by  introducing  the  idea that the territory  or group  concerned 
cannot  develop  without  outside  support.  A  purely  offensive  strategy  can 
cause  important  threats,  which  are  potentially  disastrous  for  development, 
to  be neglected. 

A technique  which  can  be used to  evaluate the strategy  of  a programme  is 
(Strengths-Weaknesses-Opportunities-Threats)  analysis.  This 
SWOT 
technique,  with  a  group  composed  of  as  many  stakeholders  as  possible, 
creates  a  formal  structure  which  helps  to  clarify  and  specify  in  detail  the 
strengths and weaknesses of a given (regional or sectoral) situation, and the 
existing or potential opportunities  and threats. The tool is described  in more 
detail  in Volume 3 of  the  MEANS  Collection. 

Example: SWOT  analysis 

SWOT  analysis  was  used  during  the  formulation  of  the  Objective  6  regional 
development  plan  in  Sweden.  The  results  produced  by  means  of  this  technique 
helped 
trends 
to  highlight  major  environmental,  structural  and  democratic 
characteristic  of  the  context  in  the  region  under  consideration.  The  major  factors 
identified  during this  analysis  are synthesised  below: 

Strengths:  Large protected  natural  areas and  substantial  natural  resources. 

Weaknesses:  Isolated  region,  low  population  density,  vulnerability  to  economic  and 
budgetary  fluctuations, structural  weaknesses  of  firms. 

Opportunities:  Tourism, quality  of  forest  resources, fresh-water  fish  farming,  mineral 
prospection. 

Threats: Population  loss,  shortage  of  jobs. 

The recommendations  drawn  from this  analysis were  used to orient  choices  made  in 
the  Regional  Development  Plan. This  tool  has since  been  used  in  many  regions. 

The evaluation team also examines the ambition of the programme in relation 
to the needs and  problems  identified. The aim is to clearly  establish  whether 
the  programme  was  designed  to  fully  satisfy  needs  or  definitively  solve 
problems  during  its  life  cycle,  or  whether  it  was  simply  considered  to  be  an 

Evaluation design and management 

initial step to  be prolonged  into the future, or a test to provide a subsequently 
generalisable  solution.  Comparable  programmes  can  equally  be  used  to 
assess  the  degree  of  ambition  of  the  programme.  It  is  important  for  the 
evaluation  to  examine  the  ambition  of  the  objectives  one  by  one. This  work 
is inseparable  from  an evaluation  of  the  achievement  of  objectives  since  it  is 
easy  to  fulfil  objectives  which  are  not  very  ambitious.  The  evaluation  of  the 
programme's  effectiveness  will,  as  a consequence,  be suitably  adjusted. 

Checking  the  clarity  and coherence  of  objectives 

Clarity  of  the  objectives 

The achievement  of an objective after  a period of time is easier to  recognise 
when  the  latter  has  been  clearly  identified.  A  perfectly  clear  objective  is 
therefore  verifiable.  In  the  first  of  the  following  examples,  the  objective  is 
relatively  easy  to  verify,  consisting  of  stimulating  the  growth  of  new 
business  creation. This objective  is associated  with  a measurable  indicator, 
of  which  the  significance  is  perfectly  clear  (rate  of  new  business  creation 
per  10,000  inhabitants)  and  which  was  fixed  in  the  form  of  a  quantitative 
value  (+  20%).  The  following  examples  show  quantified  objectives  which 
are  more  difficult  to  verify,  as  well  as  non-quantified  objectives,  some  of 
team  has 
to  verify 
which  are  easier 
systematically  to judge  the  clarity  of the  objectives. 

than  others.  The  evaluation 

Examples: Verifiable  and non-verifiable  objectives 
Easily verifiable  objectives 
At  the  end  of  the  programme,  the  ratio  of  new  business  creation  for  10,000 
inhabitants has increased by 20%. 
At the end of the programme, the main towns  in the region will have improved their 
standing when opinion polls ask the question "where would you like to live?" 
Objectives  which  are difficult  to  verify 
The programme will contribute to a 1 % increase in the per capita GDP. 
The programme will enhance the competitiveness  of regional enterprises. 

When  an  objective  is  difficult  or  even  impossible  to  verify,  the  evaluation 
team  has  to  perform  an  additional  task  before  moving  on  to  the  following 
steps  in the  evaluation. 

In the case of  an ex  ante or  mid-term  evaluation, the evaluation  team 
-4  M 
collects  al!  relevant  opinions  to  make  recommendations  which  will  help 
clarify  the  objectives. 

Performing an evaluation 

H  ►  In the  case  of  a mid-term  or  ex  post  evaluation, the  evaluation  team 
tries  to  ascertain  whether  the  programme  managers  or  operators  have 
redefined or clarified the initial objectives  during  implementation.  Interviews 
should  indicate  whether  there  is  consensus  on  the  interpretation  of  the 
programme  logic. 

Efforts  to  make  an  objective  measurable  by  means  of  an  indicator  often 
help to  clarify  it. 

Coherence  of  objectives  within  a  programme 

The  evaluation  checks  whether  the  programme  objectives  are  coherent, 
that  is, whether  the  objectives  of  the  measures  follow  on  logically  from  the 
specific  objectives  defined  for  the  axes  or  sub-programmes.  This  level 
concerns  the  internal  coherence  of  the  programme.  This  is  defined  as  an 
organisation  of  the  objectives  so  that  the  objectives  on  the  lower  level 
constitute  a logical contribution  to those on the  higher  level. This  is what  is 
understood  by the notion of  a hierarchy  of objectives that  is represented  by 
the following  diagram. 

Box 20 -  Hierarchy  of  objectives 

P r o g r a m me 

Priority  axis: 

C o m p e t i t i v e n e ss 
of  enterprises 

Technological 
services  to 
enterprises 

Technological 
c e n t r es 

y  g 

- :: 

Ε  σι 

Catching  up  the 
level of  development 

Enhanced 
competitiveness 
of  enterprises 

Enhanced 
competitiveness 
of  enterprises 

Better  technological 
services  for 
enterprises 

Better  technological 
services  for 
enterprises 

New  technological 
networks 

New  technological 
networks 

New  technological 
centres 

New  technological 
centres 

Adapted  from:  EUROPEAN  COMMISSION  (1995) 
intermediate evaluations  -  Structural  Funds 

-  Common  guidelines  for  monitoring  and 

Evaluation design and management 

To  verify  the  internal  coherence  of  the  programme,  the  objectives  tree 
technique  may  also  be  used.  This  technique  serves  to  determine  the 
contribution  of  each  objective  on a lower  level to  the global  objective  of  the 
programme.  It  consists  of  a  diagram  in  which  the  most  global  objective  of 
the  programme  is  placed  on  the  far  left,  and  the  most  detailed  objectives 
on the far  right.  Each global objective  can thus  be broken down  into  several 
sub-objectives  which,  in  turn,  are  broken  down  into  objectives  on  a  more 
specific  level. This  tool  is  also  used  to  identify  and  grade  objectives. 

In practice,  neither  the  objectives  of  each  measure  (specific  objectives)  nor 
the  relations  between  the  different  levels  of  a  complex  programme,  are 
articulated  so clearly  and explicitly. The  need to clarify  the different  levels of 
programme  objectives  is then  even  more  necessary  as the  impact  of  each 
type  of  intervention  is  uncertain. 

The objectives  tree  is used to  specify  the objectives  and to  establish  a clear 
hierarchy  between  them  by  clearly 
internal 
coherence  of the  programme,  both  vertically,  between the different  levels of 
objectives,  and  horizontally,  between  the  objectives  of  different  measures 
on the same level (see Volume 4 of the MEANS Collection as well as Volume 
5 for  numerous  thematic  examples). 

the  degree  of 

identifying 

Example: Coherent  and incoherent  objectives 
Two  measures  aimed  at  "offering  better  technological  services  to  SMEs"  and  at 
"offering possibilities of venture capital financing to SMEs" have objectives which are 
consistent  with  the  objective  on  a higher  level: "increasing  the  competitiveness  of 
enterprises". 
By contrast, these objectives would be incoherent  if the objective of competitiveness 
were stated  without  any  measure contributing  clearly towards  it, or  if  measures on 
the lower level were juxtaposed without  having anything in common. 

Although  the  objectives  tree  is  used  to  identify  and  grade  objectives,  it 
cannot  be  used  to  identify  causal  links  between  them.  In  order  to  do  so, 
another  tool  is  used:  the  "logical  diagram  of  expected  impacts".  This  tool 
schematises  the  causal  links  between  the  outputs  of  the  programme, 
situated  on  the  left,  and  the  specific  impacts  up  to  the  global  impact  or 
impacts, situated  on the  right. The logic  behind the use of  an objectives  tree 
is  that  of  the  decomposition  of  a  global  objective.  By  contrast,  the  logic 
behind  the  diagram  of  impacts  is one  of  recomposing  expected  impacts  or 
objectives  (See  Box  21  and  Box  41). 

Performing an evaluation 

Box 21 -  Example  of  logical diagram  of  expected  impacts 

Measures 

Intermediate  impacts 

Global 
impacts 

Tourism 

Agriculture 

Training 

Increased 
viability of 
enterprises 

Strengthened 
economic 
fabric 

Improved 
employability 

Improved 
productive 
system 

Structurally 
balanced 
employment 

Coherence  between  the  objectives  and  other  public  policies 

It  is  important  to  recall  that  the  programme  represents  only  a  part  of  the 
public  policies  implemented  in  a  given  region,  and  that  the  resources 
allocated  to  it  account  for  only  a small  part  of  local  public  spending.  Some 
of  the  needs  that  the  programme  can  legitimately  try  to  cover  are  also 
targeted  by  other  policies,  whether  regional,  national  or  European.  It  is 
therefore  necessary  to  ensure  that  the  programmed  activities  neither 
contradict  nor  unnecessarily  duplicate  these  other  policies.  Thus,  the 
external  coherence  of  the  programme  must  be  assessed  by  examining 
whether  the  objectives  of  the  programme  are  in  contradiction  or  not  with 
the  other  policies  of  the  European  Union,  for  example  those  relating  to 
competition  or the  organisation  of  agricultural  markets. 

•^  Community  regulations  also  stipulate  that  information  on  coherence, 
with regard to economic,  social and, where relevant, regional policies of  the 
Member  States, must  be  provided  in each  programming  document. 

H  The main  risk  for  external  coherence  is the  appearance,  midway, of  other 
interventions  which  duplicate  or  contradict  the  evaluated  programme.  If 
necessary, mid-term evaluation reviews similar policies and assesses whether 
they  extend,  complement,  overlap  or  contradict  the  programme  underway. 
This  study  consists  of  interviews  with  programme  managers  and  operators 
and with anyone else who is in a position to provide reliable  information. 

Evaluation  design and management 

Example:  External  coherence  within  the  framework  of  local  development 

interventions 

The  conditions  governing  the attribution  of  Common  Agricultural  Policy  support, as 
part  of the organisation  of agricultural  markets  aimed  at maintaining  rural  areas, are 
consistent  with the objectives  of economic  development  programmes  for rural  areas 
financed  by the  Structural  Funds.  To avoid  any  duplication,  it  is  important  for 5b 
programmes  to  be  entirely  devoted  to  the  maintenance  of  a  viable  agricultural 
community  in regions  in difficulty. This  includes the maintenance  of population  levels 
in  disadvantaged  rural  areas  and the maintenance  and development  of  agricultural 
activity  in these  areas. 

Box  22 -  Highlighting  e 
of  a  cross-imp 

Kternal  c o h e r e n ce  by  m e a ns 
tacts  matrix 

Programmed  activities 

Support  for diversification 

Support  for  productive 
investments 

Support  network 

Support  for services 
to  enterprise 

Strengthening  of 
regional R&D 

Promotion  of a new image 
of  the region 

Other  policies 

National 
promotion of 
exports 

European 
promotion of 
SMEs 

Regional 
promotion of 
foreign 
investments 

+ 

? 

-

= 

+ 

+ 

+ 

+ 

+ 

+ 

= 

= 

? 

+ 

+ 

+ 

= 

= 

=  :  no interaction 
?  :  risk  of duplication 

+  :  complementarity 
-  :  risk  of  conflict 

One  of  the  techniques  which  can  be  used  to  clarify  relations  with  other 

policies 

is  the  establishment  of  a  matrix  of  cross-impacts,  (with 

lines 

consisting  of  programme  components  and  columns  consisting  of  other 

policies).  This  matrix  will 

reveal  complementarity,  duplication  and 

contradictions  which  are likely to arise between the impacts of the  programme 

under  consideration  and  other  existing  policies  (see Volume  4  of  the  MEANS 

Collection).  Use of this tool  is shown  in Box  22. In this  example  it can  be  seen 

Performing an evaluation 

that numerous cases of synergy exist between the  programmed activities and 
the other implemented policies, for example at the national level. For instance, 
"support  for diversification" and  "promotion  of foreign  investments"  reinforce 
each  other.  On  the  other  hand,  the  latter  policy,  which  relies  on  exogenous 
inputs,  is  in  conflict  with  a programme  promoting  endogenous  development 
through the creation of a support  network  for SMEs. 

Analysing  the  modalities  of  p r o g r a m me  m a n a g e m e nt 

The  implementation  process 
involves  a  series  of  organisational  and 
budgetary  measures, i.e.: the establishment  of the budget and the  selection 
of projects, financial channels, modalities of management  and coordination, 
and  systems  of  monitoring  and  reporting. 

This  stage  of  the  evaluation 
involves  the  examination  of  budgetary 
measures,  and  operational  instructions  or  guides  disseminated  to  ensure 
that the programme  is implemented with the greatest chance of attaining  its 
objectives. The  actual  analysis  of  the  appropriateness  of  the  conditions  of 
implementation  are  made  at  a  later  stage  in  the  evaluation  process, 
particularly  when the  reasons  for  the achievement  or  not  of  a given  impact 
are  sought. 

•^  In  ex  ante  evaluation,  the  analysis  will  be  limited  to  the  first  measures 
described  in  the  programme  proposal,  that  is,  primarily  the  budget,  the 
tools  chosen  and the  main  rules  established. 

►<  ►  In mid-term  or  ex  post  evaluations,  it  is possible to  go  much  further 
and examine the details of administrative  measures, the distribution of  roles 
between  managers  and operators, the  implementation  documents  given  to 
operators, the  selection  criteria  for  projects,  etc. 

Distribution  of  budgetary  inputs 

An  evaluation  focuses  primarily  on  public  funds  allocated  by  the  authorities 
that "co-decided" the programme. Additional private investments  must not be 
put  on the  same  level; nor  must  public  funds  from  other  funding  authorities, 
which  are  expected  but  have  not  been  formally  attributed.  The  programme 
distributes funds in the form of budgets allocated to its different  components. 
Even  before  examining  how  the  inputs  are  attributed  to  agencies  and  local 
authorities  responsible  for  their  implementation, an evaluation  must  examine 
the  coherence  between  the  allocation  of  inputs  and  the  programme 

Evaluation design and management 

objectives.  It must also examine the potential  leverage or impulsion effect  that 
these  public  inputs  will  have  on  the  mobilisation  of  other  funds,  particularly 
from  private  sources. On this  point  see Volume  5 of  the  Collection. 

inputs  and 

the  strategic  priorities  of 

-4  The analysis focuses  in more detail  on the correspondence  between  the 
the 
distribution  of  budgetary 
programme.  The  evaluation  checks  whether  the  relative  importance  of 
budgets  allocated  to  the  different  objectives  effectively  reflects  the  stated 
priorities  and  goals.  At  the  programme  level,  it  applies  the  same  principle 
that  is  applies  to  all  European  cohesion  policy,  i.e.  concentration.  In  terms 
of  this  principle, the  majority  of  all  resources  must  be  directed  towards  the 
achievement  of  objectives  considered  to  be  priorities. 

Concentration  is evaluated  by  answering  questions  of  the following  kind:  to 
what  extent  is  the  programme  concentrated  on  a  few  strategic  measures, 
on  limited  areas  which  have  particularly  difficult  problems,  or  on  a  small 
number  of  public  priorities? 

If an objectives  tree was  created  during  the  preceding  stage,  it can  be  used 
at  this  stage  of  the  evaluation.  The  total  amounts  allocated 
the 
corresponding  objectives  will  simply  be  marked  on  each  branch. 

to 

Modes  of  intervention 

There  are  often  several  ways  of  meeting  an  objective.  Each  way 
corresponds  to  one  of  the  different  instruments  that  can  be  used,  for 
example  direct  subsidies, consultancy  services, training, the delivery  of  free 
services,  construction  of  infrastructure,  etc. 

In  the  domain  of  human  resources,  substantial  funds  are  allocated  to 
programmes  or  systems  which  organise  or  support  the  training  and 
occupational  reintegration  of  individuals.  The  addressees  are  reached  by 
way  of  training  organisations  or  through  their  employers.  These  types  of 
programmes  provide  support 
to  public  or  private  organisations  which 
recruit  people  who  have  been jobless  for  an extended  period  of time.  Other 
programmes  may,  for  example,  support  the  retirement  of  older  farmers  so 
that  young  farmers  can  set  up. 

In the  productive  environment  assistance  may  be  direct  (e.g. a  subsidised 
investment  or  low-interest  loan)  or  indirect  (e.g.  an  innovation  centre).  It 
may  be  in  the  form  of  an  open  grant  or  a  more  selective  programme. 
Assistance  may  be  reserved  for  physical  investments  or  immaterial  ones 

Performing an evaluation 

(marketing,  prospecting  of  international  markets,  networks,  innovation, 
quality,  design, management,  etc.). It often  involves an  improvement  in the 
supply  of  services  to  businesses.  R&D  programmes  are the  most  indirect 
form  of  support  for  productive  structures. 

-^  In  the  domain  of  basic  infrastructure,  assistance  is  decided  within  the 
framework  of  programmes  or  major  projects.  In  some  cases  a  fee  is 
charged  for  access  to  a  project.  Some  projects  are  implemented  by  public 
authorities  and others  are delegated  to  private  operators. 

For  each  objective,  an  evaluation  may  examine  alternative  instruments 
which could be used to achieve that objective, and the reasons justifying  the 
choice of one instrument  rather than another.  It may also judge the relevance 
of  the choice  of  instruments  in relation to experience  gained  elsewhere. 

Selection  criteria  for  projects 

A  programme  generally  consists  of  measures  grouping  together  a  number 
of  projects  which  have  to  be  selected  in  relation  to  certain  criteria.  These 
selection  criteria  must  be established  before the  programme  starts. 

H  As a result, it will be necessary to examine, in the evaluation of a programme, 
whether  its  management  is  clearly  oriented  towards  the  attainment  of  the 
objectives.  In order to  do  so, it will also  be necessary  to  examine  whether  the 
projects  have been chosen judiciously  (see Example, page 57). 

This search for coherence  between the selection criteria of  projects and the 
programme  objectives  will  be of  major  importance  in the case  of  mid-term 
evaluation,  when  the  programme  can  still  be  improved.  In  that  case,  the 
evaluation  will  contribute  to the choice of  criteria that  will  be a key factor  in 
the  programme's  success,  through  the  type  of  project  selected  or  given 
priority. 

Two types  of  question  must  be asked: 

•  Which  selection  criteria  are used for  projects? 

•  Are the  selection  criteria  clearly  related to  the  programme  objectives? 

Distribution  of  operational  responsibilities 

Inputs which  are at the  basis of  a  public  intervention  are not  only  financial; 
they  are  also  human  and  organisational.  These  two  components  are  more 
difficult  to  grasp  precisely,  and therefore  to  estimate. 

Evaluation design and management 

-4  H  However,  in  the  same  way  as  it  recapitulates  financial  resources,  an 
the 
evaluation  must  also  examine  who 
implementation  of 
level  (programme, 
the  programme  are,  at  each 
measures,  projects),  and  how  the  articulation  between  the  different  levels 
and  actors  is  organised. 

the  actors  responsible 

for 

An evaluation  must  examine, at each  level, whether  each objective  is clearly 
under the responsibility  of at least  one  manager. When  an objective  is  under 
the  responsibility  of  several  managers, 
it  must  examine  whether  a 
coordination  mechanism  has  been  set  up. 

in  the  region  (external  coherence). 

Because  the  European  Structural  Funds  are  implemented  jointly,  evaluation 
must  also  examine  the  consequences  of  this  partnership  on  coherence 
within  the  programme  (internal  coherence)  and  in  relation  to  other  policies 
In  such  circumstances, 
applied 
evaluation theoretically  consists  of  questioning  and  examining  the extent  to 
which  the  programme's  effects  were  better  and  more  significant  owing  to 
the  partnership.  With  this  approach  it  is  possible  to  observe  whether  the 
programme  has  had  an  effect  of  synergy  or  whether  its  effects  contradict 
those  of  policies  implemented  in the  same  region  by  the  other  partners. 

Still  with  regard  to  the  different  types  of  input,  the  evaluation  must  try  to 
ascertain  whether  major  problems  of  effectiveness  exist  as  a  result  of 
delays occurring  in financial channels, control  procedures,  decision-making 
systems,  etc. 

E x a m i n i ng  t he  p r o g r a m m e 's  i n f o r m a t i on  s y s t em 

The  "information  system"  of  the  programme  consists  of  all  data  grouped 
together  for  the  planning  of  the  programme  and  then  collected  on  a 
implementation  stage 
permanent  basis  by  the  operators  during 
(applications  for  assistance,  assessment  of  projects  for  selection,  payment 
of  grants,  project  progress  reports,  final  reports,  etc.).  Some  of  these  data 
are  aggregated,  interpreted  and  transmitted  in  the  form  of  tables  or 
synthesis  reports. Others  are  retained  for  subsequent  control  and  may  also 
be  used  by  an  evaluation. 

the 

Examination  of  expected  results 

The  breakdown  of  the  programme  logic  serves to  distinguish  several  levels 
of  objectives  linked  to  one  another.  At  each  level  (operational,  specific  and 

Performing an evaluation 

global) the  objectives  are related  to  expected  outputs,  results  and  impacts 
(see Box  19). These  notions  are presented  in more detail in Volume 2 of  the 
MEANS  Collection. 

Outputs are the product of the operators' activity. They represent that  which 
is obtained  in  exchange  for  the  public  spending. To identify  an  output,  the 
following  question  can  be  asked:  "is  it  the  condition  for  the  payment  of 
public funds?" Outputs are under the full responsibility  of the operators  who 
constantly  have  to  report  their  actions  through  the  monitoring  system. 
Examples  of  outputs  include:  roads  built,  urban  wasteland  rehabilitated, 
purification  plants  constructed, and training  paid for  by the  programme. 

Results  represent  the  immediate  advantages  of  the  programme  (or, 
exceptionally,  the  immediate  disadvantages)  for  direct  addressees.  An 
advantage  is  immediate  if  addressees  benefit  from  it  while  in  direct 
interaction  with  the  programme  outputs. The expected  results  of  a specific 
project  are  usually  recorded  during  the  monitoring  process  because  they 
generally feature in the application for assistance submitted  by the  operator 
or  direct  addressee. 

Results  correspond  to  changes  experienced  by  direct  addressees,  e.g.: 
time  saved  for  users  of  a  road;  reduced  rates  for  telephone  calls; 
qualifications  obtained  by trainees; diversification  of a farmer's activity;  new 
productive  capacity  installed  by an  industrialist;  satisfaction  of  firms  which 
received  advice. 

Identification  of  expected  impacts 

As shown  in Box 23, impacts  represent the consequences  of the direct  and 
immediate  interactions  of  the  programme  with  addressees.  Thus,  a  first 
category of impacts consists of the consequences which appear and last in 
the  medium  term  among  direct  addressees  of  the  programme  (specific 
impacts).  Examples  include: traffic  on a road one year after  its opening; the 
placement  of  trainees  after  twelve  months;  sustainable  jobs  created  in  a 
factory  built  with programme  support;  and the survival of firms created  with 
programme  support.  Some  impacts  may  be  unexpected  (side  effects)  but 
are not  part  of  the  programme  logic. 

A second category of impacts consists of all the consequences which  affect, 
in  the  short  or  medium  term,  people  or  organisations  that  are  not  direct 
addressees. These  impacts  may  occur  in close  proximity  (e.g. improvement 
of the quality  of life for  people  living near a rehabilitated  industrial  wasteland 

Evaluation design and management 

or improvement of the quality of beaches near a new purification plant). They 
may also stretch far afield to distant people or organisations as in the case 
of macro­economic impacts. 

Box 23 ­ Public organisation: feedback through evaluation 

PROGRAMME 

SOCIO­ECONOMIC 
ENVIRONMENT 

Impacts 

The mechanisms of impact propagation are often divided into two 
categories: market effects (e.g. impacts on the suppliers or subcontractors 
of the assisted businesses) and non­market effects (e.g. the positive impact 
of the improved ¡mage of the region or the negative impact of a deteriorated 
environment). Non­market effects, or externalities, are not taken into 
account in private decisions, even if they have economic consequences for 
private actors. It is therefore particularly useful to identify them in the 
context of a public programme. 

Because of the time lag or their indirect nature, impacts cannot easily be 
recognised by operators in the context of their daily management. They are 
therefore analysed mainly during evaluation work. 

An illustration in terms of resources, 
proposed in Box 24. 

outputs, results and impacts is 

■4 M An evaluation examines the system of indicators and, in particular, 
checks whether it adequately reflects objectives (the quality criteria of a 
system of indicators). If necessary, and in an ex ante or intermediate 
context, the evaluation makes recommendations for improving indicators. 

Performing an evaluation 

Box  24  -  Inputs,  outputs,  results  and  impacts  of  a  road 

In the example  of the widening  of a road to  boost  the development  of  an  isolated 
valley, the different  notions  can  be  illustrated  in the following  way: 

•  Inputs: 

•  Outputs: 

•  Results: 

•  Impacts  : 

Budgets  allocated,  laws  used  compulsorily  purchase  land, 
enterprises  mobilised, information  instruments  used, etc. 

Kilometres  of  new  roads,  enlarged  intersections,  improved 
road  surfaces,  etc. 

time 
Number  of  cars  using  the  road,  reduced 
between  two  destinations,  reduced  number  of  accidents,  but 
also  more  pollution  for  people  living  near  the  road,  etc. 

travelling 

More  rapid  economic  development  of  the  towns  served, 
the 
improved  access  to  the  leisure  activities  offered  by 
regional  capital  (positive 
impacts),  decreased  value  of 
neighbouring  land  (negative  impact),  or  a reduced  number  of 
accidents  (positive  impact  not  directly  aimed  for  by  the 
programme),  etc. 

Context-related  information 

implementation  of 

indicators  concern  all  aspects  of 

the 
Programme 
programme and also serve to  monitor  its effects  (results and  impacts), from 
the  most  direct  to  the  most  distant.  They  will,  for  example,  measure  the 
extent  to  which  a  priority  group  was  reached  or  the  degree  to  which  a 
sustainable  advantage  was  obtained  for  the  end-users  of  the  programme. 
Within the framework  of  monitoring  and evaluation, a programme  indicator 
can show  that  a particular  intervention  is a success  or failure. 

Context  indicators,  in  contrast,  apply  to  the  entire  population  without  any 
distinction  as to whether  it  has  been affected  by the  programme. They  will, 
for  example,  show  the  extent  of  the  population's  needs,  or  the  extent  of 
disparities  that  must  be  reduced.  They  can  serve  to  assess  whether  the 
objectives are sufficiently  ambitious compared to the problems faced by the 
population.  In  an  evaluation,  a  context  indicator  is  able  show  that  an 
intervention  has lost  its  rationale or  must  be  modified  because the  problem 
that  needed to  be  solved  has disappeared  or  changed. 

The meeting  point  between  impact  indicators and context  indicators  seems 
obvious. It appears that  by following these two indicators the estimation the 

Evaluation design and management 

programme's  effects  should  be  feasible.  However,  this  theoretical  facility  is 
hindered  by  the  existence  of  effects  arising  from  confounding  factors, 
which  have  to  be taken  into  account.  The  expected  final  state  can  thus  be 
broken  down  into  three  parts:  initial  state,  expected  effect  and  effect  of 
confounding  factors. 

For  each  objective,  the  evaluation  examines  whether  it  is  linked  to  an 
indicator  and  whether  initial  state,  the  expected  final  state  and/or  the 
expected  effect  are  quantified. 
the  evaluation  makes 
proposals  for  objectives  which  have  no  indicators:  it  either  proposes 
relevant  indicators  or  else a  qualitative  treatment  (see the  following  section 
and  Volume  2  of  this  Collection). 

If  necessary, 

Quantification  of  outputs  and  results 

-4  An  information  system  must  be  mobilised  ex ante  during  the  negotiation 
process  of  the  programme,  to  specify  the  success  criteria  and  quantify  the 
desired  effects.  Thereafter,  the  apparatus  set  up  for  monitoring  must  allow 
for  an examination  of  outputs  and  results,  as they  are  obtained  in the  field. 
Each operator  provides  information to the programme  managers  in the  form 
of  progress  reports  or  periodic  reviews. The  authorities  responsible  for  the 
programme  receive  this  information,  discuss  it  in  monitoring  committee 
meetings  and  react  accordingly. 

H  ►  At  the  time  of  mid-term  and  ex  post  evaluations,  data  from  the 
system  must  be  used  as  effectively  as  possible,  provided  the  quality  of 
information  provided  by  this  system  has  been  controlled. 

The  evaluation  has  to  check  whether  adequate  measures  have  been  taken 
so  that,  throughout  the  life  of  the  programme,  it  is  possible  to  ensure  that 
objectives  are  being  achieved,  that  these  objectives  remain  relevant  and 
that  decisions  will  be taken  if this  is  not  the  case. 

The  evaluation  must  also  ensure  that  adequate  measures  (monitoring-
evaluation)  have  been  taken  to  report  on  the  achievement  of  objectives.  In 
fact,  this  monitoring-evaluation  work  has  to  be  organised  according  to  a 
plan  designed  before  the  start  of  the  programme.  It  has  to  clearly  specify 
who  is  in  charge  of  the  monitoring  and  decision-making,  and  according  to 
what  schedule  it  is  to  be  carried  out.  Provision  must  have  been  made  for 
adequate  financial  and  human  resources. 

Performing an evaluation 

It is by means of this framework  that the pace of spending and the  progress 
of  the  programme  can  be  monitored  (monitoring  system),  and  that  the 
different  evaluations  likely  to  be performed  during the  programme  or  at  the 
end of it can be planned. It is also this framework that  must specify  how the 
evaluations  must  be discussed  by the  authorities  managing  them  and  how 
the  public  is to  be  informed. 

The  evaluation  must  examine  the  steps  that  were  taken  to  quantify  the 
indicators. The collection  and synthesis of  information  can  be facilitated  by 
the  creation  of  a  computerised  system,  for  new  information  technologies 
facilitate  the  collection  and  circulation  of  a  large  mass  of  information. 
However,  the  construction  of  overabundant  systems  of  indicators  must  be 
avoided.  Evaluation  may  include  in-depth  reflection  to  select  the  most 
useful  indicators. 

The  evaluation  team  must  examine  the  probability  of  the  indicators  being 
correctly  informed.  It  makes  proposals  so  that,  where  relevant,  input  and 
indicators  are  systematically  quantified  by  the  operators  and 
output 
managers,  as part  of their  responsibilities. 

A  number  of  criteria  can  be  used  to  check  the  quality  of  a  system  of 
indicators  presented  in  detail  in  the  second  Volume  of  the  Collection.  To 
briefly  summarise the content, there is a group  of  quality  criteria  applicable 
to  each  of  the  indicators  selected,  and  a  group  that  concerns  the  entire 
system. 

An indicator  is of a high quality  when  it  is quantified  at regular  intervals  and 
with  a  frequency  that  is  suited  to  the  work  considered  and  published. 
Moreover,  an  indicator  must  be  usable  shortly  after  the  collection  of  the 
data on which  its construction  was  based. It is also necessary  to check  the 
sensitivity  and  reliability  of  indicators  in  relation  to  the  phenomena  to  be 
measured.  Finally,  an  indicator  must  allow  comparisons  between  the 
measures  of  the  programme,  and  with  other  programmes  or  groups  of 
addressees. 

There  are  four  main  quality  criteria  for  a  system  of  indicators:  good 
coverage  of  the  entire  programme;  good  balance  between  outputs,  results 
and impacts;  pragmatism  as  regards  numbers; and, finally,  relevance, i.e. a 
selection  of 
the  strategic  and 
operational  importance  of the  information. 

indicators  which  takes 

into  account 

Evaluation design and management 

Qualitative  information 

Box  25  -  Qualitative  a s s e s s m e nt  grid  of  t he  environmental  i m p a ct 

of  p r o g r a m m es 

Priority  1 /  Measure  1  : Quality  sites  and  premises  for  industrial  and  commercial 
investors 

Objective:  Guaranteeing  an adequate  supply  of  quality  industrial  and  commercial 
sites  for  stimulating  local  growth  and  increasing job  opportunities. 

Project  : 

Reference  : 

Yes 

Not 
applicable 

No 

1.  If the  project  is  implemented  on  a 
rehabilitated  industrial  site, will  the 
wasteland  be  reduced  and  will  certain 
buildings  be  converted  into  factories? 

2.  Does  the  project  comply  with  instructions 
concerning  the  natural  environment  or 
urban  Heritage? 

3.  Does  the  project  include  sound  energy 
saving  practices,  the  recycling  of  water 
or  the  reduction  of  waste,  or  innovative 
environmental  measures? 

4.  Does  the  site  have  an  environmental 

management  plan  or  are there  measures 
in place  for  environmental  management? 

5.  Have  measures  been  taken  to  ensure 

that  the  site  is easily  accessible  by  public 
transport? 

6.  Has  the  development  been  designed  to 
improve  the  local  urban  environment,  for 
example  by  minimising  the  effects  of 
delinquency  or  the  lack  of  a  public  spirit? 
(this  criterion  is considered  less  important 
than  the  others,  the  score  is therefore  less 
than  1) 

7.  Does  the  development  project  include 

proposals  for  environmental 
improvements  such  as  landscape 
development? 
(see above  comment) 

Total  score 

1 

1 

1 

1 

1 

0.5 

0.5 

0.5 

0.5 

0.5 

0.75 

0.38 

0 

0 

0 

0 

0 

0 

0.75 

0.38 

0 

Performing an evaluation 

A  good  evaluation  cannot  exclude  an  objective  from  its  field  of  analysis 
simply  because  it  is thought  to  be difficult  to  measure  quantitatively.  It  may 
in fact  be one of the most  important  objectives, which cannot  be  neglected 
simply  because  of  difficulties  in defining  a quantitative  indicator. 

information 

to  an 
Well-structured  qualitative 
absence  of  information. It is therefore  advisable  to  structure  information  by 
means  of an assessment  grid,  structured to  a greater  or lesser  degree and, 
if  possible,  graduated  according  the  level  of  success  of  the  intervention 
under  consideration. 

is  obviously  preferable 

The  construction  and  use  of  these  grids  are  presented  in Volume  4  of  the 
MEANS  Collection  devoted  to  technical  solutions  adapted  to  complex 
socio-economic  programmes.  Box  25  gives  an  example  of  an  assessment 
grid established  for  the purpose  of  identifying  the environmental  impacts  of 
a  particular  measure  of  an  Objective  1  programme  in  the  U.K.  The 
association  of  economic  objectives  with  other  objectives  is  interesting  in 
this  example. 

In  all  three  cases  (semi-structured,  structured  or  graduated  grids), 
qualitative  information  can  be  transformed  into  scores  for  the  purpose  of 
comparison  within  the  programme. 

-4  H  F °r  objectives  which  cannot  be  associated  with  an  indicator  in  any 
relevant way, the evaluation  can make  proposals for qualitative treatment  in 
the  case of  ex  ante and  mid-term  evaluations. 

Evaluating the impacts of an entire programme 

Socio-economic  development  programmes  usually  concern  numerous 
domains.  Although  their  impacts  converge  towards  a  global  development 
objective,  they  materialise  through  very  different  mechanisms.  When  an 
evaluation  examines  an entire  programme,  it  is expected  to  systematically 
study  all  impacts.  Given their  diversity  and  the  number  of  possible  causes, 
the applicable  methods  are  necessarily  global. 

These  methods  are  based  on  the  use  of  secondary  data,  on  qualitative 
surveys  of  operators  and  a  limited  number  of  addressees,  and  on 
extrapolations  of  the  results  of  other  evaluations. 

Programmes  financed  by  European  cohesion  policy  aim  at  accelerating 
socio-economic  development  in one of its dimensions (income, employment, 
quality  of  life,  etc.).  The  main  impact  sought  is  a  structural  and  therefore 
lasting  impact  on  the  macro-economic  or  macro-social  level. To obtain  this 
impact,  most  programmes  adopt  multi-sectoral  strategies,  consisting  of 
measures  of  diverse  kinds  (infrastructure,  consultancy  services,  subsidies, 
training,  communication,  etc.)  and  in  diverse  domains  (transport,  research, 
tourism,  etc.).  Each  measure first  produces  results  on  its direct  addressees, 
after  which  its impacts  are materialised  in successive  stages on  increasingly 
distant  groups, through  multiple  and ramified  mechanisms. The fact  that  the 
programme  exists  leads  to  the  assumption  that  all  these  mechanisms 
converge  towards  the global  impact  of  development. 

A  programme  evaluation  can  take  one  of  many  different  forms,  depending 
on  the way  its  impacts  are to  be  analysed.  In this  Volume, three  situations 
will  be  presented  as follows: 

1. Global  evaluation,  consisting  of  an  examination  of  the  impacts  of  all  the 
measures. This is the first situation described  below (starting on page 117). 
The  entire  process  is  described,  from  the  preparation  of  the  work  to  the 
writing  of the  report, through  the collection  and analysis of data. 

2. More  in-depth  analysis  of  macro-economic  impacts.  This  is  the  second 
situation  described  below.  It  is  limited  to  the  specific  characteristics  of 
this type  of  analysis  (see page  135). 

3. More  in-depth  analysis  of  a  particular  impact  at  the  micro-economic  or 
micro-social  level. This  third  situation  is  also  described  in the  form  of  a 
supplement  to  an  evaluation,  rather  than  in  the  form  of  an  entire 
evaluation  in its own  right  (see  page  147). 

Evaluation design and management 

It  is  important  to  note  that  the  first  two  situations  (overall  evaluation  and 
macro-economic  analysis)  can  be  encountered  at  all  the  stages  of  the 
programme  cycle.  By  contrast,  the  in-depth  analysis  of  a  particular  impact 
is possible  only  in ex  post  evaluation, long  enough  afterwards  for  an  impact 
to  have  had  time  to  materialise.  This  analysis  involves  a  large  amount  of 
work  and  cannot  be  applied  to  an  entire  multi-sectoral  programme,  as 
explained  in the  following  sections. 

Choosing  a mode  of  impact  analysis 

Imputing  change  to  a  programme 

Usually,  when  an  impact  is  to  be  analysed,  all  that  is  known  is  what  has 
been  observed,  that  is, a  change  (reduced  unemployment  rate,  number  of 
businesses  created,  increase  in  the  traffic  on  a  road,  etc.).  The  impact 
accounts  for  only  the  share  of  this  change  that  is  imputable  to  the 
programme  and  only  that  share.  An  impact  is  also  called  a  net  effect  or 
additional  effect. 

All  methods  used to quantitatively  estimate  an  impact  consist,  in one way  or 
another,  of  describing  what  would  have  happened  without  the  programme, 
of  comparing  that  with  what  actually  happened,  and  of  estimating  the 
difference.  In  other  words,  the  impact  of  a  programme  is  considered  to  be 
the  difference  between  the  real changes  observed  after  the  intervention  (i.e. 
the  policy-on  situation)  and  the  fictional  changes  estimated  in  a  situation 
without  the  intervention  (policy-off  situation).  The  fictional  situation  is  also 
called  the  "counterfactual  situation". This  reasoning  is  illustrated  in the  Box 
26,  by  means  of  the  example  of  an  impact  on job  creation. 

An  imputation  technique  consists  of  materially  creating  both  situations  by 
recruiting  candidates  to  participate  in  the  programme  and  by  randomly 
selecting  a  first  group  to  participate  in  the  programme  and  a  second, 
control  group,  which  will  not  participate.  A  survey  is  then  carried  out  on 
each  of  these  two  groups,  before  and  after.  The  difference  between  the 
results of the two  surveys gives a reliable estimation  of the  impact,  provided 
that  the  two  groups  are  sufficiently  representative  (each  one  has  between 
300  and  1,000  individuals,  depending  on  the  acceptable  rate  of  error  and 
the  heterogeneity  of  the  population  surveyed).  This  technique  is  time-
consuming,  expensive,  technically  difficult  and  sometimes  contestable 
from  an ethical  point  of  view.  This  explains  why  it  is  rarely  used, despite  its 

Performing an evaluation 

very strong capacity to produce robust conclusions. Instead, techniques 
are used which artificially reconstruct the policy­off situation, such as the 
comparison group technique or that of econometric simulation, developed 
below (see page 155). 

Box 26 ­ Policy­on and policy­off situations 

ω 
en 
c 
03 

«<S& 

*>*°ío«s' 

Baseline 

Evaluation 

Impact analysis is simplified to a large extent if it is carried out directly by 
the addressees. That is what happens, for instance, when the following 
questions are put to entrepreneurs who have received assistance: What will 
the value added generated by the assistance be in your firm? How many 
jobs have been created thanks to this support? These questions can be 
asked when the projects are selected, when payments are made, or during 
an evaluation survey. The answers provide a direct estimation of the effects 
and make the construction of a policy­off situation unnecessary. In fact, it 
is the surveyed people themselves who conceive the policy­off situation 
when they answer the questions. 

When the impact is estimated by means of this method an additional analysis 
must, however, be made. Addressees tend to over­estimate programme 
effects for various reasons, especially when they want to show their gratitude 

Evaluation design and management 

to  the  administration  from  which  they  received  the  assistance.  In  this 
instance  the  evaluation  team  has  to  perform  an  analysis  of  the  real  impacts 
of  the  intervention  under  evaluation, that  is, the  "net  effects"  as  opposed  to 
these  apparent  effects  or  the  effects  declared  by addressees  (i.e. the  "gross 
effects").  It  is  necessary  to  determine  the  extent  to  which  the  assistance 
really  did  make  it  possible  to  implement  a  project,  and  how  much  the 
absence of  public  intervention  would have affected  the implementation  of  the 
project.  When  public  assistance  is granted  to  projects  that  would  have  been 
implemented  even  without  assistance,  the  term  "deadweight"  is  employed 
for those  who  have  benefited  from the  support.  The  net  effect  is obtained  by 
subtracting  the  deadweight  effect  from  the  gross  effect. 

Example: Deadweight  related to  support for  employment 
The example most frequently cited is that of a firm which decides to recruit and then 
tries to find out whether it qualifies for some form of public support. Many evaluations 
in the employment  domain  estimate the deadweight  for  employment  support  to be 
very  high  (between  40  and  80%  of  all  gross  effects).  The  public  authorities  of  a 
country or  region may, however, decide that  the social problem of unemployment  is 
such a high priority that the deadweight  effect  is the price to be paid for  stimulating 
the recruitment of the unemployed. 

In some  cases  of evaluation, it  is also  necessary  to  analyse displacement  or 
substitution  effects.  Displacement  effects  appear,  for  example,  when  a  job 
is  created  in  an  assisted  geographical  area  to  the  detriment  of  another  job 
which  is  lost  elsewhere  (a  recent  case  of  this  is  the  outcry  that  occurred 
when  a  car  manufacturer  closed  a  plant  in  Belgium  and  simultaneously 
created  jobs  in  Spain).  Substitution  effects  appear,  for  example,  when  a 
person finds a job  thanks  to a programme  but  this deprives  another  person, 
who  finds  him  or  herself  at  a  disadvantage,  of  the  job.  Substitution  and 
displacement  effects  are sometimes  intentional, for  instance when  firms  are 
encouraged  to  move  into  areas  with  a  high  unemployment  rate.  In  such 
cases  the  impact  of  the  programme  obviously  includes  these  effects.  By 
contrast,  substitution  or  displacement  effects  must  be  subtracted  from 
gross  effects  to  estimate  the  net  effect  when  they  cannot  be  justified,  e.g. 
assistance  which  enables  a jobless  person to  take  the job  of  someone  else 
who,  in turn,  becomes  jobless.  In short,  as  shown  in  Box  27, estimation  of 
the  net  effect  can  be  based  on  the  gross  effect  only  if  deadweight  and,  in 
certain  cases, the  displacement  and/or  substitution  effects  are  subtracted, 
when  these  were  not  intended  by  the  programme. 

Performing an evaluation 

Box  27 -  Deadweight,  displacement  and substitution  effects 

Net jobs 
created 

Unacceptable 
displacement  effect 
(a job is created and 
another job is destroyed 
in the area) 

Gross jobs  created 
(statements made 
by participating  enterprises) 

Unacceptable 
substitution effect 
(a jobless person is recruited 
and another one 
loses her/his job) 

Deadweight 
(a job is created which would 
have been created in any case, 
even without the aid) 
_ 

Most  displacement  and  substitution  effects  can  be  analysed  credibly  by 

means  of  interviews  with  addressees.  It  is  more  difficult,  however,  to 

analyse  deadweight  in the  same  way  because  few  people  will  admit,  even 

anonymously,  to  having  received  assistance  without  anything  concrete  in 

exchange.  A  good  analysis  of  deadweight  requires  the  use  of  comparison 

group  techniques,  as the following  example  suggests. 

Example:  The  difficulty  of  highlighting  deadweight  in  a  rural  development 
programme 

Within  the framework  of a rural  development  programme  in Denmark,  farmers  were 
to  receive a subsidy for the  purpose of diversifying  their activities. In a survey  carried 
out  as part  of the evaluation, the farmers  were  asked  whether  they  agreed  with the 
following  sentence:  "the support  received  conditioned  the  implementation  of my 
diversification  project".  About  75% of the assisted  farmers  gave  a positive  answer. 
An  initial  estimation of the deadweight  would  therefore  be around  25%. 
However,  a complementary  survey  was carried  out on farmers  who had requested 
assistance but had been refused for various reasons. The results  revealed that all the 
farmers  had  realised  their  projects  even  without  the  support.  Deadweight  can 
therefore be considered  to be 100%. This conclusion  is nevertheless  hasty,  in so far 
as  some  of  these  farmers  were  not selected  for assistance  precisely  because the 
project  selection  committee  judged  them  capable  of  realising  their  projects  without 
public  support.  In this  case, a more  in-depth  analysis,  using  a genuine  comparison 
group,  would be necessary to understand the real deadweight  effect. 

Evaluation design and management 

The  above  example  shows  that  impacts  must  be  analysed  with  extreme 
care  if they  are  to  be  valid.  Whether  the  estimation  of  impacts  is  based  on 
observed  changes  or  on  the  opinions  of  programme  participants,  the 
evaluation  team  must  never  rely  on  appearances.  On  the  contrary,  it  must 
always  make  a  thorough  analysis  and,  whenever  necessary,  verify  the 
results  in the field. A  particularly  credible form  of  verification  is that  which  is 
based  on  a survey  of  non-participants. 

Validity  of  the  analysis 

The  main  authors  of  evaluation  literature  consider  that  the  validity  of  an 
impact  analysis  has  three  forms,  described  in  Box  28:  construct  validity, 
internal  validity  and  external  validity. 

Box  28 -  Forms  of  validity  of  evaluations 

Construct 
validity 

Internal 
validity 

External 
validity 

Construct  validity  means  that  the  indicators  used  for  the  analysis  accurately 
and reliably  measure the observed changes. It is particularly  important for the 
impact  to  be expressed  in  sufficiently  precise  terms,  and  for  observations  in 
the  field  to  be strictly  coherent  with  what  has to  be  measured.  For  example, 
if  the  impact  of  support  for  innovation  is  to  be  measured  by  means  of  a 

Performing an evaluation 

survey  of  regional  SMEs, the  notion  of  innovation  must  be defined  precisely 
and the questions  asked  must correspond  precisely to these definitions (e.g. 
number  of  new  products  or  production  procedures). 

is  obtained  when  the  evaluation 

Internal  validity 
team  reaches  an 
indisputable  conclusion  on  the  fact  that  the  programme  has  had  a  given 
effect.  For example, programme  support  has doubled the innovation  rate in 
directly  or indirectly  assisted SMEs (e.g. number  of  innovations  per year  for 
1,000  jobs).  This  type  of  conclusion  is  more  credible  if  the  collection  and 
analysis  of  data  has  carefully  taken  into  account  all  the  characteristics  of 
the  programme  and  its  context  (e.g.  categories  of  businesses  assisted; 
form  of  innovation  peculiar  to the  region). 

Thus, to  improve  internal validity, the evaluation  has to be adapted to all the 
peculiarities  of  the  programme  and  its  context.  This  runs  counter  to  the 
third  form  of  validity,  external  validity,  consisting  of  producing  conclusions 
that  can  be  generalised  to  other  programmes.  To  revert  to  the  above 
example,  the  evaluation  would  have  good  external  validity  if  it  could 
conclude  that  similar  support,  attributed  to  SMEs  in  other  regions,  would 
probably  have  had  the  effect  of  doubling  their  innovation  rate.  External 
validity  is  clearly  valuable  in  a  context  in  which  many  programmes  apply 
similar  methods.  It makes it  possible to apply  lessons  learned in one region, 
to  numerous  other  regions. 

•  Levels  of  in-depth  analysis 

The preceding  sections  have shown that  an evaluation team cannot  make a 
valid  impact  analysis  without  verification  in  the  field,  surveys  on  non-
participants,  and  a  detailed  analysis  of  the 
the 
programme and its context. As a result, a valid analysis of an impact  requires 
time  and  an  adequate  budget.  This  can, in turn,  lead to  the  impossibility  of 
analysing  all 
impacts  of  a  multi-sectoral  and  multi-objectives 
programme, or at  least the  impossibility  of  analysing  them  all  in detail. 

implementation  of 

the 

The choice of a level of detail is therefore particularly important. An evaluation 
may  be  performed  on  a  purely  superficial  level  (level  1  in  Box  29)  or  may 
mobilise  huge resources  for  making  quantitative  estimations  (level  4). In this 
case, it  is generally  necessary  to  focus  the  analysis on a few  impacts. 

The most superficial treatment  of an evaluative question (level 1) consists  of 
a  simple  description  of  observable  changes  in  the  concerned  groups  or 

Evaluation design and management 

eligible area. It is not  rare for evaluations to be limited to a purely  descriptive 
treatment  of  this  kind,  even  though  it  well  known  that  such  analyses  can 
give an erroneous  idea of  programme  effects. The analysis can  be  improved 
by correcting  the observed data, so as to get  closer to the real effects of  the 
programme.  For  example,  deadweight  can  be  taken  into  account  by 
applying  a  correction  coefficient  taken  from  the  evaluation  of  another 
programme,  provided  that  the  external  validity  of  that  evaluation  has  been 
verified. Similarly,  impacts  on  employment  can  be estimated  on the  basis of 
investments  made  by  supported  enterprises 
through  the  use  of  an 
extrapolation  coefficient. 

Box  29 -  Choosing  a  level  of  impact  analysis 

Level  1. 

Descriptive  study  of  c h a n g es 

r Level  2. 

C o m p a r a t i ve  s t u dy  of  c h a n g es 

•Level  3. 

Inductive  analysis  of  causal  relations 

•Level  4. 

Quantitative  estimation  of  i m p a c ts 

Comparative  analysis  (level 2)  must  at  least  be  based  on the  observation  of 
groups  or  territories  not  affected  by  the  programme.  Analyses  at  this  level 
are  always  possible,  even  with 
limited  means,  but  are  unfortunately 
performed  far  too  seldom.  The  cost  and  time  required  to  perform  field 
studies  is  though  admittedly  high.  Without  actually  demonstrating 
the 
existence  of  causal  relations  between  the  programme  and  the  socio­
economic  changes  which  took  place,  comparative  analysis  highlights  a 
range  of  qualitative  and  quantitative  indices  on  the  nature  and  importance 
of  the  programme  effects. 

Inductive  analysis  of  causal  relations  (level  3)  must  be  considered  if  an 
understanding  of  the  mechanisms  of  impact  production  is  required,  and 

Performing an evaluation 

when  numerous  interdependencies  exist  with  the  socio-economic  context 
(confounding  causes  and  externalities).  This  level  of  analysis  is  always 
possible,  notably  through  case  studies,  but  is demanding  in terms  of  field 
surveys, evaluation  techniques  and time. 

The  most  detailed  level  of  analysis  (level  4)  is  that  which  allows  the 
quantitative  importance  of  programme  impacts,  based  on  assumptions 
about  all  the  competing  relations  of  cause  and  effect,  to  be  shown.  This 
analysis,  which  is  cumbersome  and  technically  difficult,  generally  requires 
several  months  of  work.  Furthermore,  it  is  not  always  sure  to  produce  a 
conclusion, for the  reasons  explained  below  (see  page  156). 

To conclude, the  overall  evaluation  of  a programme  systematically  focuses 
on  all the  impacts  of  all the  components  of  the  programme.  Yet, given  that 
socio-economic  programmes  are  characterised  by  the  multiplicity  of  their 
impacts,  it  follows  that  their  overall  evaluation  cannot  hope  to  go  further 
than the first  level of  analysis. The validity  of  an overall  evaluation  therefore 
depends  strongly  on the  external  validity  of  the  other  evaluations  on  which 
the estimation of  impacts  is based, and, when necessary, the performing  of 
more  in-depth  analyses. 

With more in-depth analyses (levels 2 to 4), it is possible to attain a high level 
of  validity,  but  the  cost  involved  limits  them  to  the  estimation  of  a  limited 
number  of  impacts.  It is then all the  more  important  for these evaluations  to 
be performed with  particular attention  paid to external validity. This will allow 
their  findings  to  be  used  in the  evaluation  of  other  programmes,  whether  in 
the same  region or  in regions with  similar  problems. 

Exploiting  existing  information  sources 

The estimation  of  an  impact  is always  difficult.  To produce the  most  robust 
estimation,  it  is advisable  to  use as  much  existing  information  as  possible, 
such  as  administrative  data,  research  and  analyses  of  the  domain  under 
consideration,  documents  related  to  the  life  and  management  of  the 
programme,  published  statistics,  information  on  similar  programmes,  and 
so  on.  This  information  can  be  obtained  rapidly  and  at  a  low  cost.  It  can 
help  to  provide  the  first  answers  to  some  of  the  questions  asked.  If  the 
terms of reference have been prepared correctly, this document  will  already 
contain  a list of  immediately  available  information. This  is called  secondary 
data,  as  opposed  to  primary  data  which  are  collected  specially  for  the 
evaluation. These two  categories  of  data are presented  in  Box  30. 

Evaluation design  and management 

Box  30  -  Sources  of  information  for  an  evaluation 

Evaluation 

Secondary 
data 

Primary 
data 

Monitoring 
system 

Series  of 
statistics 

Literature 
Experts 

Management 
documents 

Statistics 
institutes 

Surveys, 
Case  studies 

Research 
Former 
evaluations 
_L 

^^^^^^cc^^^^^^^^^oc^^^cc^^^^cc^^^^ 

Programme  management  documents 

Programme  management  documents  contain  information  on  planned  and 
actual  spending.  This  information  can  be  used  in  a  synthetic  form,  for 
example  in  the  form  of  financial  balance  sheets  broken  down  into  axes, 
measures  and  actions.  Elementary  data  for  each  project  are  systematically 
stored  and  kept  accessible.  It  might  be  necessary  to  refer  back  to  them, for 
example  in  the  following  cases:  (1)  to  obtain  the  percentage  of  programme 
spending  devoted  to  solving  environmental  problems;  (2)  to  obtain  the 
percentage  of programme spending  which  benefited the most  disadvantaged 
rural  or  urban  areas;  (3)  to  obtain  the  percentage  of  programme  spending 
which  benefited the smallest  enterprises;  (4) to obtain the difference  between 
the  cost  per job  created  in a rural  area and  an  urban  area, etc. 

Stored  data  on  spending  often  includes  the  name,  address  and  telephone 
number  of  the  people  and  organisations  which  received  assistance.  These 
data  are  valuable  in  cases  where  it  is  necessary  to  contact  addressees 
during  an  investigation. 

Programme  management  documents  also  contain  information  on  outputs, 
that  is,  what  has  been  obtained  in  exchange  for  public  spending.  This 

Performing an evaluation 

information  can  be used in a synthetic  form, for example through  indicators 
or  progress  reports.  Elementary  data  are  often  stored,  project  by  project, 
but  are generally  not  readily  accessible.  It could  nevertheless  be  necessary 
to  refer  back  to  them, for  example  to  obtain  details  on  certain  projects  for 
performing  case  studies. 

The evaluation team can have access to  management  documents only  with 
the  help  and  cooperation  of  those  who  possess  them.  This  could  involve 
time-consuming  work  for  managers  and  operators.  The  cooperation  of 
those  who  possess  information  is obtained  far  more  easily  when the  issue 
has  been  discussed  beforehand  with 
the  parties  concerned.  The 
commissioning  authority  is  responsible  for  ensuring  that  the  necessary 
doors are opened, for example by involving  in the steering group those  who 
have most  of the  required  information, or by  promising  to supply  them  with 
a  synthesis  of  the  first  evaluation  conclusions.  To  facilitate  access  to 
information,  it  is  also  important  for  the  evaluation  team  to  undertake  to 
maintain  the  confidentiality  of  all  personal  data. 

Statistical  sources 

Statistical  sources  concern  an entire  region, a State, or  even the  European 
Union.  They  are  used  to  examine  changes  affecting  an  entire  population, 
economy,  environment  of  a given  territory  or  particular  population  within  a 
territory.  The  most  useful  statistics  for  evaluation  are  those  which  apply 
exactly  to  the  territory  concerned  by  the  programme  or  to  the  eligible 
population. The latter  sources of statistics  provide context  indicators  which 
can be used to reveal apparent  impacts (e.g. the number of  new  businesses 
created  has  doubled)  to  assess  needs  (e.g.  the  rate  of  new  business 
creation  is far  lower  than the  European  average)  or to  examine  whether  the 
objectives  remain  relevant  (e.g.  the  rate  of  business  creation  has  now 
caught  up with  the  European  average). 

Unlike management  data which concentrate  on operators and on the  direct 
addressees  of  the  programme,  statistics  also  encompass  all the  people  or 
businesses  in an area that  have  had  no  contact  with  the  programme.  As  a 
result,  the  comparison  of  a  before-after  statistic  cannot  provide  an 
estimated  impact.  At  best,  it  gives  information  that  can  be  used  within  the 
framework  of  an  impact  analysis,  to  impute  observed  changes  to  several 
causes:  the  programme  and  exogenous  factors  (or  confounding  factors). 
Thus, for example, statistics  can show  an increase in unemployment  due  to 
a sharp  natural  rise in the working  population, even though  the  programme 
has created  many  new jobs. 

Evaluation design and management 

In  impact  analysis,  statistics  provide  useful  indications  on  the  evolution  of 
exogenous  factors,  by  measuring  various  characteristics  of  the  territory  or 
group  concerned.  They  can  be  used  to  interpret  or  qualify  observed  gross 
effects  or  apparent  impacts.  They  also  supply  extrapolation  coefficients 
which  are  often  used  in  evaluation  (see  page  126).  For  example,  if  a 
statistical  study  can  be  used  to  establish  that  the  average  size  of 
businesses  created  in the  past  two  years  is 4.5 jobs, this coefficient  can  be 
used  in  an  estimation  of  impacts,  for  measuring  support  for  business 
creation. 

Statistical  data  are  directly  accessible  from 
the  organisations  which 
produce  and publish them  (European, national and  regional  public  statistics 
institutes,  private  institutes,  etc.).  Often  these  data  have  already  been 
gathered  by  programme  managers  or  by  research  organisations  (e.g. 
regional  statistical  teams).  For  example,  many  regionalised  data  can  be 
found  at  Eurostat  in the  REGIO  database  (a data  base  with  an access  fee). 
These  data  are  presented  in  the  form  of  about  70  standard  tables  per 
domain:  demography;  economic  accounts;  unemployment;  workforce; 
energy;  agriculture;  transport;  and  R&D.  Eurostat  recently  enriched  its 
regionalised  data  by  exploiting  national  annual  surveys  of 
industrial 
structures. The  information  processed  is available  in the  ELIZE data  base (a 
data  base  which  is  not  part  of  Eurostat),  consisting  of:  the  number  of 
industrial  enterprises;  employment;  salaries  and  wages;  turnover;  and 
investments. The  data  are  broken  down  into  about  thirty  economic  sectors 
and  into  two  categories  of size (+/- 20  employees).  In the  medium  term, the 
REGIO  data  base  will  be  completed  by  the  use  of  surveys  of  agricultural 
structures. 

Use  of  statistics  by  the  evaluation  team  can  involve  several  kinds  of 
technical  problem.  First,  for  reasons  relating  to  secrecy  or  collection 
methods,  it  is  not  always  possible  to  regionalise  statistics  in  detail.  Thus, 
surveys  constructed  on  limited  samples  (10,000  to  40,000  questionnaires 
throughout  Europe)  can  give  meaningful  results  for  large  regions  only  (e.g. 
workforce  surveys, 
innovation  surveys,  Eurobarometer).  Even  when 
statistics  are  regionalised  in  detail,  their  divisions  do  not  necessarily 
coincide  with  those  of  the  programmes.  Finally,  in  the  present  state  of 
practice,  some  data  are  calculated  and  published  with  a  delay  of  two  or 
three  years.  For  suggestions  on  the  way  to  use these  different  techniques, 
see  Volume  2  of  this  Collection. 

Performing an evaluation 

Past  evaluations  and  research 

Information found  in past  evaluations  and  research  can  play a major  role in 
all  stages  of  evaluation.  For  example,  reference  to  specialised  literature 
could  help  to  suggest  a  relevant  indicator,  a  tested  observation  grid,  an 
explanatory  model  of  impacts,  an  extrapolation  coefficient,  a  reference  for 
comparison, etc. 

It  can  prove  very  useful  to  use conclusions  from  the  evaluations  of  similar 
programmes  in the  same  region  or  other  regions.  Since  the  same  causes 
can  have  the  same  effects,  referring  to  similar  programmes  often  helps  to 
rapidly  provide  answers,  at  a  low  cost,  to  an  evaluative  question.  If  the 
evaluation  team  relies  heavily  on this type  of  secondary  data, it  must  verify 
the  external  validity  of  the  evaluations  consulted, that  is, the  possibility  of 
generalising  their  conclusions  to  other  contexts. 

It  is  not  advisable  to  invest  too  much  time  in  listing  and  collecting 
exhaustive  bibliographical  data. The  most  effective  way  of  gaining  access 
to this  information  is by  consulting  experts  with  in-depth  knowledge  of  the 
domain  and/or  the  region  concerned  by the  evaluation. 

Collecting  n ew  information 

Existing  or  secondary  data,  rarely  answer  all  the  questions  that  may  be 
asked  during  an  evaluation.  In  certain  cases,  a  high  quality  monitoring 
system  is  enough  to  evaluate  results,  and  the  abundance  of  available 
research  work  makes  it  possible  to  extrapolate  impacts.  In  other  cases, 
specific  investigations  must  be  considered  and  primary  data  collected  by 
field  surveys. 

Interviews  with  managers  and  operators 

Programme  managers  are  people  or  organisations  that  contribute  towards 
the  implementation  of  the  programme.  They  allocate  and  manage  funds. 
Operators,  situated  close  to  the  ground,  implement  the  programme. 
Broadly  speaking, the addressees  consist  of all the  people or  organisations 
that  directly  or  indirectly  benefit  from the  programme.  Since these  different 
types  of  actor,  who  sometimes  merge  and  overlap  (see  Box  31),  have 
information  at  different  levels  of  the  programme,  they  will  be  approached 
during  the collection  of  information. 

Evaluation design and management 

Box  31 -  Difference  between  operators  and  addressees 

According  to  the  definitions  in  this  book,  the  project  promoter  can  be  either  an 
operator  or an addressee.  In the example  in which the project  promoter  is a group 
of  businesses  which  receive  support  for  developing  collective  actions  to  promote 
exports, two situations could exist.  If the support  is isolated  and  limited  to a single 
project,  the  group  is  considered  a  direct  addressee  and  the  firms  comprising  it, 
indirect  addressees.  If  the  group  repeats  its  actions  every  year,  receives  variable 
assistance  depending  on  its activities,  and  reports  annually  on  its actions  and  on 
the  identity  of  the  participating  firms,  then  the  group  is  considered  an  operator 
and  the  participant  firms,  direct  addressees  (e.g.  a  Chamber  of  Commerce  and 
Industry  which  organises  training  courses  for  SMEs). 

During  a  survey  for  collecting  primary  data,  the  evaluation  team  will  not  always 
approach  the  group  of  firms  in  the  same  way.  The  questions  asked  will  differ 
according  to  whether  the  group  is  acting  as  an  operator  or  an  addressee. 

Management  data  are  secondary  data  which  can  be  accessed  through 
managers.  What  is  under  consideration  here  is  different.  It  essentially 
concerns  the  opinions  of  the  managers  and  operators,  on issues of  interest 
to  evaluation,  collected  in the  form  of  interviews.  Among  the  multiple  kinds 
of  opinions  collected  for  an  evaluation,  special  mention  will  be  made  of 
those  relating  to  the  success  and  failures  of  the  programme. 

The  questions  asked  by  the  team  during  these  interviews  are  of  the 
following  nature:  which  measures,  actions  and  projects  can  be  considered 
to  be  success  stories  (or  failures  from  which  lessons  must  be  learned)? 
How can these successes and failures be explained?  How  can a  particularly 
successful  project  be  recognised?  What  are  the  criteria  spontaneously 
used  for  this judgement  (effectiveness,  efficiency,  relevance)? 

these 

information  collected  during 

The  qualitative 
interviews  usefully 
complements  management  data,  particularly  for  identifying  examples  of 
best  practice  and  for  orienting  impact  analyses  towards  situations  from 
which the  most  can  be learned. At this stage  of  its work, the evaluation  team 
must  be  very  careful  to  focus  on  an  examination  of  programme  effects. 
Managers  and  operators  often  tend  to  judge  the  success  or  failure  of  a 
project  in relation to the  degree of  difficulty  encountered  in the application  of 
the  rules  or  in  the  functioning  of  the  administrative  machine.  Evaluation, 
however,  judges  the  success  of  a  project  in  relation  to  its  beneficial  and 
sustainable  results for the addressees. These two forms  of success are  often 
related  but  a  project  may  be  problematical  from  the  administrative  point  of 

Performing an evaluation 

view  (e.g.  barely  acceptable  interpretation  of  the  texts;  conflict  between 
managers)  but  nevertheless  an exemplary  success from the  point  of view  of 
its effectiveness  (although  the opposite  is more often true!). 

The  opinions  of  managers  and  operators  are  most  often  obtained  through 
individual  interviews. The evaluation  team tries to  hold  enough  interviews  to 
meet  at  least  one  person  fully  concerned  by  each  objective  at  each  level  of 
the  programme.  It may  be interesting, from this  point  of view, to  mark  all the 
interviews  held  on  the  objectives  tree.  The  number  of  interviews  may  vary 
from a few to several dozen, depending on the complexity  of the programme. 

With the  commissioner's  agreement,  the  evaluation  team  can  complement 
individual  interviews  by  a series  of  meetings,  each  attended  by  about  ten 
managers  or  operators.  These  meetings  may  be  organised  by  town,  sub-
region,  economic  sector,  domain  of  intervention,  etc.  It  may  also  be 
interesting  to  organise  meetings  separately,  at  the  different  levels  of  the 
programme  (e.g.  operators 
in  the  field,  managers  of  the  measures, 
managers  of  the  entire  programme).  An  example  of  the  latter  approach  is 
given  in  Volume  4  of  this  Collection,  devoted  to  innovative  techniques  for 
the  evaluation  of  complex  programmes.  Volume  3,  devoted  to  evaluation 
tools,  features  several  techniques  for  running  meetings  and  for  mediation, 
designed  to  make these  meetings  more  productive  and  impartial. 

Interviews  with  addressees 

threatened  by  unemployment, 

By  definition,  socio-economic  programmes  simultaneously  cover  several 
intervention  (i.e.  several  sectors)  and  numerous,  often 
domains  of 
heterogeneous,  groups  (e.g.  SMEs,  entrepreneurs,  farmers,  independent 
workers,  wage  earners 
jobless, 
researchers,  etc.).  Consequently,  the  overall  evaluation  of  a  programme 
systematically  encompasses  all  domains  and  concerned  groups.  This 
means  that  a  way  has to  be  found  to  collect  genuinely  new  information  in 
these  highly  heterogeneous  groups, without  incurring  exorbitant  costs.  If a 
questionnaire  survey  is  chosen,  for  example,  the  cost  of  constructing  a 
representative  sample  for  each  of  the  concerned  groups  will  be  extremely 
high.  In  practice  it  can  be  done  only  for  the  in-depth  analysis  of  an 
evaluative  question  focused  on  a particular  public  (see below). 

the 

Pragmatic  techniques  of  primary  data  collection  among  very  different 
addressees  include focus groups  and case studies. These are presented  in 
the following  paragraphs. 

Evaluation design and management 

The  focus  group  technique  has  been  standardised  by  market  research 
professionals.  A  focus  group  consists  of  six  to  eight  participants  and  a 
leader,  and 
interviewer 
lasts  for  one  and  a  half  to  two  hours.  The 
encourages  a high  level of  interaction  between  the  participants  on  essential 
subjects  of  the  study.  By  playing  on  the  participants'  creativity  and  on  the 
dynamics  of  the  group,  the  technique  substantially  increases  the  capacity 
for  exploration  and  helps  to  reveal  convergent  ideas.  If  the  group  is  led 
skilfully,  the  product  of  these  discussions  is  often  far  richer  than  that  of  a 
series  of  individual  interviews. 

The  presence  of  several  participants  helps to  avoid  extreme  opinions  since 
each  argument  is  contested.  However,  clientelism  among  participants,  the 
presence  of  "leader"  personalities  or a conflictual  situation  between  certain 
people  can  spoil  the  quality  of  a  meeting.  It  is  for  this  reason  that  focus 
group 
involve  managers  and 
operators.  Problems  of this  kind are less likely to  be encountered  in  surveys 
involving  addressees,  and  can  be avoided  by organising  a series of three  or 
four  meetings.  More  information  on  this  technique  is available  in Volume  3 
of  this  Collection. 

interviews  are  more  difficult  when  they 

Analysing  all  impacts  on  addressees 

It  is  important  to  remember  that  the  framework  being  examined  here  is  the 
overall evaluation  of  a programme  which  comprises  heterogeneous  outputs 
and  produces  multiple  impacts  on  varied  publics.  This  situation  precludes 
the  use of  most  classical  impact  analysis techniques,  presented  below  (see 
page  153).  It  is  furthermore  important  to  distinguish  impacts  in  terms  of 
their  level,  that  is,  those  which  can  be  closely  monitored  among  the 
programme  beneficiaries,  and  those  which  only  materialise  on  a  regional 
scale.  The  first  level  of  impact  concerns  micro-economic  or  micro-social 
mechanisms,  while  the  second  level  concerns  macro-economic  or  macro-
social  mechanisms. These impacts are analysed  below  in a section  devoted 
to  the  subject  (see  page  135). 

Combining  of  primary  and  secondary  data 

The  evaluation  team  starts  its  impact  analysis  by  observing  changes  that 
have  taken  place  during 
their 
concomitance  with  a  policy  action.  It  is  important  to  be  aware  at  this  stage 
of the evaluation  that  it  is difficult  to  find  proof  of  causal  relations. The  team 

the  programme,  and  by  questioning 

Performing an evaluation 

will 

therefore  try  to  establish  presumed  causality  and  to  explain  the 

observed  phenomena. 

It  is  interesting,  for  example,  to  combine  statistical  data  with  management 

data  in a  before-after  perspective,  as in the following  case. 

Example: A combination  of data  which  reveals  a plausible  impact 

In  an evaluability  study  of  several  measures  of  the Objective  1 ERDF  Operational 
Programme (1994-99) in the region of Valencia (Spain), a number of evaluation criteria 
were  determined  by type  of  action.  For each  of  these  criteria,  between  three and 
eleven  indicators  accompanied  by their  sources of information  were  also  proposed. 
The  total  number  of  proposed  indicators  amounted  to  123. About  40 of them  were 
eliminated at the start  of the intermediate  evaluation  because of their  unavailability. 
The  evaluators  then  selected  those  of  the  remaining  indicators  which  provided 
contrasting  information  (Valencia  region and Spain), and those  based on information 
published  periodically  (for the study  of trends  for 1990-93)  and on unpublished but 
available information for the most recent period. The indicators thus defined, featured 
in the evaluation  report, and were analysed in depth by means of graphs. With  these 
graphs  it was possible to examine trends  (favourable or unfavourable) for the  period 
preceding  the implementation  of the programme,  and to identify  whether  there was 
a variation  in this  tendency  in the following  years.  Changes  were  classified  positive 
when they concerned an improvement  in the situation in the region, or negative  when 
they  concerned  a deterioration  in the situation.  The intensity  of  the variation was 
qualified as insignificant,  moderate or intense. 

For example, for the criterion  "diversification  and efficiency of energy resources", one 
of the  indicators  used was the number  of firms  using  natural  gas (% Valencia  region 
compared  to Spain  as a whole).  For the period  1990-93  this  indicator  revealed an 
unfavourable  trend.  From  1994, at  the start  of  the  programme  implementation,  a 
positive  upward  variation  in the trend was observed. 

M  ►  Particularly  interesting  combinations  can  be  achieved  by  means  of  a 

tool  that  is  increasingly  widely  used:  the  geographical  information  system 

(GIS).  This  technique  combines  a  computerised  data  base  capable  of 

memorising  programme  management  data  and  context  statistical  data.  A 

GIS serves to  produce  maps on which overlaps  between data are visible. For 

example,  a  map  may  superimpose  unemployment  trends  in  the  different 

urban  or  rural  areas  covered  by  the  programme,  and  information  likely  to 

explain  these  trends,  including  the  importance  of  the  funds  allocated,  the 

number  of  jobless  people  trained  and  the  number  of  firms  assisted.  For 

further  information  on this technique, see Volume 3 of the MEANS Collection. 

Evaluation design and management 

Exploring  impact  mechanisms 

Case study  surveys examine a limited number of actions, projects or  situations 
(cases) and produce a series of monographs  describing  each of these cases in 
a  few  pages.  For  each  of  the  projects  studied,  the  researcher  gathers  all 
available  management  data,  questions  the  relevant  managers  and  operators, 
and  interviews  several  direct  and  indirect  addressees.  Each  case  study 
involves  considerable  expense  but  the  number  of  case  studies  is  sufficiently 
small to  keep  the total  cost  of the  evaluation  within  reasonable  limits. 

A  case  study  survey  provides  explanations  of  the  addressees'  attitudes, 
real  needs,  reasons for  participating  in the  programme  and  ways  of  judging 
the  quality  of  the  programme,  as  well  as on  the  mechanisms  leading  to  the 
appearance  of  impacts,  to  unexpected  effects,  and  so  on.  Because  the 
case study technique  is essentially  qualitative, its main function  is to  explain 
the  functioning  of  the  programme.  It  also  helps  to  interpret  quantitative 
information  obtained  elsewhere. 

Within  certain  limits,  case  studies  can  be  used  to  produce  quantitative 
information.  In fact, each  monograph  can  produce  a quantitative  estimation 
of  impacts,  obviously 
limited  to  the  project  under  consideration.  The 
question  is  then  that  of  the  generalisation  of  lessons  learned  from  the 
cases.  For  example,  if  each  case  is  selected  as  "typical"  of  a  category  of 
projects,  its impact  will  be considered  to  be an image or an  indication  of  the 
extent  of  the  impact  of  all  the  projects  in  this  category.  A  more  interesting 
approach  would  be  to  study  two  cases  per  project  category,  one  case  of 
success  and one of failure, so as to obtain  a range of estimated  programme 
effects.  For a  more  complete  presentation  of  the  case  study  technique,  the 
reader  is 'eferred to Volume 3 of the  MEANS  Collection, and for an  example 
of  the  quantitative  use  of  case  studies,  to  Volume  4. 

Quantitative  estimation  by  extrapolation 

The quantitative  estimation  of an impact,  or at least the demonstration  of  its 
existence,  constitutes  a  central  theme  of  evaluation.  The  basic  techniques 
used  for  this  type  of  analysis  are  comparison  groups  and  regression 
analysis,  as  well  as  the  modelling  of  macro-economic 
impacts.  These 
techniques  are  presented  further  on  in this volume. This section  focuses  on 
to  the  overall 
the  only  approach  which 
evaluation  of  a  complex  programme:  the  use  of  extrapolation  coefficients 
for  estimating  impacts. 

is  pragmatically  applicable 

Performing an evaluation 

Extrapolation  coefficients  are  used, for  example, to  estimate  the  number  of 
temporary  jobs  created  per  million  euros  of  investment  in  SMEs,  or  the 
proportion of genuinely  new jobs created, reported by firms. If the  evaluation 
team  uses existing  extrapolation  coefficients,  it  has to explain  and justify  its 
choices.  A  coefficient  may  be  deduced  from  an  in-depth  study  carried  out 
during  the  same  evaluation  or  a  former  evaluation.  A  coefficient  may  be 
borrowed  from  an evaluation  or study  of  similar  interventions,  provided  that 
the  conclusions  can  be  generalised. A  coefficient  may  also  be  established 
for the  needs  of  the  evaluation  by  an expert  panel  or  a  network  of  experts, 
by using the  Delphi panel technique  (see Volume 3 of the MEANS  Collection 
for further  information on this technique for consulting experts). The different 
solutions  may  also  be combined. 

•4  Extrapolation  consists  of  applying  an  impact  coefficient  to  an  initial 
datum  which  may  be  different,  depending  on  the  case.  In  an  ex  ante 
evaluation, the extrapolation  can  be  based only  on  basic  data  on the  funds 
that 
interventions. 
Coefficients  such  as  "number  of  jobs  created  per  million  euros"  (or  the 
inverse  ratio  of  cost  per job  created)  is then  used. 

the  programme  has  earmarked  for  the  different 

H  ►  In an intermediate  or ex  post  evaluation, output  data  can  be used  as 
a  basis  for  extrapolation.  Thus,  for  example,  the  number  of  beds  in  a 
recently  built  tourist  facility  will  be  used  as a  basis  to  estimate  the  number 
of jobs  created  in terms  of  a ratio  of the  number  of jobs  per  bed. In certain 
cases, addressees  are asked  to  make  a statement  on  impacts  during  their 
contact  with the operators. They  must, for example, indicate the  number  of 
jobs  created by the time the  last  payment  was  made, or the number  of jobs 
to  be  created  when  they  applied  for  assistance.  In  this  case  it  is 
recommended  that  a correction  coefficient  be applied, to take  deadweight 
into account.  This  coefficient  may vary from  0.8 at best, to  less than 0.3  for 
certain  interventions  where  deadweight  is particularly  high. 

Analysing  the  effects  of  synergy 

A programme  is composed  of  a set  of  projects  and  measures  which  are all 
oriented  towards  the  same  objective.  These  projects  are  not  independent 
and should  not  be evaluated  by adding  or even synthesising  their  separate 
effects.  On the contrary,  the  components  of  a programme  are supposed  to 
reinforce one another  through  synergy  effects. The following  box  describes 
a number  of  mechanisms  which  produce  these  effects. 

Evaluation design and management 

Box  32 -  Examples  of  mechanisms  producing  synergy  effects 

in a  programme 

A  study  by  Cambridge  Policy  Consultants,  entitled  Program Induced  Dynamic 
Effects, proposes a list of  mechanisms likely to produce effects  of synergy: 
•  critical  mass  effect:  an  isolated  project  would  have  no  impact  without  an 

adequate critical mass; 

•  size effect:  reduced costs due to the presence of several projects  in the same 

location; 

•  demonstration effect  : multiplication of the action through example; 
•  adjustment  effect: better coordination between different  public actions; 
•  capacity  effect:  an  isolated  public  action  would  have  no  impact  without 

adequate critical mass; 

•  productivity effect: the economies of scale of the public action itself improve its 

productivity; 

•  release  effect:  the  implementation  of  a project  eases a constraint  on another 

project; 

•  Impulsion effect:  the  implementation  of  a project  is an incentive to  implement 

another  project. 

Although  synergy  is an important  mechanism  conditioning  the  achievement 
of  programme  objectives,  it  is  rarely  analysed  in  its  own  right,  especially 
because  this  would  involve  additional  costs,  even  if  the  question  can  be 
treated  only  within  the framework  of  an overall  evaluation. To overcome  this 
problem,  a  methodological  exercise  was  carried  out  within  the  framework 
of the  MEANS  programme. The  proposed  approach  is presented  in detail  in 
Volume  4  of  the  Collection.  It  starts  with  the  construction  of  a  matrix  of 
cross-impacts,  where  all 
appear 
symmetrically,  in  lines  and  columns.  By  means  of  interviews  with  the 
managers  and  operators,  the  evaluation  team  assesses  the  degree  of 
importance  of  potential  synergy  between  the  components  (on  a  scale  of 
scores  from  -3  to  +3).  These  estimates  are  used  to  fill  in  the  cells  of  the 
matrix  and  to  calculate  a  synergy  coefficient  for  each  component.  Certain 
cases  of  potential  synergy  can  then  be  verified  in  the  field,  particularly  if 
they  have  significant 
in  terms  of  an  understanding  of 
programme  effects  or  decisions  to  be taken. 

the  programme 

components 

implications 

F o r m u l a t i ng  j u d g e m e n ts  a nd  c o n c l u d i ng  t he  e v a l u a t i on 

Judgement  references 

The  proximity  of  the  word  evaluation  to  the  word  value,  implies  that  an 
evaluation  is,  by  definition,  a  value  judgement.  The  credibility  of  the 

Performing an evaluation 

evaluation  is  based  entirely  on  the  clarification  of  the  references  of  this 
judgement  -  an essential  step  in any evaluation  and one which  is too  often 
neglected. A value judgement  relates to  one or more criteria, each of  which 
may  also  be associated  with  a measurement  and  a standard  to  be  met,  as 
is symbolised  by the thermometer  in Box 33. 

Box 33 -  Reference,  norm  and  criterion 

degrees Celsius or 
% of trainees employed 
after  12 months^ 

Unit  of  measurement 

Judgement 
reference 

criterion 

temperature or 

Rate of sustainable integration 
into employment and society 

Each evaluation  and each answer  to an evaluative  question  can  be based  on 
a  specific  evaluation  criterion  chosen  during  the  design  phase.  Certain 
evaluation  criteria  predominate,  however,  because  they  are  recurrent  and 
essential. They are found in almost  all evaluations. These main criteria are the 
relevance  of  the  programme  in  relation  to  needs,  the  effectiveness  of  the 
programme  in achieving  its objectives,  and the  efficiency  of  the  programme, 
that  is, its capacity to achieve the objectives at the lowest cost ( see page 70). 

Once the criteria  have been  chosen  and their  respective  standards  set, the 
measurement  tools  that  are  selected  are  usually  indicators  and,  in  certain 
cases, scores. 

If  the  programme  could  be  evaluated  in  relation  to  a  single  criterion,  for 
example  efficiency  in terms  of  cost  per job  created, the  formulation  of  the 
judgement  could  be  based  on  a  single  technique,  in  this  case  cost-
effectiveness  analysis. The  programme  could  then  be judged  in  relation  to 
this criterion. 

Evaluation design and management 

However,  in the  evaluation  of  a programme  in which  the  impacts  are  not  all 
in-depth 
known  very  precisely,  this  type  of  technique,  more  suited  to 
evaluations  of  simple  actions,  is  not  able  to  give  very  reliable  results.  It  is 
usually  recommended 
in  such  cases  to  take  cost  as  a  criterion  of 
effectiveness  but  to  use  this  criterion  for  establishing  comparisons.  The 
average  cost  per  job  created  can  then  be  calculated  and  this  ratio 
compared  to  other  ratios  obtained  elsewhere,  in  the  context  of  other 
programmes  with  similar  characteristics.  The  "¡deal"  ratio,  generally  the 
best  one, must  be defined  and will become the "standard" or the  norm. This 
principle  of  benchmarking,  which  requires  the  identification  and  validation 
of  references,  is  also  a  technique  which  can  be  used  to  establish  a  value 
judgement  in  relation  to  a model  known  for  its  quality. The  multiplication  of 
objectives  requires  the  identification  of  multiple  cases  of  best  practice, 
each  identified  for  its  quality  in  a  particular  domain. 

Grading  criteria 

The  evaluation  team  is often  confronted  with  the  need to take  into  account 
different  criteria  and  to  choose  between  them  when  drawing  conclusions. 
For  example,  what  can  be  concluded  about  a  measure  which  is  expensive 
and  creates  few  jobs,  but  affects  a highly  disadvantaged  public?  This  type 
of  trade-off  must  be  explicit  and  justified,  or  it  could  compromise  the 
impartiality  of  the  evaluation.  In  case  of  difficulty,  it  is  better  for  the 
evaluation  team  to  ask  to  be  guided  by  the  steering  group,  in  defining 
priority  criteria.  If the  steering  group  also  has  problems  making  a choice  on 
subjects  considered  too  sensitive,  it  can  request  the  assistance  of  a 
mediator.  In this  respect,  it  is  possible  to  use  certain  evaluation  tools  such 
as  mediation  techniques.  Tools 
like  METAPLAN®  or  SWOT  analysis 
(strengths-weaknesses-opportunities-threats)  have  the  characteristic  of 
eliciting all opinions  and  of  identifying  points  of consensus, without  causing 
conflict  (see Volume  3  in this  Collection). 

It  is  also  possible  to  use  multicriteria  analysis  which  allows  several  criteria 
to  be simultaneously  taken  into account  by explaining the relations  between 
them.  Use  of  this  technique  is  frequent  in  project  evaluation  but  is  still 
considered  innovative  in  the  overall  evaluation  of  multi-decision-maker 
programmes.  An  example  of  the  application  of  this  technique  is  described 
in  Volume  4  of  this  Collection.  In  programme  evaluation,  multicriteria 
analysis  implies  the  active  participation  of  a  steering  group,  if  possible 
extended  to  include  about  ten  members.  The  analysis  is  used  to  help  the 

Performing an evaluation 

participants collectively  formulate  a comparative judgement  on the  different 
measures  of  the  programme.  Apart  from  the  capacity 
to  produce 
consensual  conclusions;  the  technique  is  also  useful  for  clarifying  and 
explaining judgement  criteria. 

Cost-benefit  analysis  is  another  approach  that  can  be  used  to  grade 
criteria.  Each  impact  is  weighted  by  the  evaluation  team,  by  means  of  a 
monetary  value  representing  what  society  or  persons  concerned  are 
prepared  to  pay  to  obtain  positive  impacts  or  avoid  negative  impacts.  Use 
of  this  technique  is often  problematical,  mainly  because  of  the  difficulty  of 
putting  certain 
incontestable  financial  terms  (especially 
environmental  impacts).  Moreover,  the  technique  is  extremely  difficult  to 
apply  to  a  large  programme  and  is  practically  limited  to  the  evaluation  of 
infrastructure  projects. 

impacts 

into 

When  it  seems  too  difficult  or  complicated,  in an evaluation, to  embark  on 
explanations  or the grading of criteria, a good alternative is to ask an expert 
panel  to  formulate  conclusions.  In  fields  such  as  research  or  education 
policies,  the  expert  panel  technique  is  used  extensively.  The  experts  are 
chosen  to  represent  all  points  of  view,  in  a  balanced  and  impartial  way. 
These experts are independent  specialists, recognised  in the domain  of the 
evaluated  programme.  They  are  asked  to  examine  all  the  data  and  all  the 
analyses  made  during  the  evaluation,  and  then  to  highlight  consensus  on 
the  conclusions  that  the  evaluation  must  draw,  and  particularly  on  the 
answers to give to evaluative questions. The panel does not fully explain  its 
judgement  references  nor its trade-off  between criteria, but the credibility  of 
the  evaluation  is  guaranteed  by  the  fact  that  the  conclusions  result  from 
consensus  between  people  who  are  renowned  specialists  and  represent 
the different  "schools  of  expertise". 

Writing  up the  conclusions  and  draft  report 

The independence  of  the evaluation  team  gives  it  a wide  degree  of  latitude 
for  formulating  the  conclusions  that  it  considers  can  be  drawn  from  the 
collected  and  analysed  data.  This  latitude  is  nevertheless  framed  by  the 
constraints  of  professionalism  which  cannot  be  ignored. 

Initially,  the  conclusions  are  constructed  and  presented  in  the  form  of 
answers to evaluative questions. These answers  are expressed  in the  same 
terms  as  the  questions  asked  in  the  terms  of  reference,  and  are  as 
conclusive  as  possible.  If one  of  the  questions  asked  is  not  answered,  the 

Evaluation design and management 

reasons  are  given  explicitly.  In addition  to  answers  to  the  questions  asked, 
the  evaluation  includes  all  the  conclusions  it  considers  to  be  valid  and 
relevant  for  programme  managers.  This  is the  case,  in  particular,  when  the 
evaluation  has  made  it  possible  to  identify  unexpected  negative  effects,  or 
if  it  has  shown  that  an objective  is  no  longer  relevant. 

In  the  case  of  overall  evaluation  of  a  programme,  conclusions  are 
formulated  at the  programme  level, that  is to  say, they  are transversal  to  the 
measures  or  actions  comprising  it.  These  transversal  conclusions  may  be 
formulated  in  the  following  terms:  the  best  results  in  employment  or  the 
environment  were  obtained  in  a  particular  part  of  the  programme;  certain 
measures  are  particularly  efficient;  certain  measures  have  become  less 
relevant  after  changes  of  context,  etc.  It  is  not  acceptable  for  the  overall 
evaluation  of  the  programme  to  result  in  the  juxtaposition  of  conclusions 
peculiar  to  each  measure.  Volumes  4  and  5  of  this  Collection  provide 
technical  proposals  for  helping  evaluation  teams  formulate  transversal 
conclusions. 

One  of  the  rules  of  professionalism  consists  of  linking  each  conclusion,  as 
explicitly  as possible, to  the  data  and  analyses  that  can  be used  to  support 
it:  statistical  or  bibliographic  sources,  facts  and  opinions  collected  during 
surveys  (respecting  anonymity  when  necessary),  results  of  analyses, 
precise  references  of  comparisons,  etc.  Methodological  limits  are  also 
discussed,  to  show  whether  the  link  between  the  data  and  conclusions  is 
robust,  and  to  what  extent  the  conclusions  can  be  generalised. 

There  is  a  difference  in  the  nature  of  conclusions  and  recommendations. 
Conclusions  follow  from  the  observations  and  analyses  of  the  evaluation 
team.  Their  value  judgements  are  based  on  clear  and  explicit  references, 
and  their  future  lies  in the  hands  of  the  political  officials  of  the  programme. 
The  independence  of  an  evaluation  requires  conclusions  to  be the  ultimate 
responsibility  of  the  evaluation  team, even  if they  can  usefully  be  discussed 
by the steering  group  in their  draft  form. Recommendations,  in contrast,  are 
more  subjective  and  political.  It  is  not  unusual  for  recommendations  to  be 
made  by  both  the  evaluation  team  and  the  steering  group.  This  is  a 
suggestion  developed  in  the  last  part  of  this  Volume  (see  pages184 
and  185). 

Performing an evaluation 

Box  34  -  Contents  of  the  evaluation  report 

The  contents  of  the  report  must  include  all  the  elements  which  allow  for  a  clear 
understanding  of  the  results  of  the  evaluation  and  the  reasoning  followed.  The 
volume  and  variety  of  information  in  the  report  makes  it  necessary  to  precisely 
distinguish contents according to their nature. Six main parts can be distinguished: 

•  Elements  of  information  on  the  evaluated  programme:  the  history  of  the 
programme  and  the  measures  comprising  it;  the  analysis  of  documents;  the 
displayed  objectives;  etc. 

•  A description of the method, which  recalls the different  stages in the  evaluation 
process, the choices made by the steering group, reports on the studies  carried 
out and their  respective  contribution  to  the evaluative questions, and the  limits 
of  the  method  used. 

•  The presentation  of information constituting the basis of the evaluation: primary 

or  secondary  data collected. 

•  The arguments, that  is, the analyses and interpretations which  make  it  possible 

to  formulate  conclusions  from the  information  collected. 

•  A  clear  and  precise  description  of  the  conclusions  which  must  answer  the 

questions  asked  in the terms  of  reference. 

•  And,  when  the  terms  of  reference  specify  it,  recommendations  reflecting  the 
opinions  of  the  evaluation  team.  These  recommendations  should  be  clearly 
graded  according  to their  importance  (in terms  of  usefulness). 

Furthermore,  any evaluation  report  should contain an  executive  summary,  that  is, 
a two  to  five  page  summary  of  the six  parts. 

Box  34  presents  the  typical  contents  of  an  evaluation  report. 

Validation  of  t he  evaluation  and  final  report 

The  commissioner  or,  more  often,  the  steering  group,  expresses  an  opinion 

on  the  validity  of  the  evaluation  team's  work  at  the  draft  final  report  stage. 

One  of  the  following  decisions  is  then  made,  based  on  this  assessment: 

•  The  evaluation 

team  has  not  fulfilled 

its  contract  and  must  perform 

additional  work  without  an  additional  contract. 

•  The  evaluation 

team  must  c o n d u ct  new 

field  studies 

to  empirically 

support  some  of  its  conclusions  or  recommendations  which  appear 

interesting  but  concern  subjects  about  which  little  is  k n o w n.  This  work 

necessitates  an  additional  contract  and  budget. 

•  The  report  is  technically  and  qualitatively  acceptable. 

Evaluation design and management 

In the  first  two  cases,  the  steering  group's  opinion  must  be  confirmed  by 
the  commissioner  of  the  evaluation.  It  is  only  after  validation  of  the 
conclusions  by the  competent  authorities  that  the  report  can  be  definitively 
adopted  and  the  work  finalised.  This  step  is  tricky  because  it  consists  of 
clearly  distinguishing  both  the  technical  quality  of  the  evaluation  team's 
work,  and  the  conclusions  themselves.  Although  it  is  normal  for  the 
commissioner  to  control  the  technical  quality  of  the  evaluation,  he  or  she 
must  avoid  interfering  with  the  conclusions  if  the  evaluation  is  to  remain 
independent.  But  the  quality  of  the  work  which  enabled  the  conclusions  to 
be  drawn  is  often  related  to  the  content  of  these  conclusions.  To  facilitate 
this final validation  stage, it is preferable to announce  in advance the  criteria 
on  which  the  technical  quality  of  the  work  will  be  judged,  for  example  by 
referring  to  the  grid  proposed  on  page  180. 

Once the  report  has been validated, the evaluation team and  steering  group 
can  classify  the  main  conclusions  in a table with the following  notes:  "valid, 
credible  and  soundly  supported  conclusion";  "conclusion  not  satisfactorily 
supported  and  may  require  more in-depth  analyses"; "conclusion  leading  to 
interesting  recommendations";  "conclusion  contributes  new  knowledge"; 
etc. 

To  conclude  these  first  two  chapters,  it  should  be  underlined  that  the 
performing  of  an overall  evaluation  of a  programme  is indispensable  insofar 
as  the  principle  of  programming  is  an  essential  part  of  Structural  Fund 
regulations. Such an approach, at the overall  programme  level, is  facilitated 
today  by  the  experience  acquired  over  recent  years. 

The  rest  of  the  process  consists  of  disseminating  the  evaluation  report  and 
organising  its use. These steps are considered further on  in this volume (see 
page  159).  Before  continuing,  it  is  important  to  mention  the  ways  in  which 
more  in-depth  impact  analyses  can  be  performed. The  following  pages  will 
be  devoted  to  this  subject. 

Analysing macro-economic  impacts 

In addition to the overall  evaluation of a programme,  it may be necessary  to 
analyse  macro-economic  impacts 

The impacts  of  Structural  Fund interventions  on a regional  or  national  scale 
depend  on  the  micro-economic 
in  all  domains  of 
intervention,  but  also  on  macro-economic  mechanisms  which  appear  only 
on a  larger  scale. 

impacts  obtained 

These  mechanisms  are  difficult  to  evaluate  credibly  by  means  of  simple 
techniques.  It  is  possible  to  understand  them  more  fully  and  to  provide  a 
schematic  representation  of  them.  Their  quantitative  estimation  involves 
modelling  techniques  which,  in  recent  years,  were  developed  specially  for 
the  evaluation  of  European  cohesion  policy. 

Can macro-economic  impacts  on a regional  or  national  scale  be  considered 
to  be the  sum  of all  micro-economic  impacts? The debate  is not  closed, but 
has nevertheless  led to the recognition of the importance of  macro-economic 
mechanisms  which  condition  the  achievement  of  global  development 
objectives. This section  is devoted  entirely to  these  mechanisms. 

The examples  in Box 35 serve to distinguish these two types of mechanism. 

Box 35 -  Examples  of  micro-  and  macro-economic  mechanisms 

-  Thanks  to  assistance  received,  a  firm  develops  (micro  level)  and  buys  more 
from its regional suppliers which, in turn, buy more from their regional  suppliers, 
etc.  (supplier  effect,  macro  level). 

-  Thanks  to  assistance  received,  a  firm  expands  (micro  level)  and  pays  out 
additional  wages  which  are  partly  spent  in  the  region.  This  triggers  more 
development,  followed  by  further  income  growth, etc.  (multiplier  effect,  macro 
level). 

-  A  public  investment  fills  the  order  book  of  a  regional  firm,  which  reduces 
competition  when subsequent  investments are made (micro level). This, in turn, 
leads  to  an  increase  in  price  levels, etc.  (macro  level). 

-  To  finance  their  contribution  to  assisted  projects  (micro 

level),  regional 

authorities  increase their  taxes,  reducing  available  income,  etc.  (macro  level). 

-  Thanks  to assistance  received, a firm  becomes  more  competitive  (micro  level), 

which  eliminates  rival regional firms  (macro  level). 

Evaluation design and management 

From  the  point  of  view  of  an  evaluation,  micro-economic  impacts  can  be 
evaluated  by  questioning  a limited  number  of  people,  i.e. direct  addressees 
(where  direct  effects  have  been  observed)  and  people  who  have  been  in 
contact  with direct  addressees  (where indirect  effects  have  been  observed). 
Significant  effects  are  undeniably  produced  beyond  these  limited  circles  of 
addressees,  as  Box  35  shows.  These  more  distant  effects  involve  macro-
economic  mechanisms  which  potentially  affect  all  markets  or  economic 
actors  in the  territory  concerned. 

A v o i d i ng  o v e r - s i m p l i f i ed  a p p r o a c h es 

indicators  such  as  the  per  capita  GDP  or 

Macro-economic  changes  in  a  region  can  be  observed  relatively  easily  by 
means  of  statistical 
the 
unemployment  rate,  the  two  most  frequently  used  indicators  of  this  kind. 
This  availability  naturally 
leads  evaluation  teams  to  make  before-after 
comparisons  of  the  regional  economic  situation.  When  a  programme 
accounts  for  only  a  few  percent  or  a  fraction  of  a  percent  of  the  total 
investments,  the  observation  of  global  developments  has  no 
regional 
significance  in  terms  of  impact  -  no  more  than  have  comparisons  with  the 
global  economic  development  of  neighbouring  regions, even  if they  are  not 
eligible  for  public  support. 

Only  in cases  of  a  massive  transferral  of  public  funds  towards  a  region  that 
is  lagging  behind  in  its  development,  can  evaluation  be  founded  on  the 
observation  of  major  economic  trends. This  is what  was  done  in the  1960s 
for  the  evaluation  of  regional  policy,  by  means  of  a  technique  called  shift-
share  analysis.  This  technique  and  all  those  presented  in  this  chapter  are 
described  in  information  sheets  in  Volume  3  of  this  Collection.  Shift-share 
analysis  is  a  top-down  evaluative  approach  which  compares  observable 
in  the  assisted 
employment 
territories.  It  breaks  down  regional  employment  into  large  sectors,  and 
applies  to  them  the  development  rates  observed  at  a  national  level  for  the 
same sectors.  In this way a counterfactual  situation  is constructed,  showing 
how  the  region would  have developed  if its development  had  corresponded 
to  the  national  average.  The  comparison  with  real  development  reveals  a 
difference  (the  residue)  which  is  entirely  imputed  to  the  effects  of  outside 
support.  The  extremely  approximate  nature  of  this  analysis  can  be  justified 
only  if external  support  is very  large. 

trends  on  a  national  and  regional  scale 

A  number  of  macro-economic  effects  can  be  estimated  by  means  of 
extrapolation  coefficients  drawn from former  research or analyses,  according 

Performing  an  evaluation 

to the same logic  as that  described  above (see page  126) for the  estimation 
of  micro-economic  impacts.  The  coefficients  of  macro-economic  effects 
apply  to  those  micro-economic  effects  which  are  supposed  to  have  been 
estimated  beforehand. The  formulae  are of  the  following  type: 

Macro  effect  =  Micro  effect  X  Coefficient. 

The supplier effect  coefficient  varies between  1.1  and  1.5, depending on the 
size and  degree  of  openness  of  the  regional  economy.  The  multiplier  effect 
coefficient  varies  within  a comparable  margin. 

The  evaluation  team  must  take  very  serious  precautions  and, if  necessary, 
call  on  experts  for their  advice,  before  using  macro  effect  coefficients.  One 
of  the  reasons  for  this  need  for  caution  is  that  the  value  of  the  coefficient 
varies  widely  from  one  region  to  the  next.  The  other  reason  is  that  these 
macro-economic  mechanisms are numerous and the unfavourable ones are 
often  overlooked. Thus, for example, eviction  or  price  increases  require  the 
use of  significant  negative  coefficients. 

the 

limits 

inherent 

the  simplest  approaches, 

two 
in 
Because  of 
complementary  directions  are  proposed  for  the  evaluation  of  macro-
economic  impacts.  The  first  consists  of  developing  a  logical  model  of 
relations  of  cause  and  effect  between  the  programme  and  its  macro-
economic  objectives.  This  qualitative  approach  emphasises  those  effects 
which  are  the  most  important  in  terms  of  development,  i.e.  structuring 
effects.  The  second  approach,  of  a  quantitative  nature,  consists  of  using 
econometric  modelling  to  simulate  regional  economic  development  with 
and  without  the  programme,  to  deduce  an  estimated  impact.  These  two 
approaches  are considered  in the  following  sections. 

Linking  effects  at  t he  micro  a nd  m a c ro  levels 

A  development  programme  aims  at  producing  sustainable  structural 
change  throughout  a  region  or  country.  The  intended  effects  have  two 
dimensions,  defined  as  follows:  effects  which  must  last  even  after  the 
public  support  is  discontinued  (notion  of  structuring  effect)  and  effects 
which  occur  at  the  macro-economic  and  macro-social  levels. 

By  contrast,  programme  outputs  are  always  micro-economic  or  micro-
social  and  are  situated  at  a  particular  point  in time. They  are  not  expected 
to  last  indefinitely,  for  this  would  implicitly  amount  to  the  failure  of  the 
programme.  An  evaluation  must  therefore  try  to  identify  and  connect  the 

Evaluation design and management 

sequences  of  causes  and  effects  which  make  it  possible  to  establish  links 
between  outputs  at  a  given  date  and  permanent  effects,  and  between  the 
micro  and  macro  levels. 

This  approach  is similar to that  of  the  logical  diagram  of  impacts  which  can 
be  applied  both  to  the overall  evaluation  of  a  programme  (see page 95)  and 
to  in-depth  analyses  (see  page  150). 

Global  impact  on  social  and  economic  cohesion 

The  diagram  in  Box  36  provides  an  overview  of  the  mechanisms  linking 
Structural  Fund  programme  outputs  to  the  global  objective  of  the  policy 
under  which  they  fall,  i.e.  economic  and  social  cohesion.  It  is  a  logical 
representation  of  the  effects  of  interventions  on  development,  through  the 
mechanisms  of  structuring  effects.  The  following  sections  study 
this 
process,  step  by  step. 

The  diagram  was  developed  by  the  authors  of  this  volume,  following  their 
work  on  the  MEANS  programme.  Other  authors  would  probably  draw 
different  diagrams  or  use  different  vocabulary.  If these  diagrams  were  used 
within the framework  of  a particular  study, the  evaluation  team  and  steering 
group  would  need  to  adapt  them  to  suit  the  programme  strategy  and 
regional  context. 

Box  36  -  Micro-macro  articulation:  an  overview 

Micro 

level 

Macro  level 

Outputs 
the  domains  of.. 

in 

...  human 
resources 

.the  productive 
environment 

...basic 
¡nfrast'ucture 

Immédiat  impact 
on  development 

\ 

Sustainable  impact 
on  development 

Reduced 
unemploy­
ment 

Increased 
income 

Better 
living 
conditions 

Economic 
and  social 
cohesion 

Performing  an  evaluation 

The objective  of economic  and social cohesion, as envisaged  in the  context 
of European  policy  and supported  by all the different  Structural  Funds, finds 
its  origins  in Article  130a  of  the  European  Union  Treaty.  There  it  is  defined 
in  terms  of  "harmonious  development"  with  a  specific  geographical 
dimension:  "reduce  the  gap  between  the  levels  of  development  of  the 
various  regions, and the  backwardness  of the  more disadvantaged  regions, 
including  rural  areas".  Agenda  2000  recalls  this  objective  and  adds  a 
dimension  related to  the  extension  of  the  Union to the  countries  of  Eastern 
the  struggle  against 
following  objectives: 
Europe,  and 
unemployment;  the  promotion  of  equal  opportunities  between  men  and 
women;  and the stimulation  of competitive  development  and of  sustainable 
development  from  an environmental  point  of  view. 

the 

to 

• 

Impact  of  investments  in basic  infrastructure 

The  impact  of  infrastructure  on  cohesion 
indirect  and 
collectively  benefits  businesses  and  people.  Relations  of  cause  and  effect 
through  which  impacts  are  produced  are  multiple. 

is  essentially 

Examples  of  infrastructure  projects  which  lead  to  sustainable  impacts  at  a 

macro-economic  level: 

(1)  A new  harbour,  road,  motorway  or  electricity  supply  network  cuts  the  production 

costs  of  regional firms  and  enhances  their  competitiveness. 

(2)  A  sewerage  or  water  supply  system  helps  to  remove  impediments  to  the 
development  of  local  enterprises;  an  airport  promotes  the  development  of 
tourism. 

(3)  Improvements  to  the telecommunications  network  allow  for  the  establishment  of 
new  relations  between  local  enterprises.  These  relations  improve  the  functioning 
of  regional  innovation  networks,  which  in  turn  improves  the  competitiveness  of 
regional  SMEs. 

Infrastructure  programmes  can benefit  existing economic  activities  but  also 
increase  the  relative  attractiveness  of  a  territory  for  outside  investors. 
Infrastructure  may also generate direct  benefits in terms of access to  public 
facilities  (e.g. shorter  travelling  time  to  a  hospital). 

Evaluation design and management 

iri 
tui 

rig  e f f e c ts  in  t he  d o m a in 
•e 

B ox  37  - Ex; 

än 
3E 

i p le  of  s t r u c ti 
sic  infrastruc 

of 

Motorway 

-►· 

Used  by 
local 
entreprises 

^w 

Macro 

level 

r 
1 

Reduced 
costs 

/ 
\ 
/ 
Increased 
market 
share 

Easier 
travelling 

N, 

/\ 

ι». 

Living 
conditions 

Attractive  for 
outside 
investers 

N 

Development 
induced 
for  other 
entreprises 

Economic 
and  social 
cohesion 

• 

Impact  of  interventions  on  the  productive  environment 

For a long time  regional  policies  were  based  on  direct  support  for  so-called 
"capacity"  investments.  This  logic  implied  that  the  supported  investments 
would  increase  the  capacity  to  produce  the  commodities  for  which  an 
unsatisfied  demand  existed. 

Today, when  support  is granted  directly  to  businesses, the assumptions  are 
often  very  different.  The  intention  is  to  encourage  or  stimulate  spending 
likely to  directly  increase the competitiveness  of the assisted firms. The  idea 
is that  European  support  helps  firms  to  establish  competitive  advantages. 
These  advantages  are said  to  be  "visible"  when  they  result  in  better  "price-
competitiveness",  owing  to  productivity  gains or  reduced  production  costs. 
The  "invisible"  or  immaterial  advantages  are  increasingly  decisive  in  world 
competition  and  are  therefore  encouraged  by  public  support.  Examples 
include:  support  for  innovation,  marketing,  quality,  etc. 

Finally,  more  and  more  public  support  is  intended  to  create  or  strengthen 
competitive  advantages  which  collectively  belong  to all the  firms  in a region 

Performing an evaluation 

or  sector.  The  productive  environment  comprises  all  the  factors  that  the 
enterprise  can  find  outside  its  own  internal  organisation,  to  optimise  its 
performance. 
In  the  example  Box  38,  the  programme  subsidised  a 
conference  centre  intended  to  enhance  the  competitiveness  of  regional 
businesses. 

Box 38 -  Example  of  structuring  effects 
in the  productive  environment 

Micro  level 

Conference 
centre 

Utilisation by 
local firms 

Macro  level 

Economic 
and social 
cohesion 

Impact  of  interventions  in the  human  resources  domain 

The  initial  effect  of  support  for  human  resources  is  that  it  enhances  and 
increases  the  addressees'  qualifications  and  capacities.  The  concept  of 
human  capital  accounts  for  the  accumulation  of  educational  and  training 
actions  "invested"  in  individuals  living  in  a  region.  This  concept  builds  a 
bridge  between  the  social  nature  of  the  results  of  a training  action  and  its 
macro-economic  effects.  Human  capital  encompasses  measurable  effects 
on  labour  productivity  and  less  palpable  effects  on  adaptability,  relevance 
of  specialisation  or the  quality  of  relational  networks. 

Structural  interventions  in favour  of  human  resources  have  a direct  impact 
on  the  concerned  groups  by  increasing  their  employability,  that  is,  their 

Evaluation design and management 

ability  to  find  a job  or  keep  one  in  the  context  of  a given  region  or  specific 
economic  circumstances. 

Support  in  favour  of  human  resources  may  have  an  impact  on  businesses 
(e.g.  a  specific  training  programme  accompanies  the  installation  of  a  new 
factory;  a  training  programme  accompanies  the  restructuring  of  a  firm). 
However,  businesses  benefit  mainly  in  an  indirect  manner,  in  the  form  of 
improved  adaptability  and  the  manpower  productivity  of  the  labour  market 
concerning  them  (e.g.  advanced  technical  training  can  favour  innovation 
and  enhance  the  advantages  of  a  region  already  specialised  in  a  dynamic 
sector). 

Finally,  interventions  in the  domain  of  human  resources  often  have  essential 
non-economic  impacts,  such  as the  reconstruction  of social  links  destroyed 
by  unemployment,  the  return to social  integration, and  a new-found  balance 
in  people's  lives. These  impacts  are  often  related  to  a return  to  employment 
but  are  worth  identifying  and,  if  possible,  evaluating  separately. 

Box  39 -  Example  of  structuring  effects  in the  domain 

of  human  resources 

Micro 

level 

Training 
courses 

Qualifications 
obtained 

Macro 

level 

Macro-economic  and  strategic  impacts 

The objective  of cohesion  implies  a reduction  in disparities  between  regions 
or  between  social  categories.  To  progress  towards  this  objective,  the 

Performing an  evaluation 

assisted  regions or categories  must have a faster  pace of development  than 
the  average  pace  of  European  economic  development.  For  this  to  be 
possible,  the  programmes  must  not  only  stimulate  socio-economic 
development,  but also boost the driving force behind this development. The 
different  types of strategy  which  have been adopted  to this end give rise  to 
very  different  expectations  regarding  impacts. 

try 

Socio-economic  development  can  be  sought  by  external  forces  such  as 
investment,  immigration  of  qualified  professionals  or  technology  transfer. 
factors  of  development. 
Many  programmes 
Attractiveness 
is  the  keyword  of  this  type  of  development,  called 
exogenous  development.  Better  attractiveness  is  a  goal  of  many  kinds  of 
programme, the  results  of  which  are expressed, for  example,  in the form  of 
infrastructure,  manpower 
lower  property  prices,  changes 
qualifications,  quality  of  urban  life or  protected  countryside. 

to  attract 

these 

the 

in 

The  relocation  of  a firm  towards  an  assisted  territory  is  not,  in  itself,  proof 
of success in terms of development  because  it does not guarantee a lasting 
effect.  Certain  investments  can  be  attracted  because  they  are mobile. This 
mobility is likely to last to the point where these firms leave the territory  after 
a  few  years  and  move  to  areas  offering 
lower  production  costs. 
Consequently,  the  chain  of  cause  and  effect  which  concerns  exogenous 
development  must  be  completed  by  a  final  link:  it  must  be  assumed  that 
recently  established  businesses  will  establish  adequate  links  with  the 
manpower  and  productive  environment  in  the  territory  so  as  to  remain 
attached  to that  territory  for  a  long time. 

Socio-economic  development  may  also  be  maintained  by  internal  causes. 
Ideally, endogenous development  is generated spontaneously  from a sound 
economic  base, that  is, a set  of  firms  and  workers  who  are  strongly  rooted 
in  their  territory,  who  manage  to  impose  themselves  on  their  respective 
markets  and whose  markets  are  buoyant. 

The strength of a territory's economic  base can rely on its specialisation  in a 
growth  sector.  A  programme  aimed  at  reinforcing  this  base  could  stimulate 
sectors of strategic  and dynamic  activities through  indirect  support: training; 
collective  services;  networking; access  to  information; innovation; research; 
etc.  Support  for  endogenous  development  presumes  that  a  growth  factor 
already  exists  in the form  of  specialisation  in certain  dynamic  activities  or in 
the form  of  a regional competitive  advantage. The  programme  supports  the 
activities  concerned and enhances their competitiveness  and specialisation, 
thus  reinforcing  the  dynamics  of these  activities. 

Evaluation design and management 

Endogenous  development  of  a  territory  depends  to  a  large  extent  on  the 
existence  of  local  non-commercial  networks.  Efforts  to  create  network 
effects  is  at  the  basis  of  assistance  to  "technopoles"  and  "innovation 
systems".  These  effects  lead  to  the  creation  of  an  economic,  social  and 
political  environment  capable  of  constructing  and  reproducing  its  own 
success  factors.  In these systems, essential  competitive  advantages  do  not 
belong  to  individual  firms, nor  do they  rely  on  factors  of  production  such  as 
manpower  or  capital.  The  key  factors  of  success  of  a  territory  belong, 
collectively  and  informally,  to  a  network  of  firms  and  local  actors.  This 
network  constitutes 
the  main  source  of  competitive  advantages 
(technology,  innovations,  know-how,  etc.).  Firms  take  advantage  of  the 
network  in formal  and  informal  ways:  professional  associations;  support  to 
local  and  regional  authorities;  informal  networks;  inter-firm  mobility  of  staff; 
parallel  information  channels,  etc.  These  means  of  interaction  are  generally 
independent  from  market  transactions,  which  means  that  they  correspond 
to  the  definition  of  the  word  externality  (economic  effects  which  do  not 
depend  on  a  customer-supplier  relationship). 

in  terms  of  purely  economic 

A programme  aimed  at encouraging  endogenous  development  must  not  be 
impacts.  The  creation  of  an 
evaluated 
economic,  social  and  political  environment 
to  construct  and 
reproduce  its  own  competitive  advantages  implies  all  sorts  of  dimensions. 
The  term  "local  potential"  represents  this  set  of  factors  which  governs  the 
renewal  of the  growth  process:  innovative  capacity;  institutional  dynamism; 
acceptability  of  development;  etc. 

likely 

P e r f o r m i ng  a  q u a n t i t a t i ve  e s t i m a t i on 

Earlier  in  this  volume,  simplified  approaches  for  quantitative  estimation, 
such as shift-share  analysis and the use of coefficients, were examined  (see 
page  136).  These  approaches  were  intrinsically  limited  in  their  analytical 
capacities.  A  theoretically  ¡deal  approach  would  consist  of  performing 
quantitative  modelling  of  all the  macro-economic  mechanisms,  as  outlined 
is  probably  not 
qualitatively 
completely  realisable,  but  considerable  efforts  have  been  made  in  that 
direction. 

in  the  preceding  pages.  This  approach 

Modelling  techniques  are  used  to  represent  the  simplified  functioning  of  an 
entire  economy  and  of  the  different  markets  composing  it  (e.g.  product 
market;  labour  market;  capital  market;  etc.).  A  model  schematises  the 

Performing an  evaluation 

functioning  of the economy  at equilibrium, but  is also capable of  simulating 
changes 
in  this  economy  as  a  result  of  an  exogenous  shock  (e.g. 
fluctuations  in the  oil  price; variation  in  exchange  rates; taxation  rates),  the 
adjustment  of  modelled  variables,  and  the  achievement  of  a  new  market 
equilibrium.  It  is  thus  possible  to  use  a  model  for  evaluation  purposes  by 
considering  that  the  programme  provokes  an  exogenous  shock  (a  micro-
economic  first  round  impact)  and  by  simulating  the  repercussions  of  this 
shock  throughout  the  economy. The  new  equilibrium  takes  into  account  all 
the  macro-economic  mechanisms,  particularly  structuring  effects,  if  these 
mechanisms  have  been  modelled. 

In  practice,  one  scenario  is  simulated  with  public  intervention  and  one 
without.  The  macro-economic 
impact  is  estimated  by  calculating  the 
difference  between  the  two.  The  models  are  entirely  quantified  and  thus 
provide  an  estimation  of  the  extent  of  the  impacts  on  all  the  macro-
economic  variables  which  have  been  modelled. 

-4  ►  The  estimation  of  effects  can  be  ex  ante.  In that  case, two  scenarios 
(with  and  without  the  programme)  are compared.  It  is  also  possible  to  use 
a  model  in  an  ex  post  evaluation  situation,  although  that  is  a  far  more 
innovative  approach.  In that  case, the  comparison  is  between  the  model's 
approximate  calculation  of  past  economic  developments,  and  a  scenario 
without  public  intervention, reconstructed  ex  post  by the  model. 

Available  macro-economic  models  are  generally  limited  to  the  national 
level.  The  models  constructed  on  a  regional  or  local  scale  are  rare, 
especially  because  economic  flows  are  very  quickly  diluted  outside  the 
territory  concerned.  Modelling  nevertheless  remains  meaningful  for  small 
economies,  provided 
insular 
economies  are  often  equipped  with  macro-economic  models.  Moreover, 
the  use  of  a  model  is  all  the  more  interesting  when  the  intervention  of  the 
programme  reaches a sufficient  mass compared  to the  modelled  economy. 
It is considered that  to reach this critical  mass, the economic  importance  of 
the  programme  must  be  1 %  of  the  GDP  of the  territory's  economy. 

they  are  relatively  closed.  That 

is  why 

Within  the  framework  of  European  structural 
interventions,  the  most 
advanced  macro-economic  evaluation  experiment  was  performed  with  the 
HERMIN  model.  This  work  is  presented  in  more  detail  in  Volume  3  of  the 
Collection,  in  parallel  with  two  other  experiments. The  HERMIN  model  was 
used  for  evaluations  in  Spain,  Ireland  and  Portugal.  These  evaluations 
introduced  an  interesting  innovation  by  explicitly  simulating  supply-side 

Evaluation design and management 

effects,  a  notion  that  is fairly  similar  to  structuring  effects  mentioned  above 
(see  page  137).  In  the  case  of  HERMIN  models,  the  effect  of  programmes 
on the  productivity  of work  and/or  other factors  of  production, as well as  on 
the  attraction  of  foreign  investments  and  the  increasing  competitiveness  of 
exports,  was  simulated. 

The  above  experiment  provides  an  interesting  example  of  current  R&D  on 
the  modelling  of  macro-economic  impacts  in  terms  of  development.  This 
work  aims  at  surpassing  the  limits  of  earlier  experiments  in  modelling, 
which simulated  only demand-side  effects.  Demand-side  effects  result  from 
the  fact  that  public  money  spent  by  the  programme  continues  to  circulate 
in  the  regional  economy  in  the  form  of  consumption  by  businesses  and 
households.  They  do  not  take  into  account  gains  in  productivity  and 
competitiveness.  Clearly,  an  approach  reduced  to  demand-side  effects 
would  be too  limited  because  it  would  implicitly  assume  that  a  programme 
had  no  more  effects  when  public  spending  ceased.  This  obviously 
contradicts 
for  socio-economic 
development,  which  is  why  it  will  be  particularly  interesting  to  follow 
developments  in the  modelling  of  supply-side  effects  in future  years. 

the  basic  hypothesis  of  support 

Carrying out more in-depth analyses 

In addition  to  the  overall  evaluation  of  a programme,  which  has  to  study  a 
large  number  of  impacts  with  its  own  particular  methods  of  analysis,  it  is 
possible  and  useful  to  carry  out  specific  analyses  of  a  small  number  of 
particular  impacts.  These  analyses  use  the  entire  range  of  evaluation 
techniques,  collect  primary  data  and  result 
in  credible  quantitative 
estimations  of  the  achievement  of  a programme's  objectives. 

A  more  in-depth  analysis  implies  undertaking  an  investigation  to  observe 
changes  experienced  by  programme  addressees,  followed  by  an  analysis 
of  causality,  aimed  at determining  the part  of change that  can  be  attributed 
to the  programme. 

H  ►  Unlike  the overall  evaluation  of  a programme  which  is intended  to  be 
exhaustive, the in-depth study of a question concerns a category  of outputs, 
a  public  and  a  mechanism  for  making  impacts  apparent.  The  overall 
evaluation  of  a  programme  may  be  accompanied  by  the  in-depth  study  of 
one  or  more  questions.  It  may  also  be  decided  to  launch  an  evaluation 
devoted  entirely  to  the  in-depth  study  of  a single  question. While the  notion 
of thematic  evaluation relates to the study of  one question, it is not the  same 
as  an  in-depth  study.  A  thematic  evaluation  examines  how  a single  impact 
(e.g. competitiveness  of  SMEs)  is obtained  by  several  kinds  of  intervention, 
or  how  interventions  financed  in  a  specific  domain  (e.g.  R&D)  produce 
several  kinds  of  impact.  Thematic  evaluations  are described  in Volume  5 of 
this  Collection.  They  are  too  complex  to  fit  into  the  simple  logic  of  this 
section,  i.e.  the  evaluation  of  a  specific  intervention  from  which  a  main 
impact  is expected  (e.g. impact  of  R&D on the  competitiveness  of  SMEs). 

The  question  treated,  thus  simplified,  makes  it  possible  to  use  all  the 
classical  evaluation  techniques,  especially  for  observing  addressees' 
behaviour  and  for  attributing  to  the  programme  its  share  of  the  changes 
observed.  Since  in-depth  study  mainly  concerns  the observation  of  effects 
in  the  field,  these  analyses  concern  mid-term  evaluation  and  ex  post 
evaluation. 

Detailed  analysis  of  the  internal  logic  of  t he  p r o g r a m me 

The overall evaluation of a programme  requires the clarification of its internal 
logic  and, in particular,  an examination  of the articulation  of  its objectives  at 

Evaluation design and management 

different  levels  (see page  95). The  logic  of  a programme  can  be  represented 
schematically  in the form of an objectives tree, the global objective  being  the 
trunk  and  the  multiple  outputs  being  the  branches.  The  analysis  of  a 
category  of outputs  and  an impact  amounts to  isolating  a branch of the tree. 
It is then  possible  to  examine  this  branch  in far  more  detail. This  is shown  in 
the  following  sections. 

Logical  framework 

impact.  The 

The  logical  framework  technique,  originally  designed  for  ex  ante  evaluation 
of  projects  in  developing  countries,  is  a  tool  which  can  be  used  to  simply 
and  rapidly  describe  the  logical  relations  between  an  output  and  an 
expected 
technique  consists  of  producing  a  matrix 
presentation  stating  the  expected  outputs  (cf.  the  bottom  left  hand  side  of 
Box  40)  and  the  main  logical  steps  separating  the  global  impact  outputs 
(left  column).  The  logical  framework  matrix  thus  describes  a  branch  of  the 
objectives  tree  by  placing  the  "branches"  at  the  bottom  (outputs)  and  the 
trunk  at the  top  (global  impact). 

At  each  logical  step,  the  evaluation  team  states  the  means  that  have  been 
or  must  be  provided  to  verify  whether  the  objectives  have  been  achieved 
(verifiable  success  indexes).  At  the  same  time,  what  must  be  considered  a 
success  or  failure  (sufficient  /  insufficient  outputs,  results  and  impacts)  are 
defined.  By  making  it  necessary  to  fill this  second  column  of  the  matrix  with 
indicators  or  precise  qualitative  descriptions,  the  technique  often  helps  to 
provide  useful  qualification  which  makes  it  possible  to  revert  to  a  more 
precise  description  in the  first  column. The  third  column, called  "conditions 
and risks", lists exogenous factors  which can facilitate or hinder the  passage 
from  one  step  to  the  next.  This  point  will  be  examined  in  more  detail  below 
(see  the  "explanatory  model").  More  details  can  be  found  on  the  logical 
framework  in Volume  3 of  this  Collection  devoted  to  evaluation  techniques. 

It  is  recommended  that  the  logical  framework  technique  be  implemented  in 
a  participative  way,  involving  the  managers  and  operators  responsible  for 
the  evaluated  outputs. 

The  column  on  the  left  of  the  logical  framework  matrix  describes  the 
sequence  of  effects  of  the  programme  in  four  steps  (outputs,  results, 
specific  impact  and  global  impact). The  column  on  the  right,  by  recalling  all 
the  external  factors  which  may  condition  the  passage  from  one  step  to  the 
next,  shows  that  the  achievement  of  impacts  is  based  on  hypotheses.  It  is 

Performing an  evaluation 

for  this  reason that  evaluation  specialists  call the  left  column  the  "theory  of 
action". The theory  of  action  consists  of  all the  hypotheses  used to  explain 
that  a given  output  is going  to  produce  a given  impact.  The  main  purpose 
of the  evaluation  is to  compare  this theory  with  reality. 

Box  40  -  Logical  f r a m e w o rk 

Statement 
of 
expectations 

Verifiable 
Sucess 
indexes 

Conditions 
and  risks 

Global 
impact 

Economic  and  social 
development 

Specific 
impact 

Savings  for  regional 
enterprises 

Result 

Less  travelling  time  on 
typical  journeys 

Output 

Improved  roads 

Logical  diagram  of  impacts 

Within  the  framework  of  in-depth  analysis,  the  four  lines  of  the  logical 
framework  matrix  are generally  not  enough  to  provide  a sufficiently  explicit 
and  detailed  theory  of  action.  In  particular,  there  are  often  grey  zones 
between  results  and  impacts. This  is usually  illustrated  by the  image  of  the 
black  box of  impact  hypotheses. This image recalls the frequent  situation  in 
which designers and operators  know  how to explain what they do and what 
the  anticipated  global  impact  is,  but  do  not  know  how  to  convincingly 
explain  the  process  that  is supposed  to  lead to  this  impact. 

The example  in Box 41  proposes the breakdown  of  impact  hypotheses  into 
four  steps.  Between  each  of  these  steps,  it  may  be  necessary  to  identify 
intermediate  impacts  and  to  refine  the  logical  diagram  of  impacts.  In 
particular,  the  diagram  is worth  detailing  each  time  a  relationship  of  cause 
intense 
and  effect  seems  obscure  or 

ill-justified.  This  necessitates 

Evaluation design and management 

interaction  with  the  managers  and  operators,  during  which  the  evaluation 
team  reveals  its  implicit  hypotheses  and  produces  a  complete  theory  by 
trying  to  progressively  eliminate  any  logical  weaknesses  in the  reasoning. 

Box 41  -  More  detailed  logical  diagram  of  impacts 

Advice 
to  SMEs 
on  exports 

Reduced 
unemployment 

Advice 
to  SMEs 
on  exports 

International 
contacts 

New  and 
sustainable 
export 
growth 

Survival 
of  SMEs 

Reduced 
unemployment 

Experience  has  shown  that  when  the  steering  group  itself  constructs  the 
logical  diagram  of  impacts,  it  considers  this  step  as significant  value  added 
for  the  evaluation. 

The  'example  in  Box  41  shows  that  by  opening  the  black  box  of  implicit 
hypotheses,  the  evaluation  work  can  be  directed  towards  certain  impact 
hypotheses  rather  than  other,  less  strategic  ones. This  example  shows  that 
the  evaluation  has  little  chance  of  reaching  a  conclusion  if  the  observation 
is  limited  to  the  number  of  days'  consultancy  services  provided  to  SMEs 
and to  unemployment  statistics.  By  contrast,  the analysis  of  impacts  will  be 
more  interesting  if the  managers  of the  assisted  SMEs  are asked  how  many 
the 
international  contacts  have  been  generated  or 
programme,  how  these  contacts  have  developed,  how  many  of  them  are 
still  active,  what  turnover  they  generated,  what  percentage  of  the  firm's 
business  they  account  for, etc. An in-depth  analysis of the  programme  logic 
is  used  to  focus  the  field  observation  on  the  right  level  of  impact. 

facilitated  by 

Performing an evaluation 

■ Carrying out observations in t he field 

The empirical and pragmatic dimension of an evaluation is based on its 
capacity to observe programme effects in the field. It is therefore through 
surveys on programme addressees that the evaluation contributes its full 
value added. The in-depth answer to an evaluative question often consists 
of a collection of primary data from persons or organisations which have 
participated in the programme, and from non-participants when the 
evaluation uses comparative techniques (see pages 110 and 155). The 
following sections describe the main survey techniques used in evaluation. 

Collecting quantitative data 

The questionnaire survey is the most widely used way of collecting 
quantifiable information. It may be carried out by mail, telephone or interview. 
The formulation of the questionnaire is a decisive step. The questionnaire 
consists of a limited number of clearly stated questions, formulated so as to 
be of interest to the interviewees. It proposes closed questions which the 
interviewees can answer by choosing among suggested answers. A few 
open questions may also be used, but the answers must then be coded 
(classified by category) after the data collection. This involves a large amount 
of work. A pre-test of the questionnaire on a limited number of persons is 
indispensable for validating the form and content of the questionnaire and 
confirming its capacity to collect information. The organisation of a survey 
may require several months and its cost will vary, depending on the size and 
accessibility of the sample, the desired rate of response, etc. The different 
surveys undertaken in the region must also be carefully coordinated so as not 
to provoke rejection by the addressees (the refusal to participate in a survey 
is increasingly frequent). 

Questionnaire surveys are used extensively for a wide variety of purposes, 
but primarily for collecting quantitative information from addressees. 
Because the questions asked are closed, the data provided by a 
questionnaire can be used for the construction of indicators. A basis for 
questionnaire surveys is the use of a sample, so that the conclusions can 
be generalised to the entire population. A sample of 200 to 300 addressees 
is often enough to obtain the correct quantification of an indicator. 

The sample must, however, be larger if the quantitative nature of the 
information provided by a survey is to be exploited to the full, particularly 
when the results of two surveys are to be compared, as in the following 

Evaluation design and management 

three  examples:  a  survey  of  participants  on  a  training  course,  given  that 
some  of them  are in employment  after  12 months, and that  a comparison  is 
to  be  made  with  a  group  of  non-participants;  a  survey  of  assisted  firms, 
given  that  a  small  number  of  them  develop  hi-tech  projects  and  that  a 
comparison  is to  be  made  with  the  results  of  a  national  survey;  a  survey  of 
farmers'  incomes,  given  that  part  of  this  income  is from  tourism  and  that  a 
comparison  is to  be  made  with the  results  of  a survey  carried  out  five  years 
earlier  on  a  partially  identical  sample.  In  each  of  these  examples,  the 
evaluation  team  wants  to  compare  the  results  of  surveys  on  different  or 
partially  different  samples.  In  this  case,  the  rules  of  statistical  validity 
preclude  comparisons  if  the  samples  are  not  large  enough.  The  minimum 
size  ranges  from  500  to  2,000  individuals,  depending  on  criteria  which  are 
independent  of  the  size  of  the  surveyed  population. A  professional  may  be 
employed  to  choose  the  right  size  of  the  sample  in  a  concrete  situation. 
Otherwise,  the  size  of  statistically  comparable  samples  is  usually  between 
900  and  1,100  individuals. 

A  particular  form  of  survey  is  that  which  is  used  to  monitor  individuals  or 
groups  of  individuals through  time, and thus to  measure changes  of  attitude 
or  situation.  These  panel  surveys  are  generally  used  in  the  case  of 
interventions  in  the  employment  domain  (the  participants  are  questioned 
before  a rd  after  training,  for  example).  They  are  spread  over  several  years 
and  are  therefore  difficult  to  launch  during  an  evaluation  which  lasts  for  a 
few  months  only.  However,  when  panel  surveys  do  exist,  evaluation  uses 
their  results  as  secondary  data. 

Qualitative  survey  techniques 

An  in-depth  analysis  generally  combines  the  collection  of  quantitative  and 
qualitative  information. Several  qualitative  survey  techniques  are  presented 
in  Volume  3  of  the  Collection,  devoted  to  evaluation  tools.  The  semi-
structured  individual  interview  and  case  study  (see  page  125)  are the  most 
common  tools.  The  evaluation  team  must  not  restrict  its  technical  choices 
on  the  basis  of  habit.  It  is  for  this  reason  that  two  techniques  which  may 
prove  highly  decisive  are  discussed  below. 

The first  is the  focus  group.  Provided  it  is conducted  by  a qualified  person, 
the  focus  group  is  used  to  create  a  dynamic  of  interaction  between  the 
participants,  to  take  advantage  of  their  creativity,  and  to  gather  far  more 
information  than  that  which  would  result  from  individual  interviews  with 

Performing  an  evaluation 

each of the  participants. The focus  group  is highly  recommended  when  the 
the 
addressees  have  a  very  vague  opinion  about 
programme  has  produced  for  them  (case  of  a SME which  participated  in a 
technological  network,  for  example).  Their  answers  to  a  questionnaire  or 
during  an  individual  interview  would  probably  be  polite  answers.  In  these 
cases  the  focus  group  is used to  obtain  the  most  reliable  answers. 

the  effects 

that 

The  second  example  is  less  typical.  This  is  ethnographic  observation, 
requiring  the  researcher  to  merge  into the group of  programme  participants 
(as  a  participant  or  assistant  on  a  training  course,  for  example).  This 
technique  has  been  successfully  used  for 
investigations  of  difficult 
populations  which  lack  motivation  and  are  sometimes  even  reluctant  to 
participate. 

Analysing  i m p a c ts 

In-depth  analyses  allow  impacts  to  be  studied  most  thoroughly  (see 
Box  29).  This  is  why  particular  attention  is  paid  to  impact  analysis  in  the 
following  sections. 

Depending  on  the  situation,  impact  analysis  combines 
two  types  of 
approach  in varying  proportions:  inductive  analysis  and deductive  analysis. 
Inductive  analysis  suggests  causal  links  based  on the observation  of  facts. 
It  is  therefore  essentially  exploratory  and  aims  at  producing  a  theory 
explaining  causes  and  effects,  based  on  observations—usually 
qualitative—in  the  field.  Deductive  analysis  considers,  on the  contrary,  that 
this  explanatory  theory  is  a  starting  point  that  needs  to  be  tested  by 
comparing  it  to  reality.  In  cases  where  the  initial  hypotheses  are  validated, 
it  may  lead  to  a  qualitative  estimation  of  impacts.  The  following  two 
sections  explain  how  to  construct  an  explanatory  theory,  how  to  estimate 
impacts,  when to  use the theory  and what  the  limits  of  impact  analysis are. 
The order  of  sections  is  merely  a convention  because an  evaluation  can  be 
performed the other way  round  if this can  be justified  by the  circumstances. 

Explanatory  theory  of  impacts 

This  volume  has,  on  several  occasions,  mentioned  relations  of  cause  and 
effect  between the  programme  and  its impacts, and the way  in which  these 
relations  can  be  represented  schematically  in the  form  of  a  logical  diagram 
of  impacts.  The  explanatory  theory  is  only  an  additional  step  in  the 

Evaluation design and management 

representation  of  the  process  of  impact  production,  but  is  nevertheless  a 
difficult  and  decisive  step  in the  evaluation. 

Box  42  shows  that  the  step  from  the  logical  diagram  to  the  explanatory 
theory  adds  all  factors  which  may  facilitate  or  hinder  the  production  of 
impacts  at  each  level.  This  is  what  the  logical  framework  technique  calls 
"conditions  and  risks"  and  what  the  evaluation  literature  calls  "exogenous 
factors"  or  "confounding  factors".  In  any  evaluation  situation,  there  is  a 
potentially  infinite  number  of  exogenous  factors.  The  role  of  inductive 
analysis  is to  determine  which  of  them  appear  most  influential  in  relation  to 
what  was  observed  during  field  surveys  on  addressees. 

Box  42 -  Theory  of  action  and  explanatory  theory 

New  and 
sustainable 
export 
growth 

Survival 
of  SMEs 

When  carrying  out  inductive  analysis,  the  evaluation  team  will  also  try  to 
identify  all  the  unexpected  effects  observable  among  the  addressees.  For 
the 
example,  several  evaluations  have  shown  a  negative  effect 
inhabitants  of  assisted  areas,  resulting  from  the  fact  that  these  areas  are 
declared  "disadvantaged"  and  consequently  become 
less  attractive 
(stigmatisation  effect).  Fortunately  there  are  also  positive  effects. 

for 

The  case  study  technique  has  an  excellent  cost-effectiveness  ratio  for 
identifying  exogenous  factors  and  unexpected  effects.  It  is  probably  the 

Performing  an  evaluation 

basic  tool  of  inductive  analysis.  A  lot  can  also  be  learned  from  a  series  of 
individual  interviews  with  people  who  have  participated  in the  programme 
and  with  others  who  have  not.  Inductive  analysis  should  not,  however,  be 
considered  to  be  qualitative  in  all  cases.  Techniques  such  as  variance 
analysis  or  factor  analysis  are  also  used  regularly  to  reveal  the  most 
influential  exogenous  factors.  Without  using  such  sophisticated  tools, 
others  such  as  tables,  maps  and  graphs,  when  they  are  well  chosen,  can 
often  reveal  overlaps  and  similarities  which  clearly  indicate  the  importance 
of  certain  exogenous  factors. 

Estimation  of  impacts 

Impact  analysis  consists  of  attributing  part  of  the  observed  changes  to  the 
programme  and  another  part  to  exogenous  factors,  or  at  least  to  the  main 
exogenous  factors.  The  latter  become  confounded  with  the  programme  to 
explain  "before-after"  changes.  This  is  why  exogenous  factors  are  also 
called  "confounding  factors". 

To estimate impacts  it is necessary first to estimate exogenous factors, then 
to  subtract  this  effect  from  observed  changes,  and  finally  to  attribute  the 
difference  to  the  programme.  The  "counterfactual  situation"  is the  result  of 
the  influence  of  exogenous  factors  only.  To  estimate  a  counterfactual 
situation,  the  most  frequently  used  technique  consists  of  constituting  a 
comparison  group  with  people  who  have  not  participated 
in  the 
programme, for  example applicants  whose  project  was  rejected,  applicants 
who  dropped  out  of  the  programme  midway  or  non-applicants  with  a 
similar  profile  to  the  participants  (matching  groups  as  in  the  following 
example). A  fictive  comparison  group  can  also  be  created  from  individuals 
who  participated 
(statistical  comparison  by 
adjustment).  Once  the  comparison  group  has  been  formed,  different 
methods  of  comparison  can  be  applied:  "after  only"  comparison,  "before-
after  comparison"  or  the  use  of  time  series  which  look  at  the  situation 
before,  during  and  after.  For  a  detailed  description  of  the  technique,  the 
reader  is  referred to  Volume  3 of  the  Collection. 

in  a  national  survey 

The  evaluation  team  must  create  a  comparison  group  that  is  sufficiently 
similar  to  the  group  of  direct  addressees  of  the  programme.  Both  groups 
must  have the same essential characteristics, which amounts to saying that 
they  must  be  influenced  by  the  same  exogenous  factors.  Differences 
between  these  factors  would  probably  explain  part  of  the  differences 

Evaluation  design  and  management 

o b s e r v ed  b e t w e en  the  t wo  g r o u p s,  w h i ch  w o u ld  distort  t he  estimation  of 

i m p a c t s.  The  c o n s t r u c t i on  or  a d j u s t m e nt  of  t he  c o m p a r i s on  g r o up  therefore 

implies  a  s o u nd  k n o w l e d ge  of  e x o g e n o us  f a c t o r s,  that  is,  an  explanatory 

theory  of  i m p a c t s. 

Example:  Impact  analysis  using  matching  groups 

A  comparative  analysis  of  matching  groups  was  carried  out  in  the  U.K.,  during  the 
evaluation  of  a  programme  offering  SMEs  consultancy  services  in  management, 
training  a rd  information.  The  impacts  of  this  programme  could  not  be  expressed  in 
terms  of jobs  created  since  the  aim  was  to  increase  productivity,  which  would  in  the 
short  term  lead  to  a  reduction  in  jobs.  It  was  therefore  necessary  to  evaluate  the 
programme  in terms  of  other  criteria  such  as  productivity,  profitability  or  investments 
induced. 

A sample  of  about  300  recipient  firms  was  constituted. Another  sample  of  about  200 
non-recipient  firms  was  also  constituted  by  means  of  the  matching  pairs  technique. 
The  following  criteria  were  used  to  match  the  pairs:  type  of  activity  of  the  firm,  size 
of  the  firm  measured  in  terms  of  number  of  employees,  age,  and  geographical 
situation. 

A  questionnaire  was  addressed  to  the  recipient  firms  and  to  those  of  the  matching 
group.  A  statistical  comparison  between 
two  groups  provided 
quantitative  indications  on  the  effects  of  the  programme  in  terms  of  turnover, 
the 
productivity  and 
questionnaire  included  numerous  and  partially  redundant  questions  (triangulated 
questionnaire).  The  survey  also  included  a search  for  deadweight. 

"biased"  statements, 

induced.  To  avoid 

investments 

in  the 

firms 

•  U se  of  e x p l a n a t o ry  t h e o ry 

Explanatory  t h e o ry  is  even  m o re  necessary  w h en  the  estimation  of  i m p a c ts 

uses  regression  analysis.  U se  of  this  t o ol  starts  w i th  the  d r a w i ng  up  of  a 

small  m o d el  to  s c h e m a t i c a l ly  represent  c a u s es  (the  p r o g r a m me  a nd 

its 

main  e x o g e n o us  factors),  effects  (observed  changes)  a nd  relations  of  cause 

and  effect.  This  m o d el 

is  t h en  a d j u s t ed 

to  represent  as  accurately  as 

possible  the  available  d a ta  ( m a n a g e m e nt  d a ta  a n d / or  survey  data).  The 

result  of  the  analysis  is  a  quantitative  e s t i m a t i on  of  t he  i m p a c t.  The  validity 

of  the  e s t i m a t i on  strongly  d e p e n ds  on  the  quality  of  t he  initial  m o d e l,  and 

particularly  on  the  f a ct  of  this  m o d el  t a k i ng  e x o g e n o us  factors  into  a c c o u nt 

correctly. 

Regression  analysis 

is  a 

t y p i c al  e x a m p le  of  d e d u c t i ve  analysis.  The 

t e c h n i q ue  necessitates 

the 

formalisation,  f r om 

the  outset,  of  a  set  of 

Performing  an  evaluation 

hypotheses  on  causes  and  effects  (which  is  why  it  is  also  referred  to  as 
hypothetico-deductive).  For  this  purpose  it  uses  an  explanatory  theory 
either  borrowed  from  the  scientific  literature  or  taken  from  an  expert's 
knowledge,  or  else  constructed  by  inductive  analysis  in  the  context  of  an 
evaluation.  In a second  stage, the  hypotheses  are compared  to  reality  and 
the  evaluation  team  draws  conclusions  on  the  existence  of  the  impact,  on 
its quantitative  importance  or  on  its  non-existence. 

Limits  of  impact  analysis 

The  evaluation  of  an  impact  is  an  exercise  which  is  similar  to  forecasting, 
even  when  the  approach  is  retrospective.  What  would  have  happened  if 
public  money  had  not  been  spent?  There  is  no  ex  post  evaluation  without 
an  answer  to  this  question.  Yet,  the  description  of  what  would  have 
happened  without 
the 
counterfactual  situation,  is  nothing  other  than  a  reverse  forecast.  Thus, 
evaluation  has  all the  same  limits  as  forecasting,  and  it  is well  known  that 
there are  many  such  limits. 

the  construction  of 

the  public  action, 

i.e. 

Certain  contexts  lend  themselves  readily  to  forecasting  and  evaluation, 
whereas  others  are  so  complex  that  neither  of  the  two  exercises  produces 
credible  results.  In  an  uncertain  context,  the  evaluation  team  cannot 
establish  the  counterfactual  situation  convincingly.  Even the  participants  in 
the  programme  are unable to  estimate the impact  that  was  produced  within 
the  framework  of  their  own  project. 

Favourable  situations  are those  in which  the  explanatory  theory  is  relatively 
simple  and  linear.  For  example,  public  investments  improve  an  airport, 
which  leads  directly  to  the  multiplication  of  air  links.  This,  in  turn,  leads 
directly to an increase in tourism. In this example the cause  is clearly  known 
and  mechanically  leads  to  a  relatively  predictable  effect.  It  is  possible  to 
estimate  impacts  in such  a situation,  provided  the  price  is  paid  in terms  of 
surveys  and  analyses. 

Unfavourable  situations  are  those  in  which  the  explanatory  theory  is 
complex,  particularly  when  it  contains  relations  of  circular  causality. This  is 
the  case  of  the  impact  of  training  programmes,  the  effects  of  which  follow 
entangled  paths.  On  the  one  hand, they  produce  new  qualifications  which 
must  fit  with  economic  needs  in  order  to  improve  the  employability  of 
trainees  and result  in high placement  rates. On the other  hand, they  provide 
firms  with  more  productive  manpower,  which  enhances  their  productivity 

Evaluation design and management 

but  also  reduces  employment.  Productivity  gains  nevertheless  have  a 
favourable  influence  on competitiveness,  which  indirectly  helps to  create  or 
safeguard  jobs.  Impact  analysis  in  this  context  becomes  very  difficult 
because  the  causal  relations  can  be  explained  in  various  ways:  the 
placement  rate  may  be  either  the  cause  or  the  consequence  of  gains  in 
competitiveness. 

The difficulty  is even more obvious when the  phenomenon  of circularity  also 
applies  to  the  policy  itself.  It  is  to  be  hoped,  for  example,  that  the 
unemployed  are  oriented  towards  training  in  areas  where  placement  rates 
are  high,  rather  than  being  oriented  randomly.  Does  this  not  mean  that  the 
policy  itself  is  influenced  by  the  impacts  it  is  supposed  to  produce?  With 
this  type  of  circular  causality,  the  evaluated  programme  can  be  seen 
alternately  as  a cause  or  an effect.  In this  situation, the  concepts  of  effects, 
impacts  and  results  lose  their  meaning. 

Thus,  certain  impact  analyses  are  impossible  to  perform  due  to  the  very 
nature  of  the  problem  posed,  no  matter  how  professional  the  evaluation 
team  may  be. The terms  of  reference  must  therefore  be drawn  up  with  care 
because  certain  questions  relating  to  programme  impacts  defy  evaluation, 
even  when  it  includes  in-depth  analyses. 

IV 
Using an evaluation 

Optimising the use of an evaluation 

If an  evaluation  is to  add  real value  in the  institutional  and  decision-making 
spheres,  its  conclusions  must  be  disseminated  correctly  and  must  be 
viewed  favourably  by  potential  users. 

The  evaluation  report  is  not  necessarily  the  only  nor  the  most  effective 
medium, particularly when users are elected representatives or citizens  who 
will receive the results through the media. These considerations  represent a 
strong  argument 
in  favour  of  the  early  establishment  of  a  plan  for 
communicating  the  results  of  an evaluation. 

After  the  dissemination  stage,  the  last  and  most  important  stage  of  the 
evaluation  is its use. The entire evaluation  process  must  be geared  towards 
obtaining the most effective  utilisation possible. It is advisable, however,  not 
to  be  over-optimistic  about  certain  forms  of  use.  Although  a  good 
evaluation  report  always  produces  new  knowledge  (cognitive  use),  it  does 
not  necessarily  lead  readers  to  revise  their  judgements  in  line  with  its 
conclusions  (normative  dimension),  nor  to  adopt  all  the  recommendations 
in  their  decisions  concerning  the  programme  (instrumental  value  added). 
Indeed,  the  public  decision-making  process  often  involves  several  actors 
and  a large  number  of  factors. 

Disseminating  t he  conclusions  of  an  evaluation 

In the following  sections, the term "user"  is employed  to  refer to any  person 
who  reads or hears the conclusions  and  recommendations  of an evaluation, 
and  for  whom  this  information  leads  to  a  change  in  their  knowledge, 
opinions  or  attitudes.  Potential  users  of  evaluations  range  from  co-
decision-makers,  evaluation  operators  and  managers,  to  readers  of  a 
newspaper  or journal  in which  an evaluation  is  mentioned. 

The  dissemination  of  an  evaluation  is an  essential  phase  which  follows  the 
production  of  the evaluation  report,  but  which warrants  being  planned  from 
the  early  stages  when  the  terms  of  reference  are  drawn  up.  It  is  preferable 
for  the  terms  of  reference  to  announce  to  whom  the  evaluation  results  will 
be addressed  and  in what  form.  Decisions  in this  respect  are confirmed  by 
the commissioner  after the submittal  of  the report,  but  by carefully  planning 
the  dissemination  strategy  the  use of  the  evaluation  can  be  optimised. 

Evaluation design and management 

Communication channels 

The dissemination strategy consists of a compromise between the objective 
of maximising the visibility of the evaluation and that of establishing a 
climate of trust in which interactions between all the evaluation stakeholders 
can take place. As far as surveys conducted by the evaluation team on 
addressees and programme operators are concerned, trust and visibility can 
be guaranteed by treating all survey results as confidential by protecting the 
anonymity of the interviewees throughout the evaluation process. 

The question of interaction between the evaluation team, the steering group 
and the commissioner may, however, be difficult. Interaction focuses on 
sensitive points such as the clarification of criteria and norms, the 
identification of examples of good and bad practice, the importance given 
to a particular question or criterion rather than another, etc. Sensitive 
questions cannot be treated properly with total openness. On the contrary, 
the evaluation team must be sufficiently discreet in its work and its report 
to enable members of the steering group to express themselves sincerely 
in their own name and, if necessary, to venture away from the official 
discourse of their institutions. In certain cases, the evaluation team writes 
its report in such a way as to maintain a high degree of discretion as regards 
its interaction with the programme managers, and adds a confidential note 
to attract their attention to more sensitive points. 

Box 43 - Communication channels 

Communication channels 

ra 

0) 
■Ό 
E  S 
o õ 
υ Ξ 

* 

Ι E 
il 
E E 

£ Β 
co Q. 
Q SS 

IV 

* 

u 
c 
CD 
ta 
3 

Commissioner 

Steering group 
Managers 
Operators 

European, 
regional and 
national, 
institutions 

Citizens 
Journalists 

* 

* 

* 

.9 ­c 

11 

(fi 
'tn 
ω 
.c 
>% 

to ύ 

cu 
υ 

E 

«Γ 
£ o 
û­O 

ω 

.C  OJ lì 

CÛ ra 

C 
g 
CO 
C 
Φ 
co 

£ 

* 

* 

* 

* 

* 

* 

iV 

Using an  evaluation 

All  too  often  the  problem  of  the  use  of  an  evaluation  is  reduced  to  the 
question: "To publish or not to  publish the final report?" when  in fact a larger 
set  of  questions  ought  to  be  taken  into  consideration.  Apart  from  the 
question  of  publication,  the  way  in  which  information  is  disseminated 
should  be  carefully  considered  and  selected.  Various 
forms  of 
communication  and  numerous  possible  media  are  presented  in  Box  43. 

The  people  involved  in the  programme  (evaluation  steering  group,  managers 
and operators) tend to use evaluation as soon as the first results are presented 
by  the  evaluation  team  to  the  steering  group  (e.g.  results  of  surveys  and 
analyses,  intermediate  reports, draft  final  report). From their  point  of view, the 
use of  evaluation  starts  before  the communication  of the  final  report. 

For the other  users  it is essential to  plan  a suitable communication  strategy 
that  combines  a  medium  and  a  deadline.  Box  43  shows  that  different 
communication  channels  are  possible  and  need  to  be  chosen  in  terms  of 
the  availability  of 
the  different  categories  of  user  ("receivers",  as 
communication  specialists  would  say). 

Communication  plan 

The  commissioner  and  the  highest  administrative  and  political  authorities 
do  not  have the time to read evaluation  reports  in detail. The  most  effective 
mode  of  communication  at  this  level  is  a  short  synthesis  in  a  few  pages, 
referring  to  certain  sections  of  the  final  report  for  further  details.  This  can 
usefully  be complemented  by an oral  presentation  by the steering group  or 
its  President, during  a seminar  or meeting  after the conclusion  of the  work. 

For  communication  to  European,  national  and  regional  institutions,  a  short 
synthesis  can  similarly  be  used.  This  is  facilitated  if  the  synthesis  is 
presented  in  the  form  of  a  brochure  containing  concrete  examples  of 
programme  outputs,  and  showing  how  these  outputs  have  been judged  by 
the  evaluation.  The  best  way  of  informing  social  partners  and  programme 
addressees  is  by  publishing  special  documents  (newsletters,  synthesis 
sheets,  etc.).  Other  documents  are  prepared  to  provide  information  for  the 
"public  at  large"  (brochures,  articles,  etc.)  and  the  media.  In this  respect,  a 
press  conference,  attended  by  the  steering  group,  may  be  arranged. 
Because 
the 
evaluation,  a particularly  strong  impact  on the journalists  can  be  obtained. 

the  members  of  the  group  enhance  the 

legitimacy  of 

In most cases, the rules of administrative transparency  require the  publication 
of  the  final  report.  This  is  nevertheless  only  one  of  many  ways  —and  not 
always  the  most  effective  way—of  making  the  conclusions  of  the  evaluation 

Evaluation design and management 

available  to  its  users. A  real  effort  to  produce  a  complete  publication  and  to 
disseminate  it  widely  will,  however,  have  the  advantage  of  exposing  the 
evaluation  to  critical  examination  by  experts,  which  is a guarantee  of  quality. 
Moreover,  if  the  commissioner  announces  that  the  report  will  be  published 
and  widely  disseminated,  the  evaluation  team  will  be  more  motivated.  The 
evaluation  will  probably  be  more  credible  because  its  organisation  is  more 
transparent,  and the  methodological  quality  of the  report  is likely to  be  better. 

On  the  other  hand, there  is a  risk  of  self-censorship  since  conclusions  and 
recommendations  will  be  more  cautious.  To  avoid  this  drawback,  which 
could  prove  to  be  a  real  handicap,  the  evaluation  team  could  be  asked  to 
produce  a confidential  note  in  its  report. 

Good  communication  to  users  of  an evaluation  also  requires  a "staggered" 
dissemination  process.  If  the  evaluation  includes  field  surveys  and  an  in-
depth  analysis  of  certain  questions,  part  of  the  results  obtained  should  be 
presented,  without  waiting  for  the  conclusion  of  the  final  report. 

The  communication  plan  must,  above  all,  be  carefully  synchronised  to 
correspond  to  the  main  phases  of  the  programme:  dates  of  the  main 
management  meetings;  dates  of  the  main  decisions;  release  of  public 
reports  on  the  policies  concerned  by  the  programme  at  the  regional, 
national  and  European  levels. 

In reality, it is not  only the communication  plan  but  also the entire  evaluation 
process  which  ought  to  be organised  in relation to the  users'  schedule. This 
concern  is  present  from  the  outset  (see  page  67)  and  is  decisive  as  far  as 
the  usefulness  of  the  evaluation  is  concerned. 

C o n s i d e r i ng  all  f o r ms  of  u se 

to  be  effective: 

Evaluation  is  frequently  judged  in  terms  of  its  own  criteria.  It  is  naturally 
is  value  added  of  an  evaluation  sufficient 
expected 
compared  to  the  financial  and  human  inputs  devoted  to  it?  To answer  this 
type  of question, a  large number  of  meta-evaluations  have  been  performed 
and  published.  Most  studies  conclude  that  there  are  several  types  of  use, 
and  that  direct  use  of  evaluation  recommendations  by  decision-makers  is 
only  one  form  among  others.  Three  main  forms  of  use  are  highlighted. 

For  raising  the  level  of  knowledge 

-4  M  ►  Any  evaluation  will,  first  and  foremost,  afford  its  users  an 
learn  and,  if 
opportunity  to  discover—sometimes 

to  their  surprise—to 

Using an evaluation 

possible, to  understand. The design stage of an evaluation  has an  important 
part  to  play  in this  respect  because  it  helps to  clarify  objectives  which  were 
badly  expressed  or  only  implicit  in the  programming  documents,  to  explain 
impact  mechanisms,  to  identify  changes  of  behaviour,  to  check  whether 
expected  effects  were  obtained  and to  highlight  unexpected  effects. 

The  evaluation  exercise  transforms  the  user's  representation  of 
the 
programme  and  its  effects. This form  of  use may  be  conceptual  when  new 
knowledge  focuses  on  the  explanatory  theory  of  impacts. 

Whatever the declared  aim, a successful  evaluation  must  have a visible  and 
recognised  cognitive  use. 

For  deciding  on  improvements 

that 

they  can  subsequently  appropriate 

M  The  evaluation  team  and  steering  group  make  recommendations  to  the 
decision-makers.  The active  participation  of the  steering  group  in this  stage 
of  an  evaluation  has the  advantage  of  lending  a more  realistic  dimension  to 
the  recommendations  and  of  taking  into  account  the  concerns  of  the 
these 
stakeholders  so 
recommendations  more  easily.  Irrespective  of  the  choice  made,  it  is 
important  for  recommendations  to  be  clearly  separated  from  conclusions, 
and  for  their  status  to  be  clearly  explained  by  answering  the  following 
questions: Does the  implementation of this recommendation  have a financial 
implication?  Can it be implemented  in the short term?  Is there consensus  on 
the  recommendation?  Even  if  there  is  never  an  obligation  to  follow  the 
recommendations  of  an  evaluation,  the  commissioner  should  find  it  worth 
while  monitoring  the  future  of  these  recommendations  for  a year  or  two. 

Example:  Instrumental  use  of  evaluation 

The  intermediate  evaluation  report  of  the  Irish  1994-99  CSF  provided  a sound  basis 
for  discussions  during  the  mid-term  review  of  that  CSF.  Following  negotiations 
between  the  European  Commission  and  the  national  authorities,  the  Monitoring 
Committee  agreed  on  a  series  of  amendments  concerning  almost  3%  of  the  total 
budget.  Part  of  the  main  changes  in  allocations,  as  well  as  certain  other  financial 
decisions,  clearly  resulted  from  recommendations  made  by  the  evaluators  of  the 
CSF. Thus, out  of  the  18 measures  or groups  of  measures  for  which  allocations  were 
called  into  question, eight  had  their  support  reduced. On  the  other  hand, five  of  the 
eight  domains 
identified  as  requiring  additional  support,  were  granted  such 
assistance.  Non-financial  decisions  relating  to  changes  of  priorities  were  also taken, 
i.e. training  and  the strengthening  of the capacity  of  small  businesses  for  expansion, 
at  the expense  of  support  to  the  private  sector  in general. 

Evaluation design and management 

At  best,  traces  of  recommendations  can  be  found  in  decisions  taken 
subsequent  to  an  evaluation.  In  such  cases,  evaluation  constitutes  a 
decision-making  tool  and  can  serve  to  improve  a  programme  midway  or  at 
the  time  of 
in  the  example.  This  use  of  evaluation 
its  renewal,  as 
conclusions  is qualified  as  "instrumental". 

Forjudging  success  and  failure 

►  Evaluation  provides  users  with  arguments  for  forming  or  modifying  their 
opinions.  It  helps  to  clarify  criteria  and  define  norms  for  comparison.  It  is 
also  used  to  draw  conclusions  on  the  success  or  failure  of  a particular  part 
of  a  programme,  and  consequently  to  identify  best  practice  and  exemplary 
projects. 

This  form  of  use  is  essential  when  the  evaluation  serves  for  accounting  to 
the  political  authorities  and  the  public  at  large.  It  is  a  use  which  is  outside 
the  programme  and  which  feeds  into  debate  on  the  justification  of  this 
programme.  In  such  cases  an  evaluation  is  similar  to  reporting  and  is 
intended  to  enable  people  outside  the  context  of  the  programme  (public 
authorities,  elected  representatives,  the  general  public)  to  form  a  global 
opinion  on the  intrinsic  value  of the  action, independently  of the opinions  of 
the  stakeholders. 

In the  same  context,  users  compare  evaluation  conclusions  with  their  own 
opinions.  They  consequently  maintain,  support  or  change  their  personal 
judgement  on the  programme. This form  of use  is called  normative  because 
it comprises  a  large degree  of  judgement. 

Not  all  evaluations  are  used  in this  way  because  normative  use  is the  most 
difficult.  Users  who  have  to  revise  their  judgement  are  likely  to  resist, 
especially  if this judgement  has  already  been  expressed  publicly. 

The  specialised  literature  usually  distinguishes  two  evaluation  practices, 
depending  on  whether  it  is  primarily  intended  for  actors  directly  concerned 
by  the  programme  or  exogenous  addressees.  An  evaluation  is  called 
"formative" when  its use, within the evaluated  action or  programme, is  likely 
to  give 
their 
behaviour,  improve  the  action  and  thus  transform  its  object. 

information  enabling 

the  stakeholders 

to  change 

them 

Evaluation  is  said  to  be  "summative"  when  it  serves  both  to  judge  a 
programme  and  to  help  the  supervisory  authorities  decide  on  its  launch, 
maintenance  or  reconstruction.  In  this  case  users  are  outside  funding 

Using an  evaluation 

authorities  or  other  social  actors  who  may  decide  on  the  appropriateness 
of  adopting  the  best  practices  identified  during  an  evaluation. 

Irrespective of the nature of the decision taken subsequent to an evaluation, 
the  process  of  use  involves  the  three  successive  forms  of  utilisation. 
the 
Decision-makers  start  by 
programme.  They  then  formulate,  strengthen  or  modify  their  judgement 
and,  finally, express  themselves  convincingly  at the  right  moment,  so  as  to 
persuade  their  co-decision-makers  to  follow  suit  (since  public  decision­
making  is almost  always  taken jointly). 

their  understanding  of 

transforming 

Thus,  numerous  logical  steps  are  involved  in  the  passage  from  evaluation 
to  decision-making,  and  the  last  of  these  steps  brings  several  actors  into 
play.  This  is  why  the  instrumental  use  of  an  evaluation  is  difficult  to 
demonstrate.  This  is  confirmed  by  meta-evaluations  which  often  conclude 
on  the  absence  of  a  direct  short-term  link  between  recommendations  and 
decisions. 

-4  It is in this sense that  ex ante evaluation  is distinguished from a feasibility 
study  of  a programme. A feasibility  study  is always  based  on the  objectives 
of  a  political  authority,  for  directly  preparing  a  decision.  In  no  way  does  it 
challenge  these  objectives.  Its  only  aim  is  to  aid  decision-making.  An 
evaluation,  by  contrast,  always  has  a  dimension  of  judgement  about  the 
appropriateness  of  the  programme,  taking 
its  political 
objectives  and  the  point  of  view  of  addressees.  Even  in  the  case  of  an  ex 
ante  evaluation  of  a  programme  proposal,  instrumental  use  is  not  the  only 
mode  of  use; cognitive  and  normative  dimensions  are  inevitably  present. 

into  account 

Evaluation also differs from basic  research from the  point  of view of  its  utility 
because  it  responds  to  an  order  by  the  programme  managers,  at  a  time 
when  they  require  information  and  answers  to  questions.  The  exercise  is 
the  scope  of 
always  highly  constrained  by  theories  of  action  and 
investigation  chosen  by  the  decision-makers.  By  contrast,  the  results  of 
evaluations  are often  accepted  more  readily  by  public  authorities  and, as a 
result,  are  used  more. 

Assessing the quality of an evaluation 

Although the quality of an evaluation  is often judged on the sole basis of the 
final  report,  it  largely  depends  on  the  way  in  which  the  entire  process  was 
managed,  from  the  commissioning  of  the  evaluation  to  the  formulation  of 
recommendations  and  the  communication  of  results. 

A fair judgement  of  the  quality  of  an  evaluation  requires  the  clarification  of 
the  criteria  used.  This  assessment  may  be  based  on  eight  criteria 
constituting  important  professional  standards:  meeting  the  needs  of  the 
commissioners;  relevance of scope of an evaluation; appropriateness  of the 
methodology;  reliability  of  data;  soundness  of  the  analysis;  credibility  of 
results; impartiality  of  conclusions;  and  clarity  of the  report. 

It  is  necessary  to  assess  the  quality  of  an  evaluation  to  verify  whether  the 
team  satisfied  the  commissioners'  requirements,  but  also  to  distinguish 
between  those  conclusions  which  are  valid  and  soundly  backed-up,  and 
those  which  are  likely  to  be  contested  or  which  necessitate  more  in-depth 
empirical  work. 

Assessing  the  quality  of  an  evaluation  is,  or  should  be,  a  preliminary  step 
preceding the  use of  its conclusions and recommendations. This is a logical 
preliminary  because only an evaluation of  a high quality can  stand  up to  the 
criticisms  that  inevitably  arise  when  judgements  on  success  or  failure  are 
made. It is also a chronological  preliminary  because quality  is often  formally 
assessed  before  the  dissemination  of  the  evaluation  report  and  before 
conclusions  and  recommendations  are taken  into  account. 

We  have  chosen  to  place  this  chapter  at  the  end  of  the  three  parts  of  the 
Volume  devoted  to  the  design  and  its  realisation  of  an  evaluation  project, 
because the quality  assessment  grid provides a checklist  of all the  proposals 
and  recommendations  made  in preceding  chapters.  It was  developed  within 
the  framework  of  the  MEANS  programme,  with  the  help  of  experts  from 
several  European  countries.  It  can  be  used  by  readers  wishing  to  check 
whether  their  own  practice  meets  prevailing  professional  standards. 

M a n a g i ng  a nd  improving  t he  quality 
of  an  evaluation  process 

With the development  of evaluation and, consequently, the general  increase 
in budgets  devoted  to  it, attention  is  being  focused  more  and  more  on  the 

Evaluation design and management 

quality  of  work  submitted  and  on  the  quality  of  the  entire  process, from  the 
commissioning  of  an  evaluation  to  its  utilisation. 

Quality  control  checks  whether the evaluation team  has fulfilled  its  contract. 
Its  very  existence  helps  to  raise  the  evaluation  team's  level  of  awareness, 
which  helps to  improve the final report.  More  indirectly,  the assessment  can 
improve  professional  practice  by  spreading  standards  of  best  practice.  It 
can  also  serve  a  broader  democratic  objective  by  enhancing  the  credibility 
of  the  information  disseminated  by  the  public  administrations. 

The  quality  criteria  presented  below  were  used  by  the  Commission  for 
assessing  a number  of  evaluation  reports  during  the  1994-99  programming 
period.  They  have  also  been  applied  by  certain  national  authorities.  Thus, 
for  example,  the  quality  of  several  dozen  evaluations  performed  in  the 
Republic  of  Ireland  during  the  period  1994-98  was  assessed  by  referring  to 
these  criteria.  This  work  resembles  exercises  in  meta-evaluation,  i.e.  the 
evaluation  of  one  or  more  evaluations. 

Quality  is  a  requirement  that  is  sufficiently  crucial  for  it  to  also  be  the 
concern  of every  commissioner  of  evaluations. Quality  assessment  involves 
co-decision-makers  of  every  programme,  in terms  of  assessing  the  quality 
of  their  calls  for  tender,  and  quality  control  of  the  work  carried  out  by  the 
evaluation  team. 

Co-decision-makers  are not,  however,  the  only  actors  to  have  an  interest  in 
the  quality  of  reports  (see Box  44). The  steering  group,  in particular,  decides 
is  sufficient  or  must  be 
whether  the  collection  and  analysis  of  data 
improved,  whether  the  conclusions  are  acceptable  in  view  of  the  proof 
gathered,  and  whether  the  report  answers  all  the  questions  in the  terms  of 
reference  or  whether  further  work  needs  to  be  undertaken. 

In general, regional, national  and  European  authorities  analyse the quality  of 
evaluations  themselves,  in  relation  to  their  own  constraints  and  criteria.  In 
the  case  of  highly  technical  evaluations  or  the  publication  of  sensitive 
conclusions, 
to 
comment  on  the quality  of the  process  and  to  assess the  robustness  of  the 
conclusions.  The  quality  criteria  presented  below  are  largely  drawn  from 
professional  standards,  developed  within  the framework  of  assessments  of 
this  kind. 

the  commissioner  may  call  on  an  outside  specialist 

Using an  evaluation 

Box 44 -  Who  assesses the quality  of  evaluations? 

Quality  assessment 
can  be  done  by  .... 

with  a  view  to .... 

•  the  steering  group  with  the  help, 
when  necessary,  of  a  specialist 

•  validating  the final  report  and  verifying 

its  robustness 

•  the  co-decision-makers  of 

•  gauging  the  reliability  of  conclusions 

the  programme 

and  recommendations 

•  national  and  European 

•  gauging  the  reliability  of  conclusions 

authorities 

and  recommendations 

•  improving  the  evaluation 
system  currently  in  use 

•  independent  experts 

•  guaranteeing  the genuineness  of  the 

conclusions  disseminated  to  the  public 

•  developing  professional  standards 

(meta-evaluation) 

The  assessment  can  be  carried  out  in  a  spirit  of  quality  assurance  of  the 
entire evaluation  process, for the final report  is merely the  result of  a system 
which  mobilises  a  number  of  people  within  a  vast  institutional  ensemble, 
and  of  an  apparatus  specially  set  up for  the evaluation  itself.  In this  case, it 
is a question  of  monitoring  and  promoting  the  quality  of the  main  stages  of 
the  evaluation.  Taking  the  course  of  evaluation,  from  its  inception  to  the 
production  of  the  report  and  the  dissemination  of  the  results,  it  is  possible 
to  improve  its quality  by  acting  on  the  following: 

•  The  quality  of  information  and the  monitoring  system. 

•  The  terms  of  reference  defining  what  has to  be  evaluated  and  the  funds 

and time  allocated. 

•  The  choice  of  an  evaluation  team  (qualifications,  references,  experience, 

etc.). 

•  The contract  signed  with the  evaluation team, which  outlines  the  work  to 

be  performed. 

•  The  frequency  and  nature  of  interactions  between  the  evaluation  team 

and the  steering  group. 

•  The  rapid  and  complete  provision  of  available  high  quality  information. 

•  The  mode  of  communicating  conclusions,  including  the  report  itself  and 

the  other  channels  of  dissemination. 

•  Use of  the  evaluation  for  reporting  to the  public  authorities  and  citizens. 

Evaluation design and management 

A system of quality assurance concerns the entire evaluation process and 
therefore covers all these points. It may be represented as follows. 

Box 45 ­ System for controlling overall quality 

Terms of 
reference (contract) 

\ 

Interaction with 
commissioners 

Competence of the 
evaluation 
team 

\ 

/ 

\ 

f 

ÎI 

Performing 
the evaluation 

Π 

Quality of 
available 
information 

1 

Responsibility 
towards the stake­
holders / public 

1 

Reports  |l 

►■ 

A n a l y s i ng t he q u a l i ty of an e v a l u a t i on r e p o rt 

In general texts devoted to evaluation, the notion of quality is a subject of 
much debate. There is no system of professional certification, anywhere in 
the world, which institutionalises rules and quality criteria. Somewhat 
disparate grids of criteria are proposed, based on the evaluation models 
elaborated by various authors, but no consensus exists in this domain. 
Moreover, the nature of the criteria mentioned does not always appear 
clearly when an attempt is made to put them into practice. 

Within the framework of MEANS, eight quality assessment criteria for 
evaluation reports were selected. The global assessment of an evaluation is 
established synthetically after the final report has been analysed in terms of 
these eight criteria. The criteria are presented below, with the reference 
situation taken to be mid-term evaluations, although the grid has a more 
general scope. It can be applied to all evaluation situations, bearing in mind 
that the formulation of certain questions and the relative weight of the 
criteria must take into account the specific nature of the exercise under 
consideration. Throughout the rest of the text such adjustments for ex ante 
and ex post evaluation are mentioned, in relation to intermediate evaluation, 
when they exist. 

Using an  evaluation 

Eight  criteria  were  defined  for  assessing  the  quality  of  evaluation  reports. 
The global assessment  of an evaluation is established  synthetically  after  the 
final  report  has  been  analysed  in terms  of  these  eight  criteria. 

Meeting  needs 

Within  the  framework  of  European  Structural  Fund  programmes,  the  main 
stakeholders  are  the  members  of  the  steering  group.  Ideally,  their  diverse 
requirements  are grouped  together  in the terms  of  reference.  The  evaluative 
questions  formulated  in  this  document  must  be  complemented  during 
preliminary  meetings  and  interviews,  in  order  to  go  beyond  what  is  already 
obvious for the commissioners. It is therefore necessary to check whether the 
evaluation has answered the questions  in the terms  of reference  satisfactorily 
and whether  the report  covers  additional  requests  for  information, as well  as 
new  questions  which  might  be essential  for  the  commissioners. 

Thus,  to  ascertain  whether  "evaluation  deals  adequately  with  requests  for 
information 
formulated  by  the  commissioners,  and  corresponds  to  the 
terms  of  reference"  (see  the  synthetic  grid  in  Box  46),  the  following 
questions  must  receive  particular  attention: 

•  Has  the  evolution  of  global  and  specific  objectives  of  the  programme 

been  analysed? 

•  Does  the  report  cover  the  entire  programme?  If  not,  is  the  selection 

justified  as  regards  the  priorities  stated  by the  commissioners? 

•  Does  the  evaluation  provide  useful feedback  for  the  managers? 

•  Does  it  include  lessons  on  successes  and  failures  which  may  be  of 

interest  to  other  regions  or  countries? 

Unlike mid-term  evaluations, for ex  post  evaluations  it is important  to  check 
whether  the  information  needs  of  the  new  generation  of  programmes  have 
been  taken  into  consideration.  It  is  therefore  necessary  to  check  whether 
the  evaluation  has  managed  to  reach  a  reasonable  compromise  between 
the following  two  contradictory  requirements:  rapidly  obtaining  information 
for  feeding 
into  the  new  programme  cycle  and  not  drawing  hasty 
conclusions  before  all the  impacts  have  been  observed. 

Relevant  scope 

The  scope  of  an  evaluation  must  cover  questions  which  are  relevant  from 
the  point  of  view  of  Structural  Fund  programmes.  The  brief  description  of 
essential  programme  characteristics  and  the  analysis  of  the  strategy  of  a 

Evaluation design and management 

programme 
understanding  the  evaluated  action  and  its  rationale. 

for  the  region  concerned  are  essential  prerequisites 

for 

In order to check  the relevance of the scope  of an evaluation, it is  necessary 
first  to  check  whether  the  essential  characteristics  of  the  programme  have 
been  well  described  and  whether  the  problems  and  successes  in  the 
implementation  of  the  programme  have  been  properly  clarified. 

Secondly,  because  the  results  and  impacts  of  the  programme  have  to  be 
analysed  in  order  to  judge  the  extent  to  which  programme  objectives  have 
been  achieved, it is necessary  to check  whether  they  have  been  included  in 
the  evaluation.  It is also  necessary  to  check  whether  the  evaluation  has  not 
overlooked  other  potential  or  future  results  or  impacts,  as  well  as  any 
unexpected  yet  significant  effects  and  results  which  may  exist. 

Finally, the  scope  of  an  evaluation  depends  on the  programme  target  which 
can  be  defined  in  terms  of  eligible  geographical  areas  or  non-localised 
target  groups  (e.g. the  long-term  unemployed).  It  is  therefore  necessary  to 
check  whether: 

•  the  limits  of  the  scope,  in  terms  of  areas  or  social  groups,  are  defined 

according  to  the  logic  of  the  intervention; 

•  the  scope  includes  peripheral  areas  or  non-eligible  groups  which  are 

nevertheless  likely  to  be  affected  by  the  evaluated  interventions; 

•  lastly,  if the  evaluation  considers  the  evaluated  programme  in  isolation  or 
includes  its  interactions  with  other  European  or  national  programmes. 

Within  the  framework  of  ex  ante  evaluations,  the  report  has  to  add  all  new 
information  which  may  be  useful  for  the  improvement  of  programme 
documents:  strategy, 
implementation  of 
choices.  It  is  necessary  to  check  whether  the  report  evaluates  in  sufficient 
detail  the  target  groups'  expectations  and  the  corresponding  relevance  of 
the  programme  objectives  and  strategy. 

internal/external  coherence, 

•  Defensible  design 

This  essential  criterion  relates  to  the  technical  qualities  of  the  evaluation. 
Methodological  choices  must  be  derived  from  the evaluative  questions  and 
demonstrations  required.  The  evaluation  must,  moreover,  make  the  best 
possible  use  of  existing  research  and  analyses.  For this  criterion  as well  as 
the  following  two,  the  commissioners  could,  if  necessary,  request  an  audit 
by  programme  evaluation  specialists. 

Using an  evaluation 

In any  case, three types  of  question  have to  be asked: 

•  Have  the  relevant  knowledge  and  existing  literature  been  collected  and 

used  wisely? 

•  Are the construction  of the  method  and the  choice of tools  really  justified 

for  answering  the  evaluative  questions  properly? 

•  Were 

the  reference  situations  chosen  (counterfactual  or  similar) 

appropriate  for  making  valid  comparisons? 

Any  evaluation  report  must  include  a  description  of  the  method  used  and 
clearly define the sources of data. Similarly, the  limits of the method  and the 
tools  used  must  be  clearly  described.  It  is necessary  to  check  whether: 

•  the  method  is described  in  enough  detail  for the  quality  to  be judged; 

•  the  validity  of  data  collected  and tools  used  is clearly  indicated; 

•  finally, the  available  data  correspond  to the tools  used. 

Because  a  causal  analysis  of  effects  is  the  most  important  question  in  ex 
post  evaluations,  the  method  used  to  analyse  these  causal  relations  is  the 
priority  in  this  type  of  evaluation.  It  is  necessary  to  check  whether  the 
evaluation  adequately  analyses  relations  of  cause  and  effect  for  the  most 
essential  questions. 

•  Reliable  data 

Evaluators  use  existing  data  (secondary  data)  from  the  monitoring  system 
and from other  sources of  information, or else primary data which they  have 
collected  for  the  evaluation.  In the  latter  case, the  method  used  to  collect 
and  process  the  data  (choice  and  application  of  the  tools  used  for  this 
purpose)  are  very  important  factors  in  the  reliability  and  validity  of  the 
results. 

In order to assess the  reliability  of the data used, it is necessary to  examine: 

•  whether  available  sources  of 

information  have  been 

identified  and 

whether  the  reliability  of  this  data  has  been checked; 

•  whether  sources  of  information  taken  from  the  monitoring  system  and 

previous  studies  have  been  used  optimally; 

•  and,  lastly,  the  extents 

the 
monitoring  system  and  its  stock  of  useful  data  and  indicators  of  use  in 
ex-post  evaluation. 

to  wich  mid-term  evaluation  reviews 

•  whether  the  techniques  used  to  collect  the  chosen  data  were  complete 

and  suitable for  answering  the  evaluative  questions; 

Evaluation design and management 

Whether  the collection  of  data used quantitative or qualitative techniques  or 
a  combination  of  both,  each  of  the  following  steps  must  be  analysed: 
definition  of  "mother  populations";  selection  of  samples;  construction  of 
tools  (the  questionnaire;  the  indicators  chosen).  It  is therefore  necessary  to 
inquire  whether: 

•  the  mixture  of  qualitative  and  quantitative  data  is  appropriate  for  a  valid 

analysis  of  the  phenomenon; 

•  the  "populations"  used  for  data  collection  have  been  correctly  defined; 

•  the  survey  samples  or  cases  studied  have  been  selected  in  relation  to 

established  criteria; 

•  the  main  data  collection 

implemented  with 
appropriate  tools  and  in such  a way  as to  guarantee  an adequate  degree 
of  reliability  and  validity  of  the  results. 

techniques  have  been 

For ex ante evaluation, analyses  are limited to existing  data  because  ottime 
constraints.  This  implies  that  the  sources  of  information  selected  are 
sufficiently  reliable  so  that  a  credible  judgement  can  be  made  of  the 
relevance  and  rationale  of  the  programme. 

Sound  analysis 

Quantitative  analysis consists  of the analysis of  data  in the form  of tables  or 
any  other  form  of  statistical  analysis.  Qualitative  analysis  consists  of  the 
systematic  comparison  and  interpretation  of  information  sources  in  the 
form  of  cross-referencing.  In  both  cases  it  is  necessary  to  assess  whether 
the  methods  of  analysis  used  are  relevant  as  regards  the  type  of  data 
collected,  and  whether  the  analysis  has  been  carried  out  according  to 
instructions  in the  relevant  technical  manuals. 

In  the  case  of  the  Structural  Funds,  relations  of  cause  and  effect  are 
complex  and  therefore  constitute  a  particular  challenge  for  evaluation.  It  is 
necessary  to  check: 

•  whether  the  relations  of  cause  and  effect  underlying  the  programme  are 
sufficiently  explicit  and  relevant  so  that  the  object  of  analysis  can  be 
focused, 

•  and  to  what  extent  the  analysis  uses  suitable  techniques. 

Within  the  framework  of  in-depth  evaluation,  a  before-after  comparison  or, 
when  similar  groups  do  not  exist,  a comparison  between  beneficiaries  and 

Using an  evaluation 

a  control  group,  is  recommended.  In  the  former  case  this  before-after 
comparison  must  be  carried  out  appropriately,  otherwise  it  must  be 
established  whether  that  would  have  been  possible.  In the  latter  case,  the 
comparative  analysis  must  be  able  to  rely  on  the  collection  of  data  within 
similar  or  control  groups. 

Furthermore,  the  exploitation  of  evaluation  work  carried  out  at 
the 
programme  level should  enable the Commission to  prepare syntheses at the 
European  level. This  implies  coherent  definitions  (e.g. calculation  of  the  net 
effect),  homogeneous  figures,  etc.,  so  that  the  results  can  be  used  to 
compare the performance  of the programme  with that  of other  programmes. 

Credible  results 

if  there 

The credibility  of  results  is defined  here as the fact  that  they follow  logically, 
and  are  justified  by,  the  analysis  of  data  and  interpretations  based  on 
carefully  presented explanatory  hypotheses. The validity  of the results  must 
be  satisfactory.  Even 
the 
fundamental  contradiction  between  internal  validity  (absence  of  technical 
bias  in  the  collection  and  processing  of  data)  and  external  validity 
(representativeness  of  results),  the  balance  between 
two  must 
nevertheless  be  justifiable,  given  the  type  of  evaluation  considered  (in-
depth  or  overall).  It  is  also  necessary  to  check  whether  the  results  of  the 
analysis  were  produced  in a  balanced  and  reliable  way. 

is  no  single  solution  for  solving 

the 

The  need  to  perform  in-depth  analyses  of  a  part  of  the  programme  poses 
the  problem  of  extrapolation,  from  case  studies,  for  the  programme  (or 
Member  State)  as  a whole.  In this  context,  it  is necessary  to  check: 

•  whether  the  interpretative  hypotheses  and  extrapolations  are  justifiable, 

and  whether  the  limits  of  validity  have  been  defined; 

•  whether  the  selection  of  cases  and  samples  makes  it  possible  to 

generalise the  findings. 

With  the  stakeholders'  participation 
in  the  discussion  of  results,  the 
relevance  of the  evaluation  and, consequently,  the  utility  of  the  programme 
can  be enhanced. The following  questions  need  to  be asked: 

•  Are  the  results  credible 

in  the  sense  of  reflecting  an  acceptable 
compromise  between  the  reality  of  the  programme,  as  perceived  by  the 
actors,  and  the  reality  described  by  the  observed  or  estimated  figures 
and  facts? 

Evaluation design and management 

•  To  what  extent  were  the  stakeholders 

involved 

in  the  process  of 

interpreting  empirical  facts? 

•  Were the  results  presented  impartially  to  the  stakeholders? 

• 

Impartial  conclusions 

Conclusions  include  suggestions  and  sometimes  even  recommendations 
which  are  more  than  results.  Whereas  results  are  "technical"  and  can  be 
analysed  without  too  much  risk  of  impartiality,  conclusions  and,  a  fortiori, 
recommendations  are  issued  on the  basis  of  value judgements.  The  quality 
of  the judgement  is thus  decisive. 

To  answer  the  question  "Are  the  conclusions  fair,  free  of  personal  or 
implemented 
partisan  considerations,  and  detailed  enough 
concretely?",  it  is  necessary  to  check: 

to  be 

•  whether  the  elements  on  which  the  conclusions  are  based  are  clear; 

•  whether  the  conclusions  are  operational  and  sufficiently  explicit  to  be 

implemented; 

•  whether  controversial  questions  are presented  in a fair and  balanced  way. 

•  Clear  report 

Evaluation  results  can  be  disseminated  and  communicated 
the 
stakeholders  in  writing  or  verbally.  The  final  report  is  only  one  means  of 
diffusion  among  others, and continual  communication  of  results  is desirable. 
The  legibility  of  the  report  will  depend  on  the  quality  of  the  presentation  of 
results  and  the  limits  of  the  work  performed.  It  is necessary  to  check: 
•  whether  the  report  was  written  clearly for  its addressees  and whether  it  is 

to 

set  out  logically; 

•  whether  specialised  concepts  are  used  only  when  absolutely  necessary 

and  whether  they  are  clearly  defined; 

•  the  extent  to  which  the  presentation,  tables  and  graphs  enhance  the 

legibility  and  intelligibility  of  the  report; 

•  and  whether  the  limits  of  the  evaluation, in terms  of  scope,  methods  and 

conclusions,  are  clearly  shown. 

In many  cases  only  the  summary  of  a report  is read.  It  is therefore  essential 
for  this  summary  to  be  clear  and  concise.  It  must  present  the  main 
conclusions  and  recommendations  in  a  balanced  and  impartial  manner.  It 

Using an  evaluation 

must  be easy  to  read  without  the  need to  refer  to  the  rest  of  the  report. 

It is also important  for conclusions to be clear as regards  broader  European 
issues,  such  as  environmental 
impacts,  equal 
opportunities,  and  so on. 

impacts,  employment 

M a k i ng  a  fair  global  a s s e s s m e nt 

Synthetic  assessments  recapitulate  all  the  quality  criteria. 
It  is  not 
necessary  to  recommend  any  particular  weighting  for  the  different  criteria 
because their  importance  varies from  one  situation  to  the  next. 

The synthetic  assessment  must  take  into  account  the  constraints  weighing 
on  the  evaluation  and  the  team  which  performed  it. The  initial  conclusion 
reached  by  strictly  applying  quality  criteria  must  therefore  be qualified.  For 
example,  a report  may  be judged  to  be  scarcely  adequate,  not  because  of 
shortcomings  related to  the  actual  evaluation  work,  but  because  the  terms 
of  reference  were  unrealistic.  Very  often,  the  time  and  financial  resources 
allocated to evaluation teams limit the scope and soundness of  conclusions 
far  too  much.  Contextual  constraints  must 
into 
consideration  for judging  the  report  in  fine. 

therefore  be  taken 

inputs,  the  time  allocated,  and 

is  concerned.  Constraints  relating  to 

Two  types  of  constraint  can  be  distinguished,  depending  on  whether  the 
commissioning  of  an  evaluation  or  the  management  of  the  evaluated 
programme 
the  commissioning 
procedure  are: clarity  and  precision  in the terms  of  reference, the volume  of 
budgetary 
the 
commissioners  during  the  implementation  of  the  evaluation.  Constraints 
relating  to  the  programme  and  its  management  may  result  from  a  lack  of 
clarity  on the  objectives,  a high  level of  heterogeneity  among the  measures 
and 
the 
monitoring system, as well as crises or ruptures which may occur during the 
implementation  and  which  impact  on the  course  of  the  evaluation. 

inadequate  quality  of  the  data  generated  by 

involvement  of 

interventions, 

the 

It  is  up  to  the  commissioners  to  limit,  if  not  totally  eliminate,  the  effects  of 
these  constraints,  and  up  to  the  evaluation  teams  to  anticipate  technical 
and  operational  problems  before  embarking  on the  evaluation  process. 

Evaluation design and management 

Box 46 -  Grid for a synthetic assessment of the quality 

of evaluation work 

With  regard to  this  criterion, the  evaluation  report  is: 

excellent 

good 

acceptable 

inacceptable 

1.  Meeting  needs: Does  the  evaluation  adequately 
address  the  requests  for  information  formulated 
by  the  commissioners  and  does  it  correspond 
to  the terms  of  reference? 

2.  Relevant  scope:  Have the  rationale  of  the 

programme,  its  outputs,  results,  impacts,  interactions 
with  other  policies  and  unexpected  effects  been 
carefully  studied? 

3.  Defensible  design: Is the  design  of  the  evaluation 
appropriate  and  adequate  for  obtaining  the  results 
(with  their  limits  of  validity)  needed  to  answer  the 
main  evaluative  questions? 

4.  Reliable  data: Are the  primary  and  secondary  data 
collected  or  selected  suitable?  Are  they  sufficiently 
reliable  compared  to  the  expected  use? 

5. Sound  analysis: Are  quantitative  and  qualitative 

data  analysed  in accordance  with  established  rules, 
and  are they  complete  and  appropriate  for  answering 
the  evaluative  questions  correctly? 

6.  Credible  results: Are the  results  logical  and  justified 
by  the  analysis  of  data  and  by  interpretations  based 
on  carefully  presented  explanatory  hypotheses? 

7.  Impartial  conclusions: Are the  conclusions  just  and 
non-biased  by  personal  or  partisan  considerations, 
and  are they  detailed  enough  to  be  implemented 
concretely? 

8.  Clear  report:  Does the  report  describe  the  context 
and  goal,  as  well  as  the  organisation  and  results  of 
the  evaluated  programme  in such  a way  that  the 
information  provided  is easily  understood? 

D  D  D  D 

D  D  D  D 

Π  D  D  D 

Π  D  D  D 

Π  D  Π  Π 

Π  Π  Π  Π 

Π  D  Π  Π 

Π  D  D  Π 

In view  of  the  contextual  constraints  bearing  on  the 
evaluation, the  evaluation  report  is considered  to  be 

Π  Π  Π  Π 

ν 
Guaranteeing the success of an evaluation 

Better preparation of an evaluation 

The programmes financed by the Structural Funds are subject to a 
particularly intense evaluation effort. Since 1989, several hundred 
programmes have been evaluated in their entirety, before, during and after 
their implementation. This is without counting transnational thematic 
evaluation and more in-depth analyses carried out in certain regions. 
Evaluation practice has rarely spread as rapidly and on such a massive 
scale. Progress still needs to be made, however, from the point of view of 
quality/cost ratios and that of utilisation. 

A number of the recommendations likely to increase the use of evaluation 
concern its preparation. Pluri-annual planning of the different evaluation 
exercises must be continued, the commissioning authorities need to make 
their decisions more visible when launching an evaluation, and more prior 
investments also need to be put into the drawing up of the terms of 
reference, if necessary in the form of an evaluability study when complex 
issues need to be addressed. 

■ Pluri-annual p r o g r a m m i ng of evaluations 

As described in the second part of this book (see Box 7) the sequence of 
programmes, period after period, is unavoidable. The programmes follow 
one another uninterruptedly and the regulations require a series of 
evaluations for each period (ex ante / mid-term / ex post). In practice, these 
different constraints have created and will continue to create problems of 
articulation between the different evaluation exercises. In particular, it is 
often difficult to learn from the implementation of a programme, at a time 
and in a form that is really adapted to issues involving decisions. 
Experience shows that it is necessary to think about a pluri-annual 
evaluation programme with the goal of improving the opportunities for 
feedback (a process which leads from the observation of results and 
impacts, to decisions to reorient the programme). 

To deal with this problem, many managers and evaluation teams wish to 
develop on-going evaluation. Experiments in this domain have been run in 
Spain (for regional operational programmes in Andalusia and the 
Independent Community of Valencia), Ireland (for about half of the 
operational programmes) and France (programmes in the Burgundy region), 
for example. These attempts warrant careful study. 

Evaluation design and management 

Many  experts  are, however,  sceptical  about  evaluation  conceived  as an on­
going  process. The  system  creates  a  risk  of  confusion  between  evaluation, 
monitoring  and  management,  which  could  well  result  in the  watering  down 
of  the  value  added  by  evaluation.  On  the  other  hand,  the  establishment  of 
a  pluri-annual  evaluation  programme  can  be  recommended  for  linking  up  a 
sequence  of  different  types  of  work  at  different  stages:  prior  appraisal, in-
depth  analyses,  mid-term  overall  evaluation,  and  ex  post  evaluation.  This 
recommendation  is inspired  by Australian  and  Canadian  experience.  It  was 
adopted  by 
internal  evaluation  system 
(SEM 2000). 

the  Commission 

its  own 

in 

This  type  of  programme  should  enable  different  exercises  to  be  linked 
together  and  to  benefit  at  any  time  from  lessons  from  past  experience, 
irrespective  of  the  programming  period  from  which  they  are  drawn. 
it  possible  to  coordinate  evaluation  work 
Moreover, 
commissioned  at  different  levels  in  the  same  territory,  and  to  avoid  the 
same  people  being  questioned  repeatedly  in  surveys. 

it  should  make 

Several  examples  exist  where  commissioners  considered  it  preferable  to 
request  a single team to  carry  out  a series of  evaluations  in the same  region 
or  domain  of 
the  advantage  of 
guaranteeing  automatic  coordination  of  evaluations  since  they  are  all 
it  cannot  be  extended 
carried  out  by  the  same  supplier.  However, 
indefinitely  without  compromising  the  independence  of  the  evaluations. 

intervention.  This  approach  has 

Involving  decision-makers  through  an evaluation  mandate 

Even  if  an  evaluation  is  not  mandatory,  the  decision  to  evaluate  (see 
page  65) remains  a political  act  resulting from  a decision  taken  by the  main 
commissioner.  Experience  has  shown  that  it  is  preferable  for  this  decision 
to  be  formalised  in  a  brief,  clear  and  visible  document,  often  called  the 
is  particularly  valuable  when 
"evaluation  mandate".  This  document 
personally  signed  by  the  highest  authority  concerned  by  the  evaluation. 

from 

The  mandate  specifies  what  is  going  to  be  evaluated  (the  scope  of  the 
evaluation),  the  motives  for  the  launching  of  the  evaluation  and  the  use 
expected 
it.  The  main  commissioner  specifies  whether  or  not 
recommendations  are expected to  be made at the end of the evaluation. He 
or  she  states  a  few  main  questions  for  which  the  evaluation  must  find 
answers,  which  generally  gives  an  initial  idea  of  the  evaluation  criteria,  and 
sets  a  deadline  for  submitting  the  report.  He  or  she  may  also  determine  a 
budget. 

Guaranteeing the success  of an  evaluation 

By  proceeding  in  this  way,  the  commissioner  sets  the  limits  which  seem 
reasonable  and  will  apply  to  the  entire  evaluation  process.  The  mandate 
must  nevertheless  avoid  restricting  the  steering  group's  and  evaluation 
team's  freedom.  It  must  not  become  an  instrument  of  preventive  censure. 
Its  main  advantage  is,  on  the  contrary,  to  concentrate  the  evaluation  on 
strategic  subjects  of  interest  at  the  highest  levels.  The  mandate  also 
ensures  that  evaluation  does  not  move  onto  politically  risky  ground  where 
it could  be totally  rejected  when  the  results  are released. 

The  mandate  aims  at  preventing  a  number  of  excesses,  as  listed  below. 

•  Strategic  issues  have  given  way  to  the  short-term  concerns  of  the 

managers. 

•  The authorities  do  not feel committed  by the conclusions, and those  high 
up  in  the  hierarchy  do  not  allow  their  subordinates  to  participate  in  the 
evaluation. 

•  The evaluation  team  acts  outside  its  ambit. 

•  The evaluation  report  arrives  too  late and  is therefore  impossible  to  use. 

•  The evaluation  tends  to  cover  too  much  and  cannot  reach  conclusions. 

Another  advantage  of  the  mandate  is that  it  establishes  clear  and  realistic 
rules  of  the  game.  In  this  perspective,  the  signatory  authority  officially 
hands  over  responsibility  for  the  evaluation  process  to  a person,  preferably 
assisted  by  a  steering  group.  At  the  same  time  the  composition  of  the 
steering  group  and  the  limits  of  its  missions  are determined. 

Investing  in  t he  preparation  of  the  t e r ms  of  r e f e r e n ce 

to  be  bureaucratic,  scarcely  orientated 

Experience  has  shown  that  the  terms  of  reference  are  often  drawn  up 
hastily, based on standard  models  or documents. This could  well cause  the 
local 
evaluation 
particularities,  badly-suited  to  the  programme  issues and  hardly  motivating 
for the  managers.  For as  long  as the  writing  up  of  the terms  of  reference  is 
considered  an  office  job,  it  will  be  difficult  to  make  the  necessary  choices 
for  it to  be  properly  adapted  to  the  needs  of  its future  users. 

towards 

The  situation  is  different  if,  from  the  outset,  the  commissioner  officially 
mandates  a  steering  group  for  the  entire  duration  of  the  evaluation.  This 
group  can  then  be  given  an  active  role  in  the  preparation  of  the  terms  of 
reference  (see page 75). This  implies the  organisation  of  one  or two  or  even 
three  meetings  which  will,  above  all,  make  it  possible  to  unambiguously 

Evaluation design and management 

formulate  evaluative  questions,  check  that  the  essential  (or  embarrassing) 
questions  have not  been forgotten, reject questions  which are more a matter 
of  auditing  or  control,  select  priority  questions,  and  specify  the  judgement 
criteria and  norms.  Experience  has shown  that,  after  this type of  preliminary 
study,  several  different  evaluation  strategies  are envisaged,  choices  have  to 
be  made,  and  the  quality  of  the  evaluation  is  finally  improved.  An  extreme 
but  salutary  choice  may  be  the  postponement  or  cancellation  of  the 
evaluation, when the steering group foresees that  its results will not  be used. 
The  preliminary  study  is not  an  additional  task  but  rather  an  alternative  way 
of  preparing  the evaluation terms of  reference. The two  meetings  devoted  to 
it by the  steering group  provide an opportunity  to  do what  is usually done  by 
the  evaluation  team  in the  definition  of  its tasks,  as  Box  47  shows. 

Box  47  -  Alternative  process for  evaluation 

Current  practice 

Terms of  reference 

Commissioner 

Steering  group 

Recommendation 

Mandate 

Preliminary  study, 
evaluability  study 
terms  of  reference 

Outputs 
conclusions 
recommendations 

Evaluation  team 

Outputs 
conclusions 

Dissemination 
utilisation 

Commissioner 

Dissemination 
utilisation 

Steering  group 

Recommendations 

This  process  consists  of  going  back  to  the  programme  rationale  and 
strategy,  recapitulating  its  objectives,  and  highlighting  the  order  of  priority 
and  the  articulation  of  these  objectives.  At  this  stage  the  steering  group 
checks  whether  there  is any  misunderstanding  about  the  programme  logic 
and,  where  necessary,  clarifies  it.  During  the  preparatory  meetings,  the 
steering  group carefully  formulates  the  evaluative  questions  and  selects  the 
most  important  ones.  It expresses  an opinion  on the judgement  criteria  and 
benchmarks  which  are  to  be  used  by the  evaluation  team. 

Guaranteeing the success  of an evaluation 

In  the  preparatory  study,  the  programme  managers  perform  certain  tasks 
which  only they  are competent  and authorised to  perform. They  must  strive 
to  avoid  the  frequent  situation  where  the  evaluation  team  reproduces,  with 
a loss of  quality,  information  already  known to the  administration. The  team 
must, on the contrary, focus  its work  on the production  and analysis of  new 
data. 

The  evaluability  study  is  a  similar  exercise  which  may  be  combined  with 
preparatory  meetings.  It  consists  of  technical  control  recommended  by 
several  North  American  authors,  and  concerns  evaluation  in  general  and 
in  particular.  The  study  reviews  the  level  of  available 
each  question 
knowledge  and  concludes  on  whether  the  technical  and 
institutional 
conditions will be met  so that  reliable and credible answers can  be found  to 
the  questions  asked. This  diagnosis  is  performed  rapidly  by  an  evaluation 
methodologist  (a  member  of  an  evaluation  unit  or  an  outside  expert).  It 
results  in the  elimination  of  certain  questions,  considered  too  difficult  or  for 
which  answers  already  exist. 

The  evaluability  study  defines  the  main  lines  of  the  evaluation  method  so 
that  the call for tenders can  be directed  towards  those teams with the  most 
appropriate  competencies.  It  also  allows  for  the  adjustment  of  the  number 
of questions  asked  and their  level of difficulty,  in relation to the financial  and 
human  resources  available  for  the  evaluation.  This,  in  itself,  is  a  good 
reason to  invest  in the  preparation  of the terms  of  reference. The  unrealistic 
ambitions  of  certain  terms  of  reference  are  frequently 
the  cause  of 
dissatisfaction  for  commissioners,  and  problems  for  evaluation  teams.  It  is 
not  rare  to  find  terms  of  reference  which  require  a  quantitative  analysis  of 
multiple impacts yet which allocate only a few  months and a limited  budget. 
A  study  on  about  30  impact  analyses  in  Ireland  showed  that  the  average 
cost  of  in-depth  evaluation was  between  50 and  100 Keuro, and that  it took 
from  six  to  12  months. 

Opening evaluation to more partners 

The concrete use of evaluation could also develop through a better  balance 
between  the  independence  of  the  evaluation  and  its  interaction  with  the 
interaction 
programme  managers.  Experience  has  shown  that  strong 
between  the  evaluation  team  and  the  managers  enhances  the  quality  and 
utility  of  the  exercise. 

It  is furthermore  preferable,  in  many  cases, to  involve  more  partners  in  the 
steering  group,  in  order  to  increase  the  legitimacy  and  relevance  of  the 
conclusions.  It  is,  however,  advisable  to  strictly  control  the  number  and 
choice  of  people  invited  to  participate  in  the  evaluation,  as  well  as  the 
efficiency  of  the  evaluation  process.  The  organisation  of  real  collective 
reflection  can  be  organised  within  a  larger  steering  group  by  using 
appropriate  working  models. 

Involving  the  m a n a g e rs  to  facilitate  appropriation 

Use  of  external  evaluation  teams  is justified  by the  need  for  competencies 
which  the  administration  does  not  always  have,  whether  they  concern 
evaluation  techniques  or  a  particular  impact  mechanism.  An  external 
evaluation also has the advantage  of allowing for a new, more objective  and 
impartial  view  of the evaluated  programme, and of  enhancing the  credibility 
of  the  results.  It  is  the  latter  advantage  which  justifies  the  demand  for 
independence,  mentioned 
texts  concerning 
evaluation. 

in  all  European  Union 

The  concern  for  independence  must  not,  however,  cause  the  evaluation 
team  to  work  in  isolation.  In  the  domain  of  European  Structural  Funds, 
experience converges to show that  interaction  between the evaluation  team 
and  the  managers  strengthens  the  managers'  motivation, avoids  problems 
of  access  to  information,  and  leads  to  a  better  use  of  the  results.  By 
contrast,  the  credibility  of  the  evaluation  team  may  be  weakened  if  its 
independence  is  accompanied  by  a  lack  of  familiarity  with  administrative 
systems, or if it is mistrusted and consequently  induces defensive  attitudes. 

Because  a  lack  of  contact  with  the  managers  is  always  prejudicial,  they 
must  be  required  to  be sufficiently  available,  as must  the  people  appointed 
to  perform  certain  tasks.  Sufficient  time  must  also  be  set  aside  for 
interaction  with  the  evaluation  team  at  each  intermediate  step. 

Evaluation design and management 

To avoid blockages, a common sense rule consists of setting up systems 
of feedback to the managers during the course of the evaluation. The 
steering group can circulate intermediate reports and the first results of 
surveys or analyses. This solution has the advantage of making it possible 
to correct factual errors which are easily detected by the managers. In at 
least two cases (the Brandebourg Land and the Aquitaine region), this 
exchange of information was computerised. 

For the evaluation exercise to be considered impartial, it is also possible to 
include several programme managers and/or operators in the steering 
group. These participants are asked to speak on their own behalf, to 
respect the confidentiality of the points of view expressed in the meetings, 
and not to act as "official representatives" of their institution. This practice 
has positive effects in terms of the quality and credibility of the work, in so 
far as the people invited to the steering group can voice the expectations 
and fears expressed by the managers. They may also attest to the 
impartiality of evaluation work. 

Another recommendation is to view evaluation not as the systematic search 
for problems and failures, but as a balanced examination which attaches 
sufficient importance to the identification of successes and best practice 
and to their explanation and development. In this way the image of 
evaluation can be enhanced. 

■ Opening up the steering group to external partners 

• Pluralistic evaluation committee 

The implementation of European Structural Funds is characterised by joint 
funding and decision-making between several administrative levels. Even if 
the final responsibility for commissioning an evaluation is almost always at 
a single level, evaluation cannot do without information from other levels 
nor disregard expectations at other levels. In this context, it would be 
dangerous and futile to consider evaluation as a unilateral exercise 
controlled by single decision-makers for their exclusive use. For this 
reason, evaluation practice often involves the constitution of a steering 
group which allows the expression of diverse points of view (see page 67). 

Whenever a steering group exists and plays an important part in the 
evaluation, its expansion can be considered. As shown above, it is useful to 

Guaranteeing the success  of an  evaluation 

include  a  few  managers  and/or  operators  in  the  group  to  make  evaluation 
more  interesting  and  credible  for  that  category  of  partners.  The  same 
approach  can  be adopted  to  include  other  partners  in the  group. 

Several  North  American  authors  recommend  including  in the evaluation  the 
terms  used  are  "stakeholders",  or 
main  partners  concerned.  The 
protagonists. Any  person who  is potentially  a user of the  recommendations, 
is  interested  in  the  information  produced  or  is  likely  to  win  or  lose  in  the 
process,  can  potentially  be  associated  with  the  evaluation. Thus, there  are 
clearly  a  large  number  of  stakeholders  concerned  in  evaluation.  More 
simply,  it  is possible to  divide  them  into  large categories,  usually  consisting 
of  funding  authorities,  managers, operators  and  concerned  groups. 

The  inclusion  of  a  category  of  stakeholders  in  the  steering  group  has  the 
same advantages  as those mentioned  above, in respect of the  participation 
of  managers.  The  people  invited  into  the  group  voice  the  hopes  and  fears 
expressed at their  level, enhance the relevance  of the questions  asked and, 
by  their  presence,  render  the  evaluation  more  credible.  Wider  diversity 
within  the  steering  group  brings  together  more  points  of  view,  facilitates 
their  comparison,  and  helps  to  raise  the 
level  of  consensus.  Each 
participant  adds  her/his  questions,  value  system,  judgement  criteria  and 
norms.  Provided  it  is able  to  take  advantage  of this  plurality,  the  evaluation 
produces  richer,  more  credible  and  more  useful  conclusions.  When  the 
expectations  of several partners are taken into account, the results are likely 
to  be  accepted  more  willingly  and  with  greater  interest.  These  advantages 
are  synthesised  in  Box  48. 

Box  48 -  The  advantages  of a  larger  evaluation  committee 

The  constitution  of  an  evaluation  steering  committee  consisting  of  the  different 
stakeholders  in the  programme  makes  it  possible  to  guarantee: 

•  better acceptance  of the evaluation  by those evaluated, by creating  relations of 

trust; 

•  easier access to  information and a better  understanding  of the facts and  events 

which  took  place while  the  programme  was  underway; 

• 

• 

interpretations  and recommendations  which take into account  all the  important 
points  of  view; 

the dissemination  of  conclusions  and taking  into  account  of  recommendations 
more  rapidly  and  informally. 

Evaluation design and management 

The  people  to  include  in  the  evaluation  may  be  academics,  company 
directors,  leaders  of  professional  syndicates,  heads  of  non-governmental 
organisations,  etc.  The  participants  are  always  invited 
in  a  personal 
capacity  and  must  undertake  to  respect  the confidentiality  of  points of  view 
expressed  during  meetings.  In  some  cases  of  evaluations  intended  for 
accounting to the general  public, the decision was  made to involve  citizens, 
selected  randomly,  or journalists.  If the steering group  is extended  to  about 
ten  people  with  differing  points  of  view,  its  role  in  the  evaluation  process 
becomes  more  visible  and  decisive.  In  such  cases  it  is  given  a  different 
name,  reflecting  its  size,  such  as  evaluation  committee  or  evaluation 
authority. 

When  an  evaluation  committee  is  formed,  it  is  important  to  list  all  the 
stakeholders  likely  to  have  an  interest  in  the  evaluation  and  its  results.  For 
this  purpose,  the  diagram  in  the  following  box,  called  a  "sociogram",  may 
be  used.  This  diagram  divides  the  stakeholders  into  three  categories: 
political  authorities  (elected  representatives,  decision-makers  or  funding 
authorities),  actors  in  the  implementation  (managers,  operators,  agencies) 
and the  concerned  groups  (direct  or  indirect  addressees).  It  is important  to 
check  whether  the  participants  are  available  (and  motivated  enough)  to 
participate  regularly  and  actively  in the  different  meetings  of  the  group  and 
to  monitor  its  work. 

The  composition  of  the  committee  must  be  as  representative  as  possible  of 
the  stakeholders,  without  being  too  large  (a  maximum  of  15  people).  In  the 
case of  European programmes, representatives of the Directorates  General as 
well  as  national  and  local  management  and  decision-making  authorities  are 
present.  Representatives  of the addressees  of the  programme  are also  asked 
to  participate  alongside  these  institutional  representatives.  The  question  of 
whether  or  not  to  include  the  managers  and  representatives  of  the  operators 
in this  body  is often  posed  in terms of the  neutrality  and  independence  of  the 
evaluation.  It  seems  that  the  advantages  of  including  them  outweigh  the 
drawbacks,  despite  the  substantial  risk  of  these  people  assuming  too  much 
power  because they are inevitably more motivated (it is their raison d'être)  and 
more  technically  competent  than  the  other  members.  A  numeric  balance 
between  the  three  categories  of  actor  provides  guarantees  in this  respect.  It 
is furthermore  possible to take the  precaution  of appointing  as chairperson  of 
the  committee  a  political  authority  which  has  no  direct  supervisory 
responsibility,  or  a  renowned  personality  (the  latter  solution  is  the  most 

Guaranteeing the success  of an  evaluation 

frequent  one  in  situations  of  multiple  partnership  because  it  avoids  all 
suspicion  of bias). 

Box  49 -  Sociogram  of  people  concerned  by an  evaluation 

Political 
authorities 

.,—"Funding 
authorities 

Actors in the 
implementation 

Direct  addressees 
Indirect  addressees 

Concerned  V  <w'™ers and losers) 
groups 

Enlarged  steering 
group, 
¡ Evaluation  committee  ! 

Example: Classification  of the  stakeholders  of  a rural  development  programme 

representatives, 

Some  actors  may  belong  to  more  than  one  of  the  three  categories  defined  above. 
for  example,  are  also  decision-makers, 
Regional  elected 
spokespersons  for  the  concerned  groups,  and  themselves  beneficiaries  of  the 
funding.  There  are  also  actors  which  are  both  operators  and  representatives  of  the 
beneficiary  group,  such  as  Chambers  of  Agriculture.  These  organisations  are 
interlocutors 
farmers' 
associations.  Local  administrations  are  similarly  part  of  the  three  categories: 
decision-makers,  operators  and  concerned  groups. 

the  public  authorities  and  have  strong 

links  with 

for 

In relation  to  the  diagram  in  Box  49,  it  is  interesting  to  pause  on  the  notion 
of  partnership,  a term  commonly  used  in  European  vocabulary  for  the  past 
decade.  In the context  of the administration  of  Structural  Funds, the idea of 
partners, very close to that  of  stakeholders,  has a fairly specific  meaning. It 
denotes  structures  (partners)  which  can 
intervene  in  the 
definition  and application  of  programmes  financed  by the Structural  Funds. 
The  distinction  made  above,  between  vertical  and  horizontal  partners,  is 
important  in so far  as only the former  can  legitimately  define the  evaluation 
mandate.  However,  they  are  all  "stakeholders"  in  the  programme  and  are 
likely to  be involved in the evaluation procedure, according to the  modalities 
defined. 

legitimately 

Evaluation design  and  management 

9  Organisation  of  collective  reflection 

When  the  evaluation  setting  consists  of  an  extended  body,  its  work  is  the 
same  as that  of  a steering  group,  i.e.  to  check  whether  essential  questions 
have  been  forgotten,  select  priority  questions,  finalise  the  writing  up  of 
questions,  specify  the  judgement  criteria  and  norms,  validate  the  terms  of 
reference,  react  to  intermediate  reports  by  the  evaluation  team,  comment 
on  conclusions,  validate  the  recommendations  of  the  evaluation  team, and 
issue  its  own  recommendations. 

Thus, the  difference  between a steering  group and an evaluation  committee 
is  not  so  much  in  the  nature  of  the  work  done  as  in  the  way  it  is  done. 
Whereas  a steering  group  is composed  of a few  administrative officials  who 
are  used  to  working  together,  an  evaluation  committee  is  more  open,  has 
more  members  and  has  to  hold  highly  organised  meetings  to  produce 
effective  work. 

The  following  setting  is  generally  chosen:  the  commissioner  appoints  the 
chairman  of  the  committee,  within  the  framework  of  her/his  mandate,  as 
well  as a  mediator.  Under  the  responsibility  of  the  chairman  who  chairs  the 
debates,  the  mediator  impartially  supervises  the  preparation,  organisation 
and  minutes  of  the  meetings.  Both  chairman  and  mediator  must  be 
sufficiently  neutral  to  be  accepted  by  all  the  stakeholders.  The  mediator 
must,  moreover,  have  sound  technical  knowledge  of  evaluation.  He  or  she 
may  belong  to  an  internal  evaluation  unit  or  be  recruited  from  outside 
experts  on the  basis  of  a few  days'  work. 

The  mediator  prepares  work  sessions,  if  necessary  by  holding  individual 
interviews  with  all  or  some  of  the  members  of  the  committee.  He  or  she 
organises  the discussion  in such a way  that  it  is productive  and  conclusive, 
and  notes  elements  of  consensus  in the  minutes.  The  work  of  a  pluralistic 
evaluation  committee  includes  phases  of  exploration  during  which  the 
expression  of  differences  of  opinion  is  encouraged  (as  regards  evaluative 
questions,  judgement  criteria,  recommendations,  etc.).  These  exploratory 
phases  have  a  maieutic  dimension,  for  the  mediator  tries  to  elicit  the 
expression  of  all  points  of  view,  without  censuring  minority  opinions. 

The phases of creativity  are followed  by phases  of  searching  for  consensus 
which,  in  a  spirit  of  impartiality,  lead  to  the  production  of  collective  work 
(finalisation  of  the  terms  of  reference,  validation  of  the  conclusions, 
production  of  recommendations).  Volume  3  of  this  Collection,  devoted  to 
techniques 
tools,  provides  two  examples  of  group  work 
evaluation 

Guaranteeing the success  of an  evaluation 

(METAPLAN  and  the  coloured  vote)  used  to  conduct  these  two  phases  of 
work  rapidly,  impartially  and  consensually. 

Must  an  evaluation  be  totally  consensual?  This  question  becomes 
important  when  the  process  is  extended  to  numerous  partners  whose 
points  of  view  differ.  Most  evaluations  try  to  produce  a  list  of  conclusions 
accepted  by all the  partners,  but  is this  not  simply  overlooking  the fact  that 
the  plurality  of  points  of  view  and  opinions  is a  basic  democratic  principle? 

In  several  recent  experiments  the  evaluation  process  was  designed  to 
accept  and  even  explain  the  differences  in  the  partners'  opinions.  One  of 
these  experiments,  run  as  part  of  the  MEANS  programme,  is  described  in 
Volume  4  of  this  Collection,  which  presents  several  technical  innovations. 
This  experiment  was  based  on  the  principle  of  multicriteria-multijudge 
analysis. As its  name suggests, the technique  recognises that the  members 
of  an  evaluation  committee  may  not  agree  on  the  same  priorities  for  the 
same  criteria,  and  allows  several  evaluation  criteria  to  be  taken 
into 
consideration.  For  example,  certain  partners  prefer  a measure  which  has  a 
favourable  short-term employment  impact, whereas others will opt for  long-
term  growth  of  competitiveness.  The  two  criteria  are,  moreover,  not 
necessarily  contradictory.  By  means  of  confidential  individual  interviews 
with  several  partners, the evaluation team  clarified  its choices  of  evaluation 
criteria.  Despite  significant  disagreement  on  the  weight  to  give  to  the 
different  criteria,  the  partners  classified  the  actions  of  the  programme  in 
very  much  the  same  way. 

By taking  the  risk  of  accepting  divergent  conclusions,  the  evaluation  made 
it  possible to  reveal  a partial  but  real area of  consensus  on certain  effective 
or  ineffective  actions.  This  outcome  seems  preferable  to  that  of  an 
evaluation  which  tries  to  reach  a  totally  consensual  conclusion,  but  one 
which  may  well  appear  artificial. 

Consolidating evaluation techniques 

Today,  the  range  of  classical  evaluation  tools  remains  ill-suited  to  the 
evaluation of socio-economic  programmes. The reasons for these  technical 
difficulties  are  rooted  in  the  specific  characteristics  of  the  programmes: 
entangled  objectives,  multiplicity  of 
interventions,  macro-economic 
dimension  of  the  main  expected  impacts,  and  expectations  of  several  co-
decision-makers  vis-à-vis  the  evaluation. 

This  collection  recapitulates  and  presents  a  series  of  short-  and  medium-
term technical  advances  which  should  make  it possible to adjust  evaluation 
methods  more  satisfactorily  to  suit  the  context  of  European  cohesion 
policy.  The  main  progress  made  involves  the  collection  of  more  reliable 
data,  a higher  level  of  impact  analysis  and  a  greater  capacity  in  evaluation 
teams  to  reach  conclusions. 

M a n a g i ng  c a s es  of  a m b i g u o us  or  e n t a n g l ed  objectives 

Socio-economic  programmes  generally  have  multiple  objectives.  At  the  ex 
ante  evaluation  stage,  but  sometimes  at  the  following  stages  as  well,  the 
evaluation  team  notes  ambiguities  or  inconsistencies  in  these  multiple 
objectives.  Should  this  pretext  be  taken  to  declare  the  programme 
unevaluable  and  to  stop  the  work  at  this  stage  with  a  critical  conclusion? 
Opting  for  the  simple  way  out  is  too  often  the  solution  adopted  by 
evaluation  teams, with  the  commissioners'  permission. 

If  the  evaluation  team  is  to  surpass  this  limit,  it  has  to  be  able  to  judge 
successes  and failures  by means of evaluation criteria which  go further  than 
the objectives,  when  these  are  not  defined  clearly  enough. A  good  solution 
consists  of  putting  the  following  question  to  the  managers  and  operators: 
"what  impacts  (positive  /  negative)  do  you  think  your  programme  will 
produce?"  The  answers  to  this  question  form  a  list  of  "expected  impacts 
which can  be used to  reconstruct  the logic"  of the  programme with the  best 
chance  of  it  being  évaluable.  This  approach  has  often  been  used  with 
success  outside the domain of  Structural  Funds.  In pilot  studies carried  out 
within  the  MEANS  programme,  a  regional  development  programme  was 
tested  in  an  evaluation  context.  This  experiment  is  presented  in  Volume  4 
of  the  Collection.  It  uses  the  impact  mapping  technique  which  makes  it 
possible  to  effectively  associate  several  dozen  managers  and  operators  in 
efforts  to  clarify  the  programme  logic. 

Evaluation design and management 

The experiment concerned the economic development programme co-
funded and co-decided by the European Commission, the Spanish 
government and the Valencia region. The evaluation started with a series of 
meetings during which the partners identified expected impacts and sorted 
them into groups (e.g. "diversification and efficiency of energy resources"). 
Each group of impacts could thus be defined precisely enough to construct 
effectiveness criteria and choose impact indicators (e.g. "number of firms 
which use natural gas"). 

The multi-sectoral and multi-objectives character of the programmes poses 
another evaluation problem - that of synergy. The measures or actions of a 
programme are generally evaluated as if they were independent from one 
another, without taking into account the fact that the impacts can reinforce 
one another (effect of synergy) or contradict one another (effect of negative 
synergy or "anti-synergy"). Synergy is rarely evaluated at the level of results 
and impacts for addressees. At best, factors of synergy are evaluated as 
the administrative coordination, the combination or the superimposition of 
interventions. 

A technical proposal was made within the framework of the MEANS 
programme. It consisted of the construction of a matrix of cross impacts, 
the search for the strongest potential synergy, and the verification of these 
hypotheses of synergy by means of a survey among addressees. This 
approach is presented in Volume 4 of this Collection. It is designed to be 
conducted without substantially increasing the volume of evaluation-related 
work. 

■ A n a l y s i ng i m p a c ts b e t t er 

• Enhancing the reliability and relevance of data 

Many evaluation teams have complained about the insufficiency of 
monitoring data, in respect of both quantity and quality. Progress can 
certainly be made in terms of consolidation of monitoring systems and 
particularly in terms of the choice and quantification of indicators. In an ex 
ante or mid-term exercise, the evaluation team always plays an important 
part by assessing the quality of the programme indicators and, if necessary, 
recommending ways of improving it. Volume 2 of the Collection considers 
problems of quantification in detail and proposes a method for assessing 
the quality of indicators. 

Guaranteeing  the success  of an  evaluation 

Considerable  efforts  have  been  made  on  a  European  scale, in the  sense  of 
regionalising statistics with a view to facilitating the quantification  of  context 
indicators. Given the size of the task  and the  investments  required, it  would 
probably  warrant  the  continuation  of  these  efforts  in  the  next  few  years. 
Particularly  promising  perspectives  are  opening  with  the  generalisation  of 
geographical  information  systems. 

Progress  could  also  be  made  in the  sense  of  a better  understanding  of  the 
roles which could  be played  by monitoring and evaluation  in data collection. 
Monitoring data are collected  nominatively, systematically,  exhaustively  and 
continuously. These  data  are expected to  be completely  reliable as  regards 
inputs  mobilised  and  programme  outputs.  It is more difficult  to  collect  such 
information  when  it  concerns  results  and, above  all, impacts.  To  make  the 
process  less  cumbersome,  obtain  more  reliable  information  and  avoid 
giving the  impression  of  bureaucracy,  it is often  preferable to  collect  impact 
data  by  means  of  selective  and  anonymous  surveys,  limited  to  small 
samples.  This  data  collection  technique  is  the  most  suitable  for  in-depth 
analyses. There is therefore a fine balance that  has to  be found  between  the 
improvement  of  monitoring  systems  and  the  development  of  in-depth 
evaluations. 

Improving  the  analysis  of  macro-economic  impacts 

The  analysis  of  macro-economic 
impact  mechanisms  can  also  be 
improved.  This  type  of  analysis  is  a  necessity  for  the  evaluation  of  large 
programmes  of  which  the  main  objectives  are  expressed  on  a  regional  or 
national  scale.  It  is  also  a  major  difficulty  because,  despite  a  decade  of 
efforts, evaluation techniques  in this field  have  not yet  reached an  adequate 
level  of  development.  Research  and  experiments  should  be  continued,  to 
schematise and model supply-side effects  (see 144) which  play an  essential 
role  in  the  production  of  sustainable  impacts  in  terms  of  development. 
These  effects  are  still  difficult  to  take  into  account  and  therefore  evaluate, 
but  promising  work  on the subject  is underway  and should, hopefully,  in the 
medium  term, result  in techniques  which  can  be  generalised. 

Developing  thematic  evaluations 

Evaluation  tends  to  favour  the  analysis  of  those  impacts  which  correspond 
to  the  objectives  of  the  programme  axes  and  measures.  It  must  not, 
however,  neglect the expectations of the political authorities at higher  levels. 

Evaluation design and management 

At the European level a number of common priorities need to be taken into 
account, e.g. the environment, employment and equal opportunities. It is 
also expected, at the European level, that the success obtained by certain 
programmes in a particular domain of intervention (e.g. transport 
infrastructure, occupational training, R&D) will be evaluated in depth, for the 
purpose of reproducibility. 

Improvements can be made in the mainstreaming of these priorities, far too 
often neglected in evaluation reports. Thematic evaluation may be 
conceived either at the programme level, in which case it synthesises all the 
possible impacts of the different programme measures in relation to a given 
theme. Otherwise the impacts of similar interventions can be analysed and 
a thematic evaluation performed in a region or country, or even several 
countries. Volume 5 of the Collection proposes technical solutions for the 
performance of this type of evaluation. 

■ H i g h l i g h t i ng o p e r a t i o n al c o n c l u s i o ns 

• Comparing performance 

Evaluation teams currently lack benchmarks for judging whether an 
objective is realistic or excessively ambitious (case of an ex ante 
evaluation), or whether a cost per job created is reasonable or excessively 
high (case of ex post evaluation). The result is evaluation reports without 
real conclusions, or with conclusions which focus on details of 
implementation and overlook questions of results and impacts. 

Progress which can still be made in the régionalisation of statistics and the 
harmonisation of indicators gives reason to hope that, in the long run, the 
heterogeneity of data used for programme evaluation will be reduced. This 
will afford interesting, albeit very new, prospects for multiplying comparisons 
between programmes and for constructing benchmarks. The advantage of 
such prospects certainly warrants continued efforts at harmonising 
indicators, particularly in relation to the results and impacts obtained. 

In the shorter term, it is also possible to progress through inter-regional 
cooperation involving two or more European regional partners. With this 
cooperation the various tools used can be compared, the actions can be 
assessed in terms of common criteria, the respective effectiveness of the 
two regions can be compared, and the evaluation can be finalised with clear 
and acceptable conclusions. This exercise, similar to the benchmarking 

Guaranteeing the success  of an  evaluation 

technique,  was  tested  in  an  interesting  way  by  the  Emilie-Romagne  and 
Pays-de-Loire  regions,  in the  domain  of  rural tourism. 

Comparisons  could  also  be made  more  frequently  within  each  programme. 
They  would  allow  each  evaluation to  conclude  more clearly  on the fact  that 
certain  measures are successful  and others  not. Volume 4 of this  Collection 
makes  several  proposals  in this  respect. 

The  first  proposal  is  that  scoring  techniques  should  be  used  more  often. 
These  techniques  are  used  extensively  in the  U.K.  for  selecting  projects  to 
be financed  by the  European Structural  Funds. Thus, scoring  is not  really an 
innovation.  Moreover, 
in  numerous 
evaluations,  including  the  evaluation  of  programmes  co-financed  by  the 
Structural  Funds. Scoring  is used to formalise, in a comparable  manner,  the 
judgement  made  of  several  measures within  a single  programme.  Provided 
it  is  based  on  explicit  criteria  and  is  impartial,  this  technique  helps  the 
evaluation  team  to  produce  real  conclusions. 

it  has  been  successfully  used 

The  second  proposal  is  that  multicriteria  analysis  be  used  to  compare 
measures  or  actions  of  the  evaluated  programme.  This  proposal  is  more 
innovative  because  multicriteria  analysis  is  generally  used  to  evaluate 
variants  of  a  project.  With  this  technique  progress  can  be  made  in  the 
overall  evaluation  of  multi-objectives  programmes.  The  approach  was 
tested  in  the  MEANS  programme  (see  Volume  4  of  the  Collection)  and  its 
use could  be further  developed. 

Considering  the  use  of  evaluation  as  a  priority 

Use of evaluations is certainly the point which could  be improved most. Since 
the  probability  of  an  evaluation  being  used  is  linked  to  its  appropriation  by 
users,  considerable  progress  could  certainly  be  made  by  further  involving 
managers  in  the  evaluation  process  and  by  opening  it  up  to  the  outside, 
towards  potential  new  users, as proposed  above  (see page 191). 

It  is  furthermore  possible  to  envisage  the  integration  of  evaluation  into  a 
more  global  perspective  of  programme  performance  management.  The 
striving for better  performance  is a legitimate concern, both at the  European 
Union level and at that  of a Member  State or within a programme. This  point 
is presented  and  developed  in Volume  2  of  the  Collection. 

Considerable  progress could probably  also be made, in the short term, from 
the  point  of  view  of  the  use  of  evaluation  as  a  tool  for  accounting  to 

Evaluation design and management 

democratic  institutions  and  citizens.  Like  the  cohesion  reports  published 
regularly  by  the  European  Commission,  each  programme  is  likely  to  use 
evaluation  work  for  recalling  which  problems  have  been  treated  and  for 
showing  the  programme's  contribution  to  their  resolution.  It  would  certainly 
be  worth  extending  this  democratic  evaluation  culture,  currently  limited  to 
northern  Europe.  In this  way  it  would  be  easier to  meet  the  expectations  of 
citizens  and 
the  public 
taxpayers,  whose  demands  with  regard 
authorities  are  constantly  growing. 

to 

Annexes 

Annex 1 
Process of ex ante, mid-term and ex post evaluation 

The tables  in this annex  make it possible to organise the evaluation  process 
from  A  to  Ζ  by  scanning  all  the  stages  described  in  this  Volume.  Three 
evaluation  situations  are  presented  in  parallel:  ex  ante,  mid-term  and  ex 
post  evaluation. 

The considerations  presented  in this annex are of a general  nature and  must 
be  completed  by  referring  to  the  notes  and  guides  disseminated  by  the 
national  and  European  authorities,  when  such  documents  exist. 

Evaluation design and management 

Deciding  on, designing  and launching  an  evaluation 

ex  ante 
< 

Page  65 

Page  66 

mid-term 
M 

Page 65 

Page 66 

ex  post 
► 

Page 65 

Page 66 

Deciding  on the  evaluation 

Defining the  scope 

Specifying  the  motives  for 

the evaluation  and the 

expected use 

e.g.  : improving  the 

e.g. 

: proposing 

e.g. 

validating 

relevance and coherence  of 

reallocation  of  funds 

best  practice 

the  programme 

Planning the  participation of 

Page 67 

Page 67 

Page 67 

the  main partners in a 

steering  grouo 

Including  managers of 

Including the spokes­

measures and  operators 

persons of the  concerned 

(Page  191 ) 

groups (Page  192) 

Formalising  the  decision to 

Page  184 

Page 184 

Page  184 

evaluate  in a mandate  and/or 

a pluri-annual  evaluation 

programme 

Drawing  up the terms of 

reference 

Particularly  useful  for 

not "forgetting" ex  post 

evaluation 

Asking the  partners  to 

Page 69 

Page 69 

Page 69 

express their  expectations: 

selecting  evaluative  questions 

Primarily  : 

Primarily  : coherence, 

Primarily  : effectiveness 

and judgement  criteria; 

rationale,  relevance 

effectiveness  and  efficiency 

and efficiency  of 

specifying  questions  to 

consider  in more  detail 

and coherence 

of the  first  results 

the  results  and  impacts 

Possibly  :  potential 

Possibly  : 

Possibly  : 

effectiveness,  particularly 

relevance, given the 

evolution of the 

macro-economic 

evolution of the  context 

concerned  groups 

Recalling the  regulatory 

Page 76 

Page 76 

Page 76 

framework  and  describing 

the  programme 

Programme  proposal 

Including  amendments 

decided  during 

the  implementation 

Listing available  knowledge 

Page 76 

Page 76 

Page 76 

Including  monitoring  data 

Including  monitoring  data 

and ex ante  evaluation 

and  mid-term  evaluation 

Checking the  evaluability  of 

Pages 72 and  77 

Pages 72 and 77 

Pages 72 and 77 

the  questions and  defining 

the main  lines of the  method 

Setting the  'ules of  conduct. 

Page 78 

Page 78 

Page 78 

the schedule  and the 

suggested  budget 

Annexes 

Deciding  on, designing  and  launching  an evaluation  (contd.) 

ex  ante 

m i d - t e rm 
M 

ex  post 

Launching the  evaluation 

Defining the  required  skills 

Page 82 

Page 82 

Page 82 

for  the evaluation team, 

indicating the structure of the 

applications to make, having 

the terms of reference 

validated by the steering 

group, putting out the call 

(or tenders 

Selecting the evaluation  team 

Planning the  evaluation 

team's work  with  particular 

concern (or the quality of the 

final  report 

Page 82 

Page  169 

Page 82 

Page  169 

Page 82 

Page  169 

Evaluation design and management 

Performing  an  evaluation 

ex  ante 
4 

mid-term 
M 

ex  post 
► 

Examining the  logic  of the  programme 

Analysing  the strategy  and 

Page 89 

Page 89 

Page 89 

assessing  its  relevance: 

analysing  the objectives  and 

The assessment of 

Examining whether the 

Reconstructing  the 

assessing their  clarity; 

the relevance  and  clarity 

strategy  remained  relevant 

objectives  which 

examining  the  internal  and 

of the objectives  and 

in view of  a changing 

were  followed 

external  coherence of  the 

of their  coherence  is 

context;  showing  how  the 

de facto 

objectives 

particularly  necessary 

managers and  operators 

understood  the objectives; 

making  proposals for 

clarification  if  necessary 

Examining  the coherence 

Page 92 

Page 92 

Page 92 

between the objectives  and 

modalities of  implementation: 

budgetary  allocations;  modes 

of  intervention;  selection 

criteria  for  projects;  division 

of  responsibilities. 

This examination  is 

especially  necessary 

Identifying  the results  and 

Page  100 

Page  100 

Page  100 

expected  impacts 

Examining the  quality  of  the 

Page  100 

monitoring  system, as well as 

Drawing  up a logical 

Drawing up a logical 

diagram  of  impacts 

diagram  of  impacts 

(Page 95) 

Page  100 

(Page 95) 

Page  100 

that of  the system of 

This examination  is  limited 

The examination  of the 

indicators 

to the  known  monitoring 

existing system of 

principles. The  examination 

indicators  is especially 

of  proposed  indicators  is 

necessary 

especially  necessary. 

Annexes 

Performing an evaluation  (contd.) 

ex  ante 
< 

mid-term 
M 

ex  post 
► 

Examining the  programme  effects 

Selecting and using  existing 

Page 117 

Page  117 

Page  117 

information 

Carrying out  additional 

surveys 

Primarily  : 

Including  monitoring  data 

Including  monitoring  data 

expertise, literature, 

statistics 

Page  121 

Page 121 

Page 121 

For a more  in-depth 

For a more  in-depth 

knowledge of  certain 

knowledge of  certain 

results and  impact 

results and  impacts 

mechanisms (Page 151) 

(Page  151) 

Estimating  results  and 

Page 124 

Page 124 

Page  124 

impacts 

Primarily  by extrapolation 

Also owing  to the  more 

Also owing to the  more 

from the impacts of 

in-depth analysis of 

in-depth analysis of 

similar  interventions 

certain results and impact 

certain results and impacts 

(Page 126) 

mechanisms 

(Pages 110 and  153). 

(Pages  110 and 153) 

Possibly  : 

Estimation of macro-

economic  impacts 

(Page  135) 

Possibly  : 

Estimation of  macro-

economic  impacts 

(Page  135) 

Formulating  and validating the  conclusions 

Interpreting  the results of 

Page  128 

Page 128 

Page  128 

surveys and analyses  with 

the  help of  the steering group; 

e.g.  : Judgement on the 

e.g.  : Judgement on the 

preparing an impartial 

ambition  of the objectives 

contribution  of  the  different 

judgement  on the value of 

and the  probability 

measures to the success 

the  programme 

of achieving  them 

of  the  programme 

Writing  up an evaluation 

Pages 131 and 200 

Pages  131 and  200 

Pages 131 and  200 

report clearly and carefully, 

and formulating  real 

conclusions by clearly 

answering  evaluative 

questions. 

Assessing the quality of  the 

Pages  133 and  169 

Pages  133 and  169 

Pages  133 and  169 

report  and validating  it 

Evaluation design and management 

Communicating  and  using the  results 

ex  ante 
< 

mid-term 
M 

ex  post 
► 

Communicating  conclusions and  recommendations 

Discussing the 

Page  164 

Page 164 

Page  164 

recommendations  formulated 

by the evaluation  team  and 

e.g.:  adjusting  objectives; 

e.g.  : improving  the 

e.g.  : highlighting  best 

improving  them 

clarifying  objectives; 

modalities of 

practice;  generalising 

improving the  system 

implementation; 

lessons  learned 

of  indicators. 

improving  the 

monitoring system, 

reallocating  funds 

Deciding on a communication 

Page  163 

Page 163 

Page 163 

plan as soon as the  report 

is  validated 

Organising  a seminar  to 

Page 164 

Page 164 

Page 164 

present  the  conclusions  and 

recommendations 

e.g.  : presentation to the 

e.g.  : presentation  to the 

partners  involved  in the 

authorities  responsible 

design of the  next 

for  the  programme 

programme 

Disseminating  a  publication 

Page 166 

Page  166 

Page  166 

and/or a brochure;  issuing 

a  press  communiqué 

e.g.  : brochure  presenting 

the  best  results 

Monitoring  the  taking  in 

to  account  of  conclusi 

ons 

Asking the  authority  managing 

Page 52 

Page 52 

the  programme  to make a 

clear  statement  on the 

Integrating  the  conclusions 

Integrating the  conclusions 

conclusions 

into the  programming 

into the  programme 

documents, including  the 

management,  including 

conclusions  on the 

conclusions on  the 

ambitions  of the  objectives 

reallocation  of funds, the 

system  of  indicators  and 

the  selection of  projects 

Integrating  the  results  into 

national  or European syntheses 

Pages  52 and  58 

Pages 52  and 58 

e.g.  : national report on 

e.g.  : European 

the  implementation  of 

Commission  report  on 

the  Structural  Funds 

social and  economic 

cohesion 

Annex 2 
Guide for drawing up the terms of reference 

is  intended 

to  give  methodological 

This  annex 
the 
elaboration  of  an  evaluation  project.  The  indications  are  based  on 
experience  gained  through  evaluations  performed  in  administrations,  and 
on  our  cooperation  with  the  Conseil  Scientifique  de  l'Evaluation  (CSE, 
1996). They  must  be adapted  to  each  particular  situation. 

indications 

for 

Origin  of  t he  evaluation 

The terms  of  reference  specify  the  regulatory  framework  of  the  evaluation, 
the  project  initiator,  and the  agent  responsible  for the  "political  mandate". 

Delimitation  and  description  of  the  s c o pe  of  the  evaluation 

The  terms  of  reference  describe  the  scope  of  the  evaluation  and  may  also 
define  its  limits.  For  example,  is  the  entire  programme  to  be  evaluated  or 
only  certain  aspects  of  it?  Are  the  effects  of  diverse  public  actions  to  be 
evaluated  with  regard  to  society  as  a whole  or  to  a  particular  social  group 
only? 

This description  of the  scope  covers  at  least  the following  three  themes: 

•  The objectives of the programme  and the way they have been defined, as 
well  as  the  social  problems  which  motivated  and  which  justify  public 
action. 

•  The  intended  or  expected  effects  and the target  groups. 

•  The  means  implemented  and the  procedures  used. 

If the objectives  are badly defined or obsolete, it has to be decided  whether 
the  evaluation  should  reformulate  the  implicit  objectives,  based  on  the 
different  logics  applied.  If  so,  the  evaluation  will  be  more  fruitful  when  a 
preliminary  synthesis  of  the  objectives,  even  a  limited  and  partial  one,  is 
presented  in the terms  of  reference. 

At  this  stage  it  is  advisable  to  attempt  to  "model"  the  evaluated  public 
action;  in other  words,  to  describe  the  official  "theory  of  action". What  are 
the  sequences  of  cause  and  effect  that  those  responsible  for  the  policy  in 
question  anticipated,  even  implicitly,  causing  them  to  recommend  the 

Evaluation design and management 

solutions finally opted for? For example, a particular financial incentive will 
induce a particular type of behaviour in firms, etc. 

The social change that the public authorities wish to promote should be 
described according to the grid, with three points: inputs, outputs and 
outcomes. The latter point should be broken down into immediate 
objectives of the action, on the one hand, and the aims or ultimate, ideal 
justification that is never directly attained, on the other. If necessary, the 
terms of reference specify that the evaluation team must reconstruct the 
programme logic as it was implemented in the field. 

■ Future use of evaluation results 

Defining the aims of the evaluation consists of clarifying what the initiator 
expects from it and why he or she believes it to be appropriate or 
necessary. Preliminary reflection and interaction is essential because the 
direct manager of the evaluated intervention does not always have the 
required objectivity to orientate the evaluation in the right direction. The 
main reason for which the evaluation is requested must be explained. Is the 
intention to change the strategy of the programme and if so, why? Is the 
intention to modify the monitoring procedures or project selection? Are 
resources to be re-allocated? 

In the same spirit, it is preferable for the terms of reference to anticipate the 
decision likely to be taken at the end of the evaluation. This anticipation is 
not intended to dictate the conclusions of the evaluation nor to limit the 
evaluation team's capacity to question the conditions stipulated in the terms 
of reference. The idea is to enable the team to have a better knowledge of 
the commissioners' requirements, thus avoiding misunderstandings as to 
their expectations. Whenever possible, the schedule for decisions to be 
made following the evaluation is to be specified. 

In order to enhance the future utility of the evaluation as much as possible, 
it is necessary to understand the context of the programme. The evaluation 
project must take into account key factors such as: degree of innovation of 
the action, the conflict arising from it, and the administrations' familiarity 
with the evaluation procedure. The interests and strategies of the different 
stakeholders must be recorded accurately to understand the motivations 
behind the actions and the conditions that need to be met if the evaluation 
is to be credible and its conclusions accepted by the actors. The key 
questions are: 

Annexes 

• Is the evaluated action a continuation of past policy or a break from it? 

• What are the areas of uncertainty and those for which the stakeholders 

have common references? 

• Who are the "legitimate decision-makers"? Who are the operators who 

implement the intervention? 

• Which groups could gain or lose from the action? 

The future users of the evaluation must also be carefully identified. The 
nature of relations between the actors (cooperation, competition, etc.) must 
be specified by distinguishing three levels: political authorities, operators 
and concerned groups. 

■ Evaluative questions and judgement criteria 

The terms of reference pose a small number of clear and concise questions, 
to which the evaluation team must provide answers in its conclusions. 

Whenever possible, each evaluative question integrates a judgement 
criterion, such as: 

• Is the action justified with regard to the social needs or problems to solve 

(relevance)? 

• Are there positive, negative or other unexpected consequences? 

• Could more beneficial effects be produced by adopting other 

implementation measures? (effectiveness) 

• Could more beneficial effects be produced with the same resources or 
could the same results be obtained with fewer resources? (efficiency) 

• Could better complementarity be achieved (coherence) or more synergy 

be produced within the programme, compared to related public 
interventions. 

Available knowledge and data 

The terms of reference mention what the existing sources of information are 
and whether impact explanation systems or comparison references are 
already available. It specifies the form of existing knowledge (e.g. raw data, 
information structured in a computerised data base, data already analysed). 
It also specifies who is responsible for gathering information (e.g. 
information made available to the evaluation team by a particular actor; 
information to be collected by the evaluation team itself). 

Evaluation design and management 

Methods 

Without determining the details of the method to use, the terms of reference 
must provide essential information for applicant teams to make reasonable 
methodological proposals, for example: 

• the required depth of impact analysis; 

• the importance of the credibility of conclusions, of the possibility of 

generalising results, and of the appropriation of these results by the 
actors; 

• if necessary, the sites that have already been chosen to carry out case 

studies and the reasons for these decisions; 

• the most important indicators from the commissioner's point of view.; 

They must also present the main interpretation hypotheses for use in 
causality analysis: 

• Why does such an intervention seem to be a success or failure? 

• What environmental or exogenous factors (related policies, local contexts, 
etc.) need to be taken into account for interpreting observable effects? 

S Evaluation setting 

The respective roles of the commissioner, the steering group and the 
evaluation team must be clarified. The terms of reference also specify the 
composition of the steering group and the modes of functioning 
(composition, chairperson, leader, secretary). 

Where relevant, the terms of reference specify whether the evaluation team 
is expected to make proposals to broaden the steering group, and to lead 
the meetings of this extended group. 

■ Schedule 

The terms of reference set the deadlines for the evaluation teams' various 
tasks (intermediate report on a particular question; seminars; final report), 
as well as the final deadline for the submittal of these documents. 

■ Inputs a l l o c a t ed 

A planned budget is determined by the commissioner. Where relevant, the 
amount is made known to all the applicant teams who will be judged 

Annexes 

primarily  on  the  quality  and  relevance  of  their  methodological  proposals. 
Otherwise,  the  terms  of  reference  include  detailed  methodological 
proposals  and selection  is based  primarily  on costs. 

Required  qualifications 

The terms of reference specify the knowledge and experience required from 
the  evaluation  team.  Where  relevant,  it  grades  qualifications  by  order  of 
importance. 

ì  Structure  of  the  proposal 

To  facilitate  the  selection  of  the  evaluation  team,  the  terms  of  reference 
oblige  the applicant  teams  to  organise their  proposals  in  a certain  way, for 
example: 

•  elements  of  the  evaluation  scope  and  context  which  the  candidate 

considers  essential 

•  reminder  of  the  questions  asked  and  possible  interpretation  of  these 

questions 

•  evaluation  method  proposed 

•  detailed evaluation  schedule 

•  nominative  composition  of the  proposed  team, modalities  of  distribution 
of work and coordination  within the team, and individual references of the 
team  members 

•  prices 

•  technical, legal and financial  references  of the contracting  organisation. 

Annex 3 
Bibliographie references 

Boyle,  R.,  Lemaire,  D.  (eds), Building  Evaluation  Capacity,  New  Brunswick 

Publishers,  (forthcoming). 

Commission  Européenne  (1997), Les  Fonds  structurels  en  1996.  Huitième 
rapport  annuel,  Luxembourg:  European  Communities  official 
publications  office, as well  as earlier  reports. 

Commission  Européenne 

la  cohésion 
économique  et sociale,  Luxembourg: European Communities  official 
publications  office. 

(1996),  Premier 

rapport  sur 

Commission  Européenne  (1997),  Evaluating  EU  Expenditure  Programmes. 
A  guide  for  Ex  post  and  Intermediate  Evaluation,  Luxembourg: 
European  Communities  official  publications  office. 

Conseil  Scientifique  de  l'Evaluation  (1996),  Petit  guide  de  l'évaluation  des 

politiques  publiques,  Paris:  CSE. 

Cook, TD.,  Levitón, L.C.  and  Shadish, W.R. (1985), 'Progam  Evaluation'  in 
:  Lindzey,  G.  and  Aronson,  E.  (Eds.),  Handbook  of  Social 
Psychology,  Vol. I. (3rd ed.), New York  : Random  House. 

European  Commission  (1995),  Common  Guide  for  Monitoring  and  Interim 
Evaluation  of  CSFs,  SPDs  and  operations,  European  Commission 
Structural  Funds. 

GAO  (1991),  Designing  Evaluations,  Program  Evaluation  and  Methodology 

Division, Washington. 

Guba,  E.G.,  Lincoln,  Y.S.  (1989),  Fourth  Generation  Evaluation,  Newbury 

Park: Sage  Publications. 

Owen,  M.O.,  Rogers,  P.  (1998),  Program  Evaluation:  Forms  and 

Approaches,  London: Sage  Publications. 

Monnier,  E,  (1992),  Evaluations  de  l'action  des  pouvoirs  publics,  Paris: 

CPE-Economica. 

Rieper,  0.,  Toulemonde,  J.  (Eds)  (1997),  Politics  and  Practices  of 
Evaluation,  New  Brunswick:  Transaction 

Intergovernmental 
Publishers. 

Evaluation design and management 

Rist,  R.C.,  Segsworth,  R.V.,  et  al.  (1990),  Program  evaluation  &  the 
management of government : patterns and prospects across  eight 
nations, New Brunswick  : Transaction publishers. 

Rossi,  P.,  Freeman,  H.E.  (1993,  5th  ed.),  Evaluation :  A  systematic 

Apprqach, Newbury  Park, CA  : Sage Publications. 

Vedung,  E.(1997),  Public Policy and Program Evaluation,  New  Brunswick: 

Transaction Publishers. 

Index 

Index 

accountability  17,  25 
additional  effect  112 
additionality  31 
addressee  121,  123,  192 
aims  of  evaluation  76 
appraisal  56 
attractiveness  143 
audit  62,  70 

Β 

benchmarking  130,  201 
black  box  149 
bottom-up  approach  56 

case  study  126 
coefficient  116 
cognitive  use  165 
coherence  72 
communication  plan  164 
Community  priorities  53,  58 
Community  value  added  25 
comparison  group  111,113,126,  155 
competitive  advantage  140 
concentration  98 
conceptual  use  165 
concerned  group  22 
conclusions  132,  161 
confounding  factor  119,  154 
construct  validity  114 
context  indicator  103,  119,  200 
control  group  110 
cost-benefit  analysis  57,  131 
cost-effectiveness  analysis  129 
counterfactual  situation  110,136,155 
criteria  129,  130 

D 

data  analysis  176 
data  collection  123 
deadweight  112,  113,  127 
deductive  analysis  153,  156 
demand-side  effect  146 
democratic  20,  2 1,  46 
direct  feedback  25 
displacement  effect  112 
dissemination  161 
double  loop  learning  25 

economic  base  143 
effectiveness  70,  129 
efficiency  70,  129 
employability  141 
endogenous  development  143 
equity  72 
evaluability  72 
evaluability  study  82,  187 
evaluanda  65 
evaluation  committee  2 1,  190,  192 
evaluation  criterion  70 
evaluation  method  174 
evaluation  report  131,  133 
evaluation  setting  67,  68,  194 
evaluative  question  69,  72,  73,  186 
evaluator  20 
ex  ante  evaluation  31, 36, 37, 38, 52, 
69,  70,  88,  89,  90,  92,  97,  98, 99, 
100,  102,  104,  107,  127,  167 
ex  post  evaluation  31, 53, 55, 88, 93, 
97,  104,  125,  127,  147,200 

exogenous  development  143 
exogenous  factor  154,  155 
expected  effects  197 
expected  impacts  101 
explanatory  hypotheses  73 
explanatory  theory  156,  165 

Evaluation  design and  management 

external coherence  95,  100 
external evaluation  189 
external validity  61,  115,  116,  121 
externality  102,  144 
extrapolation  coefficient  120,  127 

internal coherence  93,  100 
internal validity  115 

joint  decision-making  27 

feedback  183 
focus  group  124,  152 
formative evaluation  25,  166 

learning  24 
logical diagram of  impacts  94,  149 
logical framework  148,  154 

information 

geographical 
125, 199 
gross effect  112 

system 

M 

H 

hierarchy  of  objectives  93 
human  capital  141 
human  resources  141 

I 

impact  24, 62, 70, 87,  101, 126, 173 
impact  mapping  9,  197 
implementation  97 
in-depth  analysis  199 
independence  81,  189 
indicative  budget  78 
indicator  5,  7,  40,  64,  129,  198 
indirect  feedback  25 
individual  interview  152 
inductive  analysis  153 
infrastructure  projects  57 
input  87,  99 
instrumental  contribution  25 
intermediate evaluation  37, 55, 71, 81 
intermediate  impact  149 

135 

macro-economic  124, 
maieutic  194 
mainstreaming  9,  58 
major  project  60,  99 
managerial evaluation  20 
mandate  184 
matching  groups  155 
matrix of cross impacts  9, 96,128,198 
MEANS  5,  6,  40,  42 
measure  60 
measurement  tool  129 
mediator  130,  194 
meta-evaluation  170 
METAPLAN  130 
methodologist  187 
micro-economic  124,  135 
mid-term  evaluation  31, 38,  52,  88, 
92, 93, 95,97,  99,  100, 102, 104, 
107,  125,  127, 147 

monitoring  62,  70,  175 
monitoring  committee  38,  68 
monitoring  system  104 
multicriteria analysis  9,  130,  201 
multiplier  effect  135 

Index 

N 

needs  24,  70,  87,  89,  129 
net effect  112 
network effect  144 
normative use  166 

objective  87 
objectiveness  20 
objectives tree  94,  148 
officials  6 
operator  121, 192 
output  101 
overall evaluation  117,  123 

partnership  32,  41 
pluralistic  evaluation  21 
pluri-annual evaluation plan  54,  55 
pluri-annual programming of 

evaluation 183 

policy  31 
preliminary  study  186 
primary data  117,  121, 124,  175 
productive environment  141 
programme  30,  173,  177 
programme  indicator  103 
programme logic  87,  147 
programme manager  121 
programming cycle  87 
project  promoter  122 
protagonist  191 
public  61 

quality  169,  172 

raison d'être  24,  89 
rationale  89 
recommendations  66,  132 
reconstructed  logic  197 
regression analysis  126,  156 
relevance  70,  90,  129 
reproducibility  72 
result  24,  25,  62,  70,  87,  100,  101 

schedule  78 
scope  65,  184 
scoring  129,  201 
scoring card  9 
secondary data  117,  124,  175 
shift-share analysis  144 
single loop learning  25 
socio-economic  development 

programmes  6 

sociogram  192 
specific  impact  101 
stage  31 
stakeholder  191, 193 
stakeholder evaluation  21 
standard  129 
statistical sources  119 
steering group  68,  190 
step  31 
stigmatisation effect  154 
strategy  87,  90 
structuring effect  137,  146 
subsidiarity  24 
substitution  effect  112 
summative evaluation  53,  166 
supplier effect  135 
supply-side  effect  146,  199 
sustainability  72 
SWOT analysis  91, 130 
synergy  56,  61, 72,  127,  198 

Evaluation  design  and  management 

terms of  reference  68,  75,  76,  185 
thematic  evaluation  9,  61,  147 
theory of  action  149 
transparency  25,  35,  37,  53 
transversal  61,  132 

u 
unexpected  effect  154 
user  161 

Acknowledgements 

The preparation of this Volume took place over several years from 1993 to 1998. The 
preparatory work included the following steps: 

• Experts, the majority of them European, were brought together to participate in 
several workshops on the following themes: "Evaluating structuring effects", 
"Organising intermediate evaluations" and "Assessing the quality of evaluations". 

• These three workshops gave rise to the publication of a number of handbooks, the 

contents of which have been partially reused in this Volume. 

• Nearly three hundred European Commission civil servants participated in training 
seminars. The contents of these training courses gradually evolved according to 
the wishes of the participants. This Volume is built on the contents of these training 
courses. 

• Two reports were produced in the form of a memorandum about the performing of 

ex ante and ex post evaluations. The main points of the memorandum are 
presented in Annex 1 of this Volume. 

The production of this Volume was coordinated by Philippe Goybet and Anastassios 
Bougas (D.G. XVI.G.2 : Coordination of Evaluation) and involved several Directorate­
Generals of the European Commission, notably D.G. II, V, VI, XIV and XVI, and in 
particular Miguel Benito Alonso, Andrea Mairate, David Sweet and René­Laurent 
Ballaguy (ENA). 

This Volume has been written by the Centre for European Evaluation Expertise, 
notably with the participation of Pascale Ambrois and Céline Fontaine. It was 
translated into English by Liz Libbrecht. 

The authors wish to thank all those who contributed, either directly or indirectly, to 
the production of this Volume, and in particular the following experts: H. Aalbu 
(Nordlands Forskning), J. Bachtier (EPRC, Glasgow), A. Blalock (Admiralty Inlet 
Consulting, Seattle), R. Boyle (Institute of Public Administration, Dublin), J. Bradley 
(ESRI, Dublin), R. Brancatti (Ministerio del Bilancio, Rome), E. Calderón (Universidad 
Politechnica, Madrid), E. Chelimsky (American Evaluation Society), Hagens J. (TERR 
Amersfoot), P. Nijkamp (Vrije Universiteit, Amsterdam), Β. Perret (Conseil Scientifique 
de l'Evaluation, Paris), . O. Rieper (AKF, Copenhagen), L. Senn (Università Bocconi, 
Milan), E. Stern (Tavistock Institute, London), L.I. Ström (SIR, Östersund). 

European  Commission 

Evaluating  socio-economic  programmes 
Evaluation  design  and  management 

Luxembourg: Office for  Official  Publications  of  the  European  Communities,  1999 

1999  —  225  pp. —  14.8  χ 21  cm 

ISBN  92-828-6626-2 

Venta  ·  Salg  ·  Verkauf  ·  Πωλήσεις  ·  Sales  ·  Vente  ·  Vendita  ·  Verkoop  ·  Venda  ·  Myynti  · Försäljning 

Pavia Hatza 1 
HR­10000 Zagreb 
Tel (385­11481 94 1 
Fu (385­1) 481 94 1 

MAGYARORSZAG 

Euro Info Service 
Europa Haz 
M a r o si >get 
, fn 
'■a í 
H­1396 Budapest 62 
Tel I30­1) 350 80 2b 
Fax (36­1) 350 90 32 
E­mail eu ι οι nlo β mail ma la v hu 
URL. http //www eurointo hu/inde» htm 

ROY International 

4 1, Mehmar Hayarden Street 
PO Box 13056 
61130 Tel Aviv 
Tel (972-3) 649 94 69 
F ai (972-3) 648 60 39 
E­mail royil O netvision nei il 

URL· http/A*ww.royjnt co il 

Sub­agent for the Palestinian Authority 

Index Information Services 

PO Bo»19502 

BELGrQUE'BELGlE 

J » « n D « L i n n oy 
Avenue du Roi 202/Korungslaan 202 
B­1190 Bruielles.'Brussel 
Tel 132­2) 538 43 06 
Fai (32­2) 538 08 41 
E­mad («ende lannoyOrnroboardbe 
URL http//www iean­de­lannoy be 

La librairie européenne/ 
De E u r o p e e· Boekhandel 
Rue de la Lo. 244/WetstraaI 244 
B­1040 Bruxeees/Brussel 
Tel (32-2) 295 26 39 
Fu (32­2) 735 M 60 
E­mail rnailOkbeuropbe 
URL http I www hbouroç Do 

Moniteur belge/Belgisch Staatsblad 
Rue de Louvam 40­42 Leuvenseneg 40­42 
B 1000 Bru'eUes. Brussel 
Tèi (32­2) 552 22 11 
Fax (32­2) 511 01 84 

DANMARK 

J. H. Schultz Information A/S 
Herstedvang 10­12 
DK­2620 Albertslund 
Tit (45)43 63 23 00 
Fu (45) 43 63 19 69 
E­mail Schultz O schwitz dk 
URL http y/www Schultz dk 

DEUTSCHLAND 

Bundesenzeiger Verlag GmbH 
Vartneosabletrung 
Amsterdam«! Sunn« 192 
D 50735 Köln 
Tel ( 4 9 - 2 2 1 ) 97 66 80 
Fu ( 4 9 - 2 2 1 ) 97 66 82 78 
E­Mail veflnebObundesanzeiger de 
URL http //www bundesanzetger de 

ΕΛΛΑΔΑ/GREECE 

G. C. Eletlheroudakis SA 
International Bookslore 
Paneptstimiou 17 
GR­10564 Altana 
Tel (30­1)33141  Β0.Ί/2/3/4/5 
Fu  (30-1)  323 98 21 
E-mail  elobOoksOnetor  gr 

ESPANA 

Boletín Oficial def Estado 
Trafalgar. 27 
E­280 71 Madrid 
Tel (34)915 38 21 11 (Ubros) 

9 1 3 84 17 15(Suscnp | 
Fu (34) 915 38 21 21 iLibrosi 
9 1 3 84 1 7 1 4 ( S u a c n p) 

E­mail dientes O com boe es 
URL http //www boe es 
Mundi Prensa Libro«. SA 
Caste»), 37 
E­28001 Madrid 
Tel (34)914 36 37 00 
F u ( 3 4) 9 1 5 75 39 98 
E ma.! libreria Q munotpren sa es 
URL http //www muncfcpronsa Com 

FRANCE 

Journal officiel 
Senne« des pubbeabons des CE 
26 rue Dosati 
F­75727 Pans Cede« 15 
Tel (33) 140 58 77 31 
Fu (33) 140 58 77 00 
URL http //www journal ottioel gouv li 

IRELAND 

Government Supplies Agency 

Publications Sed ion 
4­5 Harcourt Road 
Dublin 2 
Tel (353­1)66131 11 
Fu (353­1)475 27 60 

ITALIA 

Licosa SpA 
Via Duca di Calabria 1/1 
Caseia poslaiu 552 
1­50125 Firenze 
Tel (39) 0 55 64 83 I 
Fu (39) 055 64 12 57 
E­mail kcosaOftbu: rt 
URL http //www Itone i/kctsa 

LUXEMBOURG 

Messageries du livre SAP.L 
5. rue ílartfeisen 
L­2Í11 Luxembourg 
Tel (352)40 10 20 
Fax (352) 49 06 61 
E­mail mailOmdl lu 
URL http ."www mdi lu 

NEDERLAND 

SOU Servicecentrum Ullaevers 
Cnristotlol Pianti|nstraai 2 
Postbjs 20014 
2500 E A Den Haag 
Tel (31 70)378 98 80 
Fu (31­70) 378 97 83 
Email sduØsdunJ 
URL: Http ./www sdu m 

Mani'sehe Vertag*­ und 
Unlverattatsbuchhandlung GmbH 
Korimarkt 16 
A­1014 Wien 
Tel. (43­1) 53 16 11 00 
Fu (43­1) 53 16 11 67 
E­Mail bestellen O m a ni co at 
URL: http /www man* al/index htm 

PORTUGAL ~ 

Distribuidora de U v r os Bertrand L d ." 
Grupo Bertrand, SA 
Rue das Terras dos Vales. 4­A 
Apartado 60037 
f 2700 Amadora 
Tel (351­1)495 90 50 
Fa» (351­11 496 02 55 

Imprensa Nacional­Cas* da Moeda. EP 
Rua Marqjf ? Sa da Bandeira. 16­A 
Ρ­1050 Lisboa Code« 
Tel ( 3 5 1 - 1 ) 3 5 3 03 99 
Fa« (351­1)353 02 94 
E­mail del mcmOmail telepic p; 
URL: http 'www mem pt 

SUOMLFINLANH 

Keskuskatu 1 .Con Ir alga lan ■ 
PL/PB 128 
FIN­0010­  ΗβΙ»-ηκιΉβΚιηαίοηι 
Ρ/Dn  (358-9)  121 44  18 
F/lax  (358-9)  121 44  35 
Sahkûposli akatilausOakatcemmoncom 
UFIL. http //www akateeminen com 

SVERIGE 

B T J AB 
Traktorvägen 11 
S 221 82 Lund 
T!n (46­46) 18 00 00 
Fa» (46­46) 30 79 47 
E post tjttou­putjCDij se 
URL: http//www blj se 

UNITED KINGDOM 

The Stationery Office Llri 
IMernabonal Rales Agency 
51 Nint E»ms Lane 
London SW8 50R 
Tel (44­171)973 90 90 
Fa» (44­171)873 84 63 
E­mail ipa enquiñes O Ineso co uk 
URL: hnp/'wwwthe­stationery­otricecouk 

ISLAND 

Bokabud Larusar Blonda! 
Skolavordustig 2 
IS­101 Reykiavik 
Tel (354) 551 56 50 
Fa» (354) 552 55 60 

NORGE 

Swets Norge AS 
Oslonjoveion 18 
Boks 6512 Erterstad 
NOOOeOskj 
Tel (47-22)97 45 00 
Fa» (47­22) 97 45 45 

SLHWEIZ.SU1SSESVIZ2ERA 

Euro Info Center Schweiz 
c/o OSEC 
Siamplenbachsuaße 85 
PF 492 
CH 8035 Zurich 
Tel (41-TJ365 53 15 
Fa» (41­1)365 5411 
E­mail ercsOosecch 
URL: hrtp/flivww osee ch/ecs 

B A L G A R UA 

Europress Euromedia L id 
59. Hvd Vilosria 
BO· 1000 S o ta 
Tel (359­2)980 37 66 
Fa» (359­2) 980 42 30 
E­mail MilenaOmoox cil bg 

CESKA REPUBLIKA 

USIS 
NI S­prod3|na 
HsWesstMfi ?2 
CZ­130 00 Praha3 
Tei (420­2)24 2 3 14 86 
Fax (420­2) 24 23 11 14 
E m a il nfcpospOdecnis cz 
URL: http usiscr cz 

CYPRUS 

Cyprus Chamber of Commerce and Industry 
PO Bo» 1455 
CV­1509 Nicosia 
Tel (357-2)66 95 00 
Fa» (357­2) 66 10 44 
E­mail demetrapOccn orgcy 

MALTA 

Miller Distributors Ltd 
Malia ime m abonai Airport 
PO Bo»25 
Luqa LOA 05 
Tel (356)66 44 88 
Fa» (356167 67 99 
E­mail gwirthOusa.net 

POLSKA 

Ars Polena 
KrakowskiP Prz ed míesete 7 
Skr pocztowa 1001 
PL­00­ 950 Warszawa 
Tel (­10­22)826 12 01 
Fax (40­22) 826 62 40 
E­mail ars_polObevy hsncom pi 

ROMANIA 

Sn G­tal Berthelol Nr 41 
RO­70749 Bucuresti 
Tel (40­1)31544 03 
Fax (40­1 314 22 66 

CCEC 
60­le try a Oktyabrya A v 9 
'17312 Moscow 
Tel (7­095) 135 52 27 
Fax (7­095) 135 52 27 

SLOVAKIA 

Centtum vn SU 
Nam Stobody. 19 
SK­81223 Bratislava 
Tel. (421­7)54 41 83 64 
Fa» (421­7) 54 41 83 64 
E­mail e u m p e t b bt sltk stuba sk 
URL http //www stik stuba Sk 

SLOVENIJA 

Gospodarskl Vestnlk 
Dunajska costa 5 
SLO­l000L|ubl|an.i 
Tel (366)613 09 16 40 
Fax 1386) 613 09 16 45 
E­mail europCgvestnik si 
URL http//www gvesinik.si 

TURKIYE 

Dünya Infolel AS 
100. Υ Ί Mahailossi  34440 
TR-80050  Bagcilai-Istanbul 
Tel  (90-2121629  4G 89 
Fu  (90-212)  629  46 27 
E-mail  mlotelOdun/a-gazeto  corn tr 

AUSTRALIA 

Humer  Ρ 
PO  Bo»  404 
3067 Abbots'ord.  Vtctona 
Tel  (61-3)94  17 53  61 
Fa» (61-3)  94  19 71  54 
E-mad  jpdavtesOozomailcomau 

CANADA 

Les  editions  La Liberté Inc. 
3020. chemin Sainte­Foy 
G1X 3V Samte­ Foy Quebec 
Tel (1­418)658 37 63 
Fax (1­600) 567 54 49 
E­mm irbenoOmedicmqcca 

Renout Publishing Co. Ltd 
536P Chemin Canotek Road Unit 1 
KU JJ3 Ottawa. Ontano 
Tel (1­613)745 2 6 65 
Fax (1­613)745 76 60 
E­mail order deptOreno­.fbooks com 
URL http //www renoufboofcs com 

EGYPT 

The Middle East Observer 
41 S hent Street 
Cairo 
Tel (20­2)392 69 19 
Fa» (20­2) 393 97 32 
E­ma4 mafoudaOmeobservercomeg 
URL: http://www meobserver com eg 

EESTI 

INDIA 

Eesll Kau ban d us ­Tôoet us koda (Estonian 
Chamber of Commerce and Industry) 
Toom-Kook 17 
EE­0001 Tillinn 
Tel ( 3 7 2 ) 6 46 02 44 
Fu (372) 6 46 02 45 
E­mail emloOkodaee 
URL http www koda ee 

EBIC India 
3rd Floor, Υ B Chavan Ceñiré 
Gen J Bhosaie Marg 
400 C21 Mumbai 
Tel (91­22)282 60 64 
Fa» (91­22) 285 45 64 
E­ma·! ebcOgtasbmOI vsnlnetir 
URL http //www eocindta com 

JAPAN 

PSI­Japan 

Asaht Sanbancho Plaza «206 
7­1 Sanbancho Chrvoda­ku 
Tokyo 102 
Tel. (81­3)32 34 69 21 
Fu (81­3) 32 34 69 IS 
E­mail: booksOpS'iap in.co |P 
URL: http:/Awww psi­japan com 

EBIC Malaysia 

Level 7. Wisma Hong Leong 
|| J.I .111 P| :,,. 
50450 KuaU Lumpur 
Tel 160­3) 262 62 96 
Fa» (60­3) 262 61 98 
E­mail ebtc­kltfmol net my 

Mundi Prensa Mexico. SA dt CV 

Rio Panuco No 141 
Colonia Cuauhtemoc 
MX 06500 Mexico, DF 
Tel ( 5 2 - 5 ) 5 33 56 58 
Fu (52-5) 514 67 99 
E­mail 101545 2361 Ocompuserve com 

EBIC Philippines 

19th Floor. PS Bank Tower 
Sen Gil J. Puvat Ave cor Tindalo Si 
Manali City 
Metro Manilla 
Tel (63­2) 759 66 80 
F ai (63­2) 759 66 90 
E­mail occpcomOglobo corn ph 
URL http //www eeep com 

EBIC Sri Lanka 

Tra n 5 Asw Hotel 
115 Sir chittampalam 
A G.iidifMir Mawallia 
Colombo2 
Tel (94­1)074 71 50 78 
Fu (94­1) 44 87 79 
E­rrail ebtcsJOilmincrxn 

EBIC Thalland 

29 Vanisse Building. Hin Floor 
E ■< Chrdtom 
Ploenchit 
10330 Bangkok 
Tel (66-2) 655 06 27 
Fax (66­2) 655 06 2b 
E­m.­il ebrcbfckOksc15th.com 
U R I: http/www ebicbkk.org 

UNITED STATES OF AMERICA 

Beman Associates 

4611­F Assembly Drive 
Lanham MD20706 
Tel (1­80O)"/"4 44 47 (mHrco telephone) 
Fax (1­800) 865 34 50 (toll free fax) 
E­mail queryObernancom 
URL. httpJnwww beman com 

Bine wenden Sie sich sn ein Büro Ihrer 
Wahl/ Please c o n t a ci the sale« office 
of your choice/ Veuillez vous »dresser 
■u bureau de vente de voira choix 

2. rue Mercier 
L­2965 Luxembourg 
Tel '352) 29 29­42455 
Fu (352) 29 29­42756 
F­maa inlo mfoOopoce cec.be 
URL http //eut­oo eu mt 

The  MEANS  collection: 
"Evaluating  socio-economic  programmes" 

This  six-volume  set  gives  wider  public  access  to  the  lessons  of  the  MEANS 
programme  which  the  European  Commission  introduced  with  a view  to  impro­
ving  and  promoting  evaluation  methods.  Its  publication  meets  a  growing 
need  for  reliable  benchmarks  to  deal  with  the  problems  of  evaluation  encoun­
tered  when  implementing  public  measures  in general and  Structural  Fund  ope­
rations  in  particular. 

In  this  context,  the  set  constitutes  a  unique  methodological  guide  that  can 
solve  the  technical  and  organisational  problems  arising  in  the  course  of 
evaluation  work  and  for  which  the  existing  literature  provides  little  help. 

The  collection  is  aimed  not  only  at  professional  evaluators,  for  whom  it  will 
provide  a valuable  handbook,  but  also  at  non-specialists,  in particular  manag­
ing  authorities  and  structural  programme  Monitoring  Committees. 

Evaluation  design  and  management 

Each  volume  is  designed  to  be  independent  of  the  others,  yet  they  are  also 
complementary.  Together,  they  deal with  all aspects  of  the evaluation  of  socio­
economic  programmes: 
I. 
II.  Selection  and  use  of  indicators  for  monitoring  and  evaluation 
III.  Principal  evaluation  techniques  and  tools 
IV.  Technical  solutions  for  evaluation  within  a  partnership  framework 
V 

Transversal  evaluation  of  impacts  on  the  environment,  employment  and 
other  intervention  priorities 

VI.  Glossary  of  300  concepts  and  technical  terms 

OFFICE  FOR  OFFICIAL  PUBLICATIONS 

OF THE  EUROPEAN  COMMUNITIES 

L-2985  Luxembourg 

EC Structural  Funds 

Evaluating socio-economic programmes 

Selection  and  use  of  indicat 

ors 
iorn 
jation 

m 

MEANS 
Collection 

Volume 2 

EC Structural Funds 

Evaluating socio-economic  programmes 

Selection  and  use  of  indicators 
for  monitoring  and  evaluation 

Volume 2 

European Commission 

MEANS  is a programme  of  the  European  Commission. 
It  is aimed  at  improving  Methods  of  Evaluating  Structural  Policies. 
The MEANS  Programme  and  Collection  was  supervised  by 
Phillipe  Goybet  and  Miguel  Angel  Benito  Alonso, 
Unit  for  the  Coordination  of  Evaluation  in  the  Directorate  for  Coordination 
and  Evaluation  of  Operations  of  DG XVI 

Coordination-evaluation@fmb.dg16.cec.be 

The realisation  of  the  Programme  was  entrusted  to  the 
Centre  for  European  Evaluation  Expertise  (C3E) 
under  the  direction  of  Eric  Monnier  and  Jacques  Toulemonde. 

Responsibility 

for  the  opinions  expressed  in  this 

book  lies  solely  with  the  authors  of  the  text: 
C3E  -  13 bis  Place  Jules  Ferry  -  F  -  69006  Lyon  -  France  -  c3e@c3e.fr 

A  great deal of  additional information on  the  European Union is available 
on the Internet. 
It can  be accessed through the Europa server (http://europa.eu.int)  and 
on the  Inforegio Website (http://inforegio.cec.eu.int). 

Cataloguing  data can be found at  the end of this  publication. 

Luxembourg:  Office for Official Publications of the European 
Communities,  1999 

ISBN 92-828-6626-2 

©  European Communities, 1999 
Reproduction  is authorized,  except for  commercial purposes,  provided 
the  source is  acknowledged. 

PRINTED  ON WHITE CHLORINE-FREE PAPER 

Printed  in Italy 

Foreword 

Developing  good  evaluation  practices 
has  become  a  priority  area  for 
the 
relevant 
Commission  and 
the  effective  use  of  Community 
to 

is  directly 

t resources. 

Indeed, the  sound  financial  management 
principles  which  the  Commission  seeks 
to  promote  via  its  SEM2000  (Sound  and 
Efficient  Management)  Programme  rely, 
specifically,  on  the  regular  evaluation  of 
the  Community  activities. 
The  Directorates  General  responsible  for 
structural  policy  have considerable  expe­
rience  in this area.  Since  1988 they  have 
made  evaluation  an 
in  the  process  of 
integral  part  of  each  stage 
Community  Structural  intervention.  Thus,  the  programme  development 
stage  includes  a  prospective  evaluation;  the  implementation  stage  a 
strengthening  of the monitoring systems and, at the end of the intervention, 
evaluation  establishes  what  has  been  achieved  in  terms  of  programme 
impacts. 
However,  the  capacity  of  evaluation  to  contribute  to  the  effectiveness  and 
transparency  of  structural  interventions  rests  ultimately  on  the  integrity  of 
the  methods  adopted  to  verify  the  credibility  of  results. 
This  is why,  four  years  ago,  on the  initiative  for  the  Directorate  General  for 
Regional  Policy  and  Cohesion  (DG  XVI),  the  MEANS  programme  was 
initiated  as a response  to  the demand  for  developing  evaluation  methodo­
logy.  This  also 
include  practices  combining  both  general  scientific 
requirements  and  the  specific  requirements  of  Community  interventions 
whilst,  at the  same time,  remaining  accessible  to  a large  audience. 
One  of  the  more  tangible  results  of  the  MEANS  programme  has  been  the 
publication  of this  six volume  set  by  a group of  independent  experts,  focu­
sing  on  the  principal  elements  of  socio-economic  programme  evaluation. 
A  short  summary  of this  work  can  be found  at the  beginning  of the  present 
volume. This collection covers the various  methodological  approaches  and 
innovations  that  have  taken  place  in  the  context  of  structural  funds  eva­
luation.  It  will  enable  those  responsible  for  evaluation  to  develop  their 
approach  in  a  flexible  and  independent  manner  and  for  the  various 

programme  partners  to  exercise  greater  control  over  the  evaluation  pro­
cess.  Attending  the  conferences  of  evaluation  specialists  and  "clients" 
convened  in  Brussels  (1995),  Berlin  (1996)  and  Seville  (1998),  I  gained 
some  impression  of  the  interest  that  new  evaluation  techniques  can  gene­
rate.  I would  stress  that  the  development  of  evaluation  needs  to  progress 
in  tandem  with  efforts  to  incorporate  evaluation  findings  in  the  decision 
making  process.  This  implies  evaluation  as  a  means  of  establishing  whe­
ther  policies  are  equitable,  effective  and  relevant  to  the  identified  needs. 
In  satisfying  these  requirements,  evaluation  should  be  seen  as  a  valuable 
instrument  affording  both  Member  State  and  Community  authorities  the 
necessary  guarantees  that  Community  funds  are  being  deployed  in  the 
best  interests  of the  EU citizen. This  is a cost  worth  paying  when  the  goals 
of  Economic  and  Social  Cohesion  and  the  European  Union  enlargement 
are  at  stake. 

Finally,  I  wish  to  extend  my  thanks  to  the  members  of  the  Committee  of 
Independent  Experts  established  at  the  launch  of  the  programme.  Their 
expertise  and  standing  have  contributed  greatly  to  the  quality  and  relevan­
ce  of  the  MEANS  programme.  I therefore  thank:  E.  Chelimsky  (USA).  ex-
President  of  the  "American  Evaluation  Association";  K.  Kennedy  (IRL), 
Director  of  the  "Economic  and  Social  Research  Institute"  (ESRI);  J.  R. 
Cuadrado  Roura  (SP),  President  of  the  "European  Regional  Science 
Association"; C. Seibel  (F), Director of the  "Statistiques  démographiques  et 
sociales  de l'Institut  National de la Statistique et des  Etudes  Economiques" 
(INSEE);  L.  Senn  (I),  Director  of  the  Department  of  Economie  Regional  of 
the  University  Bocconi;  E.  Stern  (U.K.),  Director  of  the  unit  of  evaluation  of 
the  Tavistock  Institute;  A.  Sorber  (t)  (NL),  Head  of  the  "Policy  Analysis 
Department  of  the  Ministry  of  Finance";  L.  I. Strom  (SV), Statens  Institut  för 
Regionalforsking  (SIR);  L.  Tsoukalis  (GR.),  professor  at  the  University  of 
Athens;  H.  Wollmann  (D),  professor  at  the  University  of  Humboldt  -  Berlin 
and  founder  member  of  the  "European  Evaluation  Society". 

E.  Landaburu 

The  MEANS Collection: 

"Evaluating socio-economic programmes" 

of  European  Union 

The  evaluation 
structural 
interventions  has  entered  into  a  phase  of  maturity, 
attested by the extension of work carried out by both the 
Member States and the Commission. Regulations  based 
on  decisions  by  the  Council  have  largely  contributed  to 
the  development  of  evaluation  practice  and  have 
stimulated  methodological  research. 

initiated  'by 

Over  the  past  ten  years, the  Commission  has  striven  to 
promote  and  improve  methods  for  evaluating  Structural 
Funds. The main  methodological  conclusions  have  been 
debated  and  disseminated  as  part  of  the  MEANS 
in  1991. 
programme 
Achievements  to  date  provide  enough  material  for  a 
complete  and  accessible  publication  intended  for  a 
broad public: a six volume Collection covering all aspects 
of the evaluation  of socio-economic  programmes. These 
Volumes  are  written  with  a  perspective  that  is  broader 
than  the  current  administrative  framework,  for  this  is 
likely to evolve more rapidly than  professional  standards. 

the  Commission 

information  on 

In general, the Collection is aimed at a readership of non-
specialists  seeking  essential 
the 
evaluation of socio-economic development  programmes. 
These  programmes  have  multiple  objectives,  a 
partnership  dimension  and  a  multi-sectoral  content  -  all 
characteristics which call for suitable evaluation  methods 
and  solutions  not  yet  provided  by  the 
international 
technical literature. The complete Collection consists of 6 
volumes  designed  to  be  complementary  and  to  provide 
essential 
for 
professionals  wanting  to  independently  develop  their 
own evaluation  practices. Each volume can, however,  be 
read and  understood without  having to  refer to the  other 
volumes  in the Collection. 

recommendations 

information  and 

Indicators  for monitoring  and evaluation 

The  first  Volume  in  the  MEANS  Collection  entitled 
"Evaluation  design  and  management"  serves  as  an 
introduction  both  to  the  Collection  and  to  the  evaluation 
of  socio-economic  programmes.  It  enables  readers  to 
understand  the  implications  of  evaluation  and  to  know 
what they  can and cannot  expect  from  an evaluation  at a 
given  time  in  a  particular  context.  It  covers  all  the 
elements  needed to  steer  an evaluation  project from A to 
Z:  sharing  of  responsibilities,  defining  of  the  aims  and 
method  of  an  evaluation,  establishing  a  schedule  and 
budget, assessing the quality  of the work  performed and, 
finally,  managing  the  use of  conclusions. 

The second Volume focuses on the  "Selection  and use of 
indicators  for  monitoring  and  evaluation".  It  deals  with 
the  subject  of 
indicators  used  for  monitoring  and 
evaluation.  It  shows  how  indicators  can  be  used  to 
support  policy  decisions,  improve  management  and 
measure  the  overall  performance  of  programmes.  The 
volume  proposes  a  methodological  framework  and  a 
practical  guide  for  the  use of  indicators  in these  different 
situations. 

The  third  Volume  presents  a  selection  of  "Principal 
evaluation  techniques  and  tools"  applicable  to 
the 
evaluation  of  socio-economic  programmes.  These  tools 
are taken  from  relevant  scientific  disciplines:  economics, 
sociology,  management,  geography, etc.  Each one of  the 
23 tools  is described  in the  form  of  an  information  sheet 
featuring  a  general  description,  the  principles  of  its 
implementation, a list of its strengths and weaknesses for 
the  evaluation  of  Structural  Funds,  an  example  of  its 
application  and  a list  of  references. 

A number of  methods  adapted to the evaluation of  socio­
economic  programmes  are the  object  of  a more  detailed 
description 
in  the  fourth  Volume  entitled  "Technical 
solutions  for  evaluation  within  a  partnership  framework". 
Several  pilot  evaluations  were  implemented  within  the 
MEANS  programme  to  design  and  test the following  four 
innovations,  the  application  of  which  is  described  in 

777e MEANS  Collection 

detail  in  this  Volume:  impact  mapping,  cross-impact 
matrix, score cards and  multi-criteria  analysis. 

The  fifth  Volume  deals  with  "Transversal  evaluation  of 
impacts  on  the  environment,  employment  and  other 
intervention  priorities"  of  the  European  Union  and 
Member  States.  Among  the  questions  addressed  are 
impacts  in  terms  of  equal  opportunities  between  men 
and  women  and  the  competitiveness  of  SMEs.  The 
Volume  provides the elements  required for  an analysis of 
those  impacts  which  are  not  necessarily  part  of  the 
explicit  objectives  of  all the  actions evaluated. 

Finally,  the  last  Volume  is  a  "Glossary  of  concepts  and 
technical  terms".  The  compilation  of  the  Collection  has 
necessitated  the  rigorous  standardisation  and  definition 
of  the  terms  used.  Each  of  the  first  five  volumes  has  an 
index  of terms  which, along with  hundreds  of definitions, 
are brought  together  in the  glossary. 

Contents 

Introduction 

I Terminology 15 

The notion of an indicator and its different versions 17 

The system of indicators of a socio-economic programme 23 

Application to different types of structural interventions 39 

II Fields of application 

How to read the information sheets 

Indicators for seven domains 
■ Transport infrastructure 
■ Energy infrastructure 
■ Training 
■ Tourism 
■ Fisheries 
■ Research and Technological Development 
■ Agriculture and rural development 

Indicators for five priorities 
■ Equal opportunities 
■ Environment 
■ Information society 
■ Competitiveness of SMEs and enterprises in general 
■ Urban development 

Indicators for three major aims 
■ Economic development 
■ Employment 
■ Quality of life 

III Methods of producing and using indicators 

Conclusion 

9 

45 

47 

49 
49 
61 
69 
77 
85 
93 
103 

111 
111 
115 
125 
135 
145 

155 
155 
165 
173 

179 

209 

Appendixes 

Appendix  1 -  General  bibliography 

Appendix  2  -  Lessons  from  international  experience 

Appendix  3  -  Programme  indicators  for  the  1994-99  period: 
a  state  of  the  art 

Appendix  4  -  Grid  for  assessing  the  quality  of  a  system 
of  indicators 

Appendix  5  -  List  of  examples 

Index 

213 

215 

219 

229 

233 

237 

241 

Introduction 

Introduction 

International  experience 

Extensive international  experience  in the use of  indicators  has existed for a 
long time. Its origins  lie in the success of major macro-economic  indicators 
that  currently  play  a key  part  in political  decision-making  and  in the  media. 
The  role  of  indicators  in  the  measurement  of  government  deficits  in 
candidate countries for the euro  is a clear  indication  of their  importance. In 
the  1970s,  because  of  the  success  of  economic  indicators,  many  efforts 
were  made  to  develop  social  indicators.  At  present,  there  is  not  a  single 
political  field  in  which  efforts  are  not  being  made  to  create  and  quantify 
indicators at international  level. This dynamism  mainly concerns fields such 
as  education,  science, technology,  the  environment,  health, the  quality  of 
life and the  information  society. 

Indicators  have, for  a long  time, been  associated  with statistics  since  their 
purpose  has  been  to  describe  the  context  of  a  policy  in  relation  to  the 
context of other regions or countries. In this perspective, the OECD and the 
OSCE  produce  interesting  series  of  statistical  publications  covering  many 
fields. 

In  recent  years, 'efforts  to  construct  indicators  have  also  turned  to  the 
measurement  and management  of the performance  of public activities. The 
UK is the most  advanced  European country  in this sense, going as far as to 
establish  direct  links  between  performance  and  the  allocation  of  public 
funds.  France  has also  been  making  significant  efforts  in this  direction. 

Apart  from  the  famous  example  of  macro-economic  convergence  criteria, 
the European Union uses indicators in many fields. Most of these  quantified 
data  are  available  to  the  public  through  Eurostat.  Science  and  technology 
is  the  main  area  in  which  efforts  are  being  made,  mobilising  a  team  in 
Brussels as well as a network  of organisations  in the Member  States. Every 
two  years  the  full  list  of  Science  and  Technology  (S&T)  indicators  is 
published  in the form  of  a report. 

•  Community  indicators  and  practices 

In the field of Structural  Funds, indicators are used in two types of situation: 

A large number  of  global  indicators  are used on a European scale. Most  of 
these  have  been  rigorously  defined  and  the  definitions  harmonised  (e.g. 
unemployment  rate,  population  density,  and  disparities  in  infrastructure). 
For  many  years  a  lot  of  work  has  also  gone  into  regionalising  these 

Indicators  for monitoring  and evaluation 

statistical  indicators. They  are used  when  major  political  decisions  have  to 
be  taken,  such  as the  earmarking  of  budgets  and  the  definition  of  eligible 
areas.  They  are  also  used  to  account  for  the  use  of  Structural  Funds  in 
reports  to the  European  institutions  (e.g. the triennial  "Cohesion  Report"). 

Since  the  1988  reform,  the  Commission  has  encouraged  national  and 
indicators  for  quantifying  programme 
regional  authorities  to  develop 
objectives,  monitoring  programme 
implementation  and  estimating  the 
effects  obtained. 

Lists of  indicators  have  been  proposed  to  certain  countries  by the  services 
of the EC. However, programme  managers have generally constructed  their 
own  decentralised  information  systems  adapted to their  programmes. This 
freedom  of  choice,  combined  with  the  multiplicity  and  heterogeneity  of 
programmes,  has  spawned  a  multitude  of  indicators.  Many  of  these  are 
specific  to  a  particular  domain  of  intervention  (e.g.  number  of  visits  by 
tourists,  number  of  trainee  hours,  number  of  homes  with  access  to  a 
sewerage  system). 

It  seems  that  substantial  progress  still  needs  to  be  made  in  the  use  of 
indicators for programme  management.  In monitoring and evaluation, efforts 
to  develop  quantification  must  be  continued,  particularly  for  indicators 
related  to  programme  impacts.  For  the  third  Structural  Fund  programming 
period (2000-2006), stricter methodological supervision, by the Commission, 
of systems of  indicators  in fields  of  intervention  is expected. 

This second Volume of the  MEANS  Collection thus  proposes  some  ideas for 
reflection  and  a  practical  guide  for  the  use  of  indicators.  Like the  other  five 
Volumes, it is written for  professionals  involved in the evaluation  of  European 
Structural  Funds, 
i.e.  designers  and  managers  of  socio-economic 
programmes  and  evaluation teams.  Its purpose  is to  show  that  it  is  possible 
to  define,  choose  and  quantify  useful  indicators  in  a  pragmatical  way. 
Suggestions  in this  Volume  are  consistent  with  the  decentralised  approach 
followed  until  now  by the  Commission.  In other  words, the  idea  is to  clarify 
possible  options  and  not  to  impose  a  restrictive  system.  The  autonomy  of 
programme managers  should not, however,  be the source of wide  disparities 
in  concepts  and  approaches,  nor  hinder  the  harmonisation  of  the  main 
indicators.  For  this  reason, without  claiming  to  impose  standard  indicators, 
this  manual  aims to  provide a sound  methodological  framework  that  can  be 
shared  by all programme managers and facilite interaction  between all actors 
involved  in European economic  and social cohesion  policy.  It therefore: 

Introduction 

•  proposes  a coherent  and  rigorous  conceptual  framework; 

•  demonstrates the operationality  of this framework  for the  monitoring  and 

evaluation  of  programmes; 

•  proposes  criteria for assessing the  quality  of  systems of  indicators; 

•  shows  the  advantages  of  comparability  and  makes  suggestions  in  this 

respect; 

•  suggests  ways  of  using 

indicators 

for  managing  programme 

performance; 

•  finally,  suggests  possible  links  between the  'micro'  level (outputs,  results 
and  impacts  peculiar to  measures)  and the  'macro'  level (global  impacts 
in terms  of  cohesion  and  Community  priorities). 

A  large  place  has  been  reserved  for  examples  of  indicators  relating  to 
different  fields  of  intervention  of  socio-economic  programmes.  These 
examples  are  relevant  to  programme  designers  and  managers. They  have 
been developed  by qualified experts  in the  relevant fields. They  provide the 
reader  with  a  pragmatic  set  of  indicators  suitable  for  monitoring  outputs 
and  results,  and for  evaluating  impacts. They  also  suggest  new  routes  for 
more  effectively  linking  information  systems  to  the  aim  of  economic  and 
social  cohesion. 

The  last  part  of  this  Volume  is  entirely  devoted  to  the  measurement  of 
programme  performance. It contains  numerous suggestions for  developing 
the  use of  indicators for  this  purpose.  In particular,  it  proposes  a list of  key 
indicators for developing comparisons  and facilitating  communication  on a 
European scale. 

Box  1 -  How to  use this  guide 

For  the  non-specialist  who  is  interested  in  having  a  broad  overview  of 
indicators 

Part  I  (Terminology)  provides  a  wealth  of  information  about  different  types  of 
Indicators  and  their  use  within  the  framework  of  the  Structural  Funds.  It  also 
provides  a  standard  framework  of  definitions  that  should  reduce  the  risk  of 
misunderstandings  about 
II  (Fields  of  application)  provides 
concrete  examples of the use of  indicators. A quick  reading  might  concentrate  on 
transport 
a 
infrastructure,  equal  opportunities  and  economic  development.  Part  III  (Methods 
of  producing  and using  indicators)  and the appendices  can be read by those  who 
wish to  supplement  the  information  provided  in the  earlier  parts of  the Volume. 

limited  number  of  relevant 

information  sheets,  for 

indicators.  Part 

instance: 

For  the  specialist  and  non-specialist  with  an  interest  in  particular  fields  of 
application 

The  relevant  information  sheet(s)  in  Part  II is the  first  port  of  call.  The  numerous 
examples  resulting  from  the  contribution  of  specialists  provide  concrete 
illustrations  of the  use of  indicators  in different  fields  of  application. Reference  to 
Part  I  will  then  allow  readers  to  develop  their  knowledge  of  indicators  from  a 
general  perspective  and  establish  a  standard  terminology.  Part  III  and  the 
appendices  should  be  read  by  those  who  wish  to  supplement  the  information 
provided  in the  earlier  parts  of  the Volume. 

For  decision-makers,  programme  managers,  etc. 

Part  III  provides  an  essential  starting  point  for  those  who  are  concerned  with 
managing  programmes,  commissioning  evaluations,  etc.  In the  case  of  thematic 
evaluations, the  relevant  information  sheet  in  Part  II is also  essential  reading. For 
a  more  in-depth  appreciation  of  indicators  and  associated  issues,  a  number  of 
appendices  are  provided,  along  with  a  bibliography.  Part  I has  the  essential  role 
of  facilitating  communication  by  providing  a standardised  terminology  relating  to 
indicators 

I 
Terminology 

Indicators  for monitoring  and evaluation 

The notion of an indicator and its different versions 

The  notion  of  an  indicator 

An indicator can be defined as the measurement of an objective to be met, a 
resource  mobilised,  an  effect  obtained,  a  gauge  of  quality  or  a  context 
variable. An indicator  produces  quantified  information  with a view  to  helping 
actors of  public  interventions to  communicate,  negotiate or make  decisions. 
Within the framework  of  evaluation, the most  important  indicators  are linked 
to  the  success  criteria  of  public  interventions.  An  indicator  can  be  used  to 
measure  a  fact  or  an  opinion.  The  resulting  measurement 
is  usually 
approximate. An  indicator  may  be constructed  specifically  for  an  evaluation 
by the evaluation team ('ad hoc'  indicator) or taken from a monitoring system. 

What  constitutes  a  good  indicator? 

An  indicator  quantifies  an  element  considered  to  be  relevant  to  the 
monitoring  or  evaluation  of  a  programme.  For  example:  "1200  long-term 
unemployed  received training financed  by the  programme"  or  "75% of  the 
participants  in training  courses  claim to  be satisfied  or  highly  satisfied". 

A good indicator  must provide simple information that both the supplier  and 
the  user  can  easily  communicate  and  understand.  This  is,  however,  a 
necessary  but  not  sufficient  quality.  The  following  are  examples  of 
indicators  that  are  readily  understood:  rate  of  budget  absorption; 
percentage  of  regional  firms  assisted;  number  of  net  jobs  created;  and 
number  of jobless  in the  eligible area. 

An  indicator  may  have  several  values,  depending  on  the  state  of  the 
quantitative  variable  to  be  measured.  Thus,  the  unemployment  rate 
indicator  may  have an initial value, a value for  the intermediate objective, a 
value taken  mid-way  through  the  implementation  of  a programme,  etc. 

Indicators  and  quantitative  data 

Not  all  statistical  data  or  other  quantitative  information  are  automatically 
indicators.  In this  Volume, the  term  "indicator"  applies  only  to  information 
that  programme  managers  consider  as relevant  and  necessary  for them  to 
make  decisions,  negotiate  and  communicate.  Within  the  framework  of 
monitoring  and evaluation, the  most  important  indicators  are related to  the 

Indicators  for monitoring  and evaluation 

success  criteria  of  the  programme,  such  as:  the  state  of  progress  of 
projects;  objectives  to  be  met;  beneficial  effects  to  be  obtained;  quality 
factors;  progress to  be  made; etc. 

The  measurement  given  by  an  indicator  may  be  complete  or  partial.  For 
example,  the  indicator  "rate  of  budget  absorption"  measures  all  the 
financial  resources  used  by  a programme  (this  indicator  may  be  calculated 
for  public  expenditure  at  the  Community,  national  and  local  levels, as  well 
as for  private  expenditure).  In contrast, the  indicator  "number  of  kilometres 
of motorway  built"  is a partial measurement  of output. To obtain a complete 
measurement,  not  only  would  the  length  of  the  section  built  have  to  be 
taken  into  account  but  also  other  important  elements  such  as  the  number 
of  lanes on the  motorway  or  its  reference speed. 

Some  indicators  measure  a  well-defined  variable  such  as  the  "number  of 
SMEs  assisted  by  the  programme".  In  contrast,  other  indicators  apply  to 
abstract  concepts  and  help to  define  and  measure them. That  is the  case, 
for example, of the indicator  "proportion of businesses assisted, developing 
new  activities  in  sectors  with  growing  demand".  This  indicator  provides 
both  a definition  of  sectoral  redeployment  and  a measure  of  the  impact  of 
a  programme  on  it. 

Indicators  constitute  an  essential  part  of  the  information  system  of  a 
programme,  along  with  other  more  qualitative  forms  of  information.  The 
information  system  of  a  programme  produces,  notably,  operating  reports, 
reviews,  progress  reports  and evaluation  reports. 

Indicators  and  qualitative  data 

regional  authorities 

This Volume  is devoted  entirely  to  indicators, that  is to  say, to  quantitative 
measurement.  It  does  not  cover  qualitative  measurement,  for  which  the 
is  used.  The  following  statements  are  examples  of 
term  "descriptor" 
descriptors: 
their  mode  of 
"the 
management  when  the  programme  was  implemented"  or  "the  motorway 
did  not  spoil  the  countryside".  Nor  does  this  Volume  cover  scoring.  The 
following  statement  is an example  of  scoring:  "the environmental  impact  of 
the  project  is -2  on a scale  ranging from -3  to  +3,  representing  all  possible 
levels  of  impact,  from  the  most  negative  to  the  most  positive".  In  this 
Volume, a score  is not  considered  to  be an  indicator. 

transformed 

in  Volume  4  of  the  MEANS 
Scores  and  descriptors  are  presented 
Collection. These techniques are often  used  in combination  with  indicators. 

777e notion of an indicator 

For example, the World Bank uses a scoring system that functions in 
parallel with indicators, to monitor and evaluate its projects. The British 
government manages its research grants by means of expert panels that 
use scoring systems and indicators. 

This Volume presents several typologies of indicators which are 
summarised in Box 2. These typologies are consistent with those in the 
document "Common Guidelines for Monitoring and Evaluation" (European 
Commission, 1995, Luxembourg: OPOCE). They were refined during 
discussions with about thirty experts, mainly during a seminar organised in 
Brussels on 23 January 1997. 

Box 2 - Main typologies of indicators 

>■ in relation to the processing of information: 

Elementary, derived and compound indicators (see below) 

>- in relation to the comparability of information: 

Specific or generic indicators, key indicators (see page 20) 

>- In relation to the scope of information: 

Context and programme indicators (see page 24) 

>- in relation to the phases of completion of the programme: 

Resource, output, result and impact indicators (see page 28) 

>- in relation to evaluation criteria: 

Relevance, efficiency, effectiveness and performance indicators (see 
page 34) 

>- in relation to the mode of quantification and use of the information: 

Monitoring and evaluation indicators (see page 35) 

Elementary, derived and c o m p o u nd indicators 

This first distinction separates indicators according to the extent to which 
the information provided has to be processed. 

An elementary indicator provides basic information on which other 
indicators can be built. For example, the number of jobless is an elementary 
indicator which can be used to calculate the unemployment rate (number of 
unemployed / working population), changes in the unemployment rate, etc. 
The number of kilometres of roads built, the number of businesses assisted, 
and the number of beaches complying with standards are all examples of 
elementary indicators. 

Indicators  for monitoring  and evaluation 

A  derived  indicator  (ratio  or  rate)  is  based  on  the  calculation  of  the  ratio 
between two elementary  indicators. For example, an effectiveness  indicator 
can  be  obtained  by  dividing  two  values  of  the  same  impact  indicator 
(estimated  impact  /  expected  impact).  An  efficiency  indicator  can  be 
obtained  by  dividing  the  values  of  two  elementary  indicators  (observed 
result /  budget  absorption). 

A  compound  indicator  is  the  weighted  sum  of  several  elementary  or 
derived  indicators.  A  weighting  is  attributed  to  each  indicator  to  reflect  its 
importance  in relation to  a set of  indicators. The  indicator  considered to  be 
the  most  important  is  therefore  given  the  most  weight,  and  the  one 
considered to  be  least  important  is given the  least weight. The value  of  the 
compound  indicator is the sum of all the adjusted values of these indicators. 
For  example,  the  United  Nations  "quality  of  life"  index  combines  three 
elementary  indicators:  income  (GDP),  life  expectancy  and  percentage  of 
children in school. In 1986, an index of global infrastructure endowment  was 
calculated  by  DG  XVI  for  all  European  regions. This  index  is the  weighted 
sum  of  41  ratios  of  regional  infrastructure  endowment.  The  infrastructure 
considered  ranged  from  roads  (expressed  in  km/km2)  to  nature  reserves 
(expressed  in ha/inhabitant).  Each  ratio  relates the  regional  endowment  for 
a given  infrastructure  to that  of the  best-endowed  European  region. 

Breaking  down  an  indicator 

All  derived  or  compound  indicators  can  be  broken  down  into  different 
dimensions  of  the  variable  measured, as in the following  examples:  annual 
budget  absorption  broken  down  by  category  of  intervention;  turnover  of 
assisted  businesses  broken  down  by  sector  of  activity;  number  of  jobless 
broken  down  by  age-group. 

The  term  "breakdown"  is  used  when  the  indicator  is  divided  into  different 
categories,  e.g.:  agriculture  -  industry  -  service;  workers  -  jobless;  men  -
women.  In contrast, the term  "disaggregation"  is used when the indicator  is 
divided  into  similar  categories,  e.g.: distribution  of  budgets  between  zones 
within  an  assisted  region;  distribution  of  the  number  of  trainees  between 
training  organisations,  etc.  Indicators  are  often  quantified  at  a  lower  level 
and then  summed  to  produce  an aggregated  indicator. 

Specific  and  generic  indicators,  key  indicators 

One  characteristic  of  socio-economic  programmes  is  the  diversity  of  the 
interventions  that  comprise  them.  A  single  programme  often  consists  of 

The notion of an indicator 

several  dozen  different  interventions  and  two  programmes  rarely  contain 
the  same  combination  of  interventions. This  diversity  tends  to  multiply  the 
number  of  indicators  used  and  make  them  difficult  to  compare.  It  is 
therefore  particularly  useful  to  distinguish  indicators  in  terms  of  the 
comparability  of  information. 

A  specific  indicator  is  used  in  the  case  of  an  intervention  and  is  not 
intended  to  be  used  for  comparison.  For  example,  the  proportion  of 
trainees belonging to the Catholic  minority  is a good  indicator  for a training 
programme  in  Northern  Ireland, but  is of  little  use  in  another  region  or  for 
another  intervention. 

A generic  indicator  serves to  make comparable  measurements  of  sever­
al  different  kinds  of  intervention  within  the  same  programme.  The  com­
parison  is  internal  and  allows  the  aggregation  of  data  within  the  pro­
gramme,  in the form  of a sum  or average.  Examples  of  generic  indicators 
applied to  an entire programme  are the  rate of  budget  absorption  and  the 
completion  rate. Some  generic  indicators  apply  only  to  a  part  of the  pro­
gramme:  number  of  innovations  (for  all  the  components  which  favour 
technological  development);  proportion  of  SMEs  among  the  addressees 
or  recipients  (for  all the  components  which  benefit  businesses);  cost  per 
job created (for all the components which directly or indirectly create jobs); 
percentage  of  the  surface  areas  of  protected  zones  in  the  total  surface 
area of zones meriting  protection  (a generic  indicator for actions  related  to 
biodiversity,  water  resources  and  soil quality).  Generic  indicators  are rela­
tively  difficult  to  define  and  to  quantify  when  the  components  of  a  pro­
gramme  are  heterogeneous.  Methods  for  facilitating  this  task  are set  out 
in the  last chapter  of  this Volume. 

Key  indicators  are those  which  lend  themselves  to  internal  comparison 
between  different  interventions  and  to  external  comparison  with  other 
programmes. They  can  be  used  to  establish  points  of  reference  such  as 
average  European  performance  or  cases  of  excellent  performance  to  be 
emulated.  The  main  challenge  in  the  use  of  key  indicators  is  to  ensure 
that  their  quantification  in  several  different  programmes  adheres  to  har­
monised definitions, so that the information  produced can be pooled. Key 
indicators  can  then  be  used  for  aggregation  on  a  European  scale. 
Examples  of  key  indicators  proposed  in  this  Volume  include  the  rate  of 
budget  absorption,  the  completion  rate,  and  the  cost  per  job  created. 
Key  indicators  are treated  in more  detail  in the  last  chapter. 

The system of  indicators of a socio-economic  programme 

Structural  intervention  programmes 

Indicators  are  used  from  the  beginning  of  the  programme  cycle  to  help  in 
defining  territories  eligible  for  European  assistance,  in  analysing  the 
regional context, in diagnosing economic and social problems to be solved, 
and  in  assessing  the  needs  which  the  programme  has  to  meet.  At  this 
stage,  indicators  such  as  the  unemployment  rate  or  disparities  between 
infrastructures  play  a decisive  role. 

The choice and validation of the intervention strategy constitute the second 
step  in  the  programming  cycle.  At  this  stage  the  programme  designers 
must  define  the  objectives  precisely  and  quantify  them.  Indicators  are 
inseparable  from  quantification  and  are  also  very  useful  for  clarifying 
objectives.  For  example,  strategies  for  supporting  the  competitiveness  of 
SMEs often include consultancy  services. The objectives of these  measures 
are defined in terms of the number of businesses to receive particular  types 
of  consultancy  services.  The  indicator  serves  not  only  to  quantify  the 
objective, but also to define the expected service. For example, the  number 
of  SMEs  to  be  assisted  should  only  take  into  account  large  consultancy 
missions  (e.g. 5 days  of  consulting). 

Once defined and adopted, the  programme  is implemented. It is  monitored 
and  evaluated  at  mid-term.  Here  again,  indicators  are  indispensable  for 
circulating,  in  a  simple  and  condensed  form,  information  required  by 
programme managers. For example, indicators serve to monitor the pace at 
which  budgets  are  spent,  the  extent  to  which  the  schedule  is  adhered  to, 
the proportion  of the eligible  population  reached, the  rate of  satisfaction  of 
addressees  or recipients, the  number  of jobs  created, etc. 

The  programming  cycle  ends  with  an  ex  post  evaluation,  of  which  one  of. 
the main functions  is to  report  on the programme  results and on the  extent 
to  which  aims  have  been  achieved.  The  use  of  indicators  is  strongly 
recommended  at  this  stage  in  so  far  as  it  allows  the  communication  of 
simple  information  that  is  immediately  understood  by  a  wide  public,  e.g. 
cost  per job  created or rate of  placement  of jobless  people assisted. 

Most  socio-economic  programmes  adopt  integrated  strategies,  in  other 
words, they try to solve all the  problems  affecting  a given territory  and they 

Indicators  for monitoring  and evaluation 

for 

instruments 

territory.  This 
use  all  available 
characteristic  necessarily  entails  a  multiplication  of  needs  for  indicators, 
which  would 
if  the  programmes  were  not  highly 
structured.  Programmes  financed  by  the  European  Structural  Funds  are 
usually  structured  on three  levels: 

lead  to  confusion 

intervening 

in  that 

•  the  overall  programme  level to  which  the  global  objective  is  related, e.g. 
economic  development  or  employment.  This  level  consists  of  a  small 
number  of  priority  axes  (less  than  ten)  which  break  down  the  global 
objective  into  its  main  strategic  dimensions; 

•  the measures  level (one or several dozen), corresponding to the  basic  unit 
of  programme  management.  Each  measure  has  its  own  specific 
management  apparatus.  (Actions,  which  correspond  to  the  smallest 
homogeneous  component  in the  programme  -  since  each  action  groups 
together similar projects -  are considered on the same level as measures); 

•  the  project  level  (often  a few  hundred),  which  is the  implementation  unit 
of  the  programme,  since  each  project  is  a  point  of  articulation  between 
the  programme  and  its  addressees  or  recipients. 

Programme  and context  indicators 

Context  indicators  apply  to  an  entire  territory,  population  or  category  of 
population. An  example  of  a context  indicator  is the  level of  connection  to 
digital  phone  lines  in  an  eligible  territory.  Within  the  framework  of  ex  ante 
evaluation,  programme  intervention  may  be justified  by  the  backwardness 
of  the  region  in terms  of  level of  connection. The  same  indicator  may  then 
show,  within  the  framework  of  an  intermediate  evaluation,  that  the  region 
has  caught  up  with  others  in  this  respect  and  that  the  intervention  is  no 
longer  justified.  In  this  case,  the  context  indicator  does  not  provide 
information  on  the  effectiveness  of  the  programme  since  regional 
backwardness  has  been  remedied  by  market  mechanisms.  A  context 
indicator  does  not  apply  to  the  implementation  of  a  programme  and  to  its 
effects.  It  always  applies  to  the  entire  eligible  territory  or  targeted  public, 
without  distinguishing  between  those  that  have  been  reached  by  the 
programme  and those that  have  not. 

In contrast, programme  indicators concern only the part or category of the 
public  or  the  part  of  the  territory  that  has  effectively  been  reached. 
Programme  indicators  try  to  monitor,  as  far  as  possible,  the  direct  or 
indirect  effects  of the  programme.  For example, they  measure the extent  to 

The system of indicators 

which  a target  population has been reached or the extent to which a lasting 
advantage  has been obtained  by direct addressees  or  recipients. 

Within  the framework  of  monitoring  and  evaluation, a programme  indicator 
is able to  show  that  a particular  intervention  is a success  or failure. On the 
other  hand,  a  context  indicator  may  show  that  a  particular  intervention 
remains relevant or is no longer justified. Programme and context  indicators 
are  always  related  to  each  other  and this  relationship  can  be expressed  in 
the following  way: 

Initial state of the  context 
+  All the programme  effects 
+  Change  in the  context  due to other  factors 
=  Final  state of the  context 

Box 3 -  Theoretical  links between  context 
and programme  indicators 

Exogenous 
(confounding)  factors 

•fr.  Global change  in the  context 
(Present  situation  -  Baseline) 

Context 
indicators 

'' 

/ 

.-' 

'-—1 Indirect  effects  of 

I  the  proqramme 

X 

/ 

/ 

Direct effects of 
the  programme 

/ 

Programme 
implementation 

Programme 
indicators 

-k Difficulties  in  matching 

programme  and  context 
indicators 

This  relationship  is  easier  to  describe  in  theory  than  to  use  in  practice.  A 
programme  always  has  numerous  consequences:  positive  and  negative, 
direct  and  indirect,  immediate  and  long-term.  Programme  indicators  are 
rarely able to  measure all these consequences.  Moreover, the  link  between 

Indicators  for monitoring  and evaluation 

context and programme  indicators is also problematic  because a part of the 
observed  changes  in  the  context  are  due  to  confounding  factors  (also 
factors").  Techniques  for  analysing  causality  are 
called  "exogenous 
intended  to  separate 
from  those  of 
confounding  factors.  In Volume  3  of  the  MEANS  Collection  in  which  these 
techniques  are  presented,  it  can  be  seen  that  they  are  able  to  provide  an 
estimation but  never a complete  measurement  of  impacts. Box 3 shows the 
theoretical  relationship  between  context  and  programme  indicators,  and 
the  complexities  of this  relationship  in  practice. 

the  effects  of  the  programme 

The  overall  change  in  the  context  is  measured  by  means  of  context 
indicators. The  information  provided  by a context  indicator  usually  includes 
a baseline value, also called the  initial state, and  a current  value.  Examples 
of  context  indicators  include:  the  number  of  inhabitants  of  the  region 
connected  to  a  sewerage  system  (infrastructure  endowment);  the  regional 
rate of connection to the  Internet,  as a percentage  of the  European  average 
(disparity);  and  the  percentage  of  company  directors  who  consider  that 
accessibility  by  air  is a major  problem  (needs). 

Changes  which  affect  the  territory  or  the  eligible  population  are  not  only 
explained  by  the  programme,  but  also  by  exogenous  (confounding)  fac­
tors.  Indicators  measuring  the  evolution  of  these  factors  help to  interpret 
or to qualify the  information  given  by context  indicators, when these  show 
success  or  failure  that  may  be  more  apparent  than  real.  The  following 
examples  illustrate  confounding  factor  indicators  and  their  use: the  natu­
ral  growth  of  the  working  population  may  explain  an  increase  in  regional 
unemployment  even though  the  programme  has created jobs; the  drop  in 
world  prices  may  explain  the  decline  in  traditional  local  production  even 
though  the  programme  supported  marketing  and  promotion  efforts. 

Even  if the  link  between  programme  and context  indicators  is  problematic, 
it is often  useful to  link them, as the example in Box 4 shows. In the  context 
of  an  intervention  providing  consultancy  services  to  businesses  to  help 
them  develop  their  exports,  it  is  relatively  easy  to  construct  the  initial 
indicators  of  the  intervention. 
its  direct  and 
immediate  effects  is defined: number  of  recipient  firms,  number  of  hours  of 
consultancy  services  received, number of export  projects concerned  by the 
provision of advice. It is a little more difficult to estimate intermediate  effects 
such  as:  number  of  new  international  contacts  generated  by  the  advice 
that  have  been  concretised 
given,  percentage  of 
commercially.  The  following  effects  may  be  estimated,  although  with 

Information  concerning 

these  contacts 

777e system of indicators 

difficulty,  by  means of  a survey  among company  managers: the  proportion 
of  annual  turnover  resulting  from  customers  contacted  on  the  basis  of 
advice given  (after  two years). 

Beyond  a  certain  limit,  the  monitoring  of  effects  by  means  of  programme 
indicators  is  no  longer  credible.  For  example,  it  would  be  futile  to  ask 
company  managers  how  many  jobs  had  been  created  in the  months  after 
they  benefited from  a few  hours of  advisory  services.  It is at this stage that 
it becomes essential to use context  indicators.  For example, one may  know 
the  turnover  and  number  of  jobs  in  all  the  industrial  firms  in  the  region. 
These  indicators  may  serve to calculate  a ratio  of  average turnover  per job 
and,  by  extrapolation,  to  estimate  the  number  of  jobs  generated  by  new 
exports.  This  example  illustrates  the  fact  that  programme  indicators  are 
most  useful for analysing the implementation  of the programme and its first 
effects,  and then that  the  priority  progressively  shifts  to  context  indicators 
as the focus  moves to  longer term  effects. 

The  distinction  between  context  and  programme  indicators  will  system­
atically  be  applied  in  about  fifty  examples,  throughout  this  Volume  (see 
page 237  for the  list  of  examples). As suggested  in  Box  4, the two  types 
of  indicator  are  linked at all stages  of the  programme, from  its  implemen­
tation  to  its  most  long-term  effects.  This  distinction  therefore  corre­
sponds  to  the  typology  presented  below  and  its  categories  of  outputs, 
results and  impacts.  For example, there  is a point  of articulation  between 
the  indicator  of  the  capacity  of  purification  plants  built  with  Community 
funds  (programme  indicator  /  category  of  outputs)  and  the  indicator  of 
regional  infrastructure  endowment  (context  indicator  /  category  of  out­
puts).  In cases  where  it  is difficult  to  classify  an  indicator  in terms  of  the 
"context  /  programme"  typology,  it  should  be  remembered  that  a  pro­
gramme  indicator  concerns  a closed  list  of  the  persons  or  organisations 
that  participated  in the  programme  or  were  affected  by  the  programme, 
whereas  a  context  indicator  concerns  the  entire  targeted  public,  the 
entire regional economy, and the entire natural environment, including  the 
people  or economic  agents  not  affected  by the  programme. 

Indicators  for monitoring  and evaluation 

Box 4 -  Example of links betwe 

en cont ext and programme indicators 

Programme  indicators 
(related to the  intervention 
and to  its  effects) 

Context  indicators 
(related to the  whole 
eligible  area) 

Number  of entreprises  having 
received  export  advice 

Number  of  entreprises 
in the  eligible  area 

! 

Number  of  new 
international  contacts 

%  of  exporting 
entreprises 

J 

Exports  generated 

Total  exports 

1 r 

Jobs  resulting  from 
generated  exports 

Total  jobs 

Average turnover  per  job 

Resource, output, result and impact  indicators 

A programme  mobilises  resources  (financial, human  and  institutional)  with 
a  view  to  achieving  a  global  objective  (socio-economic  cohesion).  To 
monitor  and  evaluate  the  programme,  it  is  necessary  to  break  down  its 
global  logic  and distinguish a series  of related objectives, as shown  in  Box 
5.  This  typology  is  inspired  by  the  logical  framework  technique  used  for 
the  past  few  decades  in the  field  of  development  aid. The  indicators  are 
classified  into  five  categories:  resource,  output,  result,  specific  impact 
and  global  impact.  This  typology  will  be  used  frequently  throughout  this 
Volume. 

This  box  shows that  the  different  levels  of  objective  are  related to  different 
categories  of  people  (or  organisations)  concerned  by  the  programme.  It  is 
important  to  clearly  define these  categories. 

777e system of indicators 

Box 5 -  Definition  of indicators b) 

ι  level of  objective 

Level of 
objective 

Type of 
indicator 

Resource 
{input) 

Definition 

Key  actors 

Financing 
authorities  and 
operators 

Means  made 
available  by 
financing  authorities 
and used  by 
operators  for  their 
activities 

Operational 
objective 

Output 

Product  of the 
operators'  activity 

Operators 

Immediate 
specific  objective 

Result  (immediate 
outcome) 

Immediate effect  for 
direct  addressees 
or  recipients 

Direct 
addressees 
or  recipients 

Sustainable 
specific  objective 

Specific  impact 
(sustainable 
outcome) 

Strategic 
objective 
Aim 

Global  impact 
(outreach) 

Sustainable  effect 
for  direct 
addressees 
or  recipients 

Global  effect  for  the 
entire  population 
concerned  (direct 
and  indirect 
addressees 
or  recipients) 

Direct 
addressees 
or  recipients 

Direct  and 
indirect 
addressees 
or  recipients 

The  programme  funders  are  people  or organisations  which  decide  on  its 
implementation  and  allocate  the  required  resources.  The  operators  are 
those  who  use the  resources  closest  to the field. They  are the  last  link  in 
the chain  of  implementation  of the  programme. They  are at the source  of 
information  which  feeds  the  monitoring  system.  Examples  of  operators 
are:  a  local  authority  which  distributes  direct  support  to  businesses;  a 
group  of  firms  which  distributes  aid  among  its  members;  a private  com- ' 
pany which constructs and manages a telephone network; a private com­
pany which builds a road for a regional authority; a training institute  which 
trains  people funded  by the  programme. 

The  financing  authorities  and  operators  make  up  the  category  of 
programme  actors.  Irrespective  of  the  level  on  which  they  intervene,  the 
programme actors  must account  for their activity  in a regular and  complete 
way  in the form  of  progress  reports. 

Indicators for monitoring and evaluation 

Broadly  speaking,  the  programme  addressees  or  recipients  consist  of  all 
the  people  or organisations  that  receive  benefits  or  are affected  directly  or 
indirectly  by the disadvantages  of  the  programme. The  reason  d'être  of  an 
action  is to  produce a positive  change for  addressees  or  recipients1. 

Some  individuals  or organisations  are direct  addressees  or recipients  in so 
far as they receive or personally  use the services or support  provided by the 
programme.  Examples  of  direct  addressees  or  recipients  are: the  motorist 
who  uses  a  road  built  during  the  programme;  the  family  that  uses  a 
telephone  network  improved  by  the  programme;  the  firm  that  received  an 
investment  subsidy; the trainee who  benefited from  subsidised training. 

Indirect  addressees  or  recipients  are  individuals  or  organisations  that  are 
affected  by  the  programme  without  having  participated  directly  in  its 
activities. Indirect addressees or recipients  may be affected through a direct 
addressee or recipient, e.g. the firm that employs a person trained during the 
programme.  Indirect  addressees  or recipients  may also  be affected  without 
an intermediary, e.g. inhabitants of a neighbourhood  next to a rundown area 
that  was  renovated  during  the  programme,  or  people  sensitive  to  the 
deterioration of their  natural environment, who  live near a motorway. 

The strategic  objective  of  a programme  is to  provide  its  direct  and  indirect 
addressees  or  recipients  with  an  overall  advantage.  In some  cases  certain 
people  or  organisations  may,  however,  be  disadvantaged.  The  negative 
effects  which  these  addressees  or  recipients  suffer  are  usually  unintended 
(e.g. bankruptcy  of a local firm due to the distortion of competition  in favour 
of  an  assisted  firm).  The  disadvantages  may  also  be  related  to  the 
objectives  of  the  programme  (e.g. substitution  of  a  long-term  unemployed 
person for  another  unemployed  person). 

1 Because  programme  addressees  or  recipients  do  not  always  gain  from  the 
programme, the term "beneficiaries" has not been used to identify them. Moreover, 
the Commission has chosen to use the term "beneficiary" in another sense. The term 
"final recipient" is used in the regulatory texts (and measures for financial execution) 
of the Structural  Funds to  denote the  public  or  private  operators  responsible for 
commissioning projects or for the distribution of public aid (see article 9§1-J of the 
proposed  regulation  on  general  measures  for  Structural  Funds,  Official  Journal, 
9/6/1998). The final recipient  has to submit regular and systematic  reports, and is 
therefore the programme operator in the sense defined in this Volume. 

In  contrast,  certain  evaluation  manuals  define  beneficiaries  as  the  individuals, 
groups or organisations which receive programme services or assistance, that is 
to say, direct addressees or recipients in the sense used in the present Volume. 

irte system of indicators 

Box 6 -  Implementation  of the  programme  and its  effects 

Implementation  of the  programme 

Social, Natural  and  Economic 
Environment 

Actors 

Addressees/recipients 

Funding 
authorities 

J 

Operators 

Direct 
addressees 
/recipients 

Among the people and organisations concerned by the programme, there is 
a basic difference between the actors and the addressees or recipients. The 
actors  are  within  the  operational  system  of  programme  implementation, 
whereas addressees or recipients  are outside this system. As shown  in Box 
6, the  boundary  is that  which  separates the  programme  from  its  economic, 
social and natural environment. This boundary  separates the operators from 
the  direct  addressees  or  recipients  (it  may,  however,  occasionally  happen 
that operators are also simultaneously  addressees or recipients  -  e.g. in the 
case of  investments  in  roads). These  two  categories  play  a decisive  role in 
the definition  of  resource, output,  result  and  impact  indicators. 

Resource indicators  provide information on the financial, human, material, 
organisational  or regulatory  means used  by operators for  implementing  the 
financing 
programme.  Resources  are  the  joint  responsibility  of  the 
authorities,  which  allocate  them,  and  the  operators  who  implement  them. 
Most  resource indicators are regularly quantified  by the  monitoring  system. 
Examples  of  resource  indicators  include:  the  total  budget  (quantity  of 
rate); 
resources);  annual  budget  absorption 
percentage of expected overspending; percentage of European financing in 

(resource  absorption 

Indicators  for monitoring  and evaluation 

the total  public financing; number  of  people working  on the  implementation 
of the programme; number  of organisations  involved in the  implementation. 

Output  indicators  represent  the  product  of  the  operators'  activity.  More 
precisely,  an  output  is  considered  to  be  everything  that  is  obtained  in 
exchange  for  public  expenditure.  Two  examples  in  the  field  of  SME 
consultancy  services  can  be  used  to  demonstrate  the  principle  of  an 
output,  and  help to  distinguish  an output  from  a result.  Firstly,  an  operator 
might  receive  a  fixed  sum  of  money  to  finance  the  setting  up  of  a 
consultancy  service  for  SMEs.  In  this  instance,  the  expenditure  has 
"bought"  the  establishment  of  a consultancy  service,  which  is  considered 
as the output.  On the other  hand, an operator  might  be allocated  a  budget 
of  400,000  euros  for  a SME  consultancy  project  planning  to  supply  5,000 
hours  of  consultancy  services.  However,  if  the  project  were to  deliver  only 
half  of  the  planned  hours  of  services,  the  operator  would  only  be  paid 
200,000  euros.  In  other  words,  if  an  output  is  not  realised, the  support  is 
withheld.  The  output  in  this  case  would  therefore  be  the  2,500  hours  of 
consultancy  services  effectively  delivered.  Outputs  are  under  the  entire 
responsibility  of  operators  who  have  constantly  to  report  on them  through 
the monitoring system. Examples  of output  indicators include: kilometres of 
roads  built;  progress  rate  of  the  building  of  a  road;  hectares  of  urban 
wasteland  rehabilitated;  capacity  of  purification  plants  built;  number  of 
trainees whose training was paid by the programme; and percentage of this 
training  of which the  quality  is certified. 

Result  indicators  represent  the  immediate  advantages  of  the  programme 
(or,  exceptionally,  the  immediate  disadvantages)  for  the  direct  addressees 
or  recipients. An  advantage  is  immediate  if  it appears  while the  addressee 
or  recipient  is directly  in contact  with  the  programme.  The full  results  may 
be observed when the operator has concluded the action and closed off the 
payments.  Since  result  indicators  are  easily  known  to  the  operators,  they 
are generally  quantified  exhaustively  during  monitoring. 

Result  indicators  provide  information  on  changes  which  occur  for  direct 
addressees  or  recipients,  e.g. time  saved  by  users  of  a road;  reduced  rates 
for  telephone  calls;  qualifications  earned  by  trainees;  new  tourist  activity 
generated by a farmer; use of new productive capacity created by a firm; and 
the satisfaction  of  businesses which  have received consultancy  services. 

It  is  at  the  time  that  addressees  or  recipients  receive  support  or  programme 
services  that  results  can  be  quantified.  Either  direct  measurements  are  made 

777e system of indicators 

(e.g.  by counting  the  number  of  trainees  recruited  during  their  training)  or the 
direct  addressees  or  recipients  are  asked  to  state  the  advantages  they  have 
obtained (e.g. by means of a questionnaire at the end of a consultancy mission). 

Impact indicators represent the consequences of the programme beyond its 
direct  and  immediate  interaction  with the  addressees  or recipients. An  initial 
category  of  impacts group together the consequences  for direct  addressees 
or recipients of the  programme, which appear or which last into the  medium 
term  (specific  impacts), e.g. traffic  on a road one year  after  it  is opened; the 
placement rate of trainees after twelve months; sustainable jobs created in an 
industrial  plant  built  with  programme  support;  and  the  survival  rate  of 
businesses created with programme support. Some impacts are unexpected 
(spin-offs)  but indicators are rarely created for  unexpected  impacts. 

A second  category  of  impacts consists  of  all the consequences  that  affect, 
in  the  short  or  medium  term,  people  or  organisations  that  are  not  direct 
addressees  or recipients. These  impacts  may  be similar  (e.g. improvement 
industrial 
of  the  quality  of  life  for  people  living  near  a  rehabilitated 
wasteland;  improvement  in the  quality  of  beaches  near  a  new  purification 
plant). They  may, in contrast,  spillover  to  affect  people or organisations  far 
from the  programme,  as in the case of  macro-economic  impacts. 

The  mechanisms  of  impact  propagation  are  often  separated  into  two 
categories:  market  effects  (e.g. impact  on  suppliers  or  sub-contractors  of 
the  assisted  firms)  and  non-market  effects  (e.g.  positive  impact  of  the 
improved  image  of  the  region  or  negative  impact  of  a deterioration  in  the 
environment). Because  non-market  effects  or externalities are not  reflected 
in the price system on which  individual socio-economic  actors  largely  base 
their  private  decisions,  and  because  these  decisions  have  economic 
consequences for  other  actors, it is particularly  useful to take these  effects 
into account  in the context  of  a public  programme. 

Because  of  the  time  lag  or  their  indirect  nature,  impacts  cannot  easily  be ' 
known  to  operators  during  their  daily  management  of  the  programme. 
Impact  indicators  are  therefore  quantified  from  time  to  time  only,  usually 
during evaluations. One way of establishing  impacts is to carry out a survey 
of direct addressees or recipients, for example a year after they have left the 
programme. The questions  asked  might  concern facts (e.g. how  many  new 
jobs  have been created since the  support  was  obtained?)  or opinions (e.g. 
how  many jobs  would  have  been lost  in the absence  of  support?). 

Indicators for monitoring  and evaluation 

In  this  Volume,  an  "effect"  is  taken  to  mean  any  change  caused  by  the 
implementation  of the programme, whether this change  is direct or  indirect, 
immediate  or  long-term.  Thus,  effects  include  results  and  impacts.  In  all 
cases, surveys  can  be  used to  observe  the apparent  effects  (gross  effects) 
and  not  the  real  effects  of the  programme  (net  effects). The gross  effect  is 
almost  always  over-estimated,  either  because  observations  are  partly 
affected  by  confounding  factors,  or  because  opinions  gathered  are  over-
optimistic.  In particular, people who answer questionnaires  tend to forget or 
the 
overlook  what  they  would  have  done  without 
programme. Sometimes  an addressee  or recipient  has already taken a final 
decision  (e.g. to  invest,  go  on  a training  course,  start  a business)  when  he 
or she learns that the programme can be helpful. In this case, the addressee 
or recipient takes advantage of the support  (deadweight  effect), but the real 
effect  of  the  programme  is  nil.  Deadweight  effects  must,  if  possible,  be 
estimated  and  subtracted from  gross  effects. 

the  support  of 

To  obtain  an  estimation  of  net  effects,  it  is  also  important  to  take  into 
account  the  substitution  effect  (in  the  case  of  interventions  aimed  at 
individuals  or groups  of  individuals)  or the  displacement  effect  (in the  case 
of  interventions  aimed  at  geographical  areas).  Substitution  effects  occur 
when  a result  is obtained  in favour  of  a direct  addressee  or recipient  at the 
expense  of  a  person  or  organisation  that  is  not  eligible  for  the  public 
intervention. When this effect  is unwanted, it is considered  as negative  and 
subtracted from  gross  effects. 

Displacement  effects  occur  when a result obtained  in favour of  addressees 
or  recipients  is  accompanied  by  effects  on  non-addressees  or  non-
recipients  within  the  programme  area. In general, displacement  effects  are 
not  desired  and are subtracted  from  gross  effects. 

Relevance,  e f f e c t i v e n e s s,  e f f i c i e n cy 
and  performance  indicators 

The  most  useful  indicators  for  evaluation  are  those  associated  with  a 
criterion  or judgement  such  as relevance, effectiveness  or  efficiency. 

Relevance  indicators  relate  the  programme  objectives  to  the  needs  that 
have  to  be  met.  For  example,  the  number  of  places  for  trainees  that  the 
programme can provide, in relation to the number of long-term  unemployed 
in the region; the number of planned consultancy  missions, in relation to the 
number  of  regional firms that  have never exported. 

The system of indicators 

Effectiveness  indicators relate what is obtained to what was expected. An 
effectiveness  indicator  can  therefore  be  calculated  by  dividing  two  values 
of  the  same  output,  result  or  impact  indicator,  that  is to  say, the  observed 
value  at  a  given  date  and  the  objective  initially  set.  When  talking  of 
effectiveness,  it  is preferable, for the  sake of  clarity, to  specify  whether  the 
reference is to the effectiveness  of outputs, results or impacts. Examples of 
effectiveness  indicators  are:  outputs  exceed  the  objective  by  5%;  the 
number  of  businesses  created  amounts  to  85%  of  the  objective;  the 
placement  rate of trainees  after  a year  is  10%  higher than  expected. 

indicator 

is  therefore  the  ratio  of  two 

Efficiency  indicators relate what was obtained to the resources mobilised. 
An  efficiency 
indicators:  the 
measurement  of  what  was  obtained  /  the  measurement  of  resources 
mobilised  to  obtain  it.  The  calculation  of  efficiency  can  be  based  on  an 
output, result or impact  indicator. When talking of efficiency,  it is preferable, 
for  the  sake  of  clarity,  to  specify  whether  the  reference  is to  the  efficiency 
of  outputs,  results  or  impacts. The efficiency  of  outputs  is often  called  the 
unit cost. The efficiency  of impacts is often called the cost /  efficiency  ratio. 
Examples  of  efficiency  indicators  are:  the  average  cost  of  a  kilometre  of 
motorway  built was 5 MEuro; the average cost of training a person who had 
been out  of work  for  an extended  period of time was 2 KEuro; the  average 
cost  of  a net job  created  was 30  KEuro. 

Performance  indicators,  according  to  the  definition  proposed  in  this 
Collection,  encompass  the  effectiveness  and  efficiency  of  outputs,  results 
and  impacts.  In  fact,  the  word  "performance"  is  used  in  many  different 
ways.  In  this  context,  there  is  a  large  overlap  between  the  notion  of  a 
programme indicator and that of a performance indicator. In some contexts, 
the term performance  is used within the framework of a "reorientation of the 
administration  towards  results,  service  quality  and  user  satisfaction"  (USA 
-  Government  Performance  and  Result  Act).  In  other  contexts,  the  term 
performance  applies  mainly  to  resource  management  and  outputs. That  is 
the  case,  for  example,  at  the  World  Bank  (World  Bank,  1997  Operation 
Evaluation  Department,  Lessons  and  Practices,  1997  /  10).  Here  again, 
because  the  vocabulary  is  not  stable,  performance  needs  to  be  redefined 
each time  reference  is made to  performance  indicators. 

Monitoring  and  evaluation  indicators 

Monitoring indicators are quantified  by operators when they implement  the 
programme  or  when  they  are  in contact  with  the  addressees  or  recipients. 

Indicators  for monitoring  and evaluation 

By  means  of  new  communication  technologies,  it  is  now  possible  to 
construct  a  monitoring  system  that  provides  information  on  resource 
consumption,  output  rates  and  numbers  of  addressees  or  recipients 
(trainees, assisted  firms, etc.).  It is even  possible  in many cases to  maintain 
a data base with the identity of the direct addressees or recipients (e.g. name 
of trainees, telephone number  of the contact  person  in the assisted firms). It 
is also easy to obtain statements  by the addressees or recipients  when they 
are  in contact  with  the  programme  (e.g. investments  made  by  the  assisted 
firms; trainee satisfaction  at the end  of their training  courses). 

The  operators  may  also  be  asked  to  quantify  the  impact  indicators  during 
monitoring,  particularly  with  respect  to  sustainable  impacts  on  direct 
addressees  or  recipients.  It is technically  possible to systematically  telephone 
or  write  to  all  direct  addressees  or  recipients,  for  example  a  year  after  their 
participation  in  the  programme.  In  practice,  it  may  happen  that  monitoring 
reinforced to this extent provokes rejection by the addressees or recipients who 
see  it  as  being  bureaucratic,  or  by  the  operators  who  consider  that  the 
workload  involved  is  excessive.  By  wanting  to  obtain  too  much  information 
through the monitoring system, there is the risk that the quality and reliability of 
this information will be reduced. The example in the box below shows the limits 
of the collection of exhaustive information among addressees or recipients. 

Box  7 -  Collection  of  information from addressees  or  recipients: 

monitoring  or  evaluation? 

Ireland  -  Objective  1 Human  Resources  Operational  Programme,  1994-99 

A  Human  Resources  Operational  Programme  was  set  up  in the  Irish  Republic  for 
the period  1994-1999.  Many  private and  public  training  organisations took  part  in 
the programme. Initially, each organisation was required to  indicate the number of 
people  that  had  found  jobs  after  the  training  course.  To fulfil  this  obligation,  the 
organisations concerned  conducted  exhaustive  mail or telephone surveys  among 
their  trainees. 

After this system had functioned for a while, the programme evaluation unit  noted 
that  the  data  were  neither  homogeneous  nor  reliable.  In  particular,  jobs  were 
measured  at  any  time  between  8  and  18  months  after  the  training  course. 
Moreover, 
the  system  was 
cumbersome  and  bureaucratic.  It  was  decided  to  abandon  the  exhaustive 
collection  of  data  and  to  quantify  the  placement  rate  indicator  by  means  of 
surveys  organised  on the  scale  of the  programme. 

training  organisations  complained 

that 

the 

777e system  of  indicators 

Impact  indicators are essentially  evaluation  indicators.  Evaluation  obtains 
information on impacts by means of surveys or in-depth studies. Collection 
techniques  use  sampling,  which  makes  it  possible  to  limit  the  number  of 
people  questioned  and to  avoid the  impression  of  bureaucracy.  Moreover, 
evaluation  is  rarely  carried  out  by  the  operators  and  so  does  not  add 
substantially to their workload. 

Monitoring  and  evaluation  indicators  can  also  be  distinguished  by  their 
implications  on the sharing of responsibilities. Monitoring  indicators and, in 
particular, resource and output  indicators, enable operators to report on the 
use of  resources  allocated to them and on the activities for  which they  are 
fully  responsible, e.g. building facilities without  overspending  or  exceeding 
deadlines. 

Result indicators are used either for monitoring or evaluation, depending on 
the  degree  of  decentralisation  adopted  in the  programme  management.  If 
the  programme  is  highly  decentralised  (management  by  results),  the 
operators  can  and  must  constantly  adjust  their  activity  in  relation  to  the 
results  obtained.  For  example,  they  may  notice  that  the  advice  given  to 
firms  is not satisfactory,  and decide to make extensive amendments to the 
system  of  support  for  consultancy  services.  In  this  case,  the  operators 
attentively  follow  result  indicators  in  order  to  be  able  to  adjust  the 
programme  as  necessary  and  fulfil  their  management  responsibilities.  In 
contrast,  if the operators are not free to  adjust the rules of  implementation, 
they  have no  personal  interest  in following  result  indicators which will  then 
be  used  for  evaluation.  In  practice,  a spectrum  of  intermediate  situations 
co-exist  between these two  models. 

Information  provided  by  evaluation  indicators  corresponds  to  factors  for 
which  the  operators  are  not  fully  responsible.  For  example,  the  operators 
cannot  be  expected  to  react  in  real  time  to  impact  indicators.  Evaluation 
indicators  are  intended  for  a  broad  audience,  including  financing  bodies' 
and  operators.  They  are  used  relatively  infrequently  to  make  decisions 
which  lead  either  to  changes  in the  programme  so  that  objectives  can  be 
achieved more satisfactorily, or to changes in objectives so that  experience 
can be taken  into  account. 

Box 8 specifies the role of the  different  types of  indicator  from the  point  of 
view of the  exercise of operators'  responsibility. 

Indicators for monitoring and evaluation 

Box 8 - Indicators and operators' responsibilities 

Resources —*■ 

Outputs 

—► 

Results 

—*· 

Impacts 

Operators.... 

... Are directly  responsible  for 
outputs  (monitoring) 

...  are indirectly  responsible for  results  since 
they  can  be  observed 

...  do  not  control the impacts  (evaluation) 

Application to different types of structural  interventions 

Application  to  infrastructure,  services  and  subsidies 

When the programme produces physical outputs such as infrastructure, the four-
category typology can easily be applied. Thus, in the first example in Box 9, the 
operator develops an industrial site and the firms that establish themselves on 
this site are the addressees or recipients. The outputs are clearly limited to what 
is under the operator's  responsibility  (the surface area developed). The results 
represent the advantages of the development, from the addressees or recipients' 
point  of  view  (the  attractiveness  of  the  industrial  site  as  compared  to  other 
competing  sites). The  impacts  appear  at the  same time as the addressees  or 
recipients' demand (progressive occupation of the site by firms). 

Box 9 -  Exar 

nples of  indicato 

-s in the case of  infrastructure 

Resources 

Outputs 

Results 

Impacts 

Industrial site 

Road between A & B 

Rehabilitation  of an 
urban wasteland 

Cost of the 
development 

Surface area 
developed 

Attractiveness of 
the site compared 
to  neighbouring 
sites 

Surface area 
bought by 
businesses after 
1 year 

Evolution of 
employment  in 
the  businesses 

Cost of  construction 

Cost of the  project 

Surface area 
rehabilitated 

Evolution of the 
number of inhabitants 
(and their socio-
professional 
composition), within 
a radius of  1 km 

Percentage of 
inhabitants of the 
town who want to 
continue living there 
(context) 

Length  built 

Gain in accessibility 

Number  of  vehicles 
using the  road after 
1 year 

Percentage of 
regional  businesses 
satisfied with their 
accessibility 
(context) 

Indicators  for monitoring  and evaluation 

In  the  second  example,  the  operator  who  has  built  a  road  (output)  can 
immediately  know  what the  advantage  is for the addressees  or recipients  in 
terms  of  accessibility  and time  saved  (result).  It will,  however,  be  necessary 
to wait a year or more before counting the number of vehicles using the road. 
An  expensive  survey  will  then  have to  be  launched  to  establish  how  many 
passengers  travel  in the  vehicles  using  the  road,  the  origin  and  destination 
of their journeys and the  changes  caused  by  use of the  road  (impacts). 

The third example is the case of a municipality which was helped to rehabilitate 
an industrial wasteland (output). This example is more difficult to treat  because 
there  are  no  direct  addressees  or  recipients. The  inhabitants  of  the town  are 
indirect addressees or recipients and the operator has no personal contact with 
those  who  benefit  from  the  project  (e.g.  inhabitants  of  neighbouring  areas, 
passers-by). It is therefore difficult to construct result indicators. By convention, 
it is proposed that  inhabitants  situated within a radius  of  1 km are considered 
as direct addressees or recipients. Beyond this radius, the project improves the 
overall quality of life of all the inhabitants of the town who will be more satisfied 
with life there and less tempted  to move away (impact). 

Box 10 -  Examples of indicators  in the case of  services 

Resources 

Outputs 

Results 

Impacts 

Training of the 
long-term 
unemployed 
Budget  spent  on 
training 

Number  of  days 
of  training 
financed 

Number of trainees 
who have gained 
a qualification 

Number of trainees 
who found work 
within a year 

Creation of new 
training courses in an 
emergency  industry 
Budget  spent 

Number  of  places 
offered  on the  new 
training  courses 

Percentage of places 
offered on training 
courses in growth 
sectors 

Number of trainees 
recruited into the 
new industry within 
a year 

Consultancy 
services to SMEs 

Budget  spent  on 
consultancy 
services 

Number of  hours 
of  consultancy 
services  financed 

Rate of satisfaction 
of SMEs who 
received consultancy 
services 

Percentage of SMEs 
who claim to have 
increased their 
activity within the 
first year, owing to the 
consultancy  services 

The four categories become more difficult to distinguish when the programme 
provides  immaterial  services  (e.g. training,  consultancy).  Not  only  does  the 

Application to different types of structural interventions 

term  "physical  outputs"  become  inappropriate  because  the  service  is 
immaterial,  the  very  notion  of  an  output  becomes  problematical.  It  is  not 
possible to  "produce" the service independently of those who receive it. 

The  production  of  the  service  (output  or  supply)  therefore  becomes 
practically synonymous with the fact that the addressee or recipient  benefits 
from it (result or demand). In the case of a programme  providing services, it 
is  logical  that  outputs  and  results  merge.  In  this  Volume,  however,  the 
distinction  has  been retained, on the  basis  of the  following  convention: the 
service financed by the operator is considered to be an output. The category 
"results"  is limited to other  immediate  advantages  gained by the addressee 
or recipient. This definition applies as indicated in Box 10. 

Box 11 -  Examples of indicators  in the  case of subsidies 

Setting up of 
a farm 

Laying up of a 
fishing  vessel 

Creation  of  new 
businesses 

Resources 

Amount of subsidy 

Amount  of subsidy 

Amount of  subsidy 

Outputs 

Results 

Impacts 

Amount of 
investments by 
farmers who set up 
farms 

Tonnage of fishing 
vessels laid up 

Number of 
businesses  created 

Percentage of 
farmers who set up 
farms and  comply 
with a regional 
quality  label 

Percentage of  people 
who formerly  worked 
on fishing vessels 
and who are now 
unemployed 

Number of jobs 
generated within 
a year on the farms 
created 

Reduced catches of 
threatened  species 
of fish (T/year)  in the 
fishing area 
concerned 

Gross value added 
created after  one 
year on the new 
farms 

Percentage of 
businesses  created 
in high tech sectors 

Number of jobs 
created within one 
year in the new 
businesses 

Gross value added 
in the new 
businesses within 
one year 

The typology in four categories becomes even more difficult to apply when the 
programme offers subsidies. The reference here is not to a subsidy  attributed 
to a programme operator (e.g. funds allocated to a local authority for creating 
a purification  plant or to  a training organisation for  providing courses), but to 

Indicators for monitoring and evaluation 

subsidies attributed to direct addressees or recipients. In this case, there is 
complete simultaneity between the resource (the programme absorbs the 
amount of the subsidy), the output (the operator grants the subsidy) and the 
result (in exchange for the subsidy, the addressee or recipient invests in a 
project and obtains immediate advantages). In the case of a programme 
delivering subsidies, it is logical that the concepts of resources, outputs and 
results tend to converge. In this Volume, the following language has been 
adopted: resources are subsidies granted; outputs represent key elements in 
the subsidised project (those which must absolutely be obtained if the subsidy 
is to be paid, e.g. the actual creation of a business or the maintenance of an 
existing business); the results are all other immediate effects on the addressee 
or recipient. These conventions apply as in Box 11. 

■ Application to d o m a i n s, priorities and aims 

Part II presents a series of systems of indicators in fifteen "fields" covered 
by the Community Structural Funds. The fifteen fields do not correspond to 
the classification of Structural Fund interventions determined by the 
Commission2. If this classification had been adhered to, the Volume would 
have had the following structure: 

Productive environment 

Agriculture, forestry and rural development 
Fishing 
Support for large businesses 
Support for SMEs and the crafts 
Tourism 
Research and Technological Development (RTD) 

Human resources 

Support for people 
Support for organisations and systems 

Basic infrastructure 

Transport infrastructure 
Telecommunications and information society infrastructure 
Energy infrastructure 
Environmental infrastructure 
Development and rehabilitation 
Health infrastructure 

2 This typology is taken from an internal memorandum of the Commission 
presenting a project to codify fields of Structural Funds interventions. 

Application  to different  types of structural  interventions 

The  European  Commission's  classification  creates  mutually  exclusive 
sectors of  intervention. In contrast, the fields studied in this Volume overlap 
each  other, the  notion  of  field  used  here  being  similar  to that  of  a  "theme" 
used  in  Volume  5  of  the  MEANS  Collection.  Certain  fields  correspond  to 
sectors  of  intervention  similar  to  the  categories  of  the  Commission, 
whereas  others  correspond  to  aims  of  the  Structural  Funds.  Other  fields 
have  an  intermediate  status,  some  of  them  corresponding  with  the 
Commission's  policy  priorities. 

The  first  series  of  fields  focus  on  the  outputs  of  Structural  Fund 
interventions 
in  seven  domains:  transport,  energy,  training,  tourism, 
fisheries,  research  and  technological  development,  and  agriculture  and 
rural  development.  Domains  of  intervention  are  situated  upstream  in  the 
chain  of  causality  in  relation  to  the  socio-economic  impacts  they  are 
intended to  produce. The relevant  indicators  in this category tend  therefore 
to focus on the outputs and results of  interventions. As for indicators  at the 
impact  level,  these  concentrate  on  specific  impacts,  i.e.  those  impacts 
arising  earliest  in the chain of  causality  and  which  can  be attributed  to  the 
results  of  interventions. 

The  second  series  of  fields  is devoted  to  five themes,  most  of  them  being 
Community  priorities: environment,  information  society, competitiveness  of 
in  general,  equal  opportunities  and  urban 
SMEs  and  enterprises 
development.  Themes  tend  to  be  situated  further  along  the  causal  chain 
than domains. In many instances a theme is linked to a mix of  interventions, 
some  of 
(i.e. 
telecommunication  networks  relate to the theme  "information  society")  and 
some related to the theme at a further stage in the causal chain (i.e. support 
for  new  high-tech  enterprises). 

the  output  stage 

theme  at 

related 

them 

the 

to 

Finally,  three  fields  reflect  the  overarching  rationale  of  Structural  Funds 
interventions  of  promoting  economic  development,  employment  and  the 
quality  of  life.  Indicators  in  this  category  therefore  provide  information  on 
global  impacts. 

As shown  in  Box  12, this  Volume  attempts to  establish  backward  linkages 
between  global  impact  indicators,  specific  impact  indicators,  results  and 
output  indicators, and  even  resource  indicators. These  attempts  provide  a 
fresh  and  innovative  way  of  dealing  with  the  micro-macro  problem,  i.e.  in 
linking  the daily  management  of  operations to the  global  aim of  the  policy. 

Indicators  for monitoring  and evaluation 

Box  12  -  Structure  in  fourteen  fields:  domains,  priorities  and  aims 

transport,  energy, 
training, tourism, 
fisheries,  RTD, 
agriculture  and 
rural  development 

environment, 
competitiveness  of 
enterprises 
and  SMEs, 
equal  opportunities, 
urban  development, 
information  society 

economic  development, 
employment, 
quality  of  life 

i 

/ 

/ 

Domains 

A 

, 

outputs 

results 

specific  impacts 

global  impacts 

Fields of  application 

How to read the information sheets 

Each  information  sheet  presented  in  this  section  is  constructed  to  a 
standardised  format: 

Introduction 

An  introduction  briefly  discusses  the  field  of  application  within  the 
framework  of the Structural  Funds. 

Interventions  within the  field 

The discussion  of  indicators  for  interventions  within the field of  application 
follows  the  chain  of  causality  from  resource  indicators  through  to  impact 
indicators.  This  is  followed  by  a  number  of  concrete  examples  of 
interventions  presenting  output,  result  and  impact  indicators. 

Programme indicators are systematically  supplied at all levels. At the impact 
level they cover both impacts arising directly from the results of  interventions 
and  intermediate  impacts  that  link  interventions  to  global  impacts  arising  at 
the  macro  level, which  are reflected  by the appropriate context  indicators. 

While  programme  and  context  indicators  are  linked  at  all  stages  along  the 
causal chain, the key role of context  indicators is to provide information about 
the  evolution  of  the  overall  macro-economic  situation.  However,  given  the 
usefulness  of  the  information  provided  by  context  indicators,  whenever 
possible this type of indicator has been included at the output and result level. 

Other  interventions  having  an  impact  in the  field 

An  intervention  in  one  field  often  has  an  impact  in  another.  This  section 
examines  the  possibility  of  interactions  between  fields,  giving  examples 
where  relevant. 

Availability  and  collection  of  data 

This  section  tackles  issues  associated  with  data  collection  within  the 
specific  context  of the  field concerned. 

Key  indicators 

A  number  of  key  indicators  are  provided  that 
aggregation  and  comparison. 

lend  themselves 

to 

Indicators  for monitoring  and evaluation 

Bibliography 

For readers  who  wish to  deepen  their  knowledge  of  a particular  field. 

Indicators for seven  domains 

Transport  infrastructure 

in  the  field  of  transport 

infrastructure 
Structural  Fund  interventions 
encompass  roads,  bridges  and  motorways,  railways  lines  and  stations 
(either  inter-city  or suburban), subways, airports, ports, intermodal  centres, 
etc.  Interventions 
involve  construction  as  well  as  upgrading.  Eligible 
projects  are  largely  limited  to  infrastructure  and  do  not  usually  include 
mobile  equipment  like  trains,  ferries  and  aeroplanes.  Major  funding  of 
transport  infrastructures  is  channelled  through  the  ERDF  in  Objective  1 
areas,  often  in  parallel  with  EIB  loans.  Interventions  take  various  forms: 
large  projects,  operational  programmes  (transport  specific)  or  transport 
measures  within  regional  programmes. 

Transport infrastructure has a number of specific characteristics, which shape 
the  systems of  indicators  used to  evaluate  interventions  in the field.  Projects 
are  usually  large  and  relatively  few  in  number,  and  are  implemented  over 
several years.  Recipients  directly  use some  projects  (roads), whereas  others 
(railways) need an operator and complementary investments before they reach 
their end users. Finally, a transport  infrastructure  project  is part of  a network; 
only the network  provides  benefits for the  users, not the project  by  itself. 

Interventions  within the  field 

Resource  indicators 

While the need to monitor the cost of programme and project  implementation 
is  universal, the  risk  of  overruns  in  terms  of  budget  and  schedule  of  large 
infrastructure  projects  makes  the  following-up  of  project  execution 
particularly  important.  At  the  programme  level,  resources  can  be  efficiently 
monitored  with  one  single  indicator:  "compliance  with  project  cost"  which 
can  be  defined  as  the  "latest  estimate  of  the  total  cost  of  the  project  in' 
proportion  to  the  cost  that  appeared  in  the  programming  document".  The 
progress  of  project  execution  can  be  followed  by  comparing  planned  and 
actual  project duration, and planned and actual  project  costs (See Box 13). 

Output  indicators 

Without  prejudice  to  more  detailed  levels  of  information  needed  in  lower 
levels  of  management,  the  information  needed  for  monitoring  purposes 

Indicators for monitoring and evaluation 

only needs to include the "rate of completion". In order to compute the "rate 
of completion", large projects must be broken down into many small tasks, 
every task being weighted with its planned cost. The "rate of completion" 
is the weighted percentage of completed tasks. It is easy to compute a 
weighted average "rate of completion" of all infrastructure projects within a 
programme. A drawback with using the rate of completion as an indicator 
is that its meaning comes very close to that of an indicator of budget 
absorption, at least while there is no cost overrun. 

In order to better reflect outputs, it is worthwhile computing an indicator of 
"compliance with project duration". To facilitate judgement and 
comparisons, it is essential that project duration be presented with both 
initial and current values. And it is also very important that time and money 
be presented side by side, since a project may involve overspending on 
cost overruns and underspending due to delays, the cost balance being 
close to nil. In this case, cost dimension alone would falsely suggest that all 
is running well (see Box 13). 

Box 13 - Follow-up of project execution 

Duration-Plan 

Duration-Current 

Cost-Current 

Cost-Plan 

<d 
" ιΛ 

■ 

■ s 

■ S 

\J 

\; 

υ 

V 

^l 
\) 

-rJ 

l> 

3 

10 

20 30 40 

50 

60 

% of Realisation 

Π Year Begin 

Now 

In this example the project, which was well behind schedule at the start of the 
year, has almost caught up with the planned level of realisation (upper two groups 
of histograms). The cost overrun observed at the start of the year has also been 
slightly reduced as the project progressed through the year (lower two groups of 
histograms). 

Transport infrastructure 

indicators  can  be  usefully  related  to  the  stock  of  regional 
Output 
infrastructure.  New  roads  and  railways  are  likely  to  increase  regional 
endowment  measured  in terms  of  km/km2 or  km/inhabitant. 

•  Result  indicators 

On the  road,  the  majority  of  travellers  use the  infrastructure  with their  own 
equipment  (cars). As  for  other  modes,  the  addressees  or  recipients  need 
the services of transport operators, and are thus dependent  not only on the 
quality  and  capacity  of  the  infrastructure,  but  also  on the  array  of  services 
made available  by those  operators  and on  how  well their  requirements  are 
met  by  them. This  fundamental  difference  has  consequences  not  only  for 
the  indicators  that  have to  be selected for  evaluation  of  investments  in the 
different  modes,  but  also  for  the  speed  and  geographical  diffusion  of  the 
benefits from any particular  project. This is because new road  infrastructure 
is available  from  the  first  day  it  is open  to  service  and  is  easily  integrated 
into the remaining road network. On the other hand, other modes frequently 
require additional  investments to  be made  by the operators  who  may  have 
to  buy  additional  and  improved  equipment,  reorganise  their  services  and 
market them. 

Impact  indicators 

The initial  specific  impacts  of  interventions  in this  field  are associated  with 
the  traffic  flow  on  the  new  infrastructure.  Actual  users  of  the  new 
infrastructure can be counted  at the earliest  one year after opening. By this 
date,  the  end-users  can  also  be  the  subject  of  studies  in  terms  of  travel 
purpose, travel time  saving, travel  cost, etc.  Other  impacts  that  have to  be 
considered  are  related  to  the  balance  of  inter-modal  competition.  The 
opening  of  new  transport  infrastructure  is  likely  to  change  the  market 
shares of the various modes. But there are also many instances in which the 
case  of  complementarity  arises,  through  which  additional  flows  by  one 
mode  in  a  section  of  the  network  will  be  transferred  to  another  mode  to 
cover  another  part  of the journey  within the  same  logistical  chain. 

The  improvement  of  regional  dynamism  and  competitiveness  remains  the 
infrastructure 
foundation  of  the  existence  and  dimension  of  transport 
projects  within  the  Structural  Funds.  Many  studies  have  shown  that  these 
impacts  are  always  very  difficult  to  assess.  Sometimes,  it  makes  sense  to 
apply  economic  impact  indicators  to  the  transport  system  as  a whole.  On 
the contrary, it is often  meaningless to apply economic  impact  indicators  to 
a single  project. 

Indicators  for monitoring  and evaluation 

The  increasing  sophistication  of  some types  of transport  infrastructure  has 
lead  to  a  growing  concentration  of  costs,  which  can  result  in  stronger 
discontinuities  of  provision  and  inequity  in terms  of accessibility.  Evaluation 
must  therefore  consider  not  only  the  gains  for  those  who  benefit  from  the 
use of  improved  services,  but  also  issues of  distribution  arising from  direct 
and 
from 
indirect  effects  on  persons  and  regions  not  benefiting 
interventions. 

Another  area of  significant  external  impacts  of transport  investments  is the 
environment.  Unfortunately,  a  lot  of  attention  is  given  to  the  impacts  of 
infrastructure  construction  but  much  less to  the  impacts  of the  functioning 
of transport systems and of policy decisions (or inaction) that may also have 
a very  strong  environmental  impact. 

A specific  problem arises about  how short term construction jobs should  be 
counted.  Since  they  are  part  of  the  human  resources  invested  in  the 
implementation  of  the  programme,  they  should  be theoretically  counted  at 
the  resource  level. This  may, however,  lead to  problems  of  double  counting 
since  the  creation  of  construction  jobs  is,  in  practice,  estimated  at  impact 
level  by  most  evaluation  teams. 

Transport infrastructure 

Example 1: New section of motorway connecting A and Β 

Programme Indicators 
(Related to the Intervention 
and its effects) 

Context indicators 
(Related to the assisted area) 

Compliance with project duration 
Rate of completion 

km of new motorway 
km of new lanes 

km of motorways per million 
inhabitants In the area (endowment) 

O u t p ut i n d i c a t o rs 

Progress 

Quantity 

Result indicators 

Speed 

E.S.S.(1)betweenAandB 

Average E.S.S. to and from all 
relevant urban centres in the area 

Impact indicators 

Traffic flow 

Time Saved (3) 

Safety 

Transport system 

Indirect economic effect 

Traffic flow(2) of vehicles using the new 
infrastructure after one year 

Traffic flow in the area (vehicle χ km 

/ year / inhabitant) 

Total journey time saved by users 
(hours χ vehicles χ average number of 
passengers per vehicle) after one year 

Number of traffic accidents(4) on the 
motorway after one year 

Traffic accidents in the area (number / 
year / Mio inhabitants) 

% traffic between A and Β using the 
new infrastructure 

Environment 

Number of houses suffering from traffic 
noise (6) 
Hectares of natural sites disturbed 

% of managers in the area who 
declare that road accessibility is a 
major constraint for their flrm{5) 

Number of dwellings in the area 

Hectares of natural sites in the area 

(1) E.S.S. stands for Equivalent Straight­line Speed and measures the ease of access from one point to 
another, irrespective of the distance between those points and of the path taken. It Is computed by 
dividing the straight­line distance between the two points by the duration of the quickest possible trip 
between them. 

(2) Traffic flows are expressed in vehicles per average day. When possible, one truck is counted as two 

cars. 

(3) Time saved should be measured In relation to both Inter­urban and intra­urban journeys. 

(4) Accidents are expressed as the number of accidents involving injuries per year. When possible, 

accidents involving deaths are counted as double. 

(5) Accessibility constraint is quantified through a survey among regional managers. It is expressed as 
the percentage of respondents who agree with the sentence 'road (rail, air, sea...) accessibility ¡s a 
major constraint for the development of my organisation". 

(6) Houses suffering from traffic noise must be counted in accordance to national levels of acceptable 

noise. The indicator applies to the vicinity of both the motorway and the other roads joining A and B. 

Indicators for monitoring  and evaluation 

Example  2: 

Improvement  of a  railway  line between  A and  B, 
bringing  it up to the  high speed  range 

Programme indicators 
(Related to the intervention 
and its effects) 

Context indicators 
(Related to the assisted area) 

Output  indicators 

Progress 

Quantity 

Result  indicators 

See Example 1 

km of high speed railway 

Speed 

E.I.S.S.(1 ) between A and Β 

km of high speed railway per million 
inhabitants 

Average E.I.S.S. between all relevant 
urban centres in the area 

Impact  indicators 

Traffic flow 

Time saved 

Transport system 

Indirect economic effect 

Traffic flow(2) of passengers between A 
and Β after one year 

Traffic flow to and from area 
(passengers χ km/year) 

Total journey time saved by users 
(hours χ passengers) after one year 

% traffic  between A and Β fully 
exploiting the new infrastructure 

% of managers in the area who 
declare that rail accessibility is a 
major constraint 

Environment 

See Example 1 

See Example 1 

(1) E.I.S.S.  stands  for  Equivalent  Interval  Straight-line  Speed,  an easily  transferable  indicator  which  is 
intended  to  measure  the  ease  of  access  from  one  point  to  another,  in  a  discontinuous  service, 
irrespective of the distance between those points and the path  taken. It is computed  by dividing the 
day into intervals of a width judged adequate for the distance in question (e.g. 2 hour intervals), then 
computing  the  best  E.I.S.S.  (or  departures  within  each  interval,  and  then  averaging  those  E.I.S.S. 
results over the relevant period of the day (e.g.  from 7:00 to 22:00). If there is no departure in a certain 
interval, the corresponding  E.I.S.S.  is taken as zero and Is thus entered  into the average  computing 
process.  The  computation  of  E.I.S.S.  values  must  take  into  account  not  only  the  actual  departure 
times  but  also transfer  possibilities  and timetables, whenever  we consider  the connection  between 
two points not enjoying a direct connection (at least with a departure in the interval being considered). 

(2) Traffic  flow is expressed  in passengers  /  year. 

Transport infrastructure 

Example 3:  Expansion of a regional  airport 

Output  indicators 

Progress 

Capacity 

Result  indicators 

Destinations 

Speed 

Programme  indicators 
(Related to the  intervention 
and its  effects) 

Context  indicators 
(Related to the assisted  area) 

See Example 1 

Maximum  number of passengers per 
year 

Maximum  number of passengers  per 
year  per million  inhabitants 

Number  of destinations with  at  least 
ten  direct flights  a  week(l) 

E.I.S.S. by air between  the closest 
urban centre  and the  most  relevant 
destinations  (direct  +  indirect 
connections) 

Average E.I.S.S. by any  means 
between all urban centres in the area 
and the  most  relevant  destinations 
(direct +  indirect  connections) 

Cost for  users 

Changes in airport  charges 

Impact  indicators 

Traffic  flow 

Indirect economic  effect 

Air  traffic  flow(2)  in the  regional  airport 
after  one year 

Air  traffic  flow  to  and from  area 

% of  managers  In the  area who 
declare that air accessibility  is a major 
constraint 

Environment 

See Example 1 

See Example 1 

(1) The  number  of  destinations  served  by  a regular  weekday  service  of  at  least  two  flights  per  day. 

(2) Air  traffic  flow  is  expressed  in  number  of  passengers  and  tonnes  of  freight  per  year. 

Indicators for monitoring and evaluation 

Example  4: 

Improvement  of  a container  terminal  in a sea  port 

Programme  indicators 
(Related to the  intervention 
and  its  effects) 

Context  indicators 
(Related  to the  assisted  area) 

Output  indicators 

Progress 

Capacity 

Result  indicators 

Waiting  and  turnaround 
time 

Punctuality 

Destinations 

Handling time 

I m p a ct 

i n d i c a t o rs 

Traffic  flow 

Indirect  economic  effect 

See  Example  1 

Maximum  number  of containers  lifted 
by cranes per  hour  and per  annum 

Waiting time  of  ships  before  docking 
(hours) 
Turnaround  time  of  vessels 

%  of scheduled  services arriving on time 

Number  of  shipping  lines calling  in 
lhe  port 

Average  import  dwell  time(1) 
Turnaround time of  road vehicles 

Traffic  flows  through  the  container 
terminal  (TEU (2)  /  year)  after  one year  ¡  (TEU  /  year) 

'  Traffic  flow  to and  from  area 

%  of  managers  in the area who 
declare that shipping  accessibility  is a 
major  constraint 

(1) Average  time  spent  by  import  units  in  the  port. 

(2) TEU  -  Twenty-foot  Equivalent  Units. 

Transport infrastructure 

Example 5:  Upgrading  of a railway line for 

freight  operations 

Programme  indicators 
(Related to the  intervention 
and  its  effects) 

Context  indicators 
(Related to the  assisted  area) 

Output  indicators 

Progress 

Quantity 

Quality 

Result  indicators 

Speed 

Impact  indicators 

Traffic  flow 

Transport system 

Indirect economic  effect 

See  Example 1 

Number of  km  upgraded 

% of  line allowing speeds of at least a 
desired threshold value (i.e. 100 km  /  h) 
%  of  line  length allowing trains of 
heaviest standard  load 

Typical commercial speed of  freight 
trains  on the  line 

Average  speed on  ralways  lines  in 
the  area 

Freight traffic  flows on  the  line 
(Tonnes  /  year) after  one year 

Traffic  flow  to  and from  area 
(Tonnes  /  year) 

Traffic  flow of  freight  using the new 
infrastructure  and withdrawn  from  road 
(Tonnes  /  year) 

% of  area managers who  declare that 
rail  freight accessibility  is a major 
constraint 

Example 6:  Buik 

ling an intermodal  (rail  /  road)  centre 

Programme  indicators 
(Related to the  intervention 
and its  effects) 

Context  indicators 
(Related  to the  assisted  area) 

Output  indicators 

Progress 

| See  Example 1 

Result  indicators 

Speed 

Impact  indicators 

Traffic  flow 

Typical commercial  speed of  goods 
transported  through  the  centre 

Freight traffic  flows through  the  centre 
(Tonnes  /  year) 

Freight traffic  flows within  the  area 
(Tonnes  /  year) 

Transport  system 

See Example 5 

Indicators  for monitoring  and evaluation 

Other  interventions  having  an  impact  in the  field 

Whilst  interventions  in  the  field  of  transport  infrastructure  create  important 
and widespread  external  impacts  in numerous  non-transport  fields,  impacts 
arising  from  non-transport  interventions  on  transport  infrastructures  are 
generally quite limited. When these  impacts do occur, they are most  likely to 
be  generated  by  interventions  aimed  at  supporting  economic  development 
and  enterprises.  Interventions  in  these  fields  are  likely  to  affect  transport 
system 
indicators  via  increased  economic  activity.  Similarly, 
interventions targeted at the environment  or tourism may have an impact  on 
the transport  system  (i.e. decreased or  increased  private vehicle use). 

impact 

•  Availability  and  collection  of  data 

Work  site  managers  should  be  required  to  provide  synthetic  output  data 
(rate of  completion  and compliance  with project  duration)  every six  months. 

As  part  of  their  normal  management  activity,  the  operators  should  quantify 
simple  result  and  impact  indicators  (e.g. traffic  flow,  average  speed,  etc.). 
Result  indicators  are typically  quantified  every  year. 

Quantifying  complex  impact  indicators  (i.e.  regional  component  of  the 
traffic  flow)  generally  requires  that  a  special  study  or  survey  be  launched. 
For instance, these studies may  involve surveys of firms or individuals  using 
the  improved  infrastructure.  These  studies  are  costly  and  cannot  be 
repeated  every  year.  A  good  method  involves  an  initial  study  just  before 
work  is finished  and  a second  study  three years  after  completion. 

Context  indicators  relate to the transport  system and the region as a whole, 
and  many  can  be  derived  from  statistical  sources.  Some  European 
statistics  are available  at  regional  level  as shown  in  Box  14. 

Box  14 -  Eurostat: Transport  infrastructure  and  road  safety 

Regional  statistics  (NUTS  II) 

Roads: total  (km) 
of  which  Motorways  (km) 
Rail: total  (km) 
of  which  Double track  or  more (km) 
of  which  Electrified  lines (km) 
Roads: network  density  (km  /  km2) 
Rail:  network  density  (km  /  km2) 
Deaths caused  by  road  accidents  (nb  /  Mio  inhab  /  year) 

Transport infrastructure 

There may be other available data of several types of impact (environment, 
market share by transport modes, industrial productivity) if regular 
collection of longitudinal data is made by national statistical offices or by 
regional organisations. 

Key indicators 

The key indicators presented below lend themselves to comparisons both 
between different interventions of the same programme and between those 
of different programmes, and can in certain cases be aggregated at a 
European level. 

Level 

Key indicator 

Resource 

Output(2) 

Result 

Impact 

Rate of consumption of budget (% of allocated funds) 
% of budget devoted to environmental mitigation measures (1) 

Rate of completion of project (% of objective) 
Compliance with project duration 

Average speed between principal economic centres(3) 

% of regional managers declaring that accessibility is a major 
constraint for their firm 

(1) Mitigation measures reduce or eliminate the environmental impact of supported projects. 
(2) In order to accurately measure outputs and reflect the dimension of the project they should be refined 

with additional details. 

(3) This gives an indication of the overall accessibility afforded by transport infrastructure investments. 

The basis of calculation is the E.S.S. (Equivalent-Straight-llne Speed) between the ten most Important 
economic centres of the region, using the most efficient mode of transport. 

For more information s e e-

European Commission (1997), EU transport in figures. Luxembourg: Office 

for Official Publications of the European Communities. 

Burmeister, Α., Colletis-Wahl, Κ. (1996), 'T.G.V. et fonctions tertiaires: 

grande vitesse et entreprises de services à Lille et à Valenciennes' 
in: Revue Transports Urbains, Volume 93. 

Button, Κ. (1995), 'What can Meta-Analysis tell us about the implications of 

Transport' in: Regional Studies, Volume 29(6), 507-517. 

Rephann, T.J. (1993), 'Highway Investment and Regional Economic 

Development: Decision Methods and Empirical Foundations' in: 
Urban Studies, Volume 30(2), 437-450. 

Energy  infrastructure 

Structural  interventions  in  the  energy  sector  are  aimed  either  at  energy 
production  (electric  power  production  plants)  or  at  energy  distribution 
(electric  power  distribution  networks,  methane  pipelines  or  other  gas 
pipelines).  They  involve  the  construction  of  new  infrastructure  or  the 
enlarging  and  reinforcing  of  existing  facilities.  Structural  interventions  may 
also  aim at saving energy.  Examples are the  implementation  of  systems  for 
the  recovery  of  the  heat  generated  within  various  production  processes,  or 
the production of electric power derived from renewable sources or biomass. 

The  main  objective  of  interventions  is  to  supply  energy  to  the  population 
and  to  industry.  In  the  first  instance,  the  supply  of  energy  for  final 
consumption  by  families  has  to  be  able  to  guarantee  that  the  quality  of 
living conditions  is acceptable.  In the second  case, the supply  of energy  to 
industry  and  public  institutions  has  to  be  competitive  with  regard  to 
quantity,  quality  and  costs.  This  implies  that  two  other  crucial  objectives, 
strictly related to economic  development  and quality of life, are attained: (1) 
reduced  energy  dependency  on  overseas  sources through  the  exploitation 
of  national  and  local  sources  of  energy  and  (2)  reduced  environmental 
impact  from  energy  production  and  distribution  through  the  exploitation  of 
alternative, and possibly  renewable,  resources. 

In  the  field  of  energy,  it  is  particularly  relevant  to  distinguish  between 
national  interventions  and  local  ones.  In  the  first  case,  intervention 
programmes  usually  consist  of  large  infrastructure  projects  that  provide 
energy to  addressees  or  recipients through  complex  distribution  networks; 
in other  words, addressees  or  recipients  do  not  directly  gain from  projects, 
but  from  the  distribution  networks  to  which  they  are  connected.  On  the 
other hand, local interventions are smaller in size, more numerous and  often 
aim at  producing  or  saving  energy for the  producer's  own  benefit.  Projects 
in  the  field  of  energy  generally  require  several  years  to  be  implemented. 
They  often  receive  loans  from  the  EIB  and  national  banks,  in  addition  to 
Structural  Fund  support. 

Interventions  within the  field 

Resource  indicators 

At the  programme  level, resources  can  be monitored  by  using  an  indicator 
of  "compliance  with  project  cost":  i.e.  the  most  recently  available 

Indicators  for monitoring  and evaluation 

information  on the total  cost  of  the  projects  in proportion  to  the cost  which 
appeared  in the  programming  document. 

Output 

indicators 

The  progress of  a project  is often verified  by  simply  monitoring  expenditure 
statements.  In cases  of  large  projects  (e.g. building  or  upgrading  of  electric 
power  production  plants) the verification  of  in-progress  implementation  can 
be estimated  by monitoring the  planned completion date (final testing of the 
works).  In  case  of  smaller  projects  (e.g.  building  or  upgrading  of  electric 
power  distribution  networks)  it  is  also  possible  to  quantify  the  physical 
progress of the works  in terms  of  "percentage  of  km of  network  completed 
by  a certain date  in respect to the total  km of  network  to  be completed". 

As  regards  the  final  output  of  the  intervention,  the  main  indicator  is  the 
power  installed for  production  plants (kW or MW is applicable to electricity). 
Power  installed  in  kW  /  MW  is  a  measure  that  can  apply  to  almost  all 
energy infrastructure,  but the rule is to  provide a breakdown  by each source 
of  energy  (e.g. coal,  gas  or  oil  generation  plants;  hydro-electric  system  or 
wind  system; etc.). 

Result 

indicators 

The  main  results  of  structural  interventions  in the field of  energy  principally 
concern  the  cost  and  reliability  of  the  new  service  supplied  as well  as  the 
typical  savings  generated  for  end-users. 

Impact 

indicators 

Specific  impacts  are  related  to  energy  consumption  (electricity,  gas)  after 
one  year.  Programme  and  context  indicators  are  the  following:  amount  of 
electricity  and  gas  consumed  as a  result  of  interventions,  number  of  users 
connected to the completed  networks, and degree of customer  satisfaction 
(to  be  monitored  and  evaluated  mainly  through  specific  surveys  among 
users;  alternatively,  this  could  be  captured  through  the  operators'  records 
related  to  the  reliability  of  the  service  supplied). Temporary  employment  is 
also  created  during  the  investment  phase  (measured  in jobs  χ  years). The 
following  impacts to  be considered  are: value added  (in practice,  measured 
in terms of  wages  and  salaries  plus  profits)  generated  by the functioning  of 
new  infrastructure  (production  and  distribution  of  electricity  and  gas), 
employment  created  by the  operation  of  infrastructure  and the  reduction  in 
energy  dependency  on overseas sources. Since energy  production  and the 

Energy infrastructure 

environment  are closely  linked, indicators  which  inform  on the depletion  of 
natural  non-renewable  energy  sources,  (e.g. quantity  of  renewable  energy 
produced)  or  on  air  pollution  (e.g.  quantity  of  fossil  fuels  burnt  and 
emissions produced) should be employed whenever  relevant. Most of these 
impacts can be either positive or negative, depending on the purpose of the 
projects  (energy  production  or energy  saving). 

Example  7:  New  electricity  plant using fossil fuels 

Programme  indicators 
(Related to the  Intervention 
and its  effects) 

Context  indicators 
(Related to the  assisted  area) 

Output  Indicators 

Progress 

Compliance with  project  duration 
Rate of  completion 

Capacity 

kW of  installed  power 

Overall national electric  power 
production (MW), broken down by 
energy  sources 

Investment  efficiency 

Investment  unit cost per kW of  installed 
power 

Unit cost per  kW of  installed  power 
for  similar  investments 

Result  Indicators 

Population  served 

Quality of service 

Cost for final users 

Impact  indicators 

Value added 

Estimated  number  of users  on the  basis 
of yearly  average  coefficients  of  energy 
consumption  for  domestic  use and the 
annual  production  of the  plant  (kWH) 

Number  and duration of  interruptions of 
electric  power  distribution  for the 
average  user  connected to the  network 
to which the new plant  is linked 

Estimated reduction in unit cost of 
energy  purchase  expressed  in 
euros  /  kWH 

Average  energy  consumption  per 
head per  year. 

Number  and duration of  interruptions 
of electric  power  distribution  for the 
average  national  user 

National average  unit cost of  energy 
purchase expressed  in euros  /  kWH 

Value added  generated  by the  plant 
(euros  /  year) 

Yearly value  added  in the  electric 
power  production  and  distribution 
sector  (1) at  national  level 

Permanent  employment 

Permanent  employment  created  by the 
operation of the completed  factory 

Indirect economic  effect 

Depletion of natural  fossil 
fuel  resources 

Yearly average amount  of fossil fuels 
necessary to  operate the  plant 
(Tonnes of Petroleum  Equivalent) 

% of  managers  in the area who 
declare that cost or availability of 
energy is a major  problem for their  firm 

(1)  class  40.10  of  NACE  (Nomenclature  statistique  des  Activités  économiques  dans  la  Communauté 

Européenne). 

Indicators for monitoring and evaluation 

Example 8:  Constructing or upgrading an 

electric power 

distribution network 

Programme  indicators 
(Related to the  intervention 
and  its  effects) 

Context  indicators 
(Related to the  assisted  area) 

Output  Indicators 

Progress 

Quantity 

See  Example  7 

km  of new  /  upgraded lines, broken 
down by kW  categories 

Overall  network  length in the area 
broken  down  by kW categories 

Investment  efficiency 

Investment  unit  cost per  km  of  new  line 

Unit cost per  km  of new  line for 
similar  investments 

R e s u lt 

I n d i c a t o rs 

Population  served 

Quality of  service 

Impact  indicators 

Number  of  users  connected  to the  new 
network  (or  new  network  portion), 
broken  down  in domestic  /  economic 
units 

Number  and duration of  interruptions of 
electric  power  distribution  for  the 
average  user  connected to the  new 
network  or part of  network  which  is new 

Number  and duration  of  interruptions 
of  electric  power  distribution  for  the 
average  national  user 

Value  added 

See  Example  7 

See  Example  7 

Permanent  employment 

See Example 7 

Consumption 

Overall yearly  consumption  of  electricity 
by users  connected to  the new  network 
(kWH) after  one year 

Average  yearly consumption of 
electricity  at  national  level 
(kWH  /  connected  resident, 
kWH  /  connected job). 

Indirect  economic  effect 

1 See Example 7 

Energy infrastructure 

Example 9:  Upgrading  a gas distribution  network 

Programme  indicators 
(Related to the  intervention 
and  its  effects) 

Context  indicators 
(Related  to the  assisted  area) 

Output  Indicators 

Progress 

Quantity 

See Example  7 

km of  new  /  upgraded  pipes, broken 
down  by diameter  categories 

Overall network  length  In the area 
broken down  by diameter  categories 

Investment  efficiency 

Investment  unit cost  per km of new pipe 

Unit cost per  km of  new pipe for 
similar  investments 

Result  Indicators 

Population  served 

See Example  7 

Consumption 

Quality of  service 

Overall yearly  consumption  of gas by 
users connected to the  new  network 
(m3) 

Average  yearly consumption  of gas at 
national  level (m3  /  connected 
resident, m3  /  connected job). 

Number  and duration of  interruptions 
of gas supply  for the average  user 
connected to the new  network 

Number  and duration of  interruptions 
of gas supply for the average  national 
user 

Impact  indicators 

Value added 

See Example  7 

Permanent  employment 

See Example 7 

Indirect economic  effect 

(1) class  40.20  of  NACE 

Yearly value  added  in the  gas 
production  and distribution  sector  (1) 
at  national  level 

See Example 7 

See Example 7 

Other  interventions  having  an  impact  in the  field 

All  structural  interventions  aim  at  increasing  economic  activity,  either 
directly  or  indirectly.  Consequently  they  play  a  role  in  determining  higher 
in  terms  of  electricity)  and  therefore  the 
energy  demand  (especially 
these 
expansion  of 
consequences  are an intended effect  of Structural  Funds, they do  not  need 
to be monitored  or evaluated through  specific  indicators. 

the  energy  sector.  However,  since  none  of 

Availability  and  collection  of  data 

The initial providers of information for the quantification of indicators are the 
operators  of  the  new  infrastructure  (during  the  construction  period,  at  the 
moment  of  project  completion,  during  the  management  of  the  new 
infrastructure). 

Indicators  for monitoring  and evaluation 

Specific  surveys  among  the  programme  addressees  or  recipients  are  not 

frequent  in the  energy  sector  since  it  is  difficult  to  track  consumers  benefiting 

from  new  investments  in  cases  of  complex  interconnected  networks. 

Interesting  statistics  can  be  obtained 

from  national  and 

local  bodies 

m a n a g i ng  electric  p o w er  p r o d u c t i on  plants  and  electricity  and  gas 

distribution  networks,  as  well  as  from  statistical  offices.  As  Box  15  shows, 

the  available  statistics  provide  data  for  the  calculation  of  most  of 

the 

context  indicators  previously  identified. 

Box  15  - E x a m p le  of  statistics  available  f r om  Eurostat  at  regional 

level  (NUTS  II) 

Total electricity  production  capacity 

Electricity  production  capacity  by  source 

Total gross  installed  capacity 

Gross  installed  capacity  by  source 

Energy  production  by  source 

Electricity  production  by  source 

Gas  production 

Production  of  coke,  lignite, oil 

Total electricity  consumption 

Electricity  consumption  by  sector 

Accounts  of  companies  in the energy  sector 

CORINAIR  statistics  at  national  level  and,  in  some  countries,  at  sub-national 
level 

Key  indicators 

The  key  indicators  presented  below  lend  themselves  to  comparisons,  both 

between  different  interventions  of  the  same  programme  and  between  those 

of  different  programmes,  and  can  in  certain  cases  be  aggregated  at  a 

European  level. 

Energy infrastructure 

Level 

Key indicator 

Resource 

Rate  of  consumption  of  budget  (%  of  allocated 
%  of  budget  devoted to  environmental  mitigation  measures 

funds) 

Output 

Result 

Impact 

Rate of completion  of  project  (%  of  objective) 
Compliance  with  project  duration 

Number  of  users  connected  to  the  new  infrastructures,  broken 
down  in domestic  /  economic  units 

%  of  managers who  declare that  cost  or availability  of energy is 
a  major  problem for their  firms 
Net  employment  created  or  maintained  (FTEs  of  which  held  by 
women) 

For  more  information  see... 

EPA  (1996),  Joint  EMEP  I  CORINAIR  Atmospheric  Emission  Inventory 

Guidebook.  Copenhagen: EPA. 

Eurostat  (various years), Energy - yearly statistics.  Luxembourg:  Eurostat. 

Eurostat,  Renewable  energy  sources  statistics  1989-94.  Luxembourg: 

Eurostat. 

Eurostat, Energy balance sheets  1993-94.  Luxembourg:  Eurostat. 

Training 

The  main objective  of  structural  interventions  in this  domain  is to  promote 
the  employability  of  individuals  across  the  entire  European  Union,  by 
adapting  and  modernising  the  Member  States'  systems  of  education, 
training  and  employment. 

the  obsolescence  of  their  qualifications. 

It  has  been  observed  that  workers  who  have  been  in  a  particular  area  of 
employment  over  a  long  period  have  difficulty  adapting  to  the  new  job 
market  due  to 
Industrial 
restructuring  and,  more  generally,  massive  redundancies  have  created  a 
large  number  of  long-term  unemployed. The  need  to  link  training  and  the 
search  for  new  employment  has  thus  become  very  clear.  Young  people 
looking  for their first jobs  also  experience  difficulties,  especially  those  with 
few skills or recognised qualifications. Their rate of  unemployment  is higher 
than the average  rate for  adults. 

Training  takes  many  forms:  basic  training  at  the  outset  of  a  career,  the 
renewal  or  acquisition  of  skills  throughout  working  life,  academic  or 
professional  training, alternated  or  not  with  periods  of  employment.  It  may 
also  be  dispensed  within  the  framework  of  assistance  to  young  people  or 
the  unemployed  to  support  their  social  and  professional  integration.  In  all 
cases,  it  is  aimed  at  providing  skills,  qualifications,  knowledge  and  know-
how to individuals so that they can use them as a basis for negotiation  in the 
job  market. 

All  training  activities,  irrespective  of  their  nature  and  public,  may  be 
measured  by  means  of the same  indicators.  On the  other  hand, the aim  of 
the  programme  determines  which  indicators  should  be selected. 

Interventions  within the  field 

Resource  indicators 

Resource indicators must highlight very clearly the distribution of European, · 
national,  regional  and  local  co-financing.  Training  programmes  very  often 
straddle  two  calendar  years,  creating  a situation  that  requires very  precise 
financial  statements. 

Output  indicators 

Output  indicators are peculiar to the types of training  delivered: surface area of 
the training centres, number of teachers, number of courses offered, number of 

Indicators  for monitoring  and evaluation 

trainees  (to  monitor  the  rate  of  attendance  of  the  programmes),  volume  of 
training  provided  (if  the  funding  of  the  intervention  is  directly  related  to  the 
number of hours of training and the number of participants). These indicators are 
relatively  easy to quantify  as programme  indicators. The number of  participants 
can be expressed as a proportion of eligible persons (coverage rate). 

Context  indicators  are  difficult  to  obtain  because  the  supply  of  training 
courses is not always transparent.  It may  be delivered  by operators with low 
visibility such as small companies,  associations  or institutions which do  not 
take  advantage  of  public  funds  and  do  not  systematically  declare  all  their 
training  activities.  In  contrast,  suppliers  of  training  courses  who  depend 
directly  on the  major  public  actors  occupy  most  of the field  when  it  comes 
to  making  their  activities  known. 

Result 

indicators 

The  addressees  or  recipients  of  assistance  for  training  are  individuals  who 
attend  training  courses  and  who  improve  their  qualifications,  skills  and 
immediate  employability.  The  most  suitable  indicator  is  the  "number  of 
participants  on  training  courses  who  obtained  the  required  qualification". 
Although few  analyses  have focused  on the  conclusion  of training  periods, 
the  qualification  rate  provides  a  robust  indicator  of  the  number  of  people 
who attended the training course  right  up to the end, even though  someone 
who  drops out  mid-way  may  have done so to  be recruited. Finally, the  "rate 
of  participation  during  the training"  is proposed  as an  indicator  that  can  be 
used to  evaluate  absenteeism. 

Employability  would  be  a  good  result  indicator  if  it  could  be  quantified.  In 
theory,  employability  is the ability  of  an  individual to find  a job  or to  make a 
success  of  a  professional  position.  This  ability  depends  on  individual 
aptitude  and  particularly  on  capacities  acquired  or  enhanced  during  the 
training.  It  also  depends  on  the  probability  of  a  job  offer  occurring  that 
corresponds  with  the  capacities  acquired.  The 
latter  dimension  of 
employability  is very difficult  to define  and measure. To date, no  recognised 
method  exists  for  doing  so  and  no  harmonised  definition  of  employability 
has  been  established. 

Impact 

indicators 

The  indicators  which  have  received  the  most  attention  in  many  countries 
are the  "rate of employment  after a period of training", the  "placement  rate" 
or  the  "average  period  of  unemployment  after  the  training".  Yet  these 

Training 

indicators  are  not  sufficient  for evaluating  an  impact  in terms  of  a return  to 
employment  because  the  best  trainees  often  qualify  for  the  most  effective 
training  (skimming-off  effect). 
It  is  therefore  difficult  to  say  whether 
recruitment  after training  is really  an impact  of the training  itself.  It may just 
as  easily  be  a  simple  effect  of  the  intrinsic  quality  of  trainees  who  have 
completed  the  course.  The  quantification  of  the  rate  of  placement  in 
employment  must  be carried out at a sufficiently  distant time-horizon  (6, 12 
or  18  months  after  the  end  of  the training)  and  must take  into  account  the 
sustainability  of  jobs.  Theoretically,  it  can  never  be  guaranteed  that  the 
training  was  the  direct  and  unique  cause  for  occupational  insertion. 
Conversely,  in  a situation  of  job  scarcity,  the  fact  of  not  finding  work  after 
completing  a  training  course  should  not  be  interpreted  as  a failure  of  the 
programme. 

A training course which  has a low  placement  rate may  nevertheless  have a 
significant  intermediate  impact  in terms  of  placement  of  trainees  into  more 
advanced  training. This  impact  may  be  monitored  by  a  "rate  of  transition" 
indicator  which  records  all  the  trainees  who  have  improved  their  situation 
(placement  in terms  of work  or education)  after  the training  course. 

Many  studies  focus  only  on  the  effect  on  the  placement  of  trainees  but 
there  are  other  domains  to  consider  such  as  socialisation  at  work  or  the 
ability  to  cope  with  the  everyday  requirements  of  the  workplace  (e.g. 
timekeeping, attitude, etc.). 

Some  effects  may  be  far  off  target  compared  to  initial  objectives.  For 
example,  when  an  effort  is  made  to  help  the  unemployed  to  find  jobs,  the 
result may be an influx of previously  inactive people into the job  market. The 
effect 
is  an  increase  in  the  unemployment  rate,  but  the  effect  on 
employment  may, in fact,  be positive. 

An  important  impact  occurs  for  those firms  or  other  organisations  (indirect 
addressees  or  recipients)  which  employ  individuals  who  have  qualified  on 
the training courses, and thus avoid  paying  part  or all of the costs  involved 
in recruiting and training an unskilled  person. The ultimate impacts for firms' 
are  difficult  to  measure.  They  can  be  conceived  theoretically:  better 
research  and  development,  better  quality  procedures,  better  performance, 
greater  value  added,  etc.  It  is,  however,  impossible  to  trace  the  effects  of 
training  so far downstream. 

In the  context  of  training,  a  good  indicator  can  be  analysed  by  means  of 
graphs and comparisons. A breakdown  of  individuals  in the  programmes  -

Indicators for monitoring and evaluation 

by gender, qualification or professional situation ­ can provide a lot of 
information about the success of training courses and the interest shown in 
them by different categories of people. The evolution of the "proportion of 
unemployed in the cohort of qualifying trainees" is a good indicator that is 
too often forgotten when more readily available indicators are chosen. 

Example 10: Financing of training for new trades 

for the long­term unemployed 

Programme indicators Context indicators 

(in relation to the intervention and (in relation to the entire target public) 

its effects) 

Output indicators 

Supply 

Volume of training 

Number of trainees 
enrolled 

Rate of coverage 

Result indicators 

Rate of success 

Number of training places offered 
(¡nel. number intended for women) 
Number of training courses 
(occupations) offered 

Number of hours of training delivered 
(hours χ trainees) 

Number of trainees enrolled on the 
course (incl. number of women) 

Number of trainees enrolled, as a % of 
the number of long­term unemployed 
in the region 

Percentage of trainees who qualified 
with the required capacities (incl. % 
of women) 

Immediate placement 
rate 

Rate ot recruitment during training 
(incl. number of women) 

Impact indicators 

Transition rate 

% of trainees who completed a more 
advanced course, after 12 months 

Number or long­term unemployed 

Placement rate 
12 months 

% of trainees employed after 
12 months 

Annual rate of recruitment after a 
period of long­term unemployment (1) 

Sustainable placement 
rate 

% of trainees employed after 12 months 
and who are still employed after 
24 months (incl. % of women) 

(1) Unemployment is considered to be long­term if it has lasted for over 18 months. 

Training 

Example  11:  Skills improvement  programme for young  people 

with few  qualifications^) 

Programme  indicators 
(in relation  to the  intervention  and 
its  effects) 

Context  indicators 
(in  relation to  the  entire target  public) 

Output  indicators 

Supply 

Result  indicators 

Adaptation of  training 

Success  rate 

Impact  indicators 

Number  of  trainees 
qualifying 

Salaries of the trainees 
recruited 

Number  of training  places  proposed 
by the  programme 

%  of  places  offered  corresponding  with 
growing  sectors 

%  of  young  people who are  trained 
for  growing  sectors 

Number  of trainees  qualifying  / 
number  of trainees enrolled on the 
training  course (incl. number  of  women) 

Number  of trainees  trained 

Average  monthly  salary of trainees 
employed after  12 months  (average 
for  women /  men) 

Number  of young people with  a  low 
level of  skills 

Average  monthly  salary of  young 

people 

(1) A  "young"  person  is  someone  who  is  under  25.  He  or  she  has  a  "low"  level  of  skills  if  he  or  she  has 
international 

that  can  normally  be  obtained  at  the  age  of  18.  The 

none  of  the  qualifications 
classification  of  qualifications  used  in  "workforce"  surveys  can  also  be  referred  to. 

Indicators  for monitoring  and evaluation 

Example  12: Assistance  to  businesses for  adapting 

their  employees'  skills 

Output indicators 

Activity  of  the  operator 

Number  of  assisted 
businesses 

Number  of  training 
courses funded 

Result  indicators 

Qualifications  obtained 

Impact  indicators 

Maintenance  of 
employment 

Professional  progress 

Programme  indicators 
(in  relation  to the  intervention  and 
its  effects) 

Context  indicators 
(in  relation  to  the  assisted  area) 

Number  of  projects  prepared  with the 
candidate  businesses 

Number  of  businesses  with  a  training 
programme  in the  area 

Number  of  businesses  that  have 
received  support  for  training 

Number  of  employees  who  have 
received  more than  10 days of  training 

Annual % of  employees who  benefit 
from  in-house  training  in the  area 

Number  of  employees  who  have 
obtained  a qualification  in trades  in 
growth  areas 

Number  or %  of  individuals  who 
attended  the  training  course  and  were 
not  made  redundant  within  12  months 
(incl. number  or %  of  women) 

Number  or % of  individuals  who 
attended the training  course and  who 
were  promoted  (incl. % of  women) 

Number  of  redundancies  in the  area  / 
year 

Other  interventions  having  an  impact  in the  field 

The  effects  of  some  interventions  merge  with  those  of  training,  particularly 
at  the  national  level  (e.g.  reduction  in  social  charges  for  employers,  tax 
reforms,  incentives  for  recruitment  or  any  combination  of  these  three 
financial  measures  favouring  businesses).  Training  may  also  be  closely 
linked  to  the  growth  of  the  information  society.  For  example,  a  very 
peripheral  area  might  have  access  to  occupational  training  by  means  of 
electronic  communication. 

Availability  and  collection  of  data 

Only  (temporal)  monitoring  data  used to  observe  individuals  over time  can 
allow  for the  construction  of  coherent  indicators.  No training  action  can  be 
indicators  during  or 
evaluated  solely  by  means  of  easily  available 
these  data  varies 
training.  The  accessibility  of 
immediately  after 
considerably  and  they  cover  periods  of  unequal  length  in  different  the 

Training 

countries.  However,  from  the  point  of  view  of  pragmatism,  the  cost  of 

collecting  them  is  proportionally  very  low  compared  to  the  cost  of  the 

training  itself. 

Depending  on  the  complexity  and  heterogeneity  of  the  programme,  the 

authorities  responsible 

for 

it  have  to  choose  between  exhaustive 

quantification  by  the  operators  and  quantification  based  on  surveys.  Box 

16  illustrates  a very  specific  case  in which  it  is  possible  to  collect  most  of 

the  information  through  the  operators. 

Box  16 -  Training  and  survey: proposed  schema  for  data  collection 

A  scheme  for  collecting  information  consists  of  asking  trainees  to  fill  in  a 
questionnaire  at the beginning  of the course. This  questionnaire  could  even be 
attached  to the registration  procedure  when there  is one.  It might  include all  the 
relevant questions concerning the person's  experience, qualifications  and, above 
all,  motivations  and aspirations.  These  variables  are essential  for  judging  the 
training course and enabling the trainees to achieve their  goals. 

It  is  essential  that  the trainees  also  answer  another  questionnaire  when  they 
complete  the training  course.  In  this  way  questions  can be  asked  about  the 
course  itself,  even  if it  seems  that  the information  thus  obtained  will  be of  little 
value  since  it  may  be  biased  by  enthusiasm  if  the  course  went  well  or by 
resentment if it did not. If a qualification is obtained at the end of the course, these 
questionnaires  can be handed  out before the final  exam. The main  purpose  will 
then be to see whether the addressees or recipients'  aspirations  have changed. 

During the third  phase of the  survey, 6 to 24 months  after  the end of the  training 
course, the impact  indicators  can be quantified, possibly on the basis of a sample 
of former  participants. 

While  this  survey  is  longitudinal  by  construction,  only  the third  phase  involves 
substantial costs. The best  possible use is made of the presence of addressees or 
recipients  at  the  launch  and  conclusion  of  the  training  course.  Moreover,  all 
questions  about  opinions  are meaningful  because  they  are asked  in the present 
and the people questioned have not had time to rationalise their choice of answers. 

•  Key  indicators 

The  key  indicators  presented  below  lend  themselves  to  comparisons  both 

between different  interventions  of the same programme  and between  those 

of  different  programmes,  and  can  in  certain  cases  be  aggregated  at  a 

European  level. 

Indicators for monitoring and evaluation 

Level 

Key indicators 

Resources 

Rate of real spending of available funds (% of budget allocated) 

Outputs 

Results 

Impacts 

Number of training courses financed directly (1) (incl. no. of 
women) 
Success rate in reaching the eligible public (2), 
Hours of services and training received by the addressees or 
recipients (incl. number for women)(3) 

% of trainees who belong to a priority public (e.g. jobless young 
people) 

Sustainable placement rate (4) (% of addressees or recipients 
who are employed after 12 months, incl. % of women) 
Rate of transition (% of addressees or recipients whose social 
situation has improved after 12 months, incl. % of women) 

(1) Number of trainees who started or completed the training course. 

(2) This indicator, also called the rate of coverage, is used to show whether the training effort was 

significant or not in relation to the problem requiring to be solved. For example, the percentage of 
individuals who were long­term unemployed at the start of the year and who were trained during the 
year will be measured. 

(3) The volume of training received (number of hours χ trainees) is used to compare the real importance 

of the training effort that was financed. 

(4) The sustainable placement rate is the most frequent, albeit highly Imperfect, measurement of the 

success of the training delivered. In order to facilitate comparisons, the date of quantification must be 
homogeneous. 

For more information... 

European Commission (1997), Key Figures on Vocational Training in the 
European Union. Luxembourg: Eurostat and CEDEFOR 

Grubb W.N. and P. Ryan, (1997), Plain Talk on the Field of Dreams: The 

Roles of Evaluation for Vocational Education and Training, Geneva: 
International Labour Office, Training Policies Branch. 

1RES (La revue de), 1995. 'Comparaison européenne des dispositifs d'insertion 

professionnelle des jeunes' in: La revue de l'IRES, numéro 17, hiver. 

Schmidt G., O'Reilly J., Schömann, Eds (1996) International Handbook of 

Labour Market Policy and Evaluation. Edward Elgard Publishing Ltd, 
Cheltenham 

Werquin P. (1997), 'Dix ans d'intervention sur le marché du travail des 

jeunes en France, 1986­1996' in: Économie et Statistique, Volume 4 
/ 5, 121­306. 

Tourism 

The three  Structural  Funds  intervene  in the  field  of  tourism  in  the  form  of 
infrastructure  provision, support  for  the  creation  or  improvement  of  tourist 
accommodation,  training  in  tourism-related  occupations,  as  well  as  agri-
tourism  in  rural  areas.  Most  regional  programmes  and  certain  Community 
Initiatives  (e.g. LEADER)  involve tourism.  Moreover,  roads and  airports  add 
to  the  attractiveness  of  a  territory 
for  tourists.  Other  Community 
interventions,  particularly  those  contributing  to  the  improvement  of  the 
environment  or  to  sustainable  development,  contribute  to  dynamism  of 
tourist  activities. 

Many  territories  affected  by  rural  or  industrial  decline  have  developed 
investment  strategies  in tourism. The  economic  impact  of these  strategies 
is sometimes  uncertain  and  is  worth  evaluating  carefully.  In  particular,  the 
real weight  of the different  components  of  investments  in tourism  must  be 
taken  into  account.  Accommodation  accounts  for  by  far  the  largest 
contribution  to regional economies  (an average of  75% of  investments  and 
90%  of jobs  created  in tourism). 

The  intervention  of  Structural  Funds  in tourism  has  peculiarities  that  need to 
be recalled. Assisted operations (accommodation, infrastructure, employment) 
are  usually  small,  which  means  that  actions  tend  to  be  highly  scattered. 
Eligible zones are often non-urban areas where the number of assisted sites is 
high.  The  operators  may  have  a  variety  of  very  different  status:  private 
individuals, firms, local authorities or semi-public  organisations. 

Interventions  in the  field 

Resource  indicators 

The  standard  indicator  of  "actual  expenditure  of  funds"  is  applicable  to 
tourism.  Where  relevant,  the  heading  "tourism"  should  appear  in  sectoral 
breakdowns  of  resource  indicators.  This  is  also  valid  for  the  other 
categories of indicators discussed below. This relative weight of  investment 
in tourism,  proposed  in the  programme,  may  be  compared  to  the  relative 
weight  of tourism  in the activities  of the  territory. 

Output  indicators 

A  programme  for  improving  the  supply  of  rooms  or  tourist  lodges  may  be 
carried out at the initiative of private individuals or a municipality.  In the first 

Indicators  for monitoring  and evaluation 

case,  the  private  person  is  the  addressee  or  recipient.  He  or  she  receives 
public  support  through  the  intermediary  of  a  programme  operator.  The 
construction  of the  lodge  is a decision  of the  addressee or  recipient.  In the 
second  case, the  municipality  can  be considered  either  as an operator  -  if 
it  allocates  support  to  other  organisations  or  to  private  individuals  -  or  an 
addressee or recipient -  if it is directly involved as the constructor.  Whatever 
the  case,  the  construction  of  the  lodge  is  financed  by  the  programme. 
Consequently,  it  is an  output. 

An  output  indicator  is  the  "number  of  subsidies  granted".  This  indicator 
may  be  broken  down  into  private  individuals  of  various  activities  and 
professionals  in the tourist  trade, themselves  broken  down  into  individuals, 
small  and  large  enterprises.  The  term  enterprise  should  be  taken  to  mean 
an  "economic  unit",  irrespective  of  its  status. 

Other  output  indicators  include  the  "amount  of  investments  made  by  the 
addressees  or recipients"  as well as the capacity  of these  investments (e.g. 
the  maximum  number  of  visitors  /  day).  These  indicators  are  important 
since in terms of the time required to recoup the cost of investments  -  over 
15  years  -  tourism  is  very  similar  to  heavy  industry.  Because  support  is 
always  attributed  by  operators,  output  indicators  include  the  activity  of 
these  operators  (e.g.  number  of  market  studies  carried  out;  number  of 
potential  addressees  or  recipients  contacted; rate of  selection  of  projects). 

Result 

indicators 

Result 
indicators  relate  to  the  increase  in  the  quality,  owing  to  the 
intervention,  of  what  is  on  offer  to  tourists.  Quality  can  be  measured,  for 
example, by compliance with  established standards  and by the  certification 
of  tourist  lodges.  In farming  areas, the  "number  of farmers  whose  share  of 
turnover  generated  by tourism  amounts  to  at  least  30%",  reflects  changes 
in supply  structures  through  the  diversification  of  agricultural  activities. 

Impact 

indicators 

Specific  impact  indicators  measure  the  number  of  tourist  visits  to  assisted 
facilities  or  accommodation  (number  of  nights  accommodation  sold  per 
year, number of visits per year, rate of occupation for different  periods of the 
year). Tourist  visits  must  be  measured  after  an adequate  period  of time  (12 
months  seems to  be a minimum). Indicators of the number  of visits must  be 
interpreted  with  caution  because  custom  in  the  tourist  market  tends  to 
fluctuate. The  general  economic  context,  the  weather,  the  development  of 

Tourism 

sporting  activities, and competition  from foreign destinations  at  low  prices, 
all add up to make the market unstable. It may be interesting to break  down 
the  number  of  visits  in  terms  of  the  origin  of  the  visitors  (nationals  of  the 
country  or  foreigners)  in  order  to  assess  the  programme's  contribution  to 
the attractiveness  of the  region  for  tourists. 

An  indicator  of  value  added  (in  practice  the  value  added  is  the  sum  of 
wages, salaries  and profits) can  be applied to  economic  units supported  in 
the  domain  of  tourism.  It  generally  consists  of  the  gross  value  added 
generated by projects  in euros  /  year. Value added is an interesting  indictor 
because  it  specifically  measures  economic  development.  It  allows  the 
factors  that  are  likely  to  cause  a territory  to  move  from  an  area  of  tourist 
activity  towards  a tourist  economy  to  be highlighted. 

The "number  of jobs created /  maintained" applies to tourism, provided  the 
seasonal  character  of  the  activity  is taken  into  account  (e.g.  estimated  in 
terms of job-days). It is also worthwhile evaluating the skills requirements of 
jobs  created  because  the  tourist  economy  requires  trained  and  adaptable 
personnel.  The  natural  environment  and  the  tourist  sites  are  no  longer 
enough; the gap with competing territories will widen in direct relation to the 
levels of training  of  personnel  in tourist  businesses. 

Value added and employment  indicators  can be linked to context  indicators 
such as average value added  per job  in the tourist  sector. 

Indicators  for monitoring  and evaluation 

Example  13: Support for the  creation  and  improvement 

of  accommodation 

Programme  indicators 
(in relation  to the  intervention  and 
its  effects) 

Context  indicators 
(in relation  to the  assisted  region) 

Output  indicators 

Activity  of  operators 

Number of  addressees 
or  recipients 

Number  of  contacts  with  potential 
addressees  or  recipients 

Number  of  economic  units  assisted 

Capacity 

Number  of  beds created or  improved 

%  of  beds  created  or  improved  which 
have a quality  label 

Number  of farmers  whose share of 
turnover attributable  to tourism  is at 
least  equal to  30% 

Total number  of  units of  tourist 
accommodation 

%  of  beds which  have  a quality  label 

Number  of  nights sold  per  year  and  per 
bed in assisted  accommodation,  after 
one  year 

Evolution  of  consumption  of  electricity 
in the  territory  during  the  tourist 
period 

%  of  nights  sold to foreign  tourists 

Number of  charter  flights  in the 
regional  airports 

Number  of  nights  in an average stay 
in assisted  accommodation 

Number of  nights  in an average stay 
in the  territory 

Value added generated  in euros  /  year 
/  bed, after one year 

Net number of jobs  created (incl. % 
occupied  by  women) 

Average  value  added  in euros  /  year 

/  bed 

Number of jobs  in the tourist  sector 

Result  indicators 

Quality 

Diversification 

Impact  indicators 

Number  of  visits 

Attractiveness  for 
foreign  tourists 

Length of  tourist  visits 

Value  added 

Jobs  created 

Tourism 

Example  14:  Support for the creation of tourist facilities 

(museums, amusement  parks, etc.) 

Programme  indicators 
(in relation  to the  intervention  and 
its  effects) 

Context  indicators 
(in relation  to the  assisted  region) 

Output  indicators 

Activity  of  Operators 

See Example  13 

Number  of  addressees 
or recipients 

See Example  13 

Total number of tourist  facilities 

Capacity 

Maximum  number  of visitors / day 

Result indicators 

Length of  the  visit 

Cost of the  visit 

Impact  indicators 

Number  of  visits 

Attractiveness  for  foreign 
tourists 

Normal  length of  visit  to the  facility 
(in  hours) 

Average cost of a visit to the facility 
(in euros per person) 

Number  of visits  per year  to  assisted 
facilities 

%  of visits  by foreign  tourists 

Value added 

Value added generated in euros  /  year 

Value added generated  in the tourism 
sector  In euros  /  year 

Jobs created 

See Example  13 

See Example  13 

Other  interventions  which  have  an  impact  in the  domain 

Examples  of  the  multiple  interventions  which  may  have  an  impact  on 
tourism  include:  efforts  to  reduce  pollution;  cleaning  up  of  rivers  and 
beaches;  improvement  of  the  water  and  air  quality;  rearing  of  livestock  in 
mountain  areas  for  maintaining  uphill  grasslands:  upkeep  of  roads; 
extensification  of  livestock  rearing  as  opposed  to  more  polluting  intensive 
techniques;  improvement  of  farm  buildings;  maintenance  of  wetlands, 
cleaning  up  of  industrial  wastelands;  revegetation  of  former  mines  and 
foundries, restoration of the urban and rural heritage; opening up of  isolated 
areas; improvement  of  links with  peripheral  regions  and  of  roads, railways, 
sea  links,  airports,  ports,  local  transport  networks  and,  lastly,  cultural 
interventions  such  as  RAPHAEL  and  KALEIDOSCOPE.  The  following 
example  concerns training. 

Indicators  for monitoring  and evaluation 

Example 15: Funding for language training 

Programme  indicators 
(in  relation  to the  intervention  and 
its  effects) 

Context  indicators 
(In relation  to the  assisted  region) 

Output  indicators  (in the field of training) 

Quantity 

Number  of  places  offered 

Number  of jobs in tourism 

Number of  trainees 
enrolled in the  course 

Number of  trainees  enrolled 
(incl. number  of  women) 

Volume  of  training 

Number of  hours of  training  delivered 
(hours  χ trainees) 

Result indicators  (in the field of  training) 

Number  of  trainees 
trained 

Rate  of  satisfaction 

I m p a ct  indicators 

Use 

Number  of trainees  qualifying  with  the 
required  level  (incl. number  of  women) 

%  of  trainees  satisfied or  highly 
satisfied 

Proportion  of  trainees  who  use  the 
skills  acquired  in a tourist  activity  in 
the  region, after  one  year 

Availability  and  collection  of  data  on  training 

Statistics  for  context  indicators  have  to  be  collected  from  the  relevant 
organisations  and  local, regional  and  national  authorities.  In the  future,  the 
main  effort  will  have  to  focus  on  searching  for  elements  to  measure  the 
economic  dimensions  of  tourism  more  accurately,  e.g.  by  observing  the 
evolution of turnover of  activities  directly or indirectly  related to this  activity. 
This  is currently the  case  of  hoteliers  who  have grouped  together  to  create 
management  centres. 

Tourism  is  a  sector  in  which  many  statistics  are  not  comparable  without 
analysing  the  conditions  in which  the data  were collected.  For example, as 
far  as the  "number  of tourist  visits"  is concerned, accommodation  uses the 
notion  of  "number  of  visits  sold"  or  of  "nights  sold",  tourist  bureaux  count 
the number  of  persons who visit their offices, and nature reserves count  the 
number  of  vehicles  in the  park  multiplied  by the  average  number  of  people 
per  car. 

For  the  above  indicator,  an  analytical  framework  and  criteria  should  be 
suggested  to  all the territories  concerned  defining  exactly  how  the  number 
of  visits  should  be calculated. 

Tourism 

Key  indicators 

A  number  of  key  indicators  lend  themselves  to  comparisons  between 
different  interventions  in the same programme and with other  programmes. 
They can therefore easily  be aggregated  on a European scale. 

Level 

Key  indicator 

Resources 

Rate  of  real  consumption  of  available  funds  (%  of  budget 
allocated) 

Outputs 

Number  of  economic  units  which  have  received  direct  support 
or  a service  supported  by the  programme  (including  the  size of 
the  unit:  large, medium, small, individual) 
Number of new economic  units (less than a year old) which have 
the 
received  direct  support  or  a  service  supported  by 
programme  (including  size: large, medium, small,  individual) 

Impacts 

Value added  generated  (euros  /  year  /  employee) 
Net jobs created or maintained  (in full-time  equivalent,  including 
%  occupied  by women) 

Bibliographic  references 

Guesnier B. (1996), Le patrimoine,  une valorisation touristique  à haut  risque. 
Paris:  Institut  de  Prospective  du  Tourisme  et  Fondation  Auguste 
Escoffier. 

Ministère  du  Tourisme  (1996),  'Prospective  et  Tourisme'  in:  Les  cahiers 

espaces du  Ministère  du  Tourisme France,  Décembre. 

Fisheries 

The  aim  of  Structural  Fund  interventions  in  the  area  of  fisheries  is  to 
strengthen  the  sector  which  is  going  through  a  prolonged  period  of 
restructuring  resulting  from  the  increasing  need  to  reduce  overfishing  of 
limited  natural  stocks.  The  overall  objective  is to  accelerate  the  structural 
adaptation of the fishing industry  and allow the emergence of a sustainable 
European fishing  industry. 

Interventions  are  financed  by  the  Financial  Instrument 
for  Fisheries 
Guidance  (FIFG),  introduced  in  1993. This  instrument  was  conceived  with 
the  aim  of  bringing  together  all  fisheries  oriented  structural  measures,  in 
particular  those  concerning  the  promotion  and  marketing  of  fisheries  and 
aquacultural  products.  Interventions  in  this  field  include  financial  support 
for the restructuring of the fishing fleet, such as subsidies for the withdrawal 
from service of fishing vessels and payments to fishermen wishing to retrain 
or  to  take  early  retirement.  Other  interventions  directly  related  to  fishing 
include the modernisation  of fleets  and  harbour  facilities. 

Another  source  of  funding  is the  PESCA  Community  Initiative,  which  aims 
to  cushion  the  social  and  economic  consequences  of  restructuring  by 
supporting  the  diversification  of  economic  activities.  Diversification  is 
promoted  by  grants  to  support  productive  investment,  the  provision  of 
services  for  SMEs  (management,  technology  transfer,  marketing,  quality 
control, etc.) and subsidies to  promote tourism (tourist attractions, yachting 
and other  leisure activities). 

Interventions  within  the  field 

Resource  indicators 

Standard  resource  indicators  can  be  broken  down  in  order  to  provide 
information  on interventions  aimed  at: reducing fishing  capacity,  improving 
processing  and  trading  conditions,  developing  aquaculture,  modernising 
port facilities  and measures  supporting  retraining, new  businesses,  etc. 

Output  indicators 

In the  case  of  subsidies  targeted  at  reducing  fleet  sizes,  standard  output 
indicators  focus,  firstly,  on  the  activities  of  operators  (e.g.  number  of 
briefing sessions with  trawler  owners,  number  of counselling  sessions  with 
fishermen,  etc.). Secondly,  they  estimate  the  physical  manifestation  of  the 

Indicators  for monitoring  and evaluation 

subsidy, in particular, the tonnage of fishing vessels withdrawn from service. 
The number of ex-fishermen  receiving income substitution  payments is also 
an  indicator  of  the  reduction  of  fishing  activities.  Where  direct  support  for 
investments  is  concerned,  outputs  can  be  estimated  by  using  physical 
measures  (e.g.  surface  area  of  low  temperature  storage  facilities  built  or 
improved,  additional  productive  capacity  in the aquaculture  sector,  etc.). 

Contextual  indicators  provide  information  on  national  or  regional  fishing 
capacity: the size of the fishing fleet  and the rate at which  it is being retired. 

Result 

indicators 

The  results  of  interventions  designed  to  reduce  fishing  capacity  and 
manpower  can be estimated  by the catch  of withdrawn vessels  during  their 
last  year  of  service  and  the  retraining  of  ex-fishermen  (e.g. the  number  of 
ex-fishermen  working  in the  aquaculture  and  tourist  sectors). 

In  the  case  of  the  modernisation  of  port  infrastructure  indicators  should 
focus  on  factors  that  increase  both  the  potential  of  shipside  activity  (e.g. 
improved  handling,  turnaround,  etc.)  and  portside  activity  (e.g.  increased 
supply  and  of  quality  warehousing,  cold  storage, etc.). 

Impact 

indicators 

The type of impact  indicator  used generally depends  on the objective of  the 
intervention. Measures that aim at diversifying the local economic  base, the 
retraining  of  ex-fishermen  or  the  modernisation  of  infrastructure  and 
facilities  require  the  use  of  socio-economic  indicators  which  cover  jobs 
created  or maintained  (FTEs), enterprises  created, employment  of  retrained 
ex-fishermen,  sales  turnover,  value  added  (i.e.  wages  and  salaries  plus 
profits),  etc.  Those  interventions  aimed  directly  at  fishing  activities  and 
resource  conservation  will  focus  on  the  size  of  catches,  broken  down  into 
species  of  fish. 

In  the  case  of  interventions  to  modernise  port 
impact 
indicators  should  capture  the  effects  of  improvements  on  traffic  levels, 
catches  and  portside  conditions  (e.g.  better  sanitation,  fewer  accidents, 
etc.) 

infrastructure, 

According  to the  objective  of  the  intervention, the  relevant  area for  context 
indicators  may  either  be the  port,  the  eligible  area or  the  region  (for  socio­
like  jobs  or  value  added)  or  fishing  fields  (for 
economic 
environment  indicators  like sustainability  of fish  stocks). 

indicators 

Fisheries 

Example 16: Grants for aquaculture farms 

Programme  Indicators 
(Related  to the  intervention  and 
its  effects) 

Context  Indicators 
(Related to the  assisted  area) 

Output  indicators 

Number  of  addressees 
or  recipients 

Productive  capacity 

Result  indicators 

Leverage  effect 

Reconversion 

I m p a ct  i n d i c a t o rs 

Production 

Value added 

Labour  productivity 

Direct  employment 

Number  of farms  receiving  grants 

Additional capacity  of supported  farms 
(tonnes  /  year) 

Total output  in the  region 
(tonnes  /  year) 

Accompanying  Investment  by  recipient 
farms 

Number  of ex-fishermen  investing  in 
aquaculture 

Additional  production  of the  recipient 
farms after  12 months  (tonnes  /  year) 

Catch of  similar  species  in the 
relevant  fishing  fields  (tonnes  /  year) 

Value added  in the  recipient  farms after 
12  months 

Value added  in the  region 
/ in  the  sector 

Value added  per employee  of the farms 
after  12  months 

Average value added by employee In 
the  region  /  in the sector 

Number of  net jobs  maintained or 
created  in the  recipient farms  after 
12  months 

Number of jobs  In the  region  /  In the 
sector 

Indicators for monitoring and evaluation 

Example  17: Subsidies for  laying up fishing  vessels 

Programme Indicators 
(Related to the intervention and 
its effects) 

Context Indicators 
(Related to the relevant fishing area) 

Output  indicators 

Activity of the operator 

Number of vessel owners briefed 

Fleet tonnage 

Tonnage of fishing vessels laid up 

Fleet tonnage in the related fishing 
area 

Result  indicators 

Reduction of fishing 
capacity 

Impact  indicators 

Fish stocks 

Sustainability 

Catch of laid up vessels during their 
latest year of activity 
(tonnes  /  year  /  species) 

Total catch by species In the related 
fishing fields (tonnes  /  year) 

Estimated stock in the relevant fishing 
field(s) (thousands of tonnes by 
speciosi 

% of stocks fished which are above 
the MBAL(1) of the related fishing area 

(1) MBAL -  Minimum  Biological Acceptable  Level is an estimate of the mature stock  level of  a species. 
Below this  level there is an increasing  risk  that the  reproductive  potential of  the  stock  will collapse. 
High  percentages  of  stocks  fished  that  are below  the  MBAL indicate an  increased  risk  of  resource 
depletion. 

Fisheries 

Example  18: Modernisation  of a fishing  port 

Programme  Indicators 
(Related  to the  intervention  and 
its  effects) 

Context  Indicators 
(Related  to the assisted  area) 

Output  indicators 

Progress 

Capacity 

Result  indicators 

Compliance  with  project  duration 
Rate of  completion 

Port capacity  built or  rehabilitated 
(number  of vessels and tonnage) 

Turnaround 

| Average  turnaround time  of  vessels 

Impact  indicators 

Traffic 

Catch 

Business activity 

Improved hygiene 

Tonnage of vessels  using the  port  after 
one year 

Catch landed  (tonnes  /  year  /  species) 

Surface  area  (m2) of port-side  units 
bought or rented by traders, 
wholesalers, etc. 

Reduction in the number of  incidents 
in which  hygiene  regulations  have been 
breached 

Value added 

Value added generated in the  port area 

Employment 

Number  of jobs created or  maintained 
in the  port area (FTEs of which held by 
women) 

Value added  related to the  fishing 
industry  in the town  /  region 

Number  of jobs  In the fishing  industry 

(FTEs of which held by women) 

Other  interventions  having  an  impact  in the  field 

Numerous  interventions  in  other  fields  are  likely  to  produce  effects  in  the 
field of fisheries and aquaculture. For example, a treatment  plant that  brings 
an end to direct discharges of human waste in coastal waters will contribute 
to the quality of  beaches and therefore facilitate efforts to diversify the local 
economy  towards  tourism.  It  may  also  contribute  to  the  diversification  of 
economic  activities  by  allowing  the  development  of  aquaculture  (mussel 
farms,  etc.).  Another  case  could  be  the  setting  up  of  a  data  exchange 
system  linking  up  trawler  owners,  processing  plants,  and  wholesalers 
allowing  the  local  market  for  fish  to  function  more  efficiently.  Similarly,  the 
construction  of a new road link to the national motorway  system could have 
an impact on the transport costs of local firms which  process locally  landed 
fish, or improve tourist access to a previously  poorly connected  region. The 
example  presented  below  is the  case of  financial  support  for the  retraining 

Indicators  for monitoring  and evaluation 

of  fishermen.  These  courses  could  be  targeted  at  providing  skills  suitable 
for  becoming  self-employed,  (e.g.  business  plans,  book  keeping,  taxation 
and  grants, etc). 

Example  19: Support for  retraining  of  fishermen 

Programme  Indicators 
(Related  to the  Intervention  and 
its  effects) 

Context  Indicators 
(Related  to the assisted  area) 

Output Indicators (in the field of  training) 

Quality 

Number  of  recognised  qualifications 
available 

Volume of  training 

Amount  of  training  received  by trainees 

supplied 

Take up 

(hours χ trainees) 

Number  of  ex-fishermen  attending 

Total number  of  potential  trainees 

Result indicators  (in Ui 

e field of  training) 

training  courses 

Success  rate 

%  of  trainees  successfully  completing 
the  course 

Satisfaction  rate 

%  of trainees  who  are satisfied  or very 

satisfied  with the training  received 

Impact  indicators 

Retraining 

%  of  participants  who  are  employed 
outside  the fishing  sector  after 
12  months 

Unemployment  amongst  ex-fishermen 
in  region 

Availability  and  collection  of  data 

Operators  through  their  monitoring  systems,  as  with  other  fields  examined 
in this Volume, should  systematically  collect  output  and  results data. When 
the  co-financing  of  investment  projects  is  concerned,  operators  should 
strive  to  collect  data  on  the  amount  invested  by  private  sector  partners  in 
order  to  measure  any  leverage  effects  that  may  occur. 

Impact  related  data  have  to  be  collected  through  surveys 
involving 
addressees  or  recipients  and  the  wider  population.  Whilst  the  tracking  of 
addressees  or  recipients  who  undergo  retraining  and  consequently  leave 
the region may  pose some  problems, the  impacts of other  interventions  will 
generally  be geographically  localised. 

In the  case  of  context  indicators  relating  to  fishery  resources,  estimates  of 
fish  stocks  and  MBAL  are  made  by  the  International  Council  for  the 
Exploration  of  the  Sea  (ICES).  These  are  based  on  data  on  international 

Fisheries 

catches,  fishing  effort  and  estimates  from  research  vessels  on  the 
abundance  of  stocks.  Government  Ministries  for  agriculture  and  fisheries 
provide  national  data  on  fishing  catches.  A  statistics  annual  devoted  to 
fisheries  is published  by  Eurostat. 

Key  indicators 

The  key  indicators  presented  below  lend themselves  to  comparisons  both 
between different  interventions  of the same  programme  and between those 
of  different  programmes,  and  can  in  certain  cases  be  aggregated  at  a 
European level. 

Level 

Key  indicator 

Resource 

Rate of  consumption  of  budget  (%  of  allocated  funds) 

Output 

Result 

Impact 

Number  of  hours  of  services  (training,  consultancy,  etc.) 
received  by addressees  or  recipients 
Number  of  individuals  receiving direct  assistance  or  services  as 
a result of the programme  (of whom  men /  women) 
Number  of  economic  units  (firms,  ship  owners, fish farms,  etc.) 
receiving  direct  assistance  or  services  as  a  result  of  the 
programme 
Number  of  new  economic  units  receiving  direct  assistance  or 
services as a result  of the  programme 

Satisfaction  Rate  (%  of  addressees  or  recipients  satisfied  /  very 
satisfied by services  provided) 
Immediate  success  rate (%  of  addressees  or  recipients  no  longer 
requiring services at the end of their participation in the programme) 

%  of  addressees  or  recipients  employed  after  12  months 
%  of  assisted  new  businesses  that  are  still  active  after  12  and 
36  months 
Gross  value  added  (after  12  months  in  terms  of  euros  /  year 
/employee) 

For  more  information  see... 

Commission  européenne  (1994),  Efude  comparative  et  intégration  des 
données  socio-économiques  dans  le  secteur  de  la  pêche  et  de 
la  Communauté.  Luxembourg:  Office  des 
l'aquaculture  dans 
publications  officielles  des  Communautés  européennes. 

Indicators for monitoring and evaluation 

European  Commission  (1994),  Employment 

in  the 
industries  of  UK,  Denmark  and  Portugal.  Luxembourg: 

fishing 
Official  Publications  Office  of  the  European  Commission. 

impact  assessment 

Eurostat  (1997), Pêche  -  Statistiques  annuelles.  Luxembourg:  Eurostat. 

Department  of 

the  Environment 

(1996), 

Indicators  of  Sustainable 

Development  for  the  United  Kingdom.  London:  HMSO. 

OECD (1997), Review  of  Fisheries  in  OECD  Countries  1995.  Paris: OECD. 

Research  and  Technological  Development 

Structural  Fund  interventions  in  the  field  of  Research  and  Technological 
Development  (RTD)  include  support  for  RTD  activities  in  enterprises, 
especially in SMEs, through the setting up of science and technology  parks, 
subsidies  and  loans for  the  development  of  new  products  and  processes, 
provision  of  services  such  as  technology  audits,  expert  advice  and  the 
creation  of  shared  data  sources  (e.g. to  give  enterprises  the  possibility  of 
finding suitable partners  in other European countries). They equally  concern 
the creation or  strengthening  of  research  institutes,  university  faculties  and 
public  laboratories, support for research projects, the financing of  individual 
researchers  and training  or  retraining  of  workers.  More  recently,  in order  to 
support the transfer of technologies to industry, interventions  have become 
research 
increasing  oriented 
organisations  and  enterprises,  and  developing  the  capacity  of  SMEs  to 
acquire and  profitably  use new  technologies. 

towards  encouraging 

links  between 

through 

The  aim  of  interventions  in  this  field  is  to  promote  economic  growth  and 
the  development  of  new  activities  based  on 
employment 
in  this  sense  means 
sustainable  technological  progress.  Sustainable 
creating  a  positive  and  self-reinforcing  dynamic  between 
innovatory 
capacity,  competitiveness  and  economic  growth.  In  order  to  do  this, 
interventions  generally  aim  to  assist  the  enterprises  and  workforce  of  a 
region  to  adapt  to,  and  profit  from,  the  opportunities  that  technological 
change  presents.  This 
involves  creating  and  adapting  necessary 
infrastructures,  supporting  training  and  education, promoting  collaborative 
research  where  this  is  relevant,  and  encouraging  a  region's  economic 
actors  to  move  towards  the  production  of  high  technology,  high  value 
added  products  and  services. 

Whilst the ERDF is the major source of funding, all the Structural  Funds are 
involved in funding  RTD interventions. These are also supported  by most of 
the  Community  Initiatives,  since  technological  development  represents  a 
major  precondition  for  reaching their  various  aims. 

RTD  has  a  number  of  particularities  that  shape  the  systems  of  indicators 
used to  evaluate  interventions.  Firstly,  RTD capacity  is highly  concentrated 
in a small  number  of  enterprises,  laboratories  and  universities  that  tend  to 
be clustered  in  and  around  capital  cities  and  a few  economically  dynamic 
regional  poles. Furthermore, RTD is polarised  in international terms with the 
majority  of  activity  taking  place  in a small  number  of  industrially  advanced 

Indicators  for monitoring  and evaluation 

institute-enterprise, 

nations.  The  success  of  RTD  projects  depends  to  a  large  extent  on  inter-
enterprise, 
inter-institute  and  cross-border  co­
operation.  The  diffusion  of  results  and  technology  transfer  must  occur  if 
RTD  is  to  benefit  not  only  the  institutes  and  firms  who  perform  it,  but  the 
region in which they are located. The results of  RTD activity  include not only 
products  and  processes,  but  also  knowledge  and  know-how,  which  is 
largely  embedded  in  individual  researchers  and  research  teams.  These 
"intangibles"  are  also  inputs  into  the  RTD  process.  Finally,  a  substantial 
amount  of  RTD is performed  by  public  agencies  (mainly  university  research 
centres  and  public  laboratories)  which  is  generally  an  input  into  industrial 
RTD performed  by firms. 

Interventions  within  the  field 

Resource 

indicators 

The  distribution  of  financial  resources  between  private  and  public  sector 
projects  gives  an  indication  of  the  capacity  of  a  region's  enterprises  to 
absorb  RTD funding. When  private sector  uptake  is low, this may  mean that 
the  capacity  of  enterprises  to  perform  their  own  RTD or to  profit  from  RTD 
performed 
laboratories  and  related 
institutions  is limited. The share of financial resources devoted to  innovatory 
projects  as opposed  to  institute-enterprise  or  international  /  inter-regional 
technology 
transfer  projects  gives  an  indication  of  the  capacity  of 
enterprises to acquire outside technologies.  Resource  indicators  should  be 
designed  to  reflect  these  issues. 

in  university  research  centres, 

Output 

indicators 

Outputs  of  RTD  interventions  can  be  in the  form  of  infrastructure,  services 
to  business  and  support  for  research.  In  the  first  instance,  monitoring 
focuses  on  physical  indicators  providing  information  on  the  number  and 
surface area of  high-tech  development  parks opened  or the surface  area of 
specialised  facilities,  usually  provided  for  small  high-tech  firms.  In  the 
public sector indicators  relate to laboratory and research centre floor space, 
the  amount  of  equipment  provided  (e.g. number  of  computers,  etc.). 

In the  case of services, indicators  relate both to the physical element  of the 
service  (e.g.  number  of  certified  experts  and  consultancies  providing 
technology  audits, diagnosis  and  other  services to  firms, etc.), to the  level 
of  service  activity  (e.g.  the  number  of  contacts  occurring  between  the 

Research and technological development 

operators and addressees or recipients of a programme) and to the volume 
of  services  provided  (e.g.  the  "number  of  hours  of  technology  audits 
undertaken", etc.). 

Result  indicators 

Most  benefits  arising from  RTD occur  after  a period  of  many  months, even 
years. For this reason, it is not easy to design result indicators that  measure 
immediate  effects. 

In the case of  RDT infrastructure  like science  parks,  result  indicators  could 
provide  information  on  the  quality  /  cost  ratio  of  the  equipment,  from  the 
standpoint  of  the  potential  users.  In the  case  of  services,  result  indicators 
basically  measure addressees'  or recipients' satisfaction. It  is also  useful  to 
measure  to  what  extent  the  supplied  service  matches  the  needs  of  local 
firms.  One  indicator  of  local  firms'  interest  is the  amount  of  private  money 
invested  in  the  supported  projects  (leverage  effect).  Another  interesting 
indicator  is the  proportion  of  addressees  or  recipients  launching  their  first 
RTD activity  with the support  of the  programme. 

One  of  the  problems  of  Structural  interventions  in the  field  of  RTD  is  that 
resources  tend  to  be  concentrated  in  public  research  bodies.  This  is 
because  initial  low  levels of  industrial  RTD performed  by firms  restrict  their 
ability  to  launch  RTD  projects  to  absorb  available funds.  This  exacerbates 
any mismatch that  may exist between technology  supply  and demand. The 
BERD  /  GERD3 ratio  is  a  contextual  indicator  of  industrial  RTD  intensity 
reflecting  the  capacity  of  private  sector  to  undertake  RTD  or  absorb 
funding. The  "ratio  of  researchers  (total, private,  public)  per  1,000  workers 
(FTEs)"  gives  similar  contextual  information.  Other  context 
indicators 
include  RTD as a percentage  of exports  and  as a percentage  of GDP. 

In the  case  of  technology  transfer  support,  the  "number  of  assisted  firms 
purchasing patents, or licences, or entering into collaborative  R&D ventures 
with  foreign  firms"  is  an  indicator  of  formal  cross-border 
technology' 
transfers. 

The  percentage  of  projects  approaching  the  marketing  stage (i.e. one year 
or less away) can give an indication of the proportion of the likely  economic 
benefits of the  programme. 

3  Business  Expenditure  on  Research  and  Development,  Gross  Domestic 

Expenditure  on  Research and  Development. 

Indicators  for monitoring  and evaluation 

Where  public  research  is  concerned,  results  of  interventions  to  support 
individual  researchers  can  be  measured  by the  number  of  supported  post­
graduate  researchers  successfully  completing  their  programme  of  research 
and  /  or  obtaining  a PhD. 

Impact 

indicators 

to 

rather 

to  measure 

In the  field  of  RTD evaluation  of  public  institutions  it  is customary  to  focus 
on  such  indicators  as  the  number  of  PhDs  and  bibliometric  measures, 
the  production  of 
notably  publications  and  citations, 
knowledge.  However,  Structural  Fund  interventions  in the field  of  RTD  now 
increasingly  emphasise  the  development  of  the  RTD  capacity  of  regional 
industry  rather  than  supporting  more  upstream  institute-based  research. 
Furthermore,  the  knowledge  produced  by  universities  and  research 
the  worldwide  stock  of  publicly  available 
institutes  contributes 
to  regional  economic 
knowledge 
than  contributing  directly 
development  and  social  cohesion.  It  is  in  these  circumstances 
that 
indicators  reflecting  an  academic  rather  than  an  industrial  logic  are  of 
limited  use.  Impact  indicators  for  Structural  interventions  which  do  support 
upstream  activities  should  therefore  concentrate  on  such  areas  as  human 
resource development,  industry-university  collaboration  and the  production 
of  potentially  marketable  products  and  services.  The  "number  of  co­
operative  R&D arrangements  between  institutions  and enterprises"  and  the 
"number  of  graduate  placements  in  firms"  are  indicators  of  collaborative 
effort.  Impacts  arising  from  supported  public  and  joint  research  projects 
can  be  measured  by  the  number  of  prototypes  or  demonstration  units 
produced  and the  number  of  patents  and  licences  sold. 

In the case of support for  industrial  RTD, interventions  are oriented  towards 
innovation  and  technology  acquisition.  This  can  be  measured  using  the 
the  acquired 
number  of  new  products  or  processes 
technology,  which  are marketed  by the  recipient  firm. 

incorporating 

Direct  employment  impacts  can be estimated  by using the  "number  of  R&D 
posts  created  or  maintained  within  the  firm"  and  the  "proportion  of  R&D 
personnel  employed  in  relation  to  the  total  workforce  (FTEs)".  Because  of 
the  long  lead  times  between  research,  development  and  the  marketing  of 
an 
the 
impact 
expectations  of  addressees  or  recipients. 

indicators  often  have  to  be  based  on 

innovation, 

Research and technological development 

Example 20: Sup sort for a science and technology  park for  SMEs 

Programme  Indicators 
(Related to  the  intervention  and 
its  effects) 

Context  Indicators 
(Related to the  assisted  area) 

Output  indicators 

Quantity 

Result  indicators 

Cost 

Scientific  attractiveness 

I m p a ct  i n d i c a t o rs 

Occupation 

Networking 

Direct  employment 

Surface  area (ha.) of  S&T(1)  park 
Floor  space  available  (m2)  in the  park 

Total  surface area  (ha.) of  S&T  parks 
in the  area 
Total  floor space availablelm2)  In S&T 
parks 

Cost of  establishing  a small  high-tech 
firm(2)  in the  park  (euros  /  year  /m2) 

Number of  researchers working  in the 
vicinlty(3) of the  park 

Number of  small  high-tech  firms 
establishing  themselves  in the  park 
after  one /  three years 
Number  of  research  institutes  in the 
park  (originating  from  outside  the 
region) 

Number  of  collaborative  projects 
involving two or  more occupants of 
the  park after one/three  years 

Number of  R&D posts created by park 
occupants after  one/three  years (FTEs, 
including  number  held by women) 

Number  of  small high-tech  firms  In 
the  area 

Number of  RTD posts in the area per 
1,000  workers 

(1) Science  and  Technology. 

(2) Whilst  there  is  no  agreed  single  definition  of  what  constitutes  a  high-tech  firm,  they  can  be  generally 
defined  as  enterprises  that  are active  in what  is considered  a  high tech  sector.  One  way  of  Identifying 
a  high  tech  sector  is  by  their  significantly  higher  than  average  expenditure  on  R &  D as  a  proportion 
of  sales  or  significantly  greater  than  average  proportional  employment  of  qualified  scientists  and 
engineers.  The  OECD  has  also  established  a  typology  of  sectors  using  the  ISIC  classification  that 
categorises  sectors  as  high  tech,  medium  tech,  medium  -low-tech  and  low  -tech. 

(3) Vicinity  can  be  defined  by  "a  distance  of  less  than  5  km".  Moreover,  this  definition  must  be  adapted 

to  the  practical  situation. 

Indicators for monitoring and evaluation 

Example  21:  Financial and  informational  support 
for technology  acquisition  by  SMEs 

Programme  Indicators 
(Related  to the  intervention  and 
its  effects) 

Context  Indicators 
(Related to  the  assisted  area) 

O u t p ut  indicators 

Number  of  addressees 
or  recipients 

Number  of  SMEs receiving  support 

Number  of  SMEs 

Volume of  technical 
services  provided 

Hours of  consultancy  and  training 
services  dispensed 

Result  indicators 

Leverage  effect 

I m p a ct  i n d i c a t o rs 

Internal  technology 
transfer(2) 

External  technology 
transfer(3) 

Technology  acquisition 

Economic  impact 

Investment  In acquisition  oriented  RTD 
by firms  benefiting  from financial  support 

Ratio of  BERD to  GERD(1 ) 

Number  of  collaborative  arrangements 
between  research  institutions  and 
assisted  enterprises  after  one  year 

Number  of assisted firms  purchasing 
patents, licences or entering  into R&D 
collaboration with foreign  firms  after 
one/three  years 

Number  of new  products or  processes 
incorporating the  acquired  technology 
marketed  by assisted firms  after one / 
three  years 

%  of  managers  of  recipient  firms  who 
declare that  the  Intervention  has  relaxed 
constraints  in terms of  access  to  new 
technologies 

%  of  managers  in the area who 
declare that  access to  new 
technologies  is a major  constraint 

Value added 

Value added generated after  one/three 
years  In euros  /  year 

Value added generated  in the area 
by SMEs 

(1) Business  Expenditure  on  Research  and  Development,  Gross  Domestic  Expenditure  on  Research  and 

Development. 

(2) Internal  technology  transfer  refers  to  transfers  taking  place  within  a  region  or  country,  generally 

between  research  institutes  and  enterprises. 

(3) External  technology  transfer  concerns  the  acquisition  of  technologies  resulting  from  RTD activities  by 

enterprises  outside  the  region  or  country. 

Research and technological development 

Example 22: Support for post-graduate  research 

Output  indicators 

Research  activity 

Result  indicators 

Qualifications 

Networking 

Impact  indicators 

Potential  innovatlons(l) 

Programme  Indicators 
(Related to the  intervention  and 
its  effects) 

Context  Indicators 
(Related to the  assisted  area) 

Number of  supported  research  students 
(of which  women) 
Number of  research  projects  employing 
supported  researchers 

Number  of  researchers  employed  In 
the  area per  1,000  workers 

Number  of  supported  researchers 
completing  post-graduate  research 
programmes  and obtaining  a PhD 

Number of  contacts  and  collaboration 
with regional firms  involving supported 
researchers 
%  of  supported  postgraduates  hired 
by regional  firms 

Number  of  patents  taken out for 
potential  innovations  being  developed 
with  private  sector  partners  resulting 
from  research  by supported  researchers 
-  after  one/three  years 

Annual number of doctoral students  In 
the area 

Number of  patents taken out  by  firms 
in the area 

(1) Potential  innovations  include  prototypes,  demonstration  units  and  services  under  development. 

Other  interventions  having  an  impact  in the  field 

The central  objective  of  Structural  Fund  interventions  in the  field  of  RTD is 
to  encourage  greater  RTD activity  in  regions  where  it  is  low  and  hence  to 
close  the  technology  gap  between  these  and  more  dynamic  regions. 
However, the dynamics of RTD are such that the disparities in capacity  have 
proved  to  be  highly  persistent  and  RTD  activity  remains  clustered  and 
geographically  polarised.  Interventions  supporting  the  development  of 
information  networks and improved telecommunications  systems  may  help 
to  counteract  this  effect,  as  they  permit  research  institutes  and  firms  to' 
enter  into  effective  collaborative  ventures  over  larger  distances  and 
generally  improve  networking  possibilities. 

Availability  and  collection  of  data 

Output  and  result  data  from  the  monitoring  of  projects  may  be 
complemented  by data on the satisfaction  of addressees or  recipients  with 

Indicators  for monitoring  and  evaluation 

services  provided  by  operators.  Information  on  enterprises  should  include 
their  size  in  terms  of  employees  or  turnover,  since  enhancing  RTD 
investment  and  technology  acquisition  capacity  of  SMEs  is  a  major 
objective  of  interventions.  Data  should  also  include  information  on  the 
accompanying  expenditure  of firms  on  RTD, so that  the  leverage  effects  of 
interventions  can  be estimated. 

As  in the  case  of  more  general  interventions  aimed  at  enterprises,  impact 
related  data  for  firms  have  to  be  collected  through  sample  surveys  and 
extrapolated  to allow  general conclusions to  be drawn. Given the  extended 
time  scale  over  which  RTD  impacts  occur,  some  reliance  has to  be  placed 
on  projections  made  by firms  on the  likely  impacts  of  interventions.  In such 
cases,  the  replies  of  the  respondent  are  more  likely  to  overestimate  than 
underestimate  the  potential  impacts  of  an  intervention. 

Context  indicators  relate to the  state of the science and technology  system 
of  a  region  or  regions  (e.g. number  of  PhDs,  post  PhD  researchers,  BERD, 
GERD,  number  of  R&D  employees,  etc.).  Data  for  context  indicators  are 
generally  available  at  a national  level  and the  up-to-date  statistics  for  R&D 
expenditure  and patent  applications  at the  national  and  regional  level in EU 
and the  European  Economic  Area  (EEA) are also  available. 

Key  indicators 

The  key  indicators  presented  below  lend  themselves  to  comparisons  both 
between different  interventions  of the  same  programme  and between  those 
of  different  programmes,  and  can  in  certain  cases  be  aggregated  at  a 
European  level. 

Research and technological development 

Level 

Key  indicator 

Resource 

Rate of  consumption  of  budget  (%  of  allocated  funds) 

Output 

Result 

Impact 

Selection  rate (1) (%  of  projects  accepted  in financial  terms) 
Number  of  hours  of  expert  advice  received  by  addressees  or 
recipients 

Satisfaction  rate (%  of addressees or recipients satisfied  /  very 
satisfied  by services  provided) 
Leverage  effect(2)  (private  sector  spending  occurring  as  a 
consequence  of  the  programme  in  relation  to  financial  support 
received) 

Value  added  /  sales  generated  (atteri 2  /  36  months  in  terms 
of euros  /  year  /  employee)(3)  (5) 
Net employment^)  created (FTEs of which held by women)  after 
1 2 / 36  months 

(1)The  project  selection  rate  provides  information  about  the  quality  of  applications  for  assistance, 

allowing inferences to be drawn on the absorptive capacity  of regions weak in RTD. 

(2) The  stimulation  of  private  sector  investment  is  often  an  explicit  objective  of  certain  types  of 

interventions, particularly RTD. 

(3) Given the long lead times  for  RTD projects, changes  in value added  and sales after  36 months are 

likely to  be limited. 

(4) Employment  created  by  RTD  interventions  is  more  readily  measured  in  terms  of  scientists  and 
technicians. Employment  impacts in areas not directly associated with performing RTD are generally 
likely to occur in the medium to long-term, and are therefore more difficult  to measure. 

(5) Given the long lead times for RTD projects, the number of  innovations resulting from an intervention 

are liable to be limited, even after 36 months. 

Indicators  for monitoring  and evaluation 

For  more  information  see... 

Archibugi  D.  and  Pianta  M.  (1996),  'Innovation  surveys  and  patents  as 
technology  indicators:  the  state  of  the  art'  in:  Innovation,  Patents 
and  Technological  Strategies.  Paris: OECD. 

Commission  européenne  (1994J,  Rapport  européen  sur  les 

indicateurs 
technologiques  1994,  Luxembourg:  Office  des 

scientifiques  et 
publications  officielles  des  Communautés  européennes. 

Eurostat  (1997), R&D Annual  Statistics  1997.  Luxembourg:  Eurostat. 

Lundvall  B.-A.  (ed.)  (1992),  National  Systems  of  Innovation:  Towards  a 
Theory  of  Innovation  and  Interactive  Learning.  London:  Pinter. 

OECD  (1994),  The Measurement  of  Scientific  and  Technological  Activities: 
for  Surveys  of  Research  and 

Proposed  Standard  Practice 
Experimental  Development  - Frascati  Manual  1993.  Paris: OECD. 

OST (1996), Science  & Technologie:  Indicateurs  1996.  Paris:  Economica. 

Agriculture  and  rural  development 

Structural Fund interventions in the field of agriculture and rural development 
are very  heterogeneous. They are  mainly financed  by the  EAGGF-Guidance 
and ERDF. Many measures  in this field intersect with those in the other fields 
mentioned  in  this  Volume.  Nevertheless,  four  types  of  interventions  are 
specific  to  this  field.  Firstly,  rural  infrastructure  (e.g.  village  renewal,  etc.). 
Secondly,  the  diversification  of  farms  (e.g. agro-tourism,  farm  shops,  new 
farm products, labelling, education and advice). Thirdly, the rationalisation of 
production  (e.g.  investments 
in  farms,  assistance  to  young  farmers, 
retirement  schemes,  processing  and  marketing  of  agricultural  products). 
Finally,  support  for  low  intensity  agricultural  production  (e.g.  afforestation, 
environmentally  favourable  land use). 

The  addressees  or  recipients  of  interventions  are  farmers,  former  farmers 
and  other  economic  units  in  rural  areas.  The  rationale  for  assisting  the 
agricultural  sector  and  rural areas  has several  dimensions. The  agricultural 
sector  is  in  need  of  structural  adjustment  because  the  demand  for  food  is 
stable,  while  high  rates  of  technological  development  have  resulted  in  an 
increase  in  supply,  a  surplus  of  production,  and  a  decrease  in  rural 
population and density. Income and employment  alternatives for farmers as 
well as for firms and their employees in related crafts and trades have to  be 
established  and developed. 

Single  projects  are  usually  very  small  in  terms  of  investment  volume  and 
large in number.  Interventions  are often  complex  measures  constructed  as 
local  programmes, containing  various types  of  public  and  private  projects. 

Interventions  within the  field 

Resource  indicators 

For  programme  driven  interventions,  resources  may  be  monitored  through 
the  measurement  of  project  subsidies  and  compared  with the funds  in the 
programming  document.  Sometimes  the  national  contribution  is  much 
higher  than  planned  in  the  programming  document.  If  this  is  the  case, 
additional  monitoring  of  European  Structural  Fund  financial  resources  is 
necessary. 

Output  indicators 

Outputs  of  interventions  in the  area  of  infrastructure  can  be  measured  in 
physical  units:  number  of  processing  and  packaging  plants  constructed, 

Indicators  for monitoring  and evaluation 

surface  area  of  renewed  villages,  etc.  Due  to  the  high  degree  of 
heterogeneity  of  projects,  it  is  difficult  to  consolidate  indicators  on  a  level 
higher than  measures,  except  by  using  an aggregated  rate  of  completion. 

In  the  general  case  of  subsidies  and  grants,  the  main  indicators  are  the 
"number  of  economic  units  assisted"  and  the  "number  of  private 
households  assisted".  These  indicators  might  be  combined  with  financial 
resources,  and  broken  down  into  categories  of  farms  or  households, 
especially  into  income  groups. 

Result 

indicators 

the 

limited 

Because  of 
rural 
infrastructure, addressees or recipients can usually  be identified without  too 
much  difficulty,  e.g.  inhabitants  connected  to  a  new  sewage  system, 
residents  served  by  an improved  road. 

interventions  concerning 

range  of 

The  "proportion  of  assisted  farms  and  households  in  the  defined  area  or 
group"  indicates  the  degree  of  penetration  of  the  programme  among  the 
total  number  of  farms  or  households  in the  defined  area or  group. 

A further indicator is the  "amount  of additional private expenses incurred  by 
farms  and  rural  households  in  relation  to  the  support  received".  This 
indicator  measures  the  "leverage  effect".  Due  to  a  ceiling  on  support,  the 
investments 
actual  investments  of  farmers  might  be  higher  than  the 
indicated  on  application  forms. 

Indicators  providing  information  on  changes  in  the  structure  of  the  local 
economy  focus, for example, on the diversification  of farmers'  incomes, the 
composition  of the  addressees  or  recipients  in terms  of  age and  gender. 

Impact 

indicators 

Unlike  larger  projects  in other  fields,  impacts  of  rural development  activities 
are  more  concentrated  in  the  immediate  region.  Agricultural  and  rural 
development  activities  have  impacts  that  depend  on  their  various  target 
groups. The diversification of activities has a direct impact on fanners, helping 
to  provide  new  employment  and  income  opportunities  adjacent  to  those  in 
the  agricultural  sector.  Investment  support  to  improve  processing  and 
marketing  of agricultural  products  has an indirect  impact on farms.  However, 
due to the high degree of competition  in the food sector  it will not  necessarily 
result  in  higher  producer  prices  or  larger  market  shares.  In conjunction  with 
this  support, supply  contracts  can help to  reduce farmers'  income risk. 

Agriculture and rural development 

Rural infrastructure improvement  measures  have impacts on the  residential 
attractiveness  of  the  eligible  area,  helping  to  avoid  people  moving  away 
from the region. Assistance has possible additional impacts on rural tourism 
as well. 

The time scale of  impacts  varies  between  one year (for  an impact  on agro-
tourism)  and ten years or even  more (for an impact  on  population  density). 

Besides  the  impacts  of  Structural  Funds  programmes,  rural  areas  are 
thoroughly dependent on other policies (mainly EAGGF-Guarantee).  Natural 
and  climatic  conditions  also  have  a  substantial  influence  on  farmers' 
incomes. 

Example  23: Fina 
of y 

ncial support to assist the 
sung farmers 

setting up 

Programme  Indicators 
(Related to tbe  Intervention  and 
its  effects) 

Context  Indicators 
(Related to the assisted  area) 

Output  Indicators 

Number of  addressees 
or  recipients 

Number  of  assisted young  farmers 
(incl. % of  women) 

Total  number  of young  farmers 

Result  indicators 

Leverage  effect 

Restructuring 

I m p a ct 

i n d i c a t o rs 

Survival  rate 

Total  investments  made  by  assisted 
young farmers  (broken  down  into  farm 
type  (1) 

Number  of  assisted young farmers(2) 
who  replace  retiring farmers 

Average  stock  of capital  per  farm 

Number  of farmers  retiring  per year 
Age distribution of  farming  population 
Ratio of  farmers  starting out to 
farmers  terminating  their  activity 

Survival  rate(2) of  young  farmers' 
businesses after  two  years 

Survival  rate of  businesses  in 
the agricultural  sector  in the  region 

Jobs  created 

Number  of FTE jobs on the farm  after 

Farm income 

two  years 

Income growth  in % two years 
after  investment 

Number of  FTE jobs in the  agricultural 
sector  in the  region 

Average  income per  farmer 

(1) Farm  type  is the  category  of  the  main  farm  production,  e.g.  field  crops,  dairy  cattle,  horticulture. 

(2) A  young  farmer  Is  defined  as  a  person  under  the  age  of  40  (35  in  some  countries)  taking  over  or 

establishing  a farm  for  the  first  time. 

Indicators  for monitoring  and evaluation 

Example 24:  Supported  investment  in agricultural  product 

processing  plants 

Programme  Indicators 
(Related  to the  intervention  and 
its  effects) 

Context  Indicators 
(Related to  the  assisted  area) 

Output  indicators 

Processing  capacity 

Result  indicators 

Leverage  effect 

Local supply  structure 

Impact  indicators 

Direct  employment 

Tonnes  /  m2 of  capacity  (broken  down 
into  different  products) 

Complementary  Investment  made  by 
addressees  or  recipients 

Number of supply  contracts fixed  with 
local  farmers 

Number  of  net jobs created  in  the 
recipient  firms  after  two  years  (FTEs 
incl.  %  held  by  women) 

Number  of  jobs  in the  agri-food 
industry 

Value added 

Increase  In producer  prices 

Increase  in  producer  prices for 
agricultural  products  in the area 

Knock-on  effect 

%  of the  region's eligible  agricultural 
products  bought  by recipient  firms 
after  2  years 

Example 25: Village  renewal  -  public  and private  projects 

Programme  Indicators 
(Related to the  intervention  and 
its  effects) 

Context  Indicators 
(Related to  the  assisted  area) 

Output  indicators 

Quantity 

R e s u lt  i n d i c a t o rs 

Population 

Satisfaction 

Impact  indicators 

M2  of  village squares  renovated 
km  of  village  roads  improved 
Number  of  houses  renovated 

Number  of  inhabitants  living  in  the 
vicinity  of  rehabilitated  areas  (1) 

Number  of  inhabitants  living  in  low 
density  areas 

Number  of  inhabitants  that  are  satisfied 
or  very  satisfied with the  works 

Residential  attractiveness  %  of  inhabitants  who  answer  "No"  to 

Total  migration  from  the  assisted  area 

the  question  "Do  you  wish  to stay  in 
the  area  in the  next  5  years?" 
Average  cost  of  a  house  in  the 
rehabilitated  village 

Average  cost of  a house  in  the 
assisted  area 

(1) Vicinity  should  be  defined  as  "less  than  1  k m ".  However,  that  definition  must  be  adapted  to  practical 

situations 

Agriculture and rural development 

Example 26: Support for establishing  tourist  accommodation 

on farms 

Programme  Indicators 
(Related  to the  Intervention  and 
its  effects) 

Context  Indicators 
(Related to the  assisted  area) 

Output  indicators 

Number  of farms 

Number  of  projects  granted  assistance 

Capacity 

Number  of tourist  beds created 
on farms 

Total number of tourist  beds  provided 

Result  i n d i c a t o rs 

Diversification 

Impact  indicators 

Number  of  overnight 
visitors 

Income 

Value added 

%  of  addressee or  recipient  who  enter 
in the  agro-tourism  business as a 
consequence  of the  programme 

Number  of overnight  stays  on 
addressees'  or  recipients'  farms  per 
year and after two  years 

Overnight  stays of  tourists 

Increase  in recipient farmers'  income 

Average  income  per farm 

Gross value added generated  by the 
programme (euros  /  year) after one year 

Value added generated within the 
agricultural  sector 

Other  interventions  having  an  impact  in the  field 

Farmers, and even  more so rural households, are often  part  of wider  target 
groups  in  other  fields.  Support  for  businesses  has  an  impact  on  job 
opportunities  for  rural  households  and  provides  alternative  or  additional 
income  possibilities for farmers, contributing  therefore to  structural  change 
in the agricultural  sector. 

Similarly,  interventions  of  the  field  of  transport  have  impacts  on  transport 
costs for agricultural inputs and products. Additionally, activities in this field 
improve  the  access  of  rural  households  to  job  opportunities  outside  their 
area. 

Structural  interventions  in the field  of  tourism  are  in some  cases  related  to 
tourist  attractions  in  rural areas. Visitors  may  be interested  in visiting  other 
small  rural  tourist  attractions.  They  may  stay  in  these  areas  for  longer 
periods  for  their  holiday  activities.  The  supply  of  regional  agricultural 
specialities  by  farmers  is  one  more  opportunity  to  open  up  rural  tourist 
potential. 

Training,  especially  in technologies  needed  in  firms  located  in  rural  areas, 
again increases the possibility of  members of  rural households finding jobs. 

Indicators  for monitoring  and evaluation 

Availability  and  collection  of  data 

Most  output  and  result  related  data  should  be available from  the  operators 
of  projects  funded through the Structural  Funds, who  should  be required  to 
routinely  collect  simple  data  on, for  example, the  number,  size  and type  of 
farms  assisted. 

Impact  related  data  need  to  be  collected  from  the  farmers  themselves  in 
special  surveys.  Like other types of  intervention, the involvement  of  outside 
teams to collect this  information  may be useful in ensuring that answers are 
reliable.  It  is  important  to  use  sample  surveys  and  extrapolate  from  them, 
since  when 
the  cost  of 
comprehensive  surveys  is  prohibitive. 

the  overall  number  of  projects 

is  high, 

indicators  and  calculating  the  possible 

Impacts  on  rural  households  and  attractiveness  of  the  region  can  be 
examined  through  comprehensive  surveys.  In some  cases  it  is  possible  to 
rely on earlier  analyses  of  measures. By adjusting coefficients  with  result  or 
output 
impact  of  assistance, 
sufficiently  reliable  and  cost-effective  estimations  can  be  made.  This 
method  is  often  used  to  identify  the  temporary  employment  effects  of 
village  renewal  or  to  calculate  the  employment  effects  of  agro-tourism  (in 
FTEs). 

Some  context  indicators  relate  to  the  national  or  regional  context  as 
appropriate.  Data  for  them  may  be  available  from  European,  national  and 
sometimes  regional  administrative  records, statistics  and  surveys.  Eurostat 
has  developed  a  special  database  of  annualised  regional  statistics, 
providing  Europe-wide  comparable  data.  For  agriculture  and  rural 
development  there  are  some  problems  of  statistical  coverage.  Information 
on  small  sized  on-farm  activities  (e.g.  agro-tourism)  is  not  provided  by 
statistics. Agricultural  data  on  regional  farm  structure:  age of farmers; size; 
farm  type,  is  only  available  with  a  substantial  time  lag,  which  limits  the 
possibilities 
is 
the  context 
implemented. 

in  which  a  programme 

for  monitoring 

•  Key  indicators 

The  key  indicators  presented  below  lend  themselves  to  comparisons  both 
between different  interventions of the same  programme and  between  those 
of  different  programmes,  and  can  in  certain  cases  be  aggregated  at  a 
European  level. 

Agriculture and rural development 

Level 

Key  indicator 

Resource 

Output 

Result 

Impact 

Rate of consumption of budget  (% of allocated  funds) 
%  of  projects  (in  financial 
disadvantaged  rural areas 

terms)  concerning  the  most 

Selection  rate (% of projects  in financial  terms  accepted) 
Number  of individuals  receiving  direct  assistance or services as 
a  result of the programme  (incl. % of men /  women) 
Number  of  economic  units  (farms,  etc.)  receiving  direct 
assistance  or  services  as  a  result  of  the  programme  (large, 
medium, small, individual) 
Number  of  new  economic  units  (tourist  accommodation  and 
attractions,  new  farms,  etc.)  receiving  direct  assistance  or 
services as a result of the  programme 
Coverage  (%  of  addressees  or  recipients,  for  example  young 
farmers, of the total number of potential addressees or recipients) 

recipients  situated 

%  of  addressees  or 
disadvantaged  areas 
Leverage  effect 
accompanying  the financial  support  received) 

(spending  by  addressees  or 

in  the  most 

recipients 

%  of  assisted  new  businesses  (diversified  farms,  campsites, 
farms taken over by young farmers, etc.) that are still active after 
24 / 36 months 
Gross value added generated (after  12 months in terms of euros 
/  year  /  employee) 
Net  employment  created  or  maintained  (FTEs  incl.  %  held by 
women)  after  12 months 
Residential attractiveness (% of inhabitants wishing to remain in 
the area) 

For  more  information  see... 

Grajewski,  R.,  Schrader,  H.,  Tissen,  G.  (1995),  Vorschläge 

für  die 

Wirkungsanalyse  und  Bewertung  von  Mabnahmen  zur  Anpassung 

der Agrarstruktur  und  zur Entwicklung  der  ländlichen  Räume in  den 

neuen  Bundesländern  gemäb  Ziel  5a  und  5b  der  Reform  der  EG-

Strukturfonds  (1991-1993).  Braunschweig. 

Riedel,  J.,  Scharr,  F.,  König,  M.  et  al.  (1997), 

IFO, 

Institut 

für 

Wirtschaftsforschung  Niederlassung  Dresden 

(Hrsg.):  Die 

Europäischen  Strukturfonds 

in  den  neuen 

Bundesländern. 

Indicators  for monitoring  and evaluation 

Zwischenbewertung  des  Mitteleinsatzes  von  1994  bis  1996. 
Dresden. 

Tissen,  G.,  Toepel,  K.  (1998),  Europäische  Strukturpolitik 

ländlichen 
Raum.  Zwischenbewertung  des  Ziel-5b-Programms  Schleswig-
Holstein.  Frankfurt  am Main. 

im 

Young,  N.,  Isla, M.,  Schrader,  H. et  al. (1991), Monitoring  and  Evaluation  of 

Objective  5b  Regions.  Report  Wye for the  European  Commission. 

Indicators for five priorities 

Equal  opportunities 

Support  provided  under the Structural  Policies  may eventually  have  effects 
on the entire population, though  these  are generally  not  uniform  across  the 
social  groups  and  may  even  exacerbate  the  situation  of  certain  categories 
of addressees or recipients, particularly women. It is for this reason that the 
equal  opportunity  dimension  of  employment  effects  needs  to  be 
systematically  integrated  into  evaluations. 

The  European  priority  of  the  promotion  of  equal  opportunities  between 
women  and  men  is  closely,  but  not  exclusively,  linked  to  employment 
issues.  In  this  context  it  does  not  seem  relevant  to  develop  numerous 
indicators  for  the  issue  of  equal  opportunities,  but  to  consider  equal 
opportunities  as  a  mainstreaming  issue,  which  means  that  as  many 
indicators  as possible  should  be broken down  by  gender. 

Box 17 -  Gender  mainstreaming 

The European priority  in favour of equal opportunities for women and men  should 
be  systematically  integrated  into  all  programmes,  measures  and  projects.  The 
term  "gender  mainstreaming"  means  that  the  respective  situation  and  needs  of 
women  and  men  are  actively  and  openly  taken  into  account  at  all  stages  of  the 
programming  cycle from ex ante diagnosis  to ex  post evaluation. 

In  order  to  apply  the  concept  of  mainstreaming,  those  who  design  a  system  of 
indicators  should,  when  relevant,  break  down  figures  into  the  categories  of 
women  and men. 

The  mainstreaming  principle  may  also  be  applied  to  other  policy  priorities,  such 
as SMEs. 

Interventions  within  the  field 

Interventions  having  the  promotion  of  equal  opportunities  as  their  main 
objective are relatively rare. In fact, the importance of the issue requires that 
the  promotion  of  gender  equity  affect  all sectors, rather  than focusing  on a 
limited  number  of specific  measures. 

Indicators  for monitoring  and evaluation 

Interventions  having  an  impact  in the  field 

The following  example  is drawn  from the field of  urban  development,  but  it 
could  have  referred  to  any  other  of  the  fields  presented  in this  Volume.  In 
terms of equal opportunities, the most  relevant  indicators  are situated at the 
impact  level,  for  instance  the  proportion  of  jobs  created  or  maintained 
which  are taken  by  women. However,  in order  to  more adequately  address 
this  issue,  indicators  need  to  go  further  and  provide  information  on  the 
number  of  women  managers  and  decision  makers,  as  well  as  examining 
their  pay  levels  in  relation  to those  of  male  employees. 

According to the  principle of gender mainstreaming, all indicators should  be 
broken  down  into women  /  men so as to take gender  issues into  account. 
In practice, this implies that the percentage of women should  systematically 
be  counted  for  all  relevant  indicators,  starting  from  impact  indicators  and 
going  backward  to  result  and  output  indicators,  and  even  to  resource 
indicators. This principle  is demonstrated  in the  following  example. 

Example 27: Esta blishment or re-generation of community  centres 

Programme  Indicators 
(Related  to the  intervention  and 
its  effects) 

Context  Indicators 
(Related  to the  assisted  area) 

Output  indicators 

Facilities 

Capacity 

Result  indicators 

Satisfaction 

Impact  indicators 

Employment 

Purchased, constructed  or  refurbished 
buildings  in  m:  (area  of  which  is 
devoted  to  meeting  women's  needs, 
i.e. nursery, etc.) 

Maximum  number  of  users that  can 
be served (including the  maximum 
number  of  users  of  resources 
specifically  intended for  women) 

%  of  users that  are satisfied  or  very 
satisfied  with  the  project  (of  which 
women  /  men) 

Maximum  number  of users  in 
nurseries  in the  area 

Number  of  users that  hold a  permanent 
job  (of  which  women  /  men) 

%  of  workers  in the  area  who  use 
nursery  services 

Availability  and  collection  of  data 

The  type  of  information  held  in monitoring  systems  should  be  improved  in 
order  to  enable  a  breakdown  of  indicators  according  to  gender  whenever 
this  is  relevant. 

Equal opportunities 

Reports  from  relevant  research  and  labour  market  analyses  (including  the 
results  of  previous  evaluations  of  relevant  interventions)  provide  a stock  of 
both  quantitative  and  qualitative 
information  that  may  be  used  for 
estimating  context  indicators  for  equal  opportunities.  For  instance,  labour 
force  surveys  provide  figures  in  terms  of  women  /  men  on  at  least  an 
annual  basis for  both the  EU as a whole and  individual  Member  States. 

If the  monitoring  system  provides  limited  data  (or even  no data)  about  the 
respective situations of women and men, it may be decided to collect these 
data  at  the  evaluation  stage  by  surveying  a sample  of  projects  in  order  to 
reconstruct  missing  output,  result, and  impact  indicators. 

Key  indicators 

The  key  indicators  presented  below  lend  themselves  to  comparisons  both 
between different  interventions  of the same programme and between those 
of  different  programmes,  and  can  in  certain  cases  be  aggregated  at  a 
European level. 

Level 

Key  indicator 

Resource 

Number  of  persons  mobilised  to  implement  the  interventions  in 
job years  (incl.  %  of  women) 

Output 

Result 

Impact 

Number  of  addressees  or  recipients  (incl. %  of  women) 

Number of  persons who have been immediately  hired during the 
life of the  project  (incl.  %  of  women) 

Jobs created or maintained after  12 months (FTEs, of which held 
by  women) 
Number  of  users  that  hold  a  permanent  job  (of  which  women  / 
men) 

For  more  information  see... 

European  Commission  (1998.) One hundred  words  for equality  - a glossary 
of  terms  on  equality  between  women  and  men.  Luxembourg: 
OOPEC. 

Environment 

Structural  Fund  interventions  concerning  the  environment  may  include 
environment-related 
infrastructure  projects  (e.g.  the  construction  or 
enhancement  of  waste-management  facilities),  support  of  environment 
investments  (e.g.  assistance  for  the  establishment  of  new 
related 
companies  providing  environmental  services),  or  projects  which  are 
intended  to  develop  environmental  skills  (e.g. training  managers  in  more 
environmentally  friendly  production techniques, or training the  unemployed 
in skills  relevant to  re-cycling). 

Much attention  is also  paid to  ensuring that  programmes  supported  by the 
Structural  Funds  result  in  sustainable  development  (i.e. that  development 
takes  place  in  such  a  way  that  it  does  not  damage  the  environment  or 
deplete  limited resources, therefore  allowing the needs of the present to  be 
met  without  compromising  the  ability  of  future  generations  to  meet  their 
needs).  In  the  Fifth  Action  Programme  on  the  Environment  which  was 
adopted  by  Council  in  1993  the  notion  of  sustainability  is  defined  as 
reflecting  "...a  policy  and  strategy  for  continued  economic  and  social 
development  without  detriment  to  the  environment  and  the  natural 
resources  on  which  continued  human  activity  and  further  development 
depend."  Such efforts  largely concentrate on mitigating the adverse  effects 
of  an  activity  on  the  environment.  Mitigation  consists  of  measures  that 
avoid,  reduce  or  remedy  negative  environmental  impacts,  or  lead  to  a  net 
improvement, therefore  achieving  sustainability. 

Interventions  within the  field 

Resource  indicators 

At  the  programme  level, resources  devoted  to  activities  having  an  explicit 
indicators  of 
environmental  objective  can  be  monitored  by  using 
"compliance  with  project  cost"  (i.e. the  most  recently  available  information 
on the total cost of the projects  in proportion to the cost which  appeared  in 
the  programming  document). 

Output  indicators 

Examples  of  output  indicators  include  the  capacity  and  standard  of  new 
water treatment  facilities,  number and  standard of training courses funded, 
covering  relevant  subjects,  etc.  These  indicators  are  essentially  usable  for 
monitoring  project  completion  and duration. 

Indicators  for monitoring  and evaluation 

Some  output  indicators  can  be  related  to  indicators  of  endowment.  For 
instance, the  capacity  of  new  waste water treatment  plants  may  be  related 
to  regional  endowment  in terms  of  waste  water  treatment  capacity. 

Result 

indicators 

Immediate  benefits  for  addressees  or  recipients  are  easily  measurable  in the 
case  of  training  and  support  to  private  investment.  In  these  cases  result 
indicators measure the new skills acquired or the new equipment constructed. 

In the case of  infrastructure,  result  indicators  measure  improvements  in the 
quality  of  services  (i.e.  drinking  water  respecting  European  standards) 
and/or  lowering  of  prices  charged  to  end-users. 

Impact 

indicators 

Regardless  of  the  type  of  activity  being  assisted,  its  ultimate  effect  on  the 
environment  depends  on  the  use  which  is  made  of  the  immediate  output 
after several months or years. A re-cycling facility will only have an effect  on 
the  environment  proportionate  to  the  degree  to  which  it  is  used.  Similarly, 
new  eco-industries  will  only  affect  the  environment  to  the  extent  that  they 
are surviving and developing, training  in environmentally  sensitive  business 
practices  will  only  be  influential  to  the  extent  that  a  sufficient  number  of 
trainees  apply  their  new  skills. 

The types  of  impact  which structural  interventions  have on the  environment 
are  extremely  diverse  and  their  identification  should  take  place  in  the 
programming  phase.  This  identification  is  related  either  to  the  explicit 
environmental  objectives  of  the  intervention  or  to  established  regulations 
and  norms  on the  environmental  management  of  publicly  financed  activity. 
It is important  to take  into account  the fact that  environmental  impacts  may 
occur  in  either  the  short  or  long-term  and  may  be  either  discrete  or 
cumulative. 

In  1994  a  Commission  memorandum  noted  that  the  measurement  of 
changes  in the overall environmental  context, that are clearly  attributable  to 
assisted  activity,  "is  difficult,  time  consuming  and  resource  intensive  for  a 
number  of  reasons...".  The  pressure  that  economic  activities  exert  on  the 
environment  is  easier  to  identify.  In  other  words,  since  the  environment  is 
open  to  such  a  variety  of  influences,  it  is  often  simply  impractical  to 
establish an empirical connection  between  programme outputs  and results, 
on the one  hand, and  impacts  measured  in terms of  a changed  state of  the 

Environment 

environment,  on  the  other.  For  example,  it  is  probable  that  many  general 

levels  of  atmospheric  or  water  contaminants  would  not  be  noticeably 

influenced  for  a  considerable  time  by  even  relatively  substantial  individual 

projects. 

The  problems  of  defining  impact  indicators  for  the  environment  are  less 

taxing  in  practice  than  they  might  be,  due  to  the  fact  that  the  objectives  of 

a  structural  intervention  in  the  field  of  the  environment  are  often  expressed 

in  terms  of  changing  pressures  on  the  environment.  In  the  case  of  point 

sources  of  contamination,  an  impact  indicator  is  simply  provided  by  the 

measure  of  decreased  pressure  (reduction  of  a  specified  pollutant  from  the 

source 

in  question).  Impact 

indicators  expressed 

in  terms  of  reducing 

pressures  on  the  environment  could  then  be  cross-checked  with  context 

indicators  obtained  from  established  data-series  and  also  expressed  in 

terms  of  pressure. 

Box  18  -  The  "Pressure-State-Response"  model 

is  one  of  a  number  of  models 
The  "Pressure-State-Response"  framework 
developed  for analysing  environmental  policies  (others  include the  Driving Force-
State-Response  model,  which  also  allows  agri-environmental 
linkages  and 
sustainable  agriculture  to  be  taken  into  account).  It  is  not  far  from  the  "Result-
Impact-Context"  model that  provides the  basis of this  document. 
Pressures  are  the  driving  forces  that  modify  the  environment  (pollution,  use  of 
water).  The  state  of  the  environment  at  a  given  time  results  from  the  combined 
action  of  three  forces:  the  natural  forces  that  govern  the  environmental  system, 
the  pressures  of  socio-economic  activities  and  the  responses  of  environmental 
policies. 

Both  pressures  on  and  state  of  the  environment  can  be  described  with  context 
indicators.  Responses  are the  results  and  impacts  of  interventions  that  have  an 
environmental objective. They may also be the results or impacts of the  mitigation 
of non-environmental  projects. 

Indicators for monitoring and evaluation 

Example 28:  Improvement  of water  supply  system 

Programme  Indicators 
(Related  to the  intervention  and 
its  effects) 

Context  Indicators 
(Related to the  assisted  area) 

Output  Indicators 

Progress 

Result  Indicators 

Coverage 

Quality of  service 

Impact  indicators 

Consumption 

Indirect  economic  effect 

Compliance  with  project  duration 
Rate of  completion 

Number  of  households  potentially 
served  by new  /  improved  networks 

%  of  households  served  by  water 
supply  networks 

Number  of  days with  insufficient  supply 
In quality  or quantity  (average  number 
of days per  1000  households) 

%  of  regional households  facing 
insufficient  water  supply  in quality  or 
quantity 

Volume  of  water  consumed  through 
new  /  improved  networks  (m3  /  year) 
after  one year 

Average  water  consumption  per  head 
per  year. 

Number  of economic  units  (firms, 
farms, etc.)  who declare that the  new 
water  supply  system  has released a 
major  constraint for their  development 

Water  resources available  per  annum 

Environment 

Withdrawls  In m3 per  person per 
annum 

Environment 

Example 29: Construction of wastewater treatment plants. 

Programme  Indicators 
(Related to the  intervention  and 
its  effects) 

Context  Indicators 
(Related  to the  assisted  area) 

Output  Indicators 

Progress 

Capacity 

Result  Indicators 

Quality of  treatment 

Impact  indicators 

Consumption 

Indirect economic  effect 

Environment 

See Example 28 

Number  of new waste water  treatment 
plants  constructed 
Maximum  annual throughput  (m3) 

% of waste water  undergoing  primary 
treatment 
%  of  waste water  undergoing  secondary 
treatment  (1) 

Volume of  waste  water  treated 
Number  of  households  /  businesses 
connected  to waste-water  treatment 
facilities  by programme  after  one year 

%  of  waste  water  treated 
%  of  households  /  businesses 
connected  to waste  water  treatment 
facilities 

% of waste water  samples from  point 
sources downstream  from the stations, 
showing  a specified  level of decline of 
identified  pollutants 

See Example  28 

% of  km of rivers  monitored on which 
pollution exceeds  a specified  standard 

(1) Quality  standards  for  wastewater  are  set  out  in  EC  directive  76  /  464  /  EEC  and  sister  directives. 
Primary  treatment  is  mechanical,  allowing  the  physical  separation  of  effluents;  secondary  treatment 
is  biological,  involving  the  use  of  micro-organisms. 

Indicators for monitoring and evaluation 

Example 30:  Improvement  of  solid waste  management  facilities 

Programme  Indicators 
(Related  to the  intervention  and 
its  effects) 

Context  Indicators 
(Related to the  assisted  area) 

Output  Indicators 

Progress 

Capacity 

Result  Indicators 

Coverage 

Impact  indicators 

Solid waste  collected  for 
recycling 

Solid waste  recycled for 
reuse as raw  materials 

Indirect  economic  effect 

Environment 

See Example  28 

Maximum  annual throughput  (tonnes) 

Number  of  households  potentially 
covered  by waste  recovery  collection 
services 

Amount  of  solid  household  waste 
collected for  recycling  in the  areas  of 
assisted  projects  (tonnes  /  year)  after 
one  year 

%  of  solid waste  recycled for  reuse as 
raw  materials  in the  areas of  assisted 
projects  after  one year 

%  of  unauthorised  landfill  sites 
closed  /  rehabilitated  in the areas of 
assisted  projects 

Amount  of  solid waste  produced  In 
the  area  (tonnes  /  year) 

%  of solid waste  recycled for  reuse as 
raw  materials  In the area 

See Example 28 

Number  of unauthorised  landfill  sites 
in the  area 
%  of  underground  water  sources 
suffering  from  pollution  emanating 
from  buried solid  waste 

Environment 

Example 31 : Sup 
&se 

Dort for  new  SMEs  providing environmental  goods 
rvices.'1' 

Programme  Indicators 
(Related to  the  intervention  and 
its  effects) 

Context  Indicators 
(Related  to the  assisted  area) 

Output  Indicators 

Volume  of  support 
services 

Number  of  addressees 
or recipients 

Result  Indicators 

Diversification 

Certification 

Impact  indicators 

Environment-related 
activities 

Environment  related 
innovations 

Briefings  with  experts  providing 
environmental  advice  within  the 
programme  (hrs) 

Number  of firms  receiving 
environmental  advice from  experts 
(at least  10 days or of a  minimum 
value of 3,500  euro) 

Number  of  recipient  firms  newly 
established  in the  environment  related 
sector 

%  of  firms qualifying  for  certification 
in respect to environmental  norms 

Volume of  business of  assisted firms  in 
the  environment  related sector  after 
two years of operation  (value  added 
per year) 

Number of  new  products  /  processes 
introduced by assisted firms  in the field 
of environment  after two years 

Number  of  regional firms  in the 
environment  related  sector 

Volume  of  environment  related  goods 
&  services  as a % of total of  goods 
&  services  in the  area 

(1) There  do  not  appear  to  be  any  established  standard  definitions  of  an  environmental  good  or  service. 

Other  interventions  having  an  impact  in the  field 

At the  resource level a specific  problem  appears  when  an attempt  is  made 
to  measure  the  environmental  cost  of  a  given  project,  for  instance,  a 
motorway  or an airport.  It is not  possible to  measure  mitigation  efforts  with 
separate  costings 
systems.  The 
environmental  cost  should  be  estimated  twice:  ex  ante  at  the  project 
selection  stage and  ex  post  at the  programme  evaluation  stage. 

through  standard  accounting 

Whilst  output  and  result  indicators  are  specific  to  other  fields  (e.g.  km  of 
motorways, number of passengers), their monitoring is a basis for measuring 
the pressure on the environment  and for quantifying  impact  indicators. 

It  is  important  to  bear  in  mind  that  environmental  impacts  may  be  due  to 
both  the  immediate  consequences  of  assisted  activity  (e.g.  the  damage 

Indicators  for monitoring  and evaluation 

item  of 
caused  to  natural  habitats  by  the  construction  of  a  major 
infrastructure  such  as  a  motorway)  and  the  mid-term  consequences  of 
these  or  other  actions  (e.g.  subsequent  increases  in  air  pollution  due  to 
increased  volumes  of traffic  using the  motorway). 

In  addition  to  harmful  impacts  on  the  environment  of  non-environmental 
interventions,  there  are  also  instances  where  positive  impacts  on  the 
environment  occur,  as in the  case of the construction  of an inter-urban  light 
railway. 

Example 32: Construction  of an  inter-urban  light  railway 

between  A  and Β 

Programme  Indicators 
(Related to the  intervention  and 
its  effects) 

Context  Indicators 
(Related to the  assisted  area) 

O u t p ut  indicators  (in  the  field  of  transport  infrastructure) 

Quantity 

| km  of  light  rail  network 

Result  indicators  (in  the  field  of  transport  infrastructure) 

Speed 

E.I.S.S.  between A and Β 

Average  E.I.S.S. between  all  relevant 
urban  centres  In  area 

I m p a ct  indicators  (in the  fields  of  infrastructure  and  environment) 

Traffic  flow 

Transport  system 

Number  of  passengers  using  the 
railway  line  (km  χ passengers  per  year) 
after  one  year 

%  of  passengers who  declare that  they 
would use a private  car  in the  absence 
of the  railway 
Reduction in volume of  private  vehicle 
traffic  (vehicle  χ km  per year)  after 
one year 

Environment 

Reduction in the total emission of NO2 
& volatile organic  compound (VOC) 
(K  /  year) 

Emission of  NO2 & volatile  organic 
compound  (VOC) (K  /  year) Average 
national emission  per vehicle  χ km 

Availability  and  collection  of  data 

Output  related  information  is  often  not  available  in  a  synthetic  form  for 
measures  with  an  environmental  objective.  Indeed,  data  on  the  individual 
outputs  of  specific  projects  may  themselves  need  to  be  synthesised  in 
order  to  obtain  a  single  indicator  the  "level  of  project  completion". 
Indicators  of  "compliance  with  estimated  project  duration"  should  be  a 
standard element  in monitoring  systems and  provided  as a matter of  course 
at  least  every  six  months. 

Environment 

In some cases, results can  be quantified  relatively  easily through  indicators 
of demand (e.g. energy use in agriculture, consumption of water per capita); 
in  other  cases,  ad  hoc  surveys  of  end  users  (e.g.  reduced  use  of  energy 
resulting  from  implementation  of  energy  audits  by  companies)  must  be 
undertaken. 

In  some  instances  monitoring  systems  may  include  information  on  a 
programme's  environmental  impact  in  terms  of  pressure  changes  (e.g. 
reduced  output  of  pollutants from  point  sources).  However,  because  of  the 
relatively complex  and novel nature of environmental  impact assessment, in 
many  instances  custom  surveys  or  investigations  of  the  results  of  specific 
projects  (particularly  those  for  which  a large-scale  or  concentrated  impact 
has been foreseen)  may  be necessary. Data on changes  in pressures on the 
environment  may  be available from established time-series  context  data on 
a national  or even  regional  level. 

Context  indicators  refer to the  general quality  of the environment  within the 
region. Time-series  data  providing  information  for this type  of  indicator  are 
becoming  increasingly  harmonised,  detailed  and  comprehensive.  For  the 
EU  as  a  whole  and  for  each  Member  State,  Eurostat  provides  a  growing 
number  of  data  series  in  both  printed  and  electronic  form.  The  recently 
established  European  Environment  Agency  is  also  beginning  to  produce 
relevant work. On the national level a wealth of information  may be obtained 
from  the  statistical  services  of  the  various  Member  States.  However, 
established  time  series  tend  to  be  less  available  and  less  detailed  on  the 
regional  level and  custom  surveys  less  relevant  and  practical. 

Key  indicators 

The  key  indicators  presented  below  lend themselves  to  comparisons  both 
between different  interventions  of the same programme  and between  those 
of  different  programmes,  and  can  in  certain  cases  be  aggregated  at  a 
European level. 

Indicators  for monitoring  and evaluation 

Level 

Key indicator 

Resource 

Output 

Result 

Impact 

Rate of  consumption  of  budget  (%  of  allocated  funds) 
%  of  budget  devoted  to  environmental  mitigation  measures 

Selection  rate (%  of  projects  accepted  in financial  terms) 
Rate of  completion  of  project  (%  of  objective) 
Compliance  with  project  duration 
Number  of  potential  connections  (domestic  /  economic  units) 
to  networks  of  basic  services  (e.g. water  treatment  facilities) 

%  of  domestic  /  economic  units  receiving  a  level  service 
satisfying  European  norms  through  the  network  (e.g.  drinking 
water) 

in  domestic  /  economic  units  (e.g.  water 

Number  of  users  connected  to  the  new  infrastructures,  broken 
down 
treatment 
facilities)  after  one  year 
Net  employment  created  or  maintained  (FTEs  of  which  held  by 
women) 

For  more  information  see... 

Department  of  the  Environment  (1993J,  Environmental  Appraisal  of 
Development  Plans, A good  practice  guide.  London:  HMSO. 

European  Environment  Agency  (1995), Environment  in the European  Union, 
Report for the  Fifth Environmental Action  Programme, Copenhagen: 
EEA. 

Eurostat  (1996),  Indicators  of  Sustainable  Development.  Luxembourg, 

Eurostat. 

Eurostat  (1996),  Statistiques  de  l'environnement 

1996.  Luxembourg, 

Eurostat. 

OECD  (1997), Environmental  Indicators  for Agriculture.  Paris: OECD. 

Information  society 

Structural 
include: 
interventions  related  to  the  "information  society" 
infrastructure  projects  (such  as  the  deployment  and  upgrading  of 
telecommunications  and telematics  networks), direct financial  support (e.g. 
in establishing  new  businesses  in the  multimedia  industry),  projects  aimed 
at  the  establishment  of  "permanent  conditions"  for  the  development  of 
Information  and  Communication  Technology  (ICT)  applications  (such  as  a 
telework  or  a  telemedicine  centre),  and  human  resources  development 
projects  (e.g. training  initiatives  concerning  new  technologies). 

ICT 

In  the  area  of 

interventions  and  applications 

ICT 
infrastructure 
interventions  are 
perceptibly  different. 
the 
to 
modernisation  of  a  telephone  network) 
infrastructure  interventions  in  other  fields,  such  as  transportation,  even 
though  the  services  provided  by  new  telecommunications  infrastructures 
may  not  be  as easily  identified, especially  as far  as the  advanced  services 
(e.g. interactive  multimedia  services)  are concerned. This  means that  result 
and especially  impact  indicators  are often  of  minor  significance. 

(e.g. 
they  are  quite  similar 

infrastructures 

Interventions  in  the  area  of  ICT  applications  development  (i.e.  telework, 
distance education, and telemedicine)  are usually aimed at trials rather than 
large-scale  diffusion.  This  means  that  it  is  rather  difficult  to  evaluate  the 
programme with reference to its context. For example, telework  projects are 
presently  conducted  on an experimental  basis and usually  involve a  limited 
number  of workers. As a consequence,  context  indicators  (in this case, the 
percentage  of  teleworkers  in relation to the total  number  of  employees  of  a 
given  geographic  area  and  /  or  industry)  are  usually  not  significant  and 
need not be monitored. However, an evaluation of the potential  contribution 
of a specific  project to the general context  (level of employment  in the  case 
of  telework,  level  of  qualification  in  the  case  of  professional  on-line 
education,  and so on) could  be provided, at  least  on a qualitative  basis. 

Not  all indicators  for  interventions  in the field  of the  information  society  are 
fully  accepted,  because  of  the  intangible  nature  of  ICT-based  services, the 
rapid  development  of  new  technologies  and  uncertainty  about 
their 
organisational  impact  on  complex  systems  like education  or  healthcare. A 
logic of experimentation  is thus required in the monitoring  and evaluation of 
such  projects. 

Indicators  for monitoring  and evaluation 

Interventions  within  the  field 

Resource 

indicators 

With  reference to the distinction  between telecommunications  infrastructure 
interventions  and  application  interventions  (see  above),  resource  indicators 
may  include  "compliance  with  project  costs" for  infrastructure  interventions. 
In the  case  of  application  interventions  the  use  of  financial  resources  may 
fluctuate  due  to  demand  factors  such  as  higher  or  lower  (than  estimated) 
participation  of  addressees  or  recipients  in initiatives. 

Output 

indicators 

Interventions  providing  telecommunications  infrastructure  can  easily  be 
monitored  with  physical  indicators, related both to the assisted  project  (like 
kilometres  of  fibre  deployed  or  number  of  telephone  exchanges  equipped 
with  devices  based on  new technologies)  and to the general  ICT context  of 
the  considered  region  (e.g.  percentage  of  upgraded  network).  If  service 
provision  itself  is  included  in  the  project,  a  measurement  of  the  provided 
services  (e.g.  number  of  CATV  -  cable  television  -  channels,  hours  of 
broadcast,  or  any  other  measure  of  available  services)  should  be  required 
as  an  output  indicator. 

Outputs  of  interventions  in  favour  of  applications  can  be  monitored  by 
measuring  the  range  of  services  offered,  with  a  series  of  indicators  that 
strictly  depend  on the specific  application: number  of telework  stations  in a 
telework  centre,  number  of  on-line  courses  provided,  number  of  services 
provided  through  a  network  (where  the  term  "service"  must  be  precisely 
defined  within  the  extent  of the  specific  project). 

Result 

indicators 

Immediate  benefits for end-users  of telecommunications  infrastructure  can 
be  in  terms  of  network  performance  (bit  rate,  number  of  failures,  etc.). 
Results  of  application  interventions  can  be  monitored  by  programme 
indicators, such  as the  "number  of  addressees  or  recipients  involved  in the 
level  of 
programme"  (e.g.  people  trained 
exploitation  of  technology  facilities  (e.g.  of  telework  stations  in  a  telework 
centre),  and  the  "level  of  satisfaction  of  the  addressees  or  recipients 
involved". Applying  result indicators to the context  is usually not  consistent, 
because  of  the  experimental  nature  of  applications  programmes  and  the 
limited  number  of  addressees  or  recipients  involved. 

in  new  technologies),  the 

Information society 

Impact 

indicators 

for 

impact 

indicators 

Connections  to  new  or  upgraded  networks  after  1  year  and  the  level  of 
usage  of  a  network  are 
telecommunications 
infrastructure  interventions.  While  the  indicator  related  to  the  number  of 
connections  can  quite  easily  be applied  both to the  programme  and to  the 
context  (in  order  to  measure,  for  example,  the  penetration  of  ICT  within 
households),  generally  the  usage  of  a  network  cannot  be  applied  to  the 
context.  In  fact  this  indicator  depends  on  the  type  of  services  provided 
through  the  network  (CATV, for  example,  may  or  may  not  be an  alternative 
to  traditional  TV, depending  on  marketing  strategies  of  the  broadcasters), 
and  can  be  used  only  when  the  new  network  (and  /  or  services  provided 
through  it)  can  be  compared  to  the  pre-existing  one.  Furthermore,  the 
relationship  between  the  network  and  its  usage  must  be  well  known  and 
assessed.  For  example,  a  higher  speed  of  connection  to  the  Internet 
(provided by a broadband  network)  may result  in a reduced time of use (due 
to a more efficient  and less time-consuming  interactivity)  or  in an  increased 
total  time  of  use (due to  a considerable  number  of  new  users  attracted  by 
better  connection  conditions).  These 
relationships  are  still  under 
assessment  in the technology  community  and  clearer  evidence  is  required 
before developing  appropriate  indicators. 

Programme  level  impacts  of  application  interventions  can  be  evaluated 
through  "survival"  indicators  focusing  on  how  many  addressees  or 
recipients  are  still  involved  in  the  project  after  a  given  period  of  time 
(depending  on  the  specific  project).  This  type  of  indicator  gives  a  credible 
measurement  of  the  project's  feasibility.  They  can  also  be  evaluated  by 
using  "objective"  indicators  relating to the  aims of  the specific  project  (e.g. 
transportation  time  reductions  for  teleworkers,  start-ups  of  "virtual"  firms 
and so on). 

Context  impact  indicators,  on  the  other  hand, are usually  not  significant  in 
the  case  of  application  interventions,  because  of  the  limited  number  of 
addressees  or  recipients  involved  in  the  experiment. 
In  this  case,  a 
qualitative  evaluation  regarding  the  potential  contribution  of  the  specific 
application  (e.g.  a  telework  centre)  to  the  context  (e.g.  increase  or  minor 
decrease  of  employment  in a given  region)  should  be provided. 

Indicators for monitoring  and evaluation 

Example  33:  Deployment  of a broadband  Hybrid  Fibre Coax  (HFC) 

network 

Programme  Indicators 
(Related to  the  intervention  and 
its  effects) 

Context  Indicators 
(Related to the  assisted  area) 

Output  Indicators 

Progress 

Compliance  with  project  duration 
Rate of  completion 

Quantity 

Number  of  HFC kms  laid 

%  of  HFC kms  in the  total  network 
(in the considered  region) 

Capacity 

Number  of  buildings/apartments  passed 

%  of  buildings  /  apartments  passed 

Result  Indicators 

Network  performance 

Services  supplied 

Impact  indicators(l) 

Connections 

Bit  rate  capacity 
Number  of  network  failures 

Number of  services 
Number  of TV channels 
Hours of  broadcasting  supplied 
(through the  network) 

Number  of  households  that  have 
connected  their  equipment  (TV, PC) to 
the  HFC network  (after  6  months) 

PC and  /  or  CATV(2)  penetration  In 
households  in the  assisted  area 

ICT usage 

Number  of  subscriptions  to the  services 
Total hours of  connection  /  month 

Adoption  rate of  teleservices  in 
households in the assisted  area 

(1) Classification  of  the  main  teleservices  can  be  found  in  the  most  relevant  ICT  surveys  conducted  at 

the  European  level  (e.g. Measuring  Information  Society  1997,  ISPO). 

(2) CATV  stands  for  Cable  television. 

Information society 

Example 34:  Upgrading  of  an existing  narrowband  network 

with  ADSL(1)  technology 

Programme  Indicators 
(Related to the  intervention  and 
its  effects) 

Context  Indicators 
(Related  to the  assisted  area) 

Output  Indicators 

Progress 

Quantity 

Capacity 

See Example  33 

Number  of telephone  exchanges 
equipped with ADSL devices 

%  of exchanges equipped with ADSL 
devices  in the assisted area 

Number of apartments  potentially 
served  by the  equipped  exchanges 

%  of apartments  potentially  served by 
the  equipped exchanges  in the 
assisted area 

Result  Indicators 

Network  performance 

See Example  33 

Supply  of  services 

Impact  indicators 

Connections 

ICT usage 

Number of services  provided 
(interactive TV channels, Internet 
access  provision, etc.) 

Number  of  households  that  have  asked 
for ADSL connection  (after  6  months) 

%  of  households  that  have asked  for 
ADSL connection  in the  assisted  area 

Number of  subscription to  services 
(after  6  months) 
Total hours of connection  /  month 
(after 6  months) 

(1) ADSL  stands  for  Asymmetrical  Digital  Subscriber  Loop. 

Indicators for monitoring and evaluation 

Example  35:  Setting  up of  a  suburban  telework  centre 

Programme Indicators 
(Related to the intervention and 
its effects) 

Context Indicators 
(Related to the assisted area) 

Output  Indicators 

Capacity 

Number of (tele) work-stations installed 

Result  Indicators 

Quality of services 
supplied 

Impact  indicators 

Participation 

Usage 

Transport times 

Business start-ups 

Employment 

Number of opening hours  /  week 
Number of support services provided 
(training, secretarial support, e-mail 
server, web hosting, etc.) 

Number of firms participating in the 
initiative (after 6 months) 
Number of stable / mobile 
teleworkers(l) hosted (after 6 months) 

Exploitation of (tele) work- stations, 
i.e. time usage (after 6 months) 

Reduction in transport times for the 
teleworkers involved 

Time lost due to traffic congestion 
in urban areas 

Number of start-ups of new "virtual" 
firms 

Number of new businesses 
established in the region per year 

Number of jobs (FTEs incl. % held 
by women) created by start-ups of 
"virtual" firms 

Number of jobs (FTE incl. % held by 
women) created by new businesses 
in the region 

(1) Stable teleworkers  are employees  whose workstation  is located in a telework  centre (at least  3 days 
/  week, according to the accepted definition of  telework); the telework station is usually  rented by a 
company  on  a  contractual  basis.  Mobile  teleworkers  are  workers  who  use  telework  facilities  on 
demand, for a limited period of time (from  1 hour to  1 day). 
Since  recent  experiments  have  assessed  that  a  telework  centre  is  not  yet  profitable  with  only 
telework-related activity, most telework centres also provide facilities for traditional activities, such as 
on-site training  courses  and  business  support.  The  mix  of  services  provided  by the  specific  centre 
depends  on the  nature  of  business  activity  and  on  the  market  structure  of  business  support  in the 
region, and cannot  be precisely monitored with structured and general indicators. 

Information society 

Example 36: On-line education  project  in professional  training(1) 

Programme Indicators 
(Related to the intervention and 
its effects) 

Context Indicators 
(Related to the assisted area) 

Output  Indicators 

Activity of the operators 

Volume of training 
supplied 

Number of trainees 

Result  Indicators 

Qualification 

Accessibility 

Flexibility 

Number of on-line courses 
% of on-line training in total training 
(planned) 

Number of hours of on-line training 
dispensed (hours  χ trainees) 

Total annual number of trainees in 
the region 

Number of individuals trained 
(of which women) 

Number of trainees graduating in 
relation to total number of participants 

% of participants who declare that 
they could not have attended a 
traditional on site course because of 
distance constraints 

% of participants who declare that 
they could not have attended a 
traditional on site course because of 
time constraints 

Satisfaction of addressees 
or recipients 

% of trainees satisfied or very satisfied 
with courses 

Wastage 

Wastage rate in on-line courses 

Wastage rate in traditional courses 

Impact  indicators 

Placement 

% of participants having found a job 
after  12 months (of which women) 

Annual rate of reemployment of 
unemployed persons in the region 

(1) On-line education usually refers to different  intensities of ICT use in training activity, ranging from a 
complete  on-line  course  to  simple  on-line  tutoring  complementary  to  on-site  courses.  A  certain 
proportion  of  on  site  lessons  is often  required, in  order to  allow  participants  to  get  to  know  each 
other  and thus enhance  later  on-line group  learning. The mix  between on-line and on-site  training 
depends on specific  course subjects, teaching  methodologies  (e.g. personal vs. group  interaction) 
and the technologies adopted (the use of videoconferencing, for instance, reduces the need for on-
site  lessons). These elements are usually  specified  in the  project  documentation, and can thus be 
monitored. 

Other  interventions  having  an  impact  in the  field 

Interventions  in  other  fields,  which  include  ICT  applications  (even  though 
not  ICT-centred),  may  have  a  considerable  impact  on  the  development  of 
the  information  society. 

A  typical  example  is  the  use  of  ICT  as  an  alternative  way  to  provide 
information or to contact  potential addressees or recipients (e.g. by  building 

Indicators  for monitoring  and  evaluation 

information 

a web  site).  Many  kinds of  business  support  include  intangible services  like 
information, orientation and training, that could  be provided at least  partially 
through  the  Internet.  The  same  concept  could  be  applied  to  delivery  of 
traffic 
in  urban  traffic  control  projects  and  other  similar 
interventions. Another  example  is the use of telematic  networks to  enhance 
(e.g. 
communications  among 
epidemiology  projects  could  use  telematics  to  gather  data  from  general 
practitioners). 

in  a  project 

the  subjects 

involved 

In  such  cases  evaluation  may  be  conducted  by  using  output  indicators  to 
measure  the  level  of  technological  innovation  in  service  provision  (e.g.  the 
number  of  business  support  services  provided  through  a web  site).  Result 
indicators  should  measure  the  use  of  technology  in  the  project  by  the 
addressees  or  recipients  (e.g. to  access  information,  to  contact  operators 
or to send data, as shown  in the examples above). Finally, impact  indicators 
should  measure  the  impact  of  the  project  on  the  general  acceptance, 
interest  and  use  of  ICT  (e.g. %  of  addressees  or  recipients  who  declare  to 
have  become  more  confident  after  using  IT technology  during  the  project). 

Availability  and  collection  of  data 

Output  related  information  is  usually  available  or  can  easily  be  obtained  in 
a concise  version  from the  managers  of  the  funded  projects. 

Most  result  indicators  should  be  quantified  by  the  operators,  through  the 
monitoring  of  addressees  or  recipients,  although  not  all  indicators  are  fully 
accepted,  especially  the  ones  referring  to  the  most  innovative  applications 
(like telework). 

Impact  related  data  should  be  collected  among  the  addressees  or 
recipients  through  special  surveys  or  case  studies.  This  information  is 
generally  interesting for  operators,  who  might  thus  be  motivated  enough  to 
be  involved  in  data  collection.  Case  studies  can  provide  better  and  more 
accurate  information  than  surveys,  especially  in  interventions  concerning 
ICT application  development,  which  are often  specific  and  require  in  depth 
analysis.  This  confirms  that  qualitative  data  remain  predominant  in  relation 
to  quantitative  data  in the  field  of  ITC as  in  many  innovative  policy  fields. 

Context  indicators  for  interventions  in  this  field  relate  to  the  general  ICT 
context  of the  region  or  regions  in question. In case  of  interventions  for  ICT 
application  development,  as  previously  mentioned,  context  indicators  are 
mainly  qualitative  and  should  be  included  in  case  study  results.  Context 

Information society 

indicators  for  telecommunications 

infrastructure 

interventions  may  be 

derived  from  European  (e.g. ISPO  and  Eurostat)  and  national  ICT surveys. 

Key  indicators 

The  key  indicators  presented  below  lend themselves  to  comparisons  both 

between  different  interventions  of the same  programme  and  between  those 

of  different  programmes,  and  can  in  certain  cases  be  aggregated  at  a 

European  level. 

Level 

Key  indicator 

Resource 

Rate of consumption  of budget  (% of allocated  funds) 

Output 

Result 

Impact 

Rate of completion  of project  (% of objective) 
Compliance  with  project  duration 
Number  of potential  connections  (businesses  /  households) to 
networks of basic  services(1) 
Number  of hours of training  (on-line)  received  by participants of 
courses  (incl.  % of men /  women) 

Number  of  domestic  /  economic  units  receiving  a  level  of 
service satisfying  European  norms through the network 
Satisfaction  rate  (%  of  addressees  or  recipients  of  on-line 
courses) 
Immediate  success  rate  (number  of  addressees  or  recipients 
qualifying  at the end of the  course) 

Number  of  users  connected  to the new infrastructures,  broken 
down  in domestic  /  economic  units  (e.g. HFC network) 
Gross value added  generated  (after 12 months  in terms of euros 
/  year  /  employee) 
Net  employment  created  or maintained  (FTEs,  incl.  %  held by 
women) 

(1)When  making  comparisons  between  countries,  attention  must  be paid  to  some  country-specific 
elements, such as the digitallsation level of national telephone networks or the pre-existence of cable 
TV  coaxed  networks  in  some  areas.  Differences  in  ICT legislation  (universal  access,  regulatory 
environment, data privacy laws, etc.) must also be taken into account. 

Indicators  for monitoring  and  evaluation 

For  more  information  see... 

Eurostat  (1997),  Information  and  Communication  Services 

-  Annual 

Statistics  1980-1995.  Luxembourg:  Eurostat. 

Jackson,  P.J.  and  Van  der  Wielen  (1998),  Teleworking  - 

International 

Perspectives.  London:  Routledge. 

Smith,  C.  (1995),  'Measuring  the  usability  and  effectiveness  of  advanced 
in:  Held,  P.  &  Kugemann,  W.  F.  (eds) 
learning  technologies' 
the 
Telematics 
Telematics  for  Education  and  Training  Conference,  Düsseldorf  / 
Neuss, 24-26  November  1994, Amsterdam:  IOS  Press. 

for  Education  and  Training,  Proceedings  of 

Spectrum  Strategy  Consultants,  DTI 

(1996,),  Development 

of 

the 

Information  Society  - An  International  Analysis.  London:  HMSO. 

Thorpe,  M.  (1993),  Evaluating  Open  and  Distance  Learning,  2°  edition. 

Harlow  (UK): Longman. 

Competitiveness  of  S M Es  and  enterprises  in  general 

Structural  interventions  in the field of enterprises  consist  of the  provision  of 
industrial  sites,  direct  financial  support  and  business  support  services. 
Business  support  services  include  assistance,  information,  advice  and 
training,  which  is  provided  directly  by  business  development  agencies  or 
indirectly  by  consultants,  technical  experts,  etc.  The  ERDF  and  ESF  are 
both  relevant.  Interventions  concerning  enterprises  are also  financed  from 
EAGGF and FIFG. For the 1994-1999  programming  period, 15 to 20% of all 
resources  (between  23,000 and 30,000  million euros)  are being  specifically 
devoted  to  supporting  SME  development. 

The addressees  or  recipients  of  interventions  are mainly  SMEs  and  micro-
enterprises  with  a  priority  to  indigenous  firms.  Larger  enterprises  are  also 
eligible  for  support,  though  eligibility  conditions  are  much  more  restrictive 
than  in the case of SMEs. The rationale for specific  assistance to SMEs has 
several  dimensions.  One  is  that  they  are  more  exposed  to  economic 
fluctuations  than  larger  firms,  because  of  their  narrower  financial  and 
human  resource  base  and  their  focus  on  smaller  markets.  They  also  face 
greater difficulties  in raising external finance,  penetrating  overseas  markets 
and acquiring  managerial and other key skills, so may have greater need for 
advice, guidance and training. They  are generally  more labour intensive and 
create  more  jobs  per  unit  of  output,  and  they  may  be  more  innovative, 
adaptable and flexible than  larger  firms. 

Since competitiveness  of  SMEs  is a  political  priority  for  the  European  Union, 
systems  of  indicators  should  specifically  record  the  situation  of  SMEs 
whenever it is relevant. According to the principle of  "mainstreaming" (see Box 
17), not only should output indicators  be broken down into  large/medium/small 
enterprise, but also result, impact, and even resource indicators. 

There are critical specificities  in the monitoring and evaluation of support  for 
enterprises that shape the systems of indicators. Firstly, the level of  support 
is often  small  in relation to the activities  of the  recipient  firm  and this  raises 
difficulties  in understanding  its consequences, even for the manager.  Some 
interventions  provide  indirect  support  to  a group  (a network)  of  enterprises 
and  this  leads  to  contact  with  the  addressees  or  recipients  being  lost. 
Furthermore, the evolution  of  an enterprise  in a competitive  environment  is 
very  much  driven  by  the  context,  making  it  difficult  to  track  the  specific 
consequences  of  public  interventions.  Lastly,  the  relevant  territory  for  an 
enterprise  often  has nothing  to  do  with  the eligible  area. 

Indicators for monitoring and evaluation 

• 

Interventions  within  the  field 

Resource 

indicators 

Many  business  support  services  are  demand-led,  so  take-up  and  the 
resulting indicator of financial consumption can fluctuate over time according 
to factors  such as the  business  cycle and  macro-economic  conditions. 

Output 

indicators 

Outputs  of  interventions  concerned  with  providing  physical  infrastructure 
can be monitored with a physical indicator  like "floor space (m2) of  premises 
provided  or  refurbished". 

However,  many  types  of  business  support  are  essentially  services  (e.g. 
information,  advice,  management  training  and  consultancy)  rather  than 
physical  products  (e.g.  roads  or  factory  buildings).  In  these  cases,  output 
indicators  may  measure  the  activity  taking  place,  the  scale  of  services 
provided  (e.g.  number  of  business  advisers,  number  of  one-stop  support 
offices  created),  and the  volume  of  services  financed. 

In the  general  case  of  financial  or  service  support,  the  main  result  indicator 
is the  "number  of  businesses  assisted". This  is a rather  basic  indicator  with 
limited value on  its own. However, when combined with the level of  financial 
resources  spent  on  support,  it  gives  a  useful  indication  of  intensity  of 
support, which  may be helpful for comparative  purposes. This indicator  can 
be  broken  down  into  categories  like  the  average  size  and  anticipated  size 
of the assisted firms (e.g. in terms of the number of employees), the  present 
or future  activity  of  the  assisted  firms  (e.g. in terms  of  growing  or  declining 
markets). 

Result 

indicators 

indicators  provide 

Result 
productive  structures  in terms  of  business  formation  and  diversification. 

immediate  changes 

information  about 

to 

"amount  of  expenditure 

The 
indicator 
expenditures) 
measures  the  "leverage  effect". 

incurred  by  firms 

(investment  or  other  such 
in  relation  to  the  support  received" 

The  "proportion  of  assisted  businesses  in the defined  area or group"  is also 
a  relevant  indicator  for  interventions  in this  field.  This  indicates  the  degree 
of  penetration  of  the  programme  among  the  total  number  of  businesses  in 
the  defined  area, group  or  sector. 

Competitiveness of SMEs and enterprises in general 

A further result indicator is the "proportion of addressees or recipients reporting 
high satisfaction ratings". The indicator will have to  be based on a follow up of 
addressees or recipients at the end of their contact  with the programme. 

Impact 

indicators 

Interventions  concerned  with  providing  physical  infrastructure  produce 
specific  impacts  that  can  be  measured  with  a  physical  indicator  like  the 
"floor  space  (m2)  of  premises  which  have  been  bought  or  are  being  rented 
by  recipient  firms  after  one  /  three years". 

Support  for  firms  may  have  an  impact  on  the  attitudes  and  behaviour  of 
the people working  in them, and on various aspects  of the performance  of 
their  businesses.  It  may  increase  the  knowledge  and  skills  of  owner-
managers  and  workers,  or  help 
improve  business  planning, 
management,  marketing,  products,  and  production  and  distribution 
systems.  The  ultimate  objective  is  to  assist  companies  to  expand  and 
create  (or to  prevent  decline  of)  market  shares, turn  over, value  added  (in 
practice,  measured  in  terms  of  wages  and  salaries  plus  profits)  and 
eventually  to  create  employment  impacts. 

to 

Impact  indicators  must  incorporate  the  time-scale  over  which  effects  are 
produced  (normally  one  to  three  years  after  the  intervention).  Note  that 
these time  periods  are likely to  extend  beyond  the  life  of the  overall  SPD  / 
regional  programme, creating  management  difficulties. 

Support  may  also  have  knock-on  effects  on  the  performance  of  suppliers 
(increased  output),  customers  (improved  quality  and  /  or  price)  and 
competitors (reduced sales). These need to be considered when assessing the 
overall net impact of the interventions. There may also be wider multiplier effects 
arising from the employment  and income created in assisted businesses. 

However, the  principal objective  of most firms  is to  increase profits  and few 
businesses  have  employment  as  their  prime  objective.  In  most  instances 
employment  is  a derived  demand  arising  from  the  need  to  employ  people 
to get work  done. Businesses may perceive there to be a trade-off  between 
employment  and  profits  because  of the cost  of  wages. 

In  some  cases  (e.g.  assistance  towards  technical  innovation  in  products 
and processes, or improved management  and production systems)  support 
may  improve  business  efficiency  and  profitability,  but  not  create  any 
additional  jobs, especially  in the short-term.  It is generally  hoped that  such 
improvements  in competitiveness  will  generate  wider  economic  benefits  in 
the  longer term. 

Indicators for monitoring and evaluation 

Contextual  conditions  are  also 
in  a  depressed  economic 
environment, the impact of the same level of support on output and employment 
may be considerably  less than in more buoyant  conditions. Sensitivity to these 
circumstances  is important  in monitoring and evaluating  programmes. 

important: 

Some  firms  succeed  through  low  prices,  achieved  by  keeping  costs  down 
(including  labour)  and  perhaps  sacrificing  environmental  standards,  while 
others  succeed 
innovation,  product  differentiation  and 
specialisation,  and  high  skills  and  productivity.  Different  combinations  of 
impact  indicators  should  detect  these  differences. 

through 

Example 37: Support for entrepreneurs creating new businesses'1' 

Programme Indicators 
(Related to the intervention and 
its effects) 

Context Indicators 
(Related to the assisted area) 

Output  indicators 

Number of operators 

Number of one-stop support offices 

Volume of services 
provided 

Result  indicators 

Business creation 

Satisfaction rate 

Impact  indicators 

New business survival 

Direct employment 

Number of hours of expert advice 
received by addressees or recipients 
wishing to launch a business 

Number of new businesses launched 
following assistance (¡nel. % owned 
by women) 

Business creation rate in number of 
business  /  year per  10 thousand 

'  inhabitants (incl. % launched by 
j women) 

% of addressees or recipients who are 
satisfied or very satisfied with the 
support services provided 

Survival rate(2) of assisted business 
start-ups after  18 / 24 / 36 months 
(of which created by women) 

Number of net jobs(3) in the assisted 
new businesses after  18 months 
(FTEs incl. % held by women) 

Business survival rate after 
1 8 / 2 4 / 36  months 

(1) New businesses  are defined as commercial enterprises  of various  kinds (including  self-employment) 
that  have commenced  trading and which  have been  helped to  start  under  the relevant  programme. 
They do not include people who were on the programme  but dropped  out, or who are yet to start. 

(2) The survival  rate  is the  proportion  of  new  businesses  that  are still  trading  after  18  months. This  is 
beyond the normal  subsidy  period and is sufficient  to gauge whether or not the business is basically 
viable. 

(3) Employment  should  be measured In terms of  full-time equivalent  (FTE) jobs generated as a result of 
support  provided under the  programme.  Part-time jobs  may be converted  into  FTEs on the basis of 
two to one. Jobs can be defined as permanent  If they continue after the end of the intervention  or if 
they last for a pre-defined  period after the end of the intervention. 

Competitiveness of SMEs and enterprises in general 

Example  38:  Informational  support for SMEs  to  promote  exports 

Programme  Indicators 
(Related to  the  intervention  and 
its  effects) 

Context  Indicators 
(Related to  the  assisted  area) 

Output  indicators 

Number  of  addressees 
or  recipients 

Result indicators 

Satisfaction  rate 

Geographical 
diversification 

I m p a ct  i n d i c a t o rs 

Exports(3) 

Number  of  assisted  SMEs (1) (2) 

Number  of  SMEs  in the  eligible  area 

%  of  addressees  or  recipients  who  are 
satisfied  or very  satisfied  with  the 
support  services  provided 

Number of  SMEs becoming  new 
exporters 
Number of  SMEs exporting to  new 
markets 

%  of  export  sales  in the  turnover  of 
assisted  SMEs  after  18  months 

Exports  of  SMEs related  to GDP of 
the  area 

Value  added 

Value added generated after  18  months 

Direct  employment 

Number of  net jobs  created  / 
maintained  (FTEs incl. %  held by 
women)  in firms  in relation to  export 
sales arterie  months 

Average value  added  by  employee 
in the area 

Total  number  of  unemployed  In 
the assisted area 

(1) The  European  Commission  has  agreed  a  standard  definition  of  SMEs  as:  employing  less  than  250 
people;  having  an  annual  turnover  of  less  than  40  million  euro  or  balance  sheet  assets  of  less  than  27 
million  euro,  and  being  Independent,  or  having  no  more  than  2 5%  of  their  capital  owned  by  a  larger 
firm.  Small  enterprises  are  defined  as:  employing  less  than  50  people;  having  a turnover  of  less  than 
7 million  euro or assets of  less  than 5 million euro, and  being  'independent'.  Micro-enterprises  employ 
less  than  10  people. 

(2) A  minimum  definition  of  assistance  is  required  to  avoid  five-minute  conversations  with  businesses 
being  counted  as  support.  Within  the  framework  of  UK  objective  2  programmes,  "  assistance  "  is  a 
minimum  of  5 days  of  support,  or  its  financial  equivalent  (#3,500  euro). 

(3)  Export  sales  are  defined  as  the  value  of  all  sales  and  contracts  by  businesses  to  consumers, 
companies  and  organisations  outside  the  region  concerned.  A  further  indicator  may  be  the  value  of 
sales  beyond  the  country  concerned. 

Indicators for monitoring and evaluation 

Example  39:  Subsidised  infrastructure for  the setting  up  of  firms 

Programme  Indicators 
(Related  to  the  intervention  and 
its  effects) 

Context  Indicators 
(Related to  the  assisted  area) 

Output  indicators 

Quantity 

Hectares  of  industrial  sites  made 

available  for  firms 

Total  area of  industrial  sites  in 
the  area 

Number  of  addressees 
or  recipients 

Number  of assisted  projects  (of  which 
SMEs) 

Result  indicators 

Leverage  effect 

Investment  by  recipient  firms 

Impact  indicators 

Value  added 

See  Example  38 

Direct  employment 

Knock-on  effect 

Number of  net jobs  created (FTEs of 
which  held  by women)  in the  recipient 
firms  alter  18  months 

Regional firms  as a % of suppliers  to 
recipient  firms  after  18  months 

See  Example  38 

See Example  38 

Other  interventions  having  an  impact  in the  field 

Interventions  in other  fields such  as RTD and transport  are  likely to  have an 
impact  on  the  competitiveness  of  SMEs  and  larger  enterprises.  Impacts  of 
interventions  in  the  transport  field  are  often  indirect  and  mostly  occur 
through  macro-economic  mechanisms.  They  are  therefore  very  difficult  to 
monitor,  or  even  to  evaluate,  through  indicators.  Indicators  are  more  likely 
to  be  useful  in  the  case  of  RTD  interventions.  A  frequent  situation  is  one 
where  an  intervention  is  targeted  at  research  centres,  with  the  aim  to 
indirectly  strengthen  the  innovative  capacity  of  regional  enterprises,  and  to 
ultimately  increase  regional  competitiveness.  In this  case,  it  is  possible  to 
use specific  impact  indicators  in an  evaluation  survey. 

Competitiveness of SMEs and enterprises in general 

Example 40: Support for  projects  in regional  research  centres 

Programme  Indicators 
(Related to the  intervention  and 
its  effects) 

Context  Indicators 
(Related to  the  assisted  area) 

Output indicators  (in the field of  RTD) 

Number  of  projects 
assisted 

Number  of  research  projects  assisted 

Number  of  research  centres  in 

|  the  area 

Result indicators  (in the field of RTD) 

Project  progress 

Networking 

Impact  indicators 

Economic  benefits 

Value added(1) 

%  of  projects  successfully  concluded 
with  a publication  in  a refereed  journal 

Number of  regional  enterprises 
involved  in supported joint  research 
projects (of which SMEs) 

Number  of  regional  enterprises  that 
have  been  Involved  in  supported 
research  projects  and that  declare 
positive  spin-offs  after  18  months 
(of which  SMEs) 

Total value  added arising  from 
supported  research  projects 

(1) The value  added  generated  by  a research  project  is  likely to  occur  over  a long  period. In order  to  sum 
up  the  declarations  of  several  enterprises,  annual  value  added  should  be  weighted  by  using  a 
discount  rate. The  indicator  of  value added  should  also  be weighted  by  using coefficients  of  likelihood 
when  possible  future  spin-offs  are  estimated. 

Availability  and  collection  of  data 

Most output and result-related  data should  be available from the  managers 
of  projects  funded  through  the  Structural  Funds. They  should  be  required 
routinely  to  collect  simple  data on, for  example, the  number,  size and  type 
of  businesses  assisted,  and  satisfaction  with  services  provided.  This 
should  also  apply  to  businesses  indirectly  assisted  by  the  supply  of 
industrial  premises  or  by  supporting  networks  of  firms.  Standard 
breakdowns  should  be  systematically  applied,  at  least  for  the  size  of 
businesses  (0-10  employees,  11-50,  51-250,  more)  and  for  the  sector 
(growing  /  declining  demand). 

Impact  related  data  need  to  be  collected  from  the  businesses  themselves 
in special surveys. The use of  outside teams to collect this  information  may 
be  useful  in ensuring  that  answers  are  reliable. The cost  of  comprehensive 
surveys  is likely  to  be  prohibitive,  so  it  is  important  to  use sample  surveys 
and extrapolate  from them. Due regard  should  however  be paid  to the  fact 

Indicators for monitoring and evaluation 

that  extrapolation  involves  a  risk  of  error. The  complexity  and  sensitivity  of 
the  issues  surrounding  causality  and  attribution  of  effects  mean that  focus 
groups  and  face-to-face  interviews  with  businesses  are  more  likely  to  get 
closer  to  the  truth  than  postal  questionnaire  surveys.  Telephone-based 
interviews  may  be  a  compromise.  It  is  important  that  decisions  about 
survey  techniques  are  made  locally  in  the  light  of  their  cultural  suitability 
(see Volume  3 of  the  MEANS  Collection). 

Context  indicators  refer  to  the  regional  or  national  context  as  appropriate. 
They  may  be  available  from  European  and  national  administrative  records, 
statistics  and surveys  (e.g. those  collected  for  taxation  purposes).  Eurostat 
has  developed  a  special  database  of  regional  industrial  statistics  with 
annual  data  since  1994. 

Key  indicators 

The  key  indicators  presented  below  lend  themselves  to  comparisons  both 
between  different  interventions  of the  same  programme  and between  those 
of  different  programmes,  and  can  in  certain  cases  be  aggregated  at  a 
European  level. 

Competitiveness of SMEs and enterprises in general 

Level 

Key  indicator 

Resource 

Rate of  consumption  of  budget  (%  of  allocated  funds) 

Output 

Result 

Impact 

Number  of  contacts  between  operators  and  addressees  or 
recipients  (of which  SMEs) 
Number  of  project  applications  (of which  by  SMEs) 
Selection  rate (%  of  projects  in financial  terms  accepted  and  % 
of  which  are  proposed  by  SMEs) 
Selection  rate  for  projects 
in  rapidly  growing  sectors  (in 
proportion  to  the  average  selection  rate  and  %  of  which  are 
proposed  by  SMEs) 
Number  of  hours  of  expert  advice  received  by  addressees  or 
recipients  (e.g. to  launch a  business) 
Number  of  firms  receiving  direct  assistance  or  services  as  a 
result  of  the  programme  (%  of which  SMEs) 

%  of  recipient  firms  active  in  rapidly  growing  sectors  (%  of 
which  SMEs) 
%  of  recipient  firms  involved  in  high-tech  projects  (%  of  which 
SMEs) 
Satisfaction  Rate  (%  of  addressees  or  recipients  satisfied  / 
very satisfied  by services  provided) 
Leverage  effect  (private  sector  spending  generated  by  the 
programme  in relation  to financial  support  received) 

%  of  assisted  new  businesses  that  are  still  active  after  18,  24 
and 36  months 
Value  added  generated  (after  18  months  in  terms  of  euros  / 
year  /employee) 
Net employment created  or  maintained  (FTEs, %  of  which  are in 
SMEs  /  of  which  held  by  women) 
Regional  knock-on  effects  (regional firms,  %  of which  SMEs, as 
a  %  of  suppliers  to assisted  businesses  after  18  months) 

Indicators for monitoring and evaluation 

For  more  information  see... 

Bachtler  J.  and  Michie  R.  (1995),  'A  new  era  in  EU  policy  evaluation?  The 
appraisal  of  the  Structural  Funds'  in: Regional  Studies,  Volume  29, 
745-751. 

Foley  P.  (1992),  'Local  Economic  Policy  and  Job  Creation:  A  Review  of 

Evaluation  Studies'  in: Urban  Studies,  Volume  29. 

HM  Treasury  (1991),  Economic  Appraisal 

in  Central  Government:  A 

Technical  Guide  for  Government  Departments.  London:  HMSO. 

McEldowney,  J.  (1991),  'Evaluation  and  European  regional  policy'  in: 

Regional  Studies,  Volume  25, 261 -268. 

Turok,  I.  (1997),  'Evaluating  European  Support  for  Business  Development: 
in: 

the  Structural  Funds 
Evidence 
Entrepreneurship  and  Regional  Development,  Volume  9, 335-352. 

in  Scotland' 

from 

Urban  development 

The definition  of  urban  development  employed  here  is  "the set  of  activities 
which  affect  individual  and  social  welfare through  changes  in the  nature  or 
intensity  of  land  use in urban  areas, and the  regulation  of those  changes". 

Within the framework  of the Structural  Funds, interventions  have four  major 
objectives  in this  field. 

The first objective  is to generate an economic  impact in the form of jobs  and 
enterprise  development,  based on the  premise that  every  individual  should 
in  an  income-earning  activity  and  to  find  a 
be  able  to  participate 
remunerative  job.  Every 
to 
individual  should  have  equitable  access 
opportunities,  including  employment,  health  care  and  schooling,  without 
regard to social or ethnic origin or gender. The second objective is therefore 
to  generate  a  social  impact.  A  third  objective  is  that  of  generating  an 
environmental  impact  in the sense that urban activities have an effect on the 
natural  and  built  environment  in  urban  areas  and  on  the  use  of  physical 
resources.  The  development  of  urban  activities  supported  by  an 
intervention  should  be  sustainable  (i.e.  it  should  respect  the  natural  and 
built  environment  in  urban  areas  and  not  lead  to  the  depletion  of  natural 
resources). When  negative  environmental  impacts do occur, they should  be 
the object  of mitigation  measures. Last  but not least, interventions  have the 
objective of generating  a civic  impact, allowing  individuals to take an active 
part  in the processes which fashion their  lives and their  urban  environment. 
The logic of  urban development  often mixes these four dimensions,  making 
sector-based  indicators  largely  irrelevant. 

Some  Structural  Fund  interventions  are  specifically  targeted  at  urban 
development.  This  is the case for  the  URBAN  Initiative  and the  Urban  Pilot 
Projects  and  many  Objective  2 programme  measures. 

The  aim  of  urban  development  interventions  is often  twofold: fostering  the 
development  of  assisted  towns  and  cities  and  reducing  discrepancies 
between  poor  and  privileged  areas  within  them. These  two  objectives  are 
generally  seen  as  complementary  rather  than  competing.  When  the 
objective  is  global  urban  development, 
interventions  mainly  have  an 
economic  and  social  dimension. When  the  programme  aims  at  integrating 
poor  areas  into the city, most  interventions  tend to  develop  social  activities 
and  citizenship. 

Most Structural  Fund interventions, which  may seem to  be largely  irrelevant 
to  issues  of  urban  development  at  first,  can  sometimes  have  a  significant 

Indicators  for monitoring  and  evaluation 

impact  on  urban  areas. For  instance, structural  adjustment  occurring  in the 
fisheries and agricultural sectors can significantly  affect the welfare of  major 
coastal  towns  and  important  market  towns  in rural  areas. 

The  monitoring  and  evaluation  of  support  for  urban  development  has  a 
number  of  specific  characteristics.  Firstly,  the  relevant  areas  of  urban 
development  are  often  small, 
local  and  do  not  correspond  with 
administrative  or  statistical  boundaries.  Secondly,  urban  development 
encompasses  a  very  broad  range  of  aims  and  activities,  from  the  most 
technical  (e.g. providing  premises to  new  enterprises), to  the  most  political 
(e.g. enhancing  citizenship  in depressed  areas). 

Interventions  within  the  field 

Resource 

indicators 

financial  consumption  should  be  broken  down 

Indicators  of 
into 
expenditure  benefiting  different  towns  and cities and expenditure  benefiting 
poor  areas  within  the  assisted  towns  and  cities.  However,  unless  this  has 
been  planned  from  the  beginning  of  the  programme,  this  break  down  will 
not  be quantified  by the  monitoring  system  and an estimate  will  have to  be 
provided  through  a special  study  at the  evaluation  stage. 

Output 

indicators 

indicators 

Output 
like  completion  rates  and  compliance  with  project 
timetable  are useful for physical  projects  such as the  reclamation  of  derelict 
land  or  housing  renovation.  Output  indicators  will  also  include  a  physical 
measure  of  the  project  such  as  the  "number  of  houses  renovated"  or 
"hectares  of  derelict  land  reclaimed". 

Many  interventions  targeted  at  urban  development  offer  services  (e.g. 
types  of 
schooling,  housing,  and 
interventions,  urban  development  is  monitored  by  indicators  focusing  on 
the  activity  of  the  operator  and  the  volume  of  services  provided  (e.g. 
number  of  advice  sessions  χ  hours). 

In  the  case  of 

training). 

these 

Some  output  indicators  can  be  monitored  at  the  programme  level  in  the 
form  of the  proportion  of a given output  produced  in urban areas or  in  poor 
areas  within  the  assisted  towns  and  cities.  Nevertheless,  output  indicators 
may  fail  to  convincingly  separate  out  the  urban  part  of  a  development 
programme. For instance, a training course that  is organised  in the  business 
centre  of  a city  may  recruit  trainees  in the  poorest  boroughs. 

Urban development 

Result  indicators 

In the  case  of  urban  renewal  projects,  the  distinction  between  direct  and 
indirect  addressees  or  recipients  can  be  problematical.  For  example,  the 
redevelopment  of a derelict  industrial  site or an area of poor quality  housing 
improves  the  urban  environment  of  those  inhabitants  in  the  immediate 
vicinity.  It  is  for  this  reason  that  inhabitants  within  a  given  distance  (often 
defined  as  1 km) of  the  project  are considered  as addressees  or  recipients 
of  an  intervention. Therefore,  the  number  of  neighbouring  inhabitants  is  a 
result  indicator. 

Impact 

indicators 

It  is  relatively  easy  to  build  indicators  for  the  specific  impact  of  a  given 
measure. For  instance, addressees  or recipients  may  be asked to  rate their 
feeling  of  segregation  or  the  quality  of  environment.  Nevertheless, 
difficulties  may  occur  since  urban  development  programmes  are  typically 
made up of many small measures (or even of a simple collection of  projects) 
which  are supposed  to  produce  a global  change  in the  context  rather  than 
specific 
impacts  on  addressees  or  recipients.  Consequently,  much 
attention  should  be  paid  to  the  choice  of  impact  indicators,  which  can  be 
related to  both the  programme  and the  urban  context. 

Example  41 : Cre; 
coui 

ition of  new  organisations  providing  debt 
iselling and local credit  schemes 

Programme  Indicators 
(Related to  the  intervention  and 
its  effects) 

Context  Indicators 
(Related  to the  assisted  urban  area) 

O u t p ut  i n d i c a t o rs 

Number  of  addressees 
or  recipients 

Number  of  organisations  and  schemes 
receiving  support 

Result  indicators 

Credit  cost 

I m p a ct  i n d i c a t o rs 

Use 

Indebtedness 

Average  interest  rate offered  in %  of 
standard  commercial  rates 

Number  of  local  inhabitants  using 
supported  debt  counselling  and  local 
credit  schemes  per  year  after  one year 

%  of  users who  have succeeded  in 
eliminating  their  debts after  one  /  three 
years 

Number  of  local  inhabitants 

Indicators for monitoring and evaluation 

Example 42:  Renovation  of  a  rundown  urban  area 

Programme  Indicators 
(Related to the  intervention  and 
its  effects) 

Context  Indicators 
(Related to the  assisted  urban  area) 

Number  of  housing  units  renovated  or 
built 

| Number  of  shop  units  renovated  or  built 

Number  of  housing  starts  in the  area 

Output  indicators 

Quantity 

Result  indicators 

Quality  of  urban 
environment 

Estimated  number  of  residents  located 
in  the  vicinity(1)  of  the  renovation  area 

Number  of  inhabitants  in  the  area 

Impact  indicators 

Number  of  addressees 
or  recipients 

Value added 

Employment 

Residential  attractiveness 

Health 

Number  of  inhabitants  buying  or 
renting  housing  units 
Number  of  businesses  setting  up  in 
renovated  or  new  units  after  1  year 

Value added  generated  in  local 
businesses after  one  /  three  years 

Number  of  inhabitants  in  the  area 

Total value added  in the area 

Number  of jobs created  or  maintained 
in the  local area (FTEs of which  held by 
women)  after  one  /  three  years 

Number  of jobs  created in  the 
area(FTEs of  which  held  by  women) 
after  one  /  three  years 

%  of  residents  located in the  assisted 
urban areas  who  declare that  they 
wish  to  remain  in the  area  in the  next 
5  years 

Number  of  local  inhabitants  who  have 
had diseases associated  with  poor 
quality  housing 

(1)  In  terms  of  residential  areas, vicinity  can  be  defined  by  'a  distance  of  less  than  1 kilometre".  However, 

this  definition  must  be  adapted  to  practical  situations  found  in  residential  areas. 

Urban development 

Example 43: Support for community organisations 

Programme  Indicators 
(Related to  the  intervention  and 
its  effects) 

Context  Indicators 
(Related to  the  assisted  urban  area) 

Output  indicators 

Number  of  recipient 
organisations 

R e s u lt  i n d i c a t o rs 

Focus on less  favoured 
areas 

I m p a ct  i n d i c a t o rs 

Number  of  community  organisations 
that  have had their  project  accepted 
(with  less than  5, between  5 to  25, 
more than  25 unpaid  volunteers) 

%  of assisted  organisations  that  are 
located  in  poor  urban  sub-areas  or 
whose  members  live  in poor  urban 
sub-areas 

Activity  of  community 
organisations 

Number  of  adults active  in assisted 
community  organisations  after  one  year 

Civic  participation 

Segregation /  exclusion 

%  of adults active  in assisted  community 
organisation and who  declare  that  they 
'feel segregated /  excluded' 

Number  of  adults  In the  urban  area 
active  in at  least one  community 
organisation 

%  of  voters who voted at  latest  local 
elections 

%  of all citizens who declare  that  they 
'feel  segregated or excluded' 

Other  interventions  having  an  impact  in the  field 

A  typical  example  of  an  intervention  occurring  outside  the  field  of  urban 
development,  but  which  creates  impacts  within  the  field  is  the  case  of 
training  programmes  for  young  unemployed  workers.  In this  instance,  the 
initial  round  of  impacts  can  generally  be  evaluated  with  an  indicator  of  the 
"number  of  addressees  or  recipients  finding  a  job  within  χ  months". 
However,  many  of  the  second  and  subsequent  rounds  of  impacts,  whether 
associated directly or indirectly with urban development  (e.g. respectively an 
increase  in the  income  of  addressees  or  recipients  giving  rise to  increased 
private  investment  in housing  in the target  area, or the demonstration  effect 
whereby  employers  learn  to  have  a  less  negative  stereotype  about  the 
potential  contribution  of  young  workers  who  were  previously  unemployed), 
do not lend themselves to quantification  in the form  of indicators. In order to 
be  practical,  monitoring  will  normally  be  limited  to  results  and  to  the  first 
round  of  impacts.  In  many  cases,  only  in-depth  and  qualitative  evaluation 
studies will be able to track second and third round effects and  convincingly 
assess  if urban development  goals  have been attained. 

Indicators  for monitoring  and  evaluation 

Example 44: Training programme for long-term unemployed(l) 

Programme Indicators 
(Related to the Intervention and 
its effects) 

Context Indicators 
(Related to the assisted area) 

Output  indicators  (in the field  of training) 

Number of trainees 

Number of individuals trained 

Result  indicators  (in the field  of training) 

Qualification 

Number of trainees successfully 
completing the course 

Satisfaction of addressees 
or recipients 

% of trainees satisfied or very satisfied 
with courses 

Number of long-term unemployed in 
poor urban sub-areas 

Impact  indicators 

Unemployment 

Poverty 

Number of trainees who live in poor 
urban sub-areas and who are no longer 
unemployed after one year (incl. % of 
women) 

Average proportion of long-term 
unemployed who stop being 
unemployed within one year 

Number of trainees who live in poor 
urban sub-areas and who no longer 
receive any selective welfare benefit 
after one year (incl. % of women) 

% of households in urban areas 
receiving at least one selective 
welfare benefit 

(1) Long-term is defined as a period of  twelve months or more. 

Examples of selective welfare benefits are, for instance, RMI in France, Minimex  in Belgium. 

Availability  and  collection  of  data 

There  are  five  statistical  bases  for  the  concept  of  the  town:  urban  localities 
(defined  by the town's  administrative  boundaries  or  by  its status  in law), sub­
urban  areas  (often  defined  by  statistical  offices),  urban  agglomerations  or 
urban units (embracing continuously  built-up urban centres forming either part 
of  one  administrative  unit  or  a  group  of  several  units),  urban  regions 
(comprising  a  nucleus  town  and  its  sphere  of  influence  or  employment 
catchment  area, frequently defined in terms of commuting), polynuclear  urban 
regions, or  conurbations  (these  may  be continuously  built-up  but  comprise a 
number of centres around which social and economic  activities are polarised). 

In  most  countries,  each  of  these  levels  is  chosen  for  the  collection  of  at 
least  some  statistics  on  urban  development  activities,  depending  on  their 
purpose.  Operators  are  unlikely 
to  spontaneously  adopt  common 
definitions  of  urban  areas  when  they  collect  data.  Therefore,  programme 
managers  must  make  a very  strong  effort  in terms  of  co-ordination  if  they 
want  to  develop  monitoring  indicators  for  urban  development  (output  and 

Urban development 

results).  Thus,  an  indicator  of  urban  development  should  not  be  quoted 
without  making  clear  what  definition  of  urban  has  been  chosen.  As  far  as 
possible,  urban  areas  should  be defined  in accordance  with  administrative 
units  since  the  range  of  statistics  collected  tends  to  be  greater  for  such 
units.  Moreover,  these  limits  are relevant  for the  authorities  that  participate 
in "structural  interventions"  and  in  local  policy  initiatives. 

When  evaluation  data  are  collected  through  special  surveys,  it  is  easy  to 
design  ad  hoc  territorial  units.  However,  statistical  units  must  be  strictly 
respected  if survey  data are to  be compared  with  context  indicators. 

Box  19 -  Indicators  of the  urban  audit 

Themes 

Issue 

Socio-economic 
aspects 

Income  levels 

Structure  of  unemployment 

Example  indicators 

Total GDP for  a city 
area  /  total  resident 
population 

%  of  unemployed 
population  out  of  work 
for  more than  six  months 

Civic  involvement 

Participation  in  elections 

%  turnout  in  last  election 

Levels of  training 
and  education 

Vocational  skills 

Access  to  childcare 

Environment 

Traffic  patterns 

Green areas and  population 
density 

Recreation  and 
culture 

Library  facilities 

%  of  unemployed 
undertaking  vocational 
training 

Number  of children  aged 
between  18  months  and 
school  age 
Number  of  public  and 
private  crèche  places 

%  of  labour  force 
travelling  to  work  by  car, 
by  public  transport  (bus,  . 
tram, train) and  by  foot 
and  bicycle 

Total  residential 
population  /  total  land 
area  in  hectares 

%  of  local  residents 
registered  with  libraries 

Indicators for monitoring and evaluation 

National  and  regional  statistics  are  a  source  of  context  indicators.  For 
instance,  a  club  of  30  German  cities  has  developed  a  statistical  database 
on  urban  development,  including  very  relevant  indicators  like  opinions 
about  social  and  ethnic  integration.  There  are  also  numerous  ranking 
exercises  of  European  cities  and  local  regions  across  a very  wide  range  of 
indicators4.  A  broad  range  of  contextual  indicators  is available  from  official 
publications.  Key  sources  of  indicators  include  the  following:  Eurostat5, 
DGXVI's  publications,  DGXI's  Urban  Environment  Unit  and  Local 
sustainability  database  (DGXI:  http://cities21.com/europractice),  and  the 
European  Environment  Agency. 

DGXVI  is  building  upon  this  work  with  an  "Urban  Audit"  which  compares  a 
large  number  of  European  cities.  This  type  of  work  will  provide  a  new 
impacts  of 
resource  against  which  to  benchmark  and  evaluate  the 
Structural  Fund  interventions  as  well  as  a  bridge  to  begin  linking  project 
impacts  to  wider  imperatives  (see  Box  19). 

Key  indicators 

The  key  indicators  presented  below  lend  themselves  to  comparisons  both 
between  different  interventions  of the same  programme  and  between  those 
of  different  programmes,  and  can  in  certain  cases  be  aggregated  at  a 
European  level. 

4  This  includes  the  ambitious  work  of  Paul  Cheshire,  Dennis  Hay  and  Gianni 
Carbonaro, which examines  103 functional  urban regions, using a database of  200 
variables  aggregated  into  four  indexes  on  income,  unemployment,  migration  and 
travel. The league table had  Frankfurt,  Munich  and Venice at the top  and  Glasgow, 
Sunderland  and  Liverpool  at  the  bottom. 

5  Key  Eurostat  sources  Include  Portrait  of  the  Regions  and  RegioMap. 

Urban development 

Level 

Key  indicator 

Resource 

Output 

Result 

Impact 

Rate of  consumption  of  budget  (%  of  allocated  funds) 
%  of  projects  (in  financial 
disadvantaged  areas 

terms)  concerning 

the  most 

Selection  rate  (%  of  projects  in financial terms  accepted)  (men  / 
women) 
Rate of  completion  of  project  (%  of  objective) 
Compliance  with  project  duration 
Number of hours of services (training, counselling, etc.)  received 
by addressees  or  recipients 
Number  of  individuals  receiving  direct  assistance  or  services  as 
a  result  of  the  programme  (of  which  male  /  female),  including 
the  number  living  in poor  urban  sub-areas 

Satisfaction  Rate  (%  of  addressees  or  recipients  satisfied  / 
very  satisfied  by  services  provided) 

%  of  addressees  or  recipients  not  needing  the  help  of  services 
after  18  months 
%  of  addressees  or  recipients  employed  after  18  months  (in  a 
stable  long-term  job) 
Residential  attractiveness (%  of  inhabitants  wishing to remain in 
the area) 

For  more  information  see... 

Bentham, OG.(1985),  'Which Areas  Have the Worst  Urban  Problems  ?' in: 

Urban Studies,  Volume  22  ,119-131. 

Coombes,  M.  Raybould,  S.  and  Wong,  C.(1992),  Developing  Indicators  to 
Assess  the  Potential  for  Urban  Regeneration,  Department  of  the 
Environment  Inner  Cities  Research  Programme.  London: HMSO. 

European  Commission 

(1997),  Community 

involvement 

regeneration.  Luxembourg:  Official  Publications  Office  of 
European  Community. 

in  urban 
the 

European  Commission  (1997,),  Europe's  Cities:  Community  Measures  in 
Urban  Areas.  Luxembourg:  Official  Publications  Office  of 
the 
European  Community. 

Healey and  Baker (1997), Business  Base Across  the  World.  London: Healey. 

Indicators for monitoring and evaluation 

OECD  (1997),  Seffer  Understanding  Our  Cities:  The  Role  of  Urban 

Indicators,  Paris:  OECD. 

Robson,  B.  Bradford, M.,  Deas, I.  Hall, E. Harirson, E. Parkinson,  M.  Evans, 
the 

R.  Garside,  R  Harding,  A.  &  Robinson,  E  (1994),  Assessing 
Impact  of  Urban  Policy.  London,  HMSO. 

Indicators for three major aims 

Economic  d e v e l o p m e nt 

One of the major aims of  structural  interventions  is to  promote  regional  and 
local  economic  development.  Interventions  financed  by  Structural  Funds 
also  pursue  global  objectives  like  creating  job  opportunities  and  reducing 
disparities 
in  the  quality  of  life.  However,  because  these  aims  are 
considered  as  both  social  and  economic  goals,  they  are  the  subject  of 
information  sheets. 

The ultimate  goal of economic  development  is overall  income growth of  the 
population  in assisted  areas. According  to a number  of assumptions,  some 
of  which  will  be  discussed  later,  the  growth  in  the  volume  of  regional 
economic  activity  (i.e. the  development  of  the  economy)  is  a proxy  for  the 
income  growth  of the  population. 

Economic  measures  dominate  most  structural  programmes.  As  for  non-
economic  measures, these too are often guided  by economic  considerations 
at  the  design  and  implementation  stages.  This  focus  on  the  economic 
dimension  as  the  main  catalyst  of  regional  development  gives  economic 
indicators  a  particular  status.  The  most  direct  contributions  to  economic 
development  take  place  in  fields  such  as  transport  infrastructure,  energy, 
information  society,  training,  RTD,  enterprises,  tourism,  agriculture  and 
fisheries.  Many  themes  covered  in  this  Volume  interact  with  the  economic 
sphere  and  involve  various  factors  that  drive  economic  performance  (e.g. 
attractiveness,  accessibility,  etc.).  Consequently,  most  of  the  related 
indicators  are  discussed  separately  in the  relevant  sections  of  this  Volume. 
This  section  focuses  on  economic  indicators  stricto  sensu,  which  include 
income, GDP, value added, investment, exports, etc. Since effects on regional 
and local economic  development  arise at the impact  level from  interventions 
in other fields, the section  "Interventions  in the field"  is not applicable  here. 

• 

Impacts  on economic  development 

Impact 

indicators 

Indicators  that  measure  economic  development  primarily 
focus  on 
impacts,  and  especially  the  most  indirect  impacts  that  are  of  a  macro-
economic  nature. 

Indicators for monitoring and evaluation 

In  order  to  evaluate  the  comparative  performance  of  a  region,  a  large 
number  of  context  indicators  are  available  at  the  sub-regional,  regional, 
national and  European levels. The main indicator  for appreciating  economic 
development  is  GDP  per  capita,  which  measures  the  total  value  added 
generated  by  a  region's  economic  units  in  relation  to  the  number  of 
inhabitants.  Since  value  added  is  eventually  distributed  in  the  form  of 
wages, interest, taxes and so on, GDP is considered  as a proxy for  regional 
income.  However,  regional  GDP  per  capita  does  not  correspond  exactly 
with  regional financial  resources  distributed  to  the  regional  population.  It is 
also  important  to  bear  in mind  that  inter-regional  social  transfers  occur  and 
that  a  proportion  of  shareholders,  and  to  a  lesser  extent  employees  of 
regional  enterprises,  are  not  necessarily  resident  in the  region. 

individual 
Since  GDP  is  an  aggregate  of  value  added  generated  by 
economic  units, the  use of  both  of these  indicators  creates  interesting  links 
between  programme  impacts  and  the  regional  context.  The  indicator  of 
"value  added  generated  by  supported  projects"  can  be  viewed  as  the 
micro-economic  counterpart  of  the  regional  GDP.  However,  the  micro-
macro  link  is subject  to  much  reservation. 

The different  ways  of  measuring  value  added  are  presented  in  Box  20. 

Box 20  - Value  added: a  key  impact  indicator 

Value added  is a key impact  indicator  as it not only captures the wealth  generated 
in  the  local  area  through  Structural  Fund  interventions,  but  can  also  be  used  to 
assess  the  'quality'  of  employment  opportunities.  Value  added  can  be  defined 
and  measured  in a number  of ways and  at a number  of  levels. For example, at  the 
level  of  an individual  economic  unit,  it can  be measured  as the: 

•  difference  between  the  revenue  generated  by  the  sale  of  output  and  the  value 
intermediate  goods) 

inputs  (raw  materials,  services  and 

of  non-labour 
consumed  to  generate  that  output; 

or  as the: 

•  combination  of  returns  to  factors  of  production  employed. This  comprises  the 
wages  and  salaries  paid  to  labour,  plus  the  profits  generated  for  owners,  plus 
rental  paid  for  the  use  of  land. 

In practice,  value  added  is  measured  in terms  of  wages  and  salaries  plus  profits. 

The  aggregation  of  value  added  across  all  productive  sectors  of an economy  can 
be termed  Gross Domestic  Product  (provided all payments to factors  are remitted 
within  the  region)  which  is an  indicator  of  regional  wealth  generation. 

Economie development 

An initial problem is that the expected  impact can arise a long time after  the 
intervention,  since  structural  adjustment 
is  a  long-term  process  of 
adaptation  involving  social,  technological  and  economic  dynamics.  The 
effect  on  wealth  creation  can  only  be  recorded  after  production  processes 
have  been  effectively  transformed.  Thus,  the  economic  impact  of  an 
investment  subsidy  will only  materialise after the supported  investment  has 
come  into  operation,  a  process  that  may  take  several  years.  Furthermore, 
economic  performance  depends to  a large extent  on the  prevailing  macro-
economic  conditions.  The  effectiveness  of  interventions  will  be  greater  in 
periods  of  sustained  economic  growth  than  when  the  economy  is  going 
through  a  period  of  recession.  For  example,  the 
leverage  effect  of 
investment subsidies  is likely to  be greater when a high level of demand  can 
be forecast  over  a significant  period  rather  than  when demand  is  declining 
or weak. 

Box  21  -  Regional  disparities 

How  can  indicators  give  a  clear  view  of  the  global  economic  performance  of  a 
region?  Given that  regions are the subject  of  a wide variety  of  external  pressures 
(economic  climate,  international  competition,  national  and  supra-national 
regulation  and  policy  decisions,  etc.), few  clear-cut  conclusions  can  be  obtained 
from  the  analysis  of  absolute  performance.  The  construction  of  indicators  in 
relative  terms  is  more  appropriate.  Considering  the  economic  performance  of  a 
sub-region  (for  example, a NUTS 3 region eligible for Objective 2), disparities  can 
be measured  by referring to the region  (for example, the  relevant  NUTS 2 region), 
the  relevant  national  state  and  Europe.  But  why  is  it  necessary  to  take  so  many 
referentials  into  account?  This procedure  allows, for example, observers to  see if 
the  region  is  performing  comparatively  badly  in  relation  to  both  the  national  and 
European  economies,  but  has  a  better  performance  than  the  region  to  which  it 
belongs.  More specifically,  consider  a sub-region  for  which  both  GDP  per  capita 
and  GDP  growth  are  inferior  to  these  observed  for  the  region,  the  state  and 
Europe. 
is 
implemented which  leads to a reduction  in the difference  between the  sub-region 
and  its  region  but  not  vis-à-vis  the  state  and  Europe.  Such  an  observation  can 
give  an indication  that  the first  step  is being  successfully  undertaken  in this  sub-
region  (compared  to  the  region)  and  some  improvement  can  be  expected  in  the 
near  future  (compared  to the  state and  Europe). 

its  development,  a  policy 

to  stimulate 

intervention 

In  order 

It  is  useful  to  recall  that  indirect  and  induced  quantitative  impacts  of 
measures  are  generally  larger  than  direct  ones.  Subsidies  and  services 
financed  by  the  Structural  Funds  can  be  expected  to  generate  additional 

Indicators  for monitoring  and  evaluation 

to  creating  value  added,  generating 
turnover,  which  will  contribute 
additional  incomes  in the  region and  benefiting  regional  businesses, and  so 
provide  added  impetus  to  a  new  round  of  growth  within  the  regional 
economy.  If Structural  Fund support  leads to the firm  ordering from  regional 
sub-contractors  (knock-on effect), this demand  will also generate  additional 
value  added. This  process,  by  which  an  injection  of  expenditure  creates  a 
subsequent  series  of  waves  of  investment  and  consumption,  is  known  as 
the  "Keynesian  multiplier".  Only  econometric  models  or 
input-output 
analysis are able to assess the  links between the value added  generated  by 
the  programme  at the micro  level and GDP  at the  macro  level. Because  this 
type of analysis is time consuming and expensive, related indicators  cannot 
be quantified  on  a  regular  basis. 

Since  the  development  of  regional  economies  is  influenced  by  a  large 
variety  of  external  pressures,  few  clear-cut  conclusions  can  be  obtained 
from  the  analysis  of  their  absolute  performance.  The  construction  of 
indicators  allowing  the  relative  evolution  of  regional  disparities  to  be 
compared  is  more  appropriate  (see Box  21). 

Result,  output  and  resource  indicators 

The  micro-macro  problem  can  be addressed  in a more  operational  fashion 
through  the  following  proposal.  Instead  of  summing  up  individual  impacts 
to  more and  more  macro  levels, a reverse approach  can  be adopted  that  is 
based  on  the  mainstreaming  principle.  This  principle  has  been  applied  in 
the field  of  equal opportunities.  Its use implies that focus  is primarily  put  on 
a given strategic  priority.  Indicators  are broken  down  in accordance with the 
priority  when this  is relevant.  Context  indicators  take  into account  a  priority 
at  the  macro  level  and programme  indicators  at  micro  level. In the  example 
of the promotion of economic  diversification  (illustrated  in Box 22), activities 
in rapidly  growing  sectors are taken  into  account  by  breaking  down  data  at 
each  stage  of  the  causal  chain.  Consequently  the  resources  devoted  to 
interventions  in  rapidly  growing  sectors,  and  the  outputs,  results  and 
impacts  produced  in these  sectors  are clearly  identifiable. 

Economie development 

Box 22 - Appi 
theí 

ying the  principle  of  mainsti 
itrategic  priority of  economi 

earning: An example  of 
c  diversification 

Programme  Indicators 

Context  Indicators 

%  of  regional  GDP in  rapidly 
growing  sectors 

Impact 

Result 

Output 

Resource 

Value added  generated  after 
one year  (of which  in  rapidly 
growing  sectors) 

Leverage  effect  (of which  in 
rapidly  growing  sectors) 

Number  of  economic  units 
supported  (of which  in  rapidly 
growing  sectors) 

Total  budgetary  expenses 
(of  which  to  support  projects 
in rapidly  growing  sectors) 

Depending  on  the  economic  development  strategies  selected  by  various 
programme  designers,  the  mainstreaming  principle  can  apply  to  various 
priorities.  Below  are presented  three frequently  cited  priorities. 

Many  programmes  are  built  upon  a  strategy  of  economic  diversification. 
Financial  assistance from  Structural  Funds  is aimed at  reducing  the  weight 
of  declining  sectors  as  a  proportion  of  regional  economic  activity.  In  such 
instances,  it  can  be useful to  break  down  major  programme  indicators  into 
growing  /  declining  sectors  (e.g.  amount  of  subsidies,  number  of 
beneficiaries, amount  of leveraged  private investment). One of the  problems 
with this  approach  is that  there  is  no  standard  typology  available  to  make 
this  distinction.  Furthermore  the  classification  of  activities  in  terms  of 
growing  /  declining  sectors  is  a  rather  delicate  distinction  to  make  given 
the complex  way  in which  markets  are segmented  and the  cyclic  nature  of 
certain activities. Another  problem  is that within declining  markets there are 
always  rapidly  growing  niches.  In  the  case  of  programme  indicators,  it 
seems  possible  to  categorise  supported  projects  by  using  the  following 
questions:  "What  is  the  most  relevant  market  for  the  project?"-"Has  this 
market  grown  significantly  during  the  last five years?"-"ls  this  market  likely 
to grow  within  the  next  five years?" 

Economic  development  programmes  also  include  a  mix  of  measures 
local  initiatives  and  attracting  new  productive 
targeted  at  promoting 
activities.  A  high  degree  of  inward  investment  compared  to  local  projects 

Indicators for monitoring and evaluation 

located  outside  the  region.  As  a  consequence 

increases  economic  dependency  on  activities  managed  from  decision 
centres 
the  regional 
economy  is  likely  to  become  more  sensitive  to  external  shocks.  The 
stimulation  of  local  entrepreneurship  is  a  process  that  produces  a  less 
visible  impact,  but  these  impacts  are  more  firmly  rooted  in  the  regional 
territory,  and  are  more  likely  to  be  longer  lasting.  For this  reason  it  may  be 
considered  as  a  strategic  priority.  In  such  a  case,  it  is  interesting  to  break 
down  as  many  indicators  as  possible  into  projects  of  an  endogenous  / 
exogenous  nature. 

Some  economic  units  are active  in markets  that  are  open  to  European  and 
worldwide  competition  (e.g.  automobile  or  computer  producers),  while 
others  are located  in smaller  regional or  local markets  (e.g. housing  or  retail 
trade).  This  means  that  the  latter  are  to  a  certain  extent  sheltered  from 
external  competition.  Public  support  for  the  sheltered  sector  can  prove  to 
be  effective  in the  short  run,  but  only  the  growth  of  the  competitive  sector 
can  generate  long  term  economic  development.  If  a  strategic  priority  is 
given to  internationally  competitive  activities, it  is interesting to  break  down 
programme  indicators  into  projects  situated  in the  competitive  /  sheltered 
sectors. 

Economic  development  and diversification  are  primarily  measured  in  terms 
of  impacts  However,  according  to  the  principle  of  mainstreaming  the 
strategic  priority  should  also  be  monitored  in terms  of  results,  outputs  and 
resources. 

Economie development 

Example  45: Support  for  the  construction  of  a  conference 

and  exhibition  centre 

Programme  Indicators 
(Related to the  intervention  and 
its  effects) 

Context  Indicators 
(Related to  the assisted  area) 

Output  indicators 

Progress 

Compliance  with  project  duration 
Rate of  completion 

Physical  output 

Floor space (m2) 

Result  indicators  for  e n d o g e n o us  economic  development 

Usage 

Number  of conferences  and  exhibitions 
organised  (% of  conferences  and 
exhibitions that  are mainly  linked  with 
local economic  activities) 

Locally  owned and managed firms as 
a % of  enterprises  in the area 

I m p a ct  indicators  for  e n d o g e n o us  e c o n o m ic  d e v e l o p m e nt 

Commercial  contacts 

Usage 

Number  of  new  commercial  contacts 
(incl. % for  locally  owned  and  managed 
firms)  due  to the  activity  of  centre  after 
one year 

Number of firms  having used the  centre 
for  promoting their  sources after one 
year  (%  of which  locally owned and 
managed) 

Example  46:  Direct  support  for  capital  investment  in  enterprises 

Programme  Indicators 
(Related to  the  intervention  and 
its  effects) 

Context  Indicators 
(Related to  the assisted  area) 

Output  indicators  for  diversification 

Number of addressees 
or  recipients 

Number of firms  receiving  grants 
(% of which  In rapidly  growing  markets  / 
sectors) 

Result  indicators  for  diversification 

Leverage effect 

Additional private  investment (% of 
which  rapidly growing  martlets / sectors) 

Impact  indicators  for  diversification 

Diversification 

Annual value  added  that  has  been 
generated  by  investments  in euros  / 
year  after  one year  (%  of  which  in 
rapidly  growing  markets  /  sectors) 

%  regional  GDP generated  in  rapidly 
growing  markets  /  sectors 

Indicators for monitoring and evaluation 

Example  47: Venture  capital  scheme 

for  small  business  development 

Programme  Indicators 
(Related  to the  Intervention  and 
its  effects) 

Context  Indicators 
(Related to  the  assisted  area) 

Output  indicators for  competitiveness 

Number of  addressees 
or  recipients 

Number of  SMEs having  had at  least 
one loan  backed by  venture  capital 
(incl. %  of  which  In non-sheltered  sectors) 

Result  indicators  competitiveness 

Leverage 

Additional  private  investment  that  is 
generated  by  loans  (¡nel. % of  which  in 
non-sheltered  sectors) 

Impact  indicators for  competitiveness 

Value  added 

Exports 

Annual  value  added  that  has  been 
generated  by venture  capital  backed 
loans  (¡nel. % of  which  In  non-sheltered 
sectors) 

Exports that  have  been generated  by 
assisted SMEs after  one year  (of  which 
in  non-sheltered  sectors) 

%  of  non-sheltered  sectors 
in  regional  GDP 

Exports  as a %  of  regional GDP 

Availability  and  collection  of  data 

Information  for  programme  indicators  at  the  output  and  result  level,  whilst 
largely  available  from  the  monitoring  systems  of  operators,  are  not 
automatically  broken  down  into  the  categories  (i.e.  growing  or  declining 
sectors,  exogenous  or  endogenous  development,  competitive  or  sheltered 
activities). 
therefore  be 
undertaken  as  an  integral  part  of  an  evaluation.  As  for  data  to  provide 
information  for  gross  impact  indicators,  these  will  often  have  to  be  based 
on  addressees  or  recipients'  expectations.  Net  impact  indicators  will  have 
to  rely  upon  complementary  surveys. 

If  necessary,  this  classification  of  data  will 

Many  economic  indicators  at  the  context  level, except  income  per  capita, 
are  presently  available  for  a  majority  of  regions  and  published  in  the 
Eurostat  Regional  statistical  yearbook.  Examples  of  available  indicators 
include GDP per capita,  productivity, value added  and  investment.  Most  of 
these context  indicators  are broken  down  into sectors  such as: agriculture, 
energy,  manufacturing  industry,  construction,  market  services,  non-market 
services. 

Economie development 

As these  indicators  are vital  for  assessing  regional  economic  performance, 

some  additional  efforts  should  be devoted  to  collect  them  for  all the  NUTS 

2  regions  and  accelerate  their  updating.  Given  that  the  targeting  of  the 

European  regional  policy  also descends  as far as the  NUTS 3 level, it will  be 

of  interest to  select the  most  representative  indicators,  at  least  one  in  each 

category  and  whenever  possible  the  most  representative,  and  collect  them 

at this  level. 

Key  indicators 

A number  of  key  indicators,  presented  below,  are  suitable  for  aggregation 

and  comparison  of  Structural  Fund  interventions  in  the  field  of  economic 

development 

Level 

Key  indicator 

Resource 

Output 

%  of budget  devoted to projects of locally owned  and  managed 
firms 
%  of budget  devoted to projects  in rapidly  growing  markets 
%  of budget  devoted to projects  in non-sheltered  sectors 

Number  of  economic  units  receiving  direct  assistance  or 
services  as  a  result  of  the  programme  (of  which  involved  in 
locally  owned  and /  or  managed  firms,  with  rapidly  growing 
markets,  in non-sheltered  sectors) 

Impact 
(programme) 

Value added  generated  by the programme  after  18 months in 
terms of euro /  year  /  employee (of which  generated by locally 
owned  and /  or  managed  firms,  by  firms  in  rapidly  growing 
markets,  by firms  in non-sheltered  sectors) 

Impact 
(context) 

Investment  /  capita,  GDP /  capita,  Value  added  /  employee. 
Exports  in  %  of  regional  GDP, %  of  regional  GDP in  locally 
owned  and  managed  firms,  rapidly  growing  markets,  non-
sheltered  sectors 

Employment 

The promotion of viable employment  is a major component  of the  Cohesion 
Policy  and  activity  to  this  end  is  supported  by  all  of  the  Structural  Funds. 
For  example,  training  activities  or  support  for  new  enterprises  will  have 
direct employment  effects, while investment  in health infrastructure or in the 
protection  of  the  environment  will  also  create  jobs,  although  in  a  more 
indirect  manner.  However,  it  remains the case that  virtually  all of the  activity 
assisted  under  the  Structural  Policies  may  eventually  have  some 
employment  effect,  though  these  are  not  generally  uniform  across  the 
population and  may  in fact exacerbate the situation  of certain categories  of 
addressees or recipients, for instance women or the long term unemployed. 

The aims of combating  unemployment  and promoting employment  have for 
long been an implicit  strand of European Community  policy. However,  since 
the mid-1990s these aims have become  increasingly  explicit  and  pervasive 
across  a  wide  range  of  Community  policies,  including  the  Structural 
Policies. Beginning with the White Paper on Development,  Competitiveness 
and  Employment  and  the  conclusions  of  the  subsequent  Essen  Council, 
activity 
in  this  policy  area  proceeded  with  proposals  made  by  the 
Commission  in  its  paper  "Action  for  Employment  in  Europe"  (1996).  Policy 
continues to develop with the formulation of annual Employment  Guidelines 
by  the  Council  and  national  employment  action  plans  by  each  Member 
State.  These  efforts  are  intended  to  respond  to  the  process  of  structural 
change, which  the  European  economy  has  experienced  over  the  past  two 
decades, with a set of structural labour market  policies that will enhance the 
creation  of  viable employment.  Strategic  priorities  have  been  progressively 
refined  and 
improving 
employability,  developing  entrepreneurship,  encouraging  adaptability  of 
businesses  and  employees  to  economic  changes,  and  strengthening 
equality  of  opportunity  (Council  Resolution  on  the  1998  Employment 
Guidelines  -  December  1997). The  last  priority  is the  subject  of  a  separate 
information  sheet  in this  Volume. 

they  now  concentrate  on  four  main 

lines: 

Impacts  on  employment 

Because  of  the  horizontal  and  pervasive  nature  of  employment  concerns, 
the  commentary  provided  in this  section  describes  interventions  that  have 
isolate 
to  artificially 
an  impact  on  employment 
employment  interventions  and  discuss  indicators  related to  them. 

rather  than  trying 

Indicators for monitoring and evaluation 

Fields  from  which  the  following  examples  are  drawn  are  respectively, 
training and tourism,  but any other field  could  have been referred to  if other 
examples  had  been  chosen. 

indicators  are  of  particular 

Some  resource 
interest  for  the  field  of 
employment.  These  indicators  measure  "human  resources"  used  by  the 
programme,  i.e.:  the  mobilisation  of  workforce  in the  implementation  of  an 
intervention. For example, the  "number  of trainers employed  by an  assisted 
training  course"  is  an  indicator  of  the  human  resources  invested  in  the 
implementation,  but  it  is  also  a  component  (although  a  minor  one)  of  the 
employment  related to the  project.  Investment  in physical  infrastructure  will 
in most  cases itself  have short-term consequences  for employment  through 
construction  activities.  Since  this  type  of  job  tends  to  be  temporary  in 
nature, units  of  measurement  should  be  expressed  in job  χ years to  clearly 
show  its transitory  nature. Strictly  speaking, this type of employment  should 
be  considered  as  a  resource  rather  than  an  impact.  In  order  to  simplify 
matters,  all  employment  created  is  often  grouped  together  and  quantified 
with  a single  indicator. 

The outputs  of  interventions  having  an  impact  on  employment  tend  to  take 
the  form  of  enhancing  factors  that  have  been  identified  as  promoting  job 
creation.  Consequently,  output 
targeted  on 
employment  do  not  focus  directly  on  employment  itself  but  rather  on  the 
infrastructure  and  services  which  are  needed  to  enhance the  employability, 
adaptability  or  entrepreneurship  of  target  groups.  Interventions  that  have 
job  creation  as  an  output  are  certainly  exceptional  (e.g. a  measure  which 
consists  of training personnel who would then  become trainers, would  have 
job  creation  as  an  output  indicator). 

for  actions 

indicators 

In  many  instances,  the  creation  or  maintenance  of  jobs  may  be  a  result  of 
an intervention. This occurs when  employment  comes about as a direct  and 
immediate  consequence  of  assisted  activity  (e.g.  job  of  a  new  assisted 
entrepreneur,  job  of  a  new  assisted  aquaculture  farmer). 

Employment  becomes  an  impact  when  it is created or  maintained as a  mid­
term  and  /  or  indirect  consequence  of  the  intervention.  For  instance, 
employment  impacts  may  consist  of  lasting  benefits  for  those  who  have 
received  assistance  (e.g.  trainees  whose  jobs  have  been  maintained, 
trainees  who  are  employed  after  12  months).  Jobs  may  be  created  in  the 
(e.g.  ports, 
management  and  maintenance  of  new 
telecommunications  networks,  and  waste  disposal  plants).  All  of  these  are 

infrastructures 

Employment 

examples  of  employment  that  is  a  direct  mid-term  effect  of  structural 
interventions.  Employment  impacts  may  also  consist  of  indirect  effects, 
mainly  through  the  development  of  addressees'  or recipients'  activities  (e.g. 
jobs  generated  in  assisted  firms  after  12  months,  jobs  generated  in  the 
suppliers  of  assisted  firms,  jobs  generated  in  the  region  through  income 
multiplier effects). Employment  impacts may arise in a more indirect way  due 
to: the growth of economic  activity which  is either generated  or sustained  by 
the  existence  of  new  infrastructure,  the  enhanced  competitiveness  of 
assisted  firms,  or  the  enhanced  competitiveness  of  companies  benefiting 
from  a  better  qualified  management  and  /  or  workforce.  It  may  be  argued 
that  analysis  of  indirect  effects  is extremely  complex  and that  concern  with 
such  impact  indicators  is  more  the  role  of  qualitative  in-depth  studies,  or 
should the  occasion  arise, macro-economic  analysis. 

Of  the  utmost  importance  is  the  need  to  make  the  distinction  between 
gross  and  net  employment  effects.  Employment 
lend 
themselves  to  easy  quantification  are  generally  gross  indicators  like  the 
number  of  jobs  existing  in  a  new  assisted  company  after  12  months, 
number  of  jobs  related  to  the  operating  of  new  infrastructure,  number  of 
jobs that  assisted  companies  declare  have been safeguarded  thanks to  the 
intervention. 

indicators  that 

Gross  indicators  do  not  provide  a  definitive  view  of  employment  effects 
because  of  the  influence  of  confounding  factors  in  the  micro  and  macro-
economic  contexts.  Since  causal  analysis  is always  difficult,  it is  unrealistic 
to  seek  completely  definitive  measures  of  net job  creation.  Nonetheless,  a 
number  of  steps  can  be  taken  to  ensure  that  indicators  are  selected  and 
interpreted  in a fashion  which  increases  their  relevance  and  accuracy. 

limited  by  ensuring 

First,  underestimation  can  be 
indirect 
consequences  of  employment  interventions  are  taken  into  consideration, 
including  any  which  arise  from  an  increase  in  demand  related  to  the 
intervention,  for  instance  by  using  standard  coefficients  for  supplier  and 
multiplier  effects.  Second,  overestimation  can  be  prevented  by  taking 
account  of  the  possibility  of  deadweight  (changes  in  employment  which 
would  have  occurred  even  had the  intervention  not  been  made). 

that  the 

Displacement  and  substitution  effects  are  more  difficult 
issues. 
Displacement  is  the  creation  of  employment  in  one  location  merely  at  the 
expense  of  economic  activity  in  another  location. A  case  of  substitution  is 
the  hiring  of  an  assisted  unemployed  individual  at  the  expense  of  another 

Indicators for monitoring and evaluation 

worker  whose  job  has  disappeared.  Some  displacement  or  substitution 
effects  may  be  intended  (e.g.  creating  jobs  in  an  assisted  area  at  the 
expense of a large dynamic  capital  city; facilitating  the return to work  of the 
long-term  unemployed,  even  at  the  expense  of  other  workers,  etc.).  It  is 
always  a  specific  decision  to  take  or  not  to  take  displacement  and 
substitution  into  account. 

Finally,  the  net  employment  effects  of  an  intervention  are  defined  as  the 
gross  employment  effects 
to 
circumstances,  less displacement  and  /  or  substitution 

less  deadweight  and,  according 

It  is  often  important  to  make  some  attempt  to  identify  the  quality  of  the 
employment,  which  arises from  an  intervention. At  its  most  basic,  this  may 
be reflected  in income  levels  but it might  also be related to the  sustainability 
of  employment. 

Example  48:  Return-to-job  programme  for  long-term  unemployed 

Programme  Indicators 
(Related to  the  intervention  and 
its  effects) 

Context  Indicators 
(Related to the  assisted  area) 

Output  indicators 

Number  of  addressees 
or  recipients 

Result  indicators 

Rate of  immediate 
transition 

I m p a ct  i n d i c a t o rs 

Gross  placement  rate 

Net  placement  rate 

Quality of jobs 

j  Number  of  long-term  unemployed 
¡  participating  in the  programme 
1  (of  which  women) 

Number  of  addressees  or  recipients 
who  are  either  hired  or  in a  better 
social  situation  at  the  end  of  their 
participation  in the  programme 
(incl. %  of  women) 

Number  of  addressees  or  recipients 
hired  after  12  (incl. %  of  women) 

Gross rate  less estimated  deadweight 
(trainees  who would  have been  hired 
without  assistance)  less  unacceptable 
substitution  (trainees  whose  hiring  has 
led another worker  to  long-term 
unemployment). 

%  of  hired addressees  or  recipients 
(incl. % of women) who's job  is full-time 
and permanent  after  12  months 

Number  of  long-term  unemployed 

in  the  area 

%  of jobs  in the area that are 
permanent  and full-time 

Employment 

Example 49: Assistance for the  setting  up 
of an equestrian  event  centre 

Programme  Indicators 
(Related  to the  intervention  and 
its  effects) 

Context  Indicators 
(Related to the  assisted  area) 

Output  indicators  (in the field of  tourism) 

Compliance with 
project  duration 

Date of  expected  completion  of  project 

Result indicators  (in the field of tourism) 

Planned utilsab'on 

Impact  indicators 

Gross mid-term  direct 
employment  effect 

Indirect  employment 
effect 

Number  of  visitors 

Number of event planned in the first year 
Number  of visitors expected in the first 
year 

Numbers employed  in administration 
of  centre and provision of services after 
18 months (FTEs, % of  held by women) 

Estimated  number  of jobs (% of 
held  by women) generated  by the  stay 
of tourists attracted  by the  centre 

Average  number of jobs (ITEs)  in 
the tourism  sector  for  1,000  visitors 
χ nights 

Number  of visitors to events  held at 
centre (incl. % which comes  from  out of 
the region) after  1 year 

Number  of visitors  in the  area 
(nights per year) 

Availability  and  collection  of  data 

The  type  of  information  held  in  management  information  systems  can 
generally  serve  as  a  basis  for  evaluating  employment  effects.  Monitoring 
indicators  can  often  produce  information  about  temporary  jobs  and  about 
gross  permanent jobs  when  employment  effects  are direct  and  short term. 

Reports  from  relevant  research  and  labour  market  analyses  (including  the 
results  of  previous  evaluations  of  relevant  interventions)  provide  a stock  of 
both  quantitative  and  qualitative 
information  that  may  be  used  for 
estimating  indirect  job  creation.  Interesting  possibilities  also  arise  from  the 
use  of  job  ratios  (e.g.  number  of  jobs  generated  by  1,000  tourist  visits, 
number  of jobs  generated  by  1,000  euros  of  value  added  in a given  sector, 
etc.).  Even deadweight  can  be derived  from  previous  evaluations  of  similar 
interventions  in similar  contexts. 

Nevertheless,  only  an  in-depth  evaluation  can  provide  a  satisfactory 
estimate of net job creation or maintenance. Such an evaluation involves an 
effort  to  track  the  effects  by  surveying  addressees  or  recipients.  It  also 
involves  a judgement  about  whether  displacement  and  substitution  effects 
are acceptable  or  not. 

Indicators  for monitoring  and  evaluation 

The  most  general  sources  of  context  indicators  on  employment  and  labour 
market  issues  are  the  permanent  statistical  surveys  and  time  series  data 
that  are  comparatively  well  established  at  the  European,  national  and 
regional  levels.  Labour  force  surveys  are  available  on  at  least  an  annual 
basis  for  both  the  EU as a whole  and  individual  Member  States,  as  is  data 
on  various  aspects  of  unemployment.  One  problem  associated  with 
information  of  this  kind  is  that  it  is  often  not  as  frequently  available  on  a 
regional  level  as  it  is  on  the  national  or  Community  level.  Moreover,  the 
regions  on  which  data  is  provided  frequently  do  not  coincide  with  the 
regions  which  are  eligible  for  assistance  from  interventions.  A  second 
problem  with  this  type  of  information  is that  its  value  for  the  evaluation  of 
interventions  is limited  by the time  lags that  are inevitable  in producing  data 
for  standard  statistical  surveys. 

Key  indicators 

The  key  indicators  presented  below  lend  themselves  to  comparisons  both 
between different  interventions  of the same  programme  and  between  those 
of  different  programmes,  and  can  in  certain  cases  be  aggregated  at  a 
European  level. 

Level 

Key indicator 

Resource 

Number  of  persons  mobilised to  implement  the  interventions  (in 
job  years,  incl.  %  of  held  by  women) 

Result 

Impact 

Number  of  persons who  have been  immediately  hired during  the 
life of  the  project  (incl.  %  of  women) 

Number  of  gross  direct  jobs  created  or  maintained  (FTEs,  incl. 
%  held  by  women)  -  number  of  individual  addressees  or 
recipients  who  are at  work  after  12  months 
Number  of  net  direct  jobs  created  or  maintained  (FTEs  incl.  % 
held  by  women:  created  or  maintained)  -  Gross  number  less 
estimated  deadweight,  less  unacceptable  displacement  and 
substitution 
Total number  of gross  /  net jobs  (FTEs, incl.  %  held  by  women) 
-  direct  employment  plus supplier  and  multiplier  effect 
Quality  of jobs  -  %  of full-time  and permanent  jobs  (incl.  %  held 
by  women) 

Employment 

Since  the  employment  impacts  of  an  intervention  may  include  both  part-
time  and  full-time  jobs,  this  distinction  should  be  recognised  in  any 
indicators  used  and  both  aggregations  and  comparisons  may  be  made  by 
converting raw data on numbers employed into full-time equivalents (FTEs). 
Dates  of  measurement  (6,  12  or  18  months)  should  also  be  subject  to 
the  purpose  of  European  comparisons  and 
standardisation 
aggregations.  The  standard  proposed  date  is  12  months.  A  more  difficult 
prerequisite  for  comparisons  is that  displacement  and  substitution  should 
be  treated  in  a  standard  way.  The  standard  proposed  treatment  is  the 
following:  displacement  at  the  expense  of  any  area  eligible  to  Structural 
Funds  should  be  deducted  from  the  gross  effects;  substitution  at  the 
expense  of  any  person  targeted  by  Structural  Funds  should  be  deducted 
from the  gross  effects. 

for 

For  more  information  see... 

C3E  (1996),  Evaluating  the  Contribution  of  the  Structural  Funds  to 

Employment.  Brussels:  European  Commission. 

European  Commission  (1996),  Counting  the  Jobs:  how  to  evaluate  the 
employment  effect  of  Structural  Intervention.  Brussels:  European 
Commission. 

European  Commission  (1996),  Regional  Success  Stories:  profiles  of  36 

projects  in  Europe.  Luxembourg:  European  Commission. 

Schmid,  G.  et  al.(1997),  Progress 

in  Evaluation  Research.  Berlin: 

Wissenschaftszentrum  Berlin für  Sozialforschung. 

The  fifth  Volume  of  the  MEANS  Collection  includes  a  chapter  devoted 
entirely to  the  evaluation  of  employment  effects. 

Quality  of  life 

The  term  quality  of  life  has  been  widely  used  in  recent  times.  It  describes 
the  well  being  of  the  population  and  as  such  has  an  environmental,  social 
and economic  dimension and  is closely  linked to sustainable  development. 
In  this  context,  sustainable  development  can  be  described  as  achieving 
good quality  of  life while  not  degrading the  environment. 

At its broadest, all aspects of Structural Fund intervention have  implications 
for  quality  of  life. This  may even  be the case where it  is defined  in  relatively 
narrow  terms,  where  the  main  focus  is  on  an  improvement  in  living 
conditions.  Any  intervention  that  improves  local  living  conditions  has  an 
impact  on  the  residents'  continued  willingness  to  live,  study  and  invest  in 
the  assisted  area  (or  regional  attractiveness). This  willingness  to  live  in  the 
area  is  an  objective  that  is  particularly  sought  after  in the  case  of  regions 
with very  low  population  densities. 

The  rationale  for  incorporating  direct  interventions  in the  field  of  quality  of 
life  into  a  development  programme  is  that  they  contribute  significantly  to 
the  attractiveness  of  the  assisted  area.  Interventions  specific  to  this  field 
consist  of  creating  facilities  that  increase  the  attractiveness  of  a  region  by 
promoting  better  health,  improved  education,  culture  etc.  The  funding  of 
such  facilities,  however,  is  not  frequent  within  the  framework  of  the 
Structural  Funds. 

Because  interventions  in  this  field  generally  concern  a  large  variety  of 
different  types  of  projects, an off-the-shelf  approach to  indicators  needs  to 
be  pursued  with  caution.  Ideally  lists  of  indicators  should  emerge from  the 
particular  context  in  which  they  are  being  used  and  reflect  the  relevant 
concerns  and  issues  involved. 

Interventions  within the  field 

Resource 

indicators 

Interventions  in the field of  quality of  life, such as investment  in health, or in 
other  local  facilities  and  services,  are  generally  assessed  by  using  the 
standard  resource  indicator  of  budget  consumption. 

Output 

indicators 

Outputs  in this field can firstly  be  measured  by standard  indicators  such  as 
"rate of completion"  and  "compliance with  project duration". The volume of 

Indicators for monitoring and evaluation 

outputs  can  often  been  measured  in  terms  of  capacity.  Therefore,  in  the 
case  of  support  for  local  hospital  facilities,  outputs  could  be  measured  by 
the  "number  of  beds  available".  For  an  assisted  theatre  or  library,  relevant 
output  indicators  could  be  the  "number  of  seats"  and  the  "number  of 
publications".  However,  such  indicators  are very specific  and of  limited  use 
for  the  monitoring  and  evaluation  of  the  programme  as a whole. 

Result 

indicators 

There  are  immediate  benefits  of  new  equipment  or  facilities  when  the 
potential  users can gain access to given services in a reduced time or  when 
the  quality  of  given  services  is  improved  or  safeguarded 

Impact 

indicators 

Specific  impact  indicators  in  this  field  generally  focus  on  the  degree  to 
which  the  output  is  taken  up  after  one  /  three  years.  Indicators  might 
include the  number  of visitors to  a library, the  number  of  people taking  part 
in  a  cultural  event,  the  number  of  patients  using  a  hospital.  Indicators  are 
expressed  in  number  per  year,  or  preferably  in  percent  of  the  facility's 
capacity. 

Impact  indicators  go  well  beyond  direct  provision  and  the  use  of  facilities 
and  should  be  designed  to  measure  how  users  are  affected  by  the 
experience,  e.g.:  is the  supported  facility  continuing  to  be  regularly  used? 
Has  it  reduced  the  sense  of  isolation  of  the  users?  Has  it  increased  their 
confidence  in the area and will it change their future willingness to  live in the 
area? 

Impact  indicators  are the  most  interesting features  of  projects  but the  most 
difficult  to  measure. Impact  on quality  of  life tends to  be more complex  than 
other  types  of  impacts;  as  a  consequence  they  are  suited  to  qualitative 
techniques  of  data  analysis. 

Quality of life 

Example 50:  Construction  of a  new  hospital 

Programme  Indicators 
(Related to the  Intervention  and 
its  effects) 

Context  Indicators 
(Related to  the  assisted  area) 

Output  indicators 

Capacity 

Result  indicators 

Access to  hospital 

Number  of  patients  that  can  be treated, 
according  to type  of  care 

Number  of  hospital  beds  per 
1,000  population 

Average  distance of  patients  house 
from  hospital 

Distance  of  patient's  house  from 
hospital  in  area 

Quality 

Number  of  different  care  specialities 
in the  hospital 

Average  number  of  care  specialities 
in national  hospitals 

Impact  indicators 

Need 

Usage 

Occupancy  rates of  beds after one year 

%  of  inhabitants  who  rank  health 
services as one of  the  major  problems 
with  their  living  conditions. 

Occupancy  rates  In hospitals 
elsewhere  in the area 

Other  interventions  having  an  impact  in the  field 

Although  some  interventions  can  be  considered  as  specific  to  the  field  of 
quality  of  life,  almost  all  Structural  Fund  interventions  that  belong  to  other 
fields  may  be  seen  as  aiming  to  eventually  improve  the  quality  of  life.  For 
instance,  motorways  and  airports,  suburban  transport  systems,  village 
renewal,  reclamation  of  derelict  lands, training, etc. 

One  aspect  of  Structural  Fund  intervention,  which  fulfils  a  number  of  the 
objectives  of  the  funds,  is  to  enhance  the  attractiveness  of  a  region, 
particularly  to  stimulate  interest 
in  the  area  among  businesses  and 
potential  residents.  In addition to  standard  resource  indicators  like  budget 
consumption, it may be useful to develop a special  indicator  measuring the 
amount  of funds  of all types of interventions  allocated to areas where  living 
conditions  are  the  most  difficult.  The  indicator  would  be  expressed  as  a 
percentage of the total funds. Delineating the  most  difficult  areas would  be 
a  useful  but  difficult  work,  both  from  a technical  and  political  standpoint. 

In addition to  standard  result  indicators  such  as the  "number  of  addressees 
or  recipients"  and  the  "satisfaction  of  addressees  or  recipients",  there  is a 
the 
point  for  developing  a  special 
programme is concentrated on residents in areas where living conditions are 

indicator  measuring  how  much 

Indicators for monitoring and evaluation 

the  most  difficult.  The  indicator  would  be  expressed  as a  percentage  of  the 
total  number  of addressees  or  recipients. 

Improved  attractiveness  of  the  assisted  area  is a major  expected  impact  in 
the  field  of  quality  of  life.  Attractiveness  is  a  multidimensional  concept, 
applying  to  individuals  as well  as to firms.  It  applies  to  external  actors  who 
may  be  attracted  by  the  area  (potential  residents,  firms,  customers,  and 
tourists)  as  well  as  internal  actors  who  may  be  retained  in the  area.  Direct 
measures  of  attractiveness  can  be  derived  from  questionnaire  surveys 
asking  questions  like  "do  you  intend  to  leave  the  area  within  the  next  five 
years"  or  "assuming  that  you  have  decided  to  move,  would  you  accept  to 
live 
from 
in  this  area?"  More  accurate  measures  can  be  derived 
comparisons  through  questions  like  "please  rank  this  area  among  these 
other  areas  as the  best  places  you  would  like to  live  in". Indirect  measures 
are  also  available  through  the  cost  of  property  and  land.  Measures  of 
immigration  and  emigration  flows  are  generally  much  too  delayed  to  be 
useful  in the  context  of  monitoring  and  evaluation. 

Example  51:  Upg 

rading  a  bus  system  in neglected  urban  areas 

Programme  Indicators 
(Related  to the  Intervention  and 
its  effects) 

Context  Indicators 
(Related to  the  whole  urban  area) 

Output  indicators 

Focusing  on 
the  problem  area 

km  of  new  bus  routes 

Result  indicators  (in tt 

ie field of  transport  infrastructure) 

Access 

Impact  indicators 

Attractiveness 

Usage 

Average  time  for  residents  of  neglected 
urban  areas to  reach ten  major  urban 
services  (i.e. university, swimming  pool, 
hospital,  library) 

Average  time  for  all  residents  in 
the  town  to  reach  ten  major  urban 
services 

Average  ranking  given  to the  neglected 
area  by  residents  who  use the  bus 
system 

%  of  users  who are residents of  areas 
where  living conditions  are a  problem 
(neglected  areas) 

%  of  town  citizens  living  in 
the  different  areas 

Availability  and  collection  of  data 

Resource,  output  and  result  indicators  will  usually  emerge from  information 
systems  used  in  the  day-to-day  management  of  projects.  These  are  often 
comprised  of financial information  systems and systems  used for  managing 

Quality of life 

the  use  of  services.  For  instance,  facilities  such  as  libraries  use  statistical 
packages  that  provide  details  on  expenditure  and  income  associated  with 
books  and  link  them  back  to  a  cost  centre. They  may  also  through  swipe 
card  technology  have  another  set  of  indicators  which  offer  non-financial 
information  such  as  the  number  of  users,  types  of  items  borrowed  and 
fluctuations  of  library  use across  different  days and times of the day. These 
information  systems  may  be brought  together  for  purposes  of  analysis. 

Impact  data  may  not  be quantitative  in nature although  they  may  be set  up 
to  add  a  quantitative  element  to  qualitative  responses.  For  example, 
questionnaire  surveys  involve  the  coding  of  responses  to  open-ended 
questions  according  to  key  words.  It  can then  be  said  that  X%  mentioned 
libraries  as an essential  aspect  of  local  quality  of  life. 

impact  of  specific  Structural  Fund 
To  help  understanding  of 
interventions  it  is  necessary  to  know  the  context  in  which  they  operate. 
Context  indicators  that  are  critical  for  quality  of  life  are  notably  long-term 
unemployment  and  income. 

the 

Key  indicators 

Indicators in the field of quality of life are not yet stable enough to enable key 
indicators to be proposed. The introduction of the dimension of quality of life 
in the  concept  of  development  would  dramatically  change  basic  concepts 
such  as  regional  disparities  or  economic  growth.  One  such  indicator,  the 
"Indicator  of  Human  Development"  (IHD), has been used  by the World  Bank 
and the United Nations for over ten years. A similar indicator  is the "Index for 
Sustainable  Economic  Welfare" which  is presented  in Box  23. 

The  Index  for  Sustainable  Economic  Welfare  (ISEW),  like  GDP,  is  an 
aggregate  measure.  It  makes  18  adjustments  to  consumption  expenditure 
(the  basis of GDP) so  that: 
•  social and  environmental  costs  are deducted; 
•  long-term  environmental  damage  and  the  depreciation  of  natural  capital 

is deducted; 

•  average  personal  consumption  is adjusted  to  take  into  account  patterns 

of  income  distribution; 

•  a value  is imputed  (and added)  for  unpaid  household  labour. 

Aggregated  indicators  such  as  GDP  and  the  ISEW,  which  bring  together 
disparate  entities  through  the  common  measure  of  money,  are  somewhat 
controversial,  partly  because  applied  monetary  values  have  a  degree  of 

Indicators for monitoring and evaluation 

subjectivity. However, aggregation can help to link different levels, both 
micro and macro, at which quality of life interventions take place. 
Significant work is required to further develop such indicators. 

Box 23 - The index for Sustainable Economic Welfare 

This index shows that between 1950 and 1996 the United Kingdom experienced 
a growing divergence between ISEW and GDP. This indicates that the quantity of 
economic wealth (GDP) does not equate with the quality of life (as proxied by 
ISEW). 

GDP and ISEW: 1950-1996 

1990 
pounds 
sterling 

10,000 
9,000 
8,000 
7,000 
6,000 
5,000 
4,000 
3,000 
2,000 
1,000 
0 

— ■ * "~ 

GDP 

ISEW 

1950 1955 1960 1965 1970 1975 1980 1985 1990 1995 2000 

Source: Jackson, T,. Marks, N., Ralls, J. & Stymne, S., Sustainable Economic 
Welfare in the UK: 1950-1996,1997, University of Surrey Centre for Environmental 
Strategy/New Economics Foundation. 

For more information s e e-

Cheshire, P. et al., (1996), 'Problems of Urban Decline and Growth in EEC 

Countries: or measuring degrees of elephantness' in: Urban Studies, 

Volume 23(2), 131-149. 

MacGillivray, A. and Zadek, S. (1995), Accounting for Change: indicators for 

sustainable development. London: NEE 

New Economics Foundation / World Wide Fund for Nature UK (1994), 

Indicators for Sustainable Development: Strategies for use of indicators 
in national reports to the Commission on Sustainable Development and 
in the EC Stnjctural Funds process. NEF / WWF UK. 

New Economics Foundation / World Wide Fund for Nature UK (1997), 

Signals of Success: A users' guide to indicators. NEF / WWF UK. 

Ill 
Methods of producing and using indicators 

Involving  suppliers  and  users  of  information 

Experience  has shown  that  a system  of  indicators  has  much  more  chance 
of  functioning  when  the  suppliers  and  users  of  the  information  have  been 
involved  in  its  creation.  In  contrast,  a  closed  group  of  specialists  will  be 
tempted  to  construct  an  expensive,  technically  ideal  system  which  may 
never  be operational. 

Thus,  the  design  of  a  pragmatic  system  of  indicators  requires  the 
involvement  of  both  users and  suppliers  of  information. As far  as the  users 
are  concerned,  explicit  support  from  the  highest  level  of  the  authority 
managing  the  programme  has to  be  assured.  It is then  advisable  to  create 
a steering  group  composed  of  future  users  of  the  system, and  to  give  this 
steering  group  the  job  of  defining  the  indicators.  The  composition  of  this 
group  will  probably  not  be  very  different  from  that  of  the  group  which  will 
have to  conduct  the  evaluation. 

A team should  then  be appointed  to  run this steering  group  and to  provide 
it with  a secretarial  service.  If this team  is to be effective  its  members  must 
belong  to  the  authority  managing  the  programme  and  have  the  required 
human  and  financial  resources  to  fulfil  its  function.  It  must,  in  particular, 
ensure  that  the  system  of  indicators  clearly  reflects  the  programme 
objectives and ensures comparability.  It is preferable for the same team that 
is responsible for creating  indicators to subsequently  be responsible for the 
implementation  of the system. 

The  public  may  also  be  involved  in  designing  the  system.  Thus,  an 
American  experience  started  with  a  series  of  focus  group  interviews,  with 
representatives  of  regional  enterprises.  The  work  of  these  groups  made  it 
possible to select  indicators  most  likely to  attract the public's  attention  and 
to  be  understood  by  it  (see Box  24). 

The suppliers  of  information  must, as stated, also  be  involved  in  designing 
indicators.  The  main  suppliers  are  the  operators  who  implement  the 
programme in the field. Their  participation  is likely to ensure that the system 
is  pragmatic  because  they  are  familiar  with  the  practical  possibilities  and 
limits of  data  collection. 

Time  permitting, it is advisable to  involve the  operators  in a preliminary  test 
of  the  system  of  indicators.  The  recommended  procedure  starts  with  the 
selection  of  a few  volunteer  operators  who  will  participate  in the  design  of 
the  system.  These  volunteers  must  represent  all  the  components  of  the 
programme. They  help to choose the  indicators, to  define them  and to  plan 

Indicators for monitoring and evaluation 

the channels for collecting  information. They express their needs in terms  of 
information feedback  (frequency  and form  of  information fed  back to them). 
The  test  comprises  an  initial  quantification  of  all the  indicators  by  voluntary 
operators.  The  normal  duration  of  such  a  test  is a  year.  Depending  on  the 
conclusions  of  the  test,  and  after  introducing  the  necessary  modifications, 
the  system 
is  validated.  The  definitions  and  the  data  collection  and 
restitution  procedures  are  clearly  established, and  a manual  is written. 

Box 24 -  Involving addressees or recipients in the choice of indicators 

Political authorities in the Benton Harbor region (USA) set up a system of context 
indicators  with  a  view  to  assessing  the  economic  development  of  the  Berrien 
metropolitan area. The key elements in the description of economic development 
were  based  on  a  series  of  focus  group  interviews  involving  the  leading 
entrepreneurs in the region. The indicators chosen were, for example: 
•  For the availability of qualified human capital: spending on education per child; 

the teacher-student  ratio; the school dropout rate. 

•  For  the  growth  and  diversification  of  the  economy:  per  capita  income; 
employment rate in the industrial sector; value added in the trade sector; index 
of  sectoral  diversity;  rate  of  employment  in  SMEs;  value  and  number  of 
residences that were issued building permits. 

•  For the quality of life: the cost of living (relative), the rate of property crimes and 

crimes against the person. 

Erickcek  G.A.  (1996)  The  Benton  Harbor  Area  Benchmarking  Data  System 
Michigan: WE Upjohn Institute 

Information  relating  to  the  context  is  drawn  from  statistics.  It  is  therefore 
advisable  to  involve  an  expert  with  recent  and  complete  knowledge  of 
exploitable  statistical  data,  in  designing  the  system.  Depending  on  the 
case,  this  expert  will  belong  to  a  statistics  institute  or  a  university  or,  if 
possible, to  the  institution  running  the  programme. 

S e l e c t i on  of  t he  m o st  relevant  indicators 

Each  of  the  programme  actors  have  their  own  responsibilities,  their  own 
areas  of  decision-making  and  therefore  their  own  information  needs.  As  a 
result, all indicators  are not useful at all levels. On the contrary, it is generally 
accepted  that  each  actor  requires  an operating  report  with  a small  number 
of  indicators,  selected  as  the  most  relevant  in  relation  to  the  nature  of  the 
decisions  that  have  to  be  made.  It  has  been  shown  that  in  a  situation  of 

Producing and using  indicators 

decision-making,  a  person  cannot  take  into  account  more  than  about  ten 
indicators  at  once  (Innés  de  Neufville,  1994).  When  there  are  too  many 
indicators the decision-makers  are swamped with an excess of information. 

The  problem  of  heterogeneity  of  programmes 

The  experience  of  the  Structural  Funds  has  shown  that  it  is  difficult  to 
choose  indicators  that  are  absolutely  necessary  for  the  monitoring  and 
evaluation  of  a  programme.  Because  the  programmes  are  multi-sectoral 
and  multi-objective,  managers  sometimes  attempt  to  measure  everything 
and design systems of  indicators which are so heavy that it is impossible  to 
make  them  work.  For  example,  the  initial  proposal  for  the  Objective  5b 
Programme  in  Burgundy  (F)  consisted  of  over  200  indicators.  In  the  end, 
only  about  fifty  of  them  were quantified. 

The  following  example  clearly  illustrates  this  problem.  It  concerns  a 
programme  (1994-1999)  for  the  industrial  reconversion  of  a  French  region 
(Franche-Comté).  This  programme  comprises  48  actions  of  various  kinds, 
e.g. several types of direct support to enterprises; specific support to  certain 
sectors,  to  groups  of  firms,  to  professionals  in  the  tourist  business,  to 
creators  of enterprise; support  for the improvement  of the environment;  etc. 
These  actions  are  managed  by  different  operators  who  all tend  to  measure 
the  effects  of  their  actions  in  terms  of  their  own  logic  (e.g.  number  of 
subsidies;  number  of  participants  in  collective  projects;  surface  area  of 
industrial sites improved). All this information provided by the operators is too 
heterogeneous  to  be  synthesised  at  the  programme  level. Apart  from  the 
indicator of  budget  absorption, no indicator applies to more than 20% of the 
programme.  For  a  programme  like  this,  a  complete  information  system 
would  be unwieldy  because  it would  have to  include the following data: 

•  one or two  generic  indicators for the  monitoring  of  resources  (e.g. rate of 
into  actions;  expected  rate  of 

budget  absorption  broken  down 
overspending); 

•  about  fifty  output  indicators,  which  amounts  to  an  average  of  one 

indicator  per  action; 

•  about  fifty  result  indicators, which  amounts  to  one  indicator  per  action; 

•  about  one  hundred  specific  impact  indicators,  or  an  average  of  two 
indicators per action  (e.g. survival  rate of assisted businesses;  placement 
rate of trainees; statements  made by firms on their  intentions to recruit  or 
invest;  number  of  innovations  generated; number  of  visits  by tourists); 

Indicators  for monitoring  and  evaluation 

•  s o me  g e n e r ic 

i m p a ct 

i n d i c a t o rs 

(e.g. 

j o bs  c r e a t e d;  value  a d d ed 

generated  in  the  assisted  firms,  etc.); 

•  s o me  c o n t e xt  indicators  (regional  u n e m p l o y m e nt  rate;  number  of  SMEs 

in  the  region;  average  value  a d d ed  per  employee  in  regional  firms;  etc.). 

In  total,  the  n u m b er  of  indicators  of  this  c o m p l e te  a nd  obviously  theoretical 

s y s t em  can  be  estimated  at  200.  In  practice,  it  is  impossible  to  p r o d u ce  and 

regularly  use  such  a  large  amount  of 

information.  In  several  European 

regions,  recent  evaluations  have  s h o wn  that  a  few  dozen  indicators  are 

e n o u gh 

to  meet 

the 

information  needs  of  the  authorities 

running 

the 

p r o g r a m me  (See  Box  25).  This  does  not  m e a n,  however,  that  additional 

indicators  may  not  be  required  to  fill  the  operators'  information  needs. 

Box  25  - T he  r e c o m m e n d a t i on  of  an  evaluation: 

reducing  t he  n u m b er  of  indicators  f r om  3 30  to  52 

A socio-economic  programme  was financed  by the European Structural  Funds for 
the  period  1994-99  in  Northern  Ireland. An  intermediate  evaluation  was  made  of 
this  programme,  by  synthesising  six  separate  evaluations  of  sub-programmes. 
The  programme  designers  had  chosen  330  indicators  for  the  monitoring.  These 
indicators  had  been  grouped  into  a single data  base that  proved  to  be difficult  to 
use and  created  problems  of  manipulability  and  availability  of  information.  These 
problems  considerably  reduced  the  use  of  the  indicators.  The  evaluation  team 
recommended  a solution  consisting  of: 

•  choosing  a  number  of  context 

indicators  situated  between  the  macro-
economic  indicators  and  the  sub-programme  indicators.  These  indicators, 
intended  to  reflect  the  global  impact  of  the  programme,  are  divided  into  three 
categories:  economic  growth, internal  cohesion  and  external  cohesion. 

•  choosing  a small  number  of  programme  indicators  by  limiting them  to the  main 

results  and  impacts; 

•  delegating  the  quantification  and  use of  the  other  indicators  to the  operators. 

In  this  way,  the  size  of  the  system  could  be  reduced  to  52  indicators  directly 
related  to  the  main  objectives  of  the  programme.  The  recommendations  were 
successfully  applied. 

Colin  Stutt  Consulting,  1997,  Northern  Ireland  Single  Programme  1994-99;  Mid 
Term  Review  External  Evaluation. 

S u g g e s t i o ns  f or  limiting  t he  size  of  s y s t e ms  of  indicators 

This  section  takes  the  example  of  the  French  region  cited  above  and 

describes  the  suggestions,  m a de  during  the  intermediate  evaluation,  for 

Producing and using  indicators 

reducing  the  number  of  programme  indicators  without  losing  relevant 
information  at this  level. As Box 26 shows, these suggestions  are based  on 
the  use  of  generic  indicators  or  on  the  grouping  together  of  the  indicators 
by category  of  addressee  or  recipient. 

Box 26  - Example  of an information  system 

using a  reduced  number  of  indicators 

Number  of  actions 
(elementary  components  of the  programme)  48 
Programme  indicators 
...  of  resources 
...of  outputs 

1 
2 

of  results  and specific  impact 

45 

Context  indicators 

Total number  of  indicators 

51 

(Rate of budget  absorption) 
(Output  rate, rate of  compliance 
with  project  duration) 
(Average of  5 indicai 
of  the 9  categories  of  addressees 
or  recipients) 
(income,  unemployment, 
attractiveness) 

for  each 

Firstly, the evaluation  suggested that  all the  resource  and output  indicators 
be grouped  together  in the  form  of three  generic  indicators:  rate of  budget 
absorption;  completion  rate;  rate  of  compliance  with  project  duration. 
These  three  indicators  can  easily  be  quantified  for  each  action,  on  the 
condition  that  the  operators  have  their  own  specific  indicators.  The  three 
indicators  can then  be aggregated  at the  level of  each  measure and that  of 
the  programme  as a whole. 

Secondly,  it  was  noted  that  the  48  actions  of  the  programme  could  be 
grouped  together  into  nine categories  of  addressee  or  recipient:  actions  in 
favour  of  businesses,  of  creators  of  businesses,  of  adults  undergoing 
training, of  professionals  in tourism, etc. For each category of addressee or 
recipient,  it  was  possible  to  choose  a  small  number  of  result  and  specific 
impact  indicators.  For  example,  for  adults  in training, two  indicators  could 
be  chosen  (success  rate  in  reaching  the  target  public  and  placement  rate 
after  12 months), and the data could  be broken down in three ways  (women 
/  men; employees  /  the  unemployed;  growing  sectors  /  declining  sectors). 
Given the conventions  adopted  in this Volume, this  represents  a total of  six 
indicators.  In  total,  about  forty  indicators  could  suffice  for  providing 
information  about  results and  specific  impacts. 

Indicators for monitoring and evaluation 

Some basic context  indicators such as the per capita income  in relation to the 
national average or the regional unemployment  rate, as well as an indicator on 
the attractiveness of the region (an indicator which is clearly needed but which 
still needs to be constructed), could be added to these programme indicators. 
By  following  these  recommendations,  it  should  be  possible  to  limit  the 
information  system  of this  programme  to a reasonable  number  of  indicators. 

A  lighter  system  limits  the  collection  and  circulation  of  information  to  the 
most  essential  elements  at  the  programme  level.  On  the  other  hand,  this 
means that  the  progress  and  results  of  each  action will  not  be  monitored  in 
a  detailed  and  centralised  manner.  It  also  means  that  the  system  focuses 
less on the decisions to  be made by the operators  and more on those to  be 
made  by the  authorities  managing  the  programme. 

Finding  generic  impact  indicators 

Impact  indicators  are  indispensable  in  evaluation  but  they  are  difficult  to 
design and quantify. Apart from the number  of jobs created, it is  particularly 
rare to  find  generic  impact  indicators  in programming  documents. 

Box 27  -  Examples  of generic  impact  indicators 

ACTIONS 

Industrial 
investments 

Modernisation  of 
craft  industries 

Venture  capital 
for  SMEs 

Internationalisa­
tion  of  SMEs 

Technological 
development  of 
enterprises 

GROUPS  OF 
IMPACTS 

GENERIC 
INDICATORS 

Value  added 
generated  in 
growth  sectors 

Number  of 
innovations  in 
assisted 
economic  units 

The  MEANS  programme  has  developed  a  method  for  creating  generic 
impact  indicators.  This  method  is  presented  in  Volume  4  of  the  MEANS 
Collection  which  includes  several  methodological  innovations.  It  functions 
in four  steps  in the  following  way: 

Producing and using  indicators 

•  The  evaluation  team  gathers  the  main  documents  relating  to 

the 
reports, 
programme 
brochures)  and  identifies  all the  sentences  describing  the objectives,  the 
performance  and the expected  or  real  impacts. 

(programming  document, 

reviews,  progress 

•  From  the  quotations  taken  from  the  documents,  the  evaluation  team 

selects those concerning  impacts. 

•  It  organises  one  or  more  seminars  with  the  programme  managers  and 
operators.  During  these  seminars,  the  participants  complete  the  list  of 
impacts  and  group  them  into  families.  Two  techniques  can  be  used  to 
facilitate  this  work:  impact  mapping  and  Metaplan  (for  a  description  of 
these techniques  see Volume 3 of  the  MEANS  Collection). 

•  Together, the  participants  of the seminar  identify  the signification  of  each 
family of impacts, name them and choose corresponding  indicators. They 
check  that  these  indicators  are  generic,  that  is to  say,  that  they  can  be 
applied to  numerous  actions  within the  programme. 

Guaranteeing  quantification 

Data  collection  by the  operators 

Information  collected  by  the  monitoring  system 
is  drawn  from  the 
management  of  funds,  the  operation  of  projects,  the  processing  of 
applications  for  subsidies,  etc.  Each  operator  provides  information  to  the 
programme  managers  in  the  form  of  progress  reports,  periodic  reviews  or 
execution  reports.  These  data  should  be  collected  and  synthesised  in  the 
form of operating reports for each monitoring meeting. The project  managers 
may  be asked to  provide exhaustive  information on the following  themes: 

•  Resources  -  public  budget  absorption  broken  down  by  source  of 

funding  and  overspending. 

•  Outputs  -  rate of progress of  projects  underway;  rate of compliance  with 
project  duration;  volume  of  outputs  in  relation  to  resources  (efficiency  of 
the outputs);  number  of  addressees  or  recipients  broken  down  into a few 
large categories (e.g. heavy or light vehicles; large, medium-sized  or small 
enterprises;  women  or  men,  etc.);  volume  of  services  obtained  by  the 
addressees  or recipients  (e.g. hours of training; advice received, etc.). 

•  Results  -  declarations  made  by  the  addressees  or  recipients  (e.g. 
investments  to be made, jobs to be created); rate of addressee or  recipient 
satisfaction;  volume  of  results  in  relation  to  the  corresponding  resources 
(efficiency  of  results). 

Indicators for monitoring and evaluation 

Whenever possible, the monitoring system must ensure that the operators 
retain the possibility of contacting the addressees or recipients at a later stage 
for an evaluation survey (name and address of trainees; name and telephone 
number of a contact in the assisted firm). This precaution is also useful for 
avoiding double counting. For example, it will ensure that a firm receiving 
several forms of assistance from the programme is not counted several times. 

A system of indicators has far more chance of functioning if it does not 
disturb existing information channels. It must therefore request information 
closest to the source. In Box 28, the quantification of result indicators is 
part of the operators' standard management procedures. 

Box 28 - Collection of information by the operators 

The form presented in this box applies to programmes for helping foreign firms 
who invest in the region. Several operators are responsible for providing services 
to these firms in various parts of the region. Upon finalisation of an investment 
project with a firm, all operators must fill in this standard questionnaire: 

Name of the person who can be contacted in the firm: 

Telephone: 

Value of investments to be made in the framework of the assisted project: 

Number of jobs created or maintained in the framework of the assisted project: 

Quality of the services obtained by the firm: 

excellent [ 1 good Π average D bad 1 ] 
Usefulness of the services obtained by the firm: 

excellent ! good C average D bad □ 

Inspired by Hatry H. P. & al.1990 

In the context of programmes financed by the Structural Funds, a particular 
difficulty arises from the co-existence of systems of indicators peculiar to 
each funding body and which do not necessarily have the same definitions. 
For example, in the case of a training programme, a Commission 
representative wanted to obtain information on the number of trainees, 

Producing and using  indicators 

considering  a trainee to  be any person who  attended  a course, even if only 
for a few days. However, the national operators were used to recording only 
those  training  courses  which  lasted  for  more  than  a  month.  In  cases  of 
differences  between  definitions  chosen  for  programme  indicators,  and  the 
operators'  usual  definitions,  it  may  be  preferable  to  quantify  indicators  in 
terms of traditional definitions which  do  not disturb the operators, and then 
to  make  corrections  at 
(harmonisation-
) by  means  of  a correction  coefficient. 

the  programme 

level 

The collection and synthesis of information  may be facilitated  by the  setting 
up  of  a  computerised  system  as  the  example  in  Box  29  shows.  This  tool 
must,  however,  be  used  with  caution.  New 
technologies 
facilitate  the  collection  and  circulation  of  a  mass  of  information  and  the 
construction  of systems  of  overabundant  indicators. Computerisation  must 
always  be  preceded  by  thorough  reflection  as  a  basis  for  the  selection  of 
the  most  useful  indicators. 

information 

Box 29 -  Computerisation  of the  monitoring  system 

France,  Objective  5b  Regional  Programme  1994-99 

The  monitoring  system  of  the  rural  development  programme  in  the  region  of 
Burgundy has been in operation  since the beginning of  1996. The operators  report 
to the five authorities which distribute the programme funds. These authorities  are 
connected  via  a  telematic  network  to  a  central  data  base  which  receives 
quantified  information for about  1,500  projects  per year. A computer  program was 
specially  designed  for  monitoring  the  programme  and for  processing  information 
in  real  time.  The  information  is  synthesised  at  a  regional  level  in  an  operating 
report  comprising  40  indicators.  These  indicators  are  presented  and  discussed 
before each  monitoring  committee  meeting. 

To encourage  operators  to  supply  the  required  information,  it  is  normal  to 
guarantee  them  feedback  in the  form  of  information  with  substantial  value 
added.  For  example,  each  operator  receives  an  annual  report  on  their 
individual  results, compared to  the average for  all the  operators. 

Apart  from these  incentives, it is also  usual to formally  establish  a contract 
for  data  collection.  For  example,  the  decision  to  accept  a  project  and  to 
grant  funds  to  it  must  be  accompanied  by  an  appendix  specifying  the 
information to be produced, the date on which it must be produced  and the 
person /  organisation  to  whom  it  must  be given. The  programme  manager 
must  of  course  take  all  necessary  steps  to  ensure  that  the  data  collected 

Indicators for monitoring and evaluation 

are  not  altered. The  procedures  for  collecting  and synthesising  information 
must  therefore  be  rigorous, transparent  and  checked  periodically. 

Finally, the  most  decisive  incentive  consists  of  linking  the  allocation  of  new 
funds  to the  provision  of  data  related  to  the  preceding  subsidy. 

Collecting  information  by  way  of  a  survey 

To  quantify  the  sustainable  impact  on  direct  addressees  or  recipients,  a 
common  approach  consists  of  selecting  a sample  of  people  or  businesses 
which  have  participated  in the  programme,  and  of  conducting  a  postal  or 
telephone  survey. This approach  is facilitated  by the fact  that  the  people  to 
be  questioned  are  known  to  the  operators,  especially  if  the  survey  is 
planned  from  the  start  of  the  programme.  The  survey  will  be  more  difficult 
if  the  operators  have  not,  in  the  course  of  their  work,  been  able  to  keep 
track  of  the  public  concerned  by  the  impacts  (e.g.  inhabitants  of  a 
rehabilitated  neighbourhood;  the  competitors  of  an  assisted  business; 
tourists  who  have visited  a protected  site). In this case, the quantification  is 
far  more  difficult  and  requires  a  tracking  exercise,  which  may  be  difficult 
and  time-consuming.  Thus,  feasiblity  constraint  eventually  prevents  the 
quantifying  of  progamme  indicators.  Beyond  a  certain  limit  it  is  no  longer 
possible  to  observe  impacts  but  only  to  estimate them  by  extrapolation. 

Box 30  -  Use  of  surveys  for  quantification 

Management 

system 

Survey 

Statistics 

based  on 
the  list  of 
addressees 
or  recipients 

on the 
entire 
population 

regular 
quantification 

quantification 
during  an  evaluation 

regular 
quantification 

X 

X 

X 

X 

Resources 
and  outputs 

Results 

Impacts 
(programme 
indicators) 

Impacts 
(context 
indicators) 

X 

X 

X 

Producing and using  indicators 

Surveys  of  addressees  or  recipients  must  be  used  with  caution  when  net 
effects are being estimated. For example, the evaluation team of an  industrial 
reconversion  programme  in  a  region  in  England  had  to  assess  the 
employment  impact  of  a  measure  aimed  at  encouraging  an  entrepreneurial 
spirit  and  training  within  firms.  The  evaluation  team  sent  a  postal 
questionnaire to several hundred assisted firms, in which their anonymity was 
guaranteed. The standard  questions  asked were:  "Would you  have  recruited 
additional  employees  even without  the  subsidy?"  or  "Do  you  know  whether 
the  new  employee  was  unemployed  before  being  recruited?".  Based on  the 
survey  results, the evaluation team estimated  a coefficient  of  net job  creation 
in  relation  to  the  assistance  received,  for  the  firms  in  the  sample.  By 
generalising the conclusions of the survey to  all addressee or recipient firms, 
a net job creation indicator could be quantified by extrapolation. It is, however, 
important  in such cases to  be attentive to the quality  of the  survey  because 
some  answers  may  be  biased  due  to  the  respondents'  favourable  opinions 
about an intervention from which they  benefit. 

The  organisation  of  a  telephone  questionnaire  survey  of  a  thousand 
individuals  may require several months and cost  between  10 and 40 KEuro. 
The  different  surveys  undertaken  in  a  region  must  also  be  carefully  co­
ordinated  so  as  not  to  swamp  the  addressees  or  recipients.  For  all  these 
reasons surveys cannot  be repeated frequently.  During the interval  between 
two  surveys,  it  is  nevertheless  possible  to  estimate 
indicators  by 
extrapolation, based  on the  data provided  by the  monitoring  system. 

The  Ministry  of  Labour  in  France,  for  instance,  relies  on  the  results  of 
numerous  surveys  carried  out  over  the  past  decade,  for  assessing  the 
impact  of  employment  incentives.  Each  of  these  empirical  studies  has 
enabled it to estimate  impact coefficients  in various situations. Owing to the 
accumulation  of  past  knowledge,  experts  at the  ministry  have  constructed 
a grid that  is used  to  estimate  impacts  on the basis of  monitoring  data  and 
of  the  main  characteristics  of  a  measure.  Since  1992,  meetings  between 
experts  are  periodically  organised  to  improve  and  adjust  this  grid  of 
coefficients. 

Using  statistical  sources 

Context  indicators  are  generally  quantified  on  the  basis  of  data  obtained 
from statistics institutes. For example, many regionalised data can be found 
at Eurostat,  in the  REGIO data  base.  REGIO data are presented  in the  form 

Indicators  for monitoring  and  evaluation 

of  about  70 standard tables  per domain: demography;  economic  accounts; 
unemployment;  workforce;  energy;  agriculture; transport;  and  R&D. 

Eurostat  uses  data  derived  from  a  few  major  surveys  to  produce  regional 
statistics.  In the  employment  domain,  numerous  available  data  are  derived 
from  the  "workforce"  and  "household  panel"  surveys.  Regionalised  data 
have  recently  been  enriched  through  the  exploitation  of  national  annual 
surveys  of  industry.  The  information  processed  is  available  in  the  ELIZE 
data  base  which  covers: the  number  of  industrial  enterprises;  employment; 
salaries  and  wages; turnover;  and  investments.  The  data  are  broken  down 
into  about  30  economic  sectors  and  two  classes  of  size  (+  /  -  20 
employees).  In the  middle term, the  REGIO data  base will  be completed  by 
an exploitation  of  the  survey  results  for  agriculture. 

There  is  also  the  QUID  data  base,  managed  by  DG  XVI,  which  groups 
together  context  indicators  adapted to the Structural  Funds. This data  base 
is  partly  based  on  Eurostat  information. 

The availability  of  statistics  can  become a problem when the eligible  area of 
a  programme  is  an  administrative  rather  than  a  statistical  zone.  The  best 
way  of  dealing  with  this  problem  is  the  "best  correspondence"  approach 
which  has  been  applied  for  the  Scottish  system  of  indicators,  as  shown  in 
Box 3 1. 

Box 31  -Correspondence  between  a statistical  unit and an  eligible 

area  -  United  Kingdom, Objective  2 Programme  1994-99 

The  development  programme  in  the  East  of  Scotland  has  an  eligible  area  that 
concerns  several  statistical  units  ("local  government  region  level")  but  there  is a 
lack of correspondence  in the way that  they are partitioned. The data provided  by 
the  Central  Statistical  Office  is  therefore  not  suitable  for  directly  constructing 
context 
indicator  of  GDP  was  nevertheless  estimated  by 
aggregating  data  for  the  territories  of  the  "central  regions"  and  for  Fife  and 
Tayside, of  which  over  half the inhabitants  are in an area eligible for the  Structural 
Funds.  The  territory  of  Lothian,  less  than  half  of  which  is  eligible  for  Structural 
Funds,  was  disqualified. 

indicators.  An 

A  good  estimation  of  context  indicators  can  also  be obtained  by  means  of 
the  interpolation  method,  as  in  the  following  (theoretical)  example.  One  of 
the aims of  an industrial  reconversion  programme  is to  enable the  region  to 
catch  up  with  others  in  terms  of  innovation.  A  situation  indicator  would 
need to  be created to help quantify this  objective. A possible indicator  is the 

Producing and using  indicators 

innovations 

annual  number  of 
in  the  region.  The  annual  number  of 
innovations  for  1,000  jobs,  on  a  European  scale,  is  provided  by  a 
Community  survey  carried  out  regularly  on  40,000  firms.  Based  on  the 
results  of this  survey, an  innovation  indicator  can  be created  for  the  region 
in the following  way: (1) choose  an  indicator  of  structural  composition (e.g. 
number of jobs, broken down into sector of activity) for which the values are 
known  at  both  a  European  and  a  regional  level;  (2)  note  the  value  of  the 
annual  innovation  rate  at the  European  level, with  its  breakdown  by  sector 
of  activity,  and  (3)  estimate  the  innovation  indicator  in  the  region  by 
multiplying  regional jobs,  sector  by sector,  by  European  innovation  rates. 

Eurostat  has  recently  created  an  information  system  (GISCO)  which 
cartographically  presents  all the statistical  series existing  in  REGIO, as well 
as other  regionalised  data. The  GISCO  system  also  includes  the  outline  of 
all the areas  eligible for  Structural  Funds.  It can therefore  easily  be  used to 
break  down  or  build  up  statistical  information  on  different  geographical 
scales. It is potentially  a very  powerful tool for estimating  context  indicators 
by  interpolation. 

Assessing  the  quality  of  a  system  of  indicators 

The  use  of  indicators  will  be  far  greater  if  their  quality  is  constantly 
improved. If evaluation has an uncontested  role, this is to assess the quality 
of systems of  indicators and to  recommend  ways of enhancing  it.  Although 
there  is  no  standard  method  for  this  quality  control,  an  approach  is 
proposed  here  that  is  based  on  the  following  11  criteria,  which  can  be 
divided  into  two  groups:  quality  criteria  applicable  to  each  indicator  and 
quality  criteria  applicable  to the  entire system. 

Quality  criteria  applicable  to  each  indicator 

The  first  quality  criterion  for  an  indicator  is  its  actual  existence,  that  is  to 
say,  it  must  be  quantified  at  regular  intervals.  Sometimes  one  or  more 
indicators  featured  in  the  programming  documents  have  never  been 
quantified  and  for  all  practical  purposes  do  not  therefore  exist. The  actual 
availability  of  data  is  therefore  the  very  first  quality  criterion.  Monitoring 
indicators  must  be  quantified  at  each  monitoring  meeting,  that  is  to  say, 
every  six  to  twelve  months.  Evaluation  indicators  are  quantified  less 
frequently,  every  three to  six  years. 

Indicators for monitoring and evaluation 

Once  an  indicator  has  been  quantified,  it  may  take  several  months  or  even 
years  before  the 
information  can  really  be  used  for  monitoring  and 
evaluation. This  is particularly  true for certain context  indicators  drawn  from 
annual statistical  publications. The freshness  of  information  then  becomes 
an important  quality criterion. Sometimes  statistics are published  two  years 
or  more  after  the  collection  of  the  data. 

A  programme  indicator  must  vary  significantly  when  the  programme  is 
implemented  and  produces  effects.  This  is what  is called  its  sensitivity.  In 
the  example  of  an  intervention  supporting  exports, the turnover  of  assisted 
businesses  is  an  indicator  that  is  not  sensitive  enough.  A  better  indicator 
would  be  the  turnover  relating  only  to  new  customers  contacted  with  the 
support  of the  programme. 

The  programme  actors  must  trust  the  information  produced.  Reliability  is 
therefore  a  quality  criterion.  Reliability  can  be  defined  as  the  fact  that  the 
same  measurement,  taken  by  two  different  people,  will  produce  the  same 
value  for  the  indicator.  In  reality  this  definition  is  only  applicable  where 
counts  are concerned.  Each time the  indicator  is quantified  on the  basis  of 
questions  put  by  one  person  to  another,  the  reliability  can  no  longer  be 
defined  mechanically.  Reliability  and  credibility  are  two  notions  that  then 
tend  to  merge. 

The  usefulness  of  an  indicator  depends  largely  on  whether  it  allows  for 
internal  comparisons  between  different  measures  of  the  programme  or 
inter-regional  external  comparisons.  The  comparability  of  the  indicator  is 
therefore  a criterion  of  its  quality.  This  criterion  is  important,  but  difficult  to 
meet. 

The  last  quality  criterion  of  an  indicator  is  similar  to  comparability  but  with 
an additional  requirement:  normativity.  Indicators  must  include a  reference 
so  that  the  outcome  can  be  judged  to  be  satisfactory  or  not.  Any  value 
observed  by  means of the  indicator  must therefore  be compared  to a norm, 
for example: objective to  be met;  norm  to  be surpassed; European  average 
to  be attained;  etc. 

A  good  indicator  must  be  understood  without  ambiguity  by  everyone  who 
has  to  use  it.  In  the  minds  of  both  decision-makers  and  the  public,  the 
meaning  of  the  indicator  must  be  the  same  as  for  the  programme 
managers.  It  must  accurately  reflect  the  concept  to  be  measured.  That  is 
what  is called  the  validity  of  construction. 

Producing and using  indicators 

The  example  in  Box  32  is  an  attempt  to  assess  Portuguese  operational 
programme  indicators  for  the  "transport"  sector  and  for  the  period  1994-
1999. 

Box 32  -  Example  of an assessment  of the quality of  indicators 

Criterion 

availability 

freshness 

sensitivity 

reliability 

comparability 

normativity 

meaning 

Portugal,  Objective  1 transport  operational  programme,  1994-99 

Assessment 

Comments 

+ 

++ 

+ 

++ 

-

= 

++ 

most  of the  indicators  are quantified  with 
the  usual  frequency 

the  main  output  and  result  indicators  are 
quantified  in real time 

the  main  programme  indicators  are 
highly  responsive  to  its  implementation 

the  indicators  are well  accepted  by  the 
programme  actors 

the  system  has few  internal  comparisons 
and  no external  comparisons 

only  half  of the  indicators  have a 
reference  judgement 

the  meaning  of  the  indicators  is fully 
understood  by  the  programme  actors 

Quality  criteria  applicable  to  the  entire  system 

The  indicators  selected  must  cover  a  sufficiently  large  proportion  of  the 
programme  measures.  This  coverage  must  be  equal  to  or  greater  than 
three-quarters  of the  planned  expenditure. 

The  system  must  consist  of  a  good  balance  between  indicators  in  the 
different  categories.  In  particular,  result  and  impact  indicators  must  be  the 
most  numerous. 

A  system  of  indicators  must  be  simple.  The  selectivity  criterion  requires 
that  the  programme  managers'  capacity 
information  be 
respected. The information must therefore be limited to a maximum of a few 
dozen figures. 

to  absorb 

The  relevance  of  the  system  implies  that  the  indicators  be  developed 
primarily  for  those  measures  or themes  which  have significant  implications 

Indicators for monitoring and evaluation 

in terms of decision-making (measures with a very high budget; innovative 
measures; themes considered to be strategic). 

Box 33 -Assessment of the quality of a system of indicators 

(continuation) 

coverage 

balance 

selectivity 

relevance 

Portugal, Objective 1 transport operational programme, 1994-99 

++ 

the system covers 100% of the measures 

++ 

+ 

there are no impact indicators; the result 
indicators cover only 4 6% of the 
programme measures 

the system comprises 26 indicators 

the system has more indicators for those 
measures and projects with the largest 
budgets 

Key indicators 

Comparisons between programmes 

Comparison between the different programme indicators is always an 
important step, but the difficulties involved must not be underestimated. 
These are due to various factors, such as the diversity of interventions 
within a programme, the diversity of regional contexts, or the incompatibility 
of definitions. For example, depending on the regions and programmes, 
tourist fréquentation is counted in terms of the number of visits or the 
number of nights; trainees are counted in terms of the number of 
participants or in hours χ trainees; and environmental protection is 
measured in terms of the number of projects, the number of sites or the 
number of hectares protected. 

Comparability may be sought and obtained through exchanges between 
managers in different regions or countries. The Interreg programmes create 
numerous opportunities for this type of comparison. In a different context, 
the Pays-de-la-Loire (France) and Emilie-Romagne (Italy) regions held 
bilateral meetings in 1997-1998 to discuss their evaluation work in the 
domain of rural tourism. In two two-day seminars, the regions analysed 
their respective actions and jointly defined a logic diagram of outputs, 
results and impacts. Common indicators were proposed, with a view to 
making inter-regional comparisons. 

Producing and using  indicators 

Yet  comparability  is  often  easier  to  obtain  and  more  profitable  at  all  levels  if 

it  results  from  a  co-ordinated  collective  effort  on  a  higher  geographical 

level.  This  approach  has  the  advantage  of  multiplying  the  possibilities  for 

comparison,  and  also  allowing  for  the  aggregation  of  indicators  at  the 

regional  or  national  level.  That  is  what  the  Scottish  Office  did  w h en  it 

created  a  series  of  standard 

indicators  applicable 

to  seven  Scottish 

programmes.  The  British  Audit  Commission 

is  another  example  of  this 

approach  (see Appendix  2). 

Box  34  -  A  set  of  standard  indicators  for  several  p r o g r a m m es 

Within  the  framework  of  its  responsibility  for  monitoring  and  evaluating  seven 
programmes  co-financed  by  the  European  Union, the  Scottish  Office  developed 
a  series  of  standard  indicators.  Some  of  these  indicators  concern  the  context: 
rate  of  employment  and  unemployment;  productivity;  manpower;  average 
income;  etc.  Some  output  and  result  indicators  have  also  been standardised,  as 
shown  by  the  following  examples,  applicable  to  the  infrastructure  measures  for 
enterprises: 

•  New  buildings  built 

•  Buildings  renovated 

•  Development  of  new  sites 

•  Improvement  of  existing  sites 

•  New /  improved  road access 

•  Surface  area for  which the  access  roads 

were  built  or  improved 

•  Rate of  occupation  of the  new  buildings 

square  metres 

square  metres 

hectares 

hectares 

kilometres 

hectares 

percentage  after  one year 
percentage  after  three  years 

•  Number  of training  places  created 

number 

To  obtain  a  high  level  of  comparability,  the  Scottish  programme  managers 
organised  several  meetings  attended  by  representatives  of  all  the  programmes. 
Standardisation  was  therefore  the  culmination  of  a  long  process  of  collective 
discussion. 

Public  c o m m u n i c a t i on 

The  utility  of  a  system  of  indicators  is  often  reduced  to  decision-making.  In 

reality,  a  system  of  indicators  is  also  useful  for  accountability  purposes,  for 

example 

to  the  European  or  national  parliaments,  to  regional  or 

local 

elected  representatives,  to  socio-economic  partners,  to  journalists  and, 

through  them,  to  citizens  and  taxpayers. 

Indicators  for monitoring  and  evaluation 

If they  are to  serve as a basis for  public  communication,  a small  number  of 
indicators  which  can  immediately  be  understood  by  lay  people  must  be 
selected,  quantified  and  published. These  are  referred to  as  key  indicators. 
The publication  of  key  indicators  is normally organised  in the form of  simple 
commented 
in  an  annual  review.  More  detailed 
information  can  also  be  made  available  through  an  "observatory"  open  to 
the  public,  or through  a system  of  consultation  on the  Internet. 

tables,  for  example 

In  defining  good  key  indicators,  the  priority  must  be  on  generic  indicators 
(applicable  to  many  different  actions  within  the  same  programme)  and 
standard  indicators  (allowing  for  comparisons  between  programmes  in 
different 
indicators  must  be 
understood  by  all  the  partners  without  long  explanations  and  without  any 
misinterpretation  of  their  meaning. 

regions  or  countries).  Moreover, 

these 

Proposals  for  key  indicators 

Returning  to  the fifteen  information  sheets  presenting  the  different  fields  of 
application,  it  is  possible  to  propose  a  series  of  indicators  that  might  be 
used  as  key  indicators.  This  list  is  neither  closed,  nor  directive,  nor  stable. 
It  is  limited  to  programme  indicators.  The  indicators  have  been  given 
scores, from  *** to  *, in decreasing  order  of  interest. 

Producing and using indicators 

Box 35  -

Proposals for  key indicators:  Resources 

Interest 

Indicator 

See  example 

Human  resources 

Number  of operators  (public  and  private 
organisations  responsible  for  providing 
assistance to addressees  or  recipients) 

Number  of advisors  (FTEs) mobilised to  provide 
advice to  addressees  or  recipients 

Financial  resources 

Rate of  budget  absorption 
(%  of  allocated  funds) 

%  projects  (in financial terms)  especially 
benefiting  women 

·· 

%  projects  (in financial terms)  in rapidly  growing 
markets /  sectors 

%  of  budget  devoted to  environmental 
mitigation  measures 

%  projects  (in financial terms)  concerning 
the  most  disadvantaged  areas 

Indicators for monitoring and evaluation 

Box  36  -

Proposals  for  key  indicators:  Outputs 

Interest 

Indicator 

See  example 

*** 

** 

** 

** 

** 

* 

* 

M 

*** 

*** 

*** 

** 

Progress  of  works 

Rate  of  completion  (%  of  objective) 

Compliance  with  project  duration 

Capacity  of  finished  works 

Number  of  potential  connections 
(business  /  households)  to  networks  of  basic 
services  (broken  down  by  services) 
Activity  of  the operators  in terms  of  attracting 
and  selecting  participants 

Selection  rate (%  of  projects  accepted  as a 
proportion  of  eligible  projects) 

Coverage  rate  (Penetration):  %  of  the  target 
population  who  have  been  (should  be) 
participants  in the  programme 

%  of  addressees  or  recipients  belonging  to 
priority  groups  (e.g. long-term  unemployed, 
early  school  leavers) 

%  of  addressees  or  recipients  situated  in  the 
most  disadvantaged  areas 

%  of  addressees  or  recipients  involved  in 
rapidly  growing  markets  /  sector 

%  of  women  in addressees  or  recipients 

%  of  SMEs  in addressees  or  recipients 

Services  funded  by  the  programme 

Number  of  individual  addressees  or  recipients 
having 
received  services,  advice,  training 

Number  of  economic  units  (enterprise, farm, 
ship  owner,  fish  farm, tourism  professional) 
having  received  services, advice,  training 

1 , 2 , 3,  4 , 5 , 6,  7,  8, 
9,  18,28,  29,30, 
33, 34, 45 

1,2,3,  4 , 5 , 6,  7,  8, 
9,  18,28,  29,30, 
33, 34, 45 

28,  33, 34 

10,30 

10,15,  19, 36, 
44,48 

2 1 , 3 1 , 38 

Number  of  hours  of  training  /  advice  provided 
to  addressees  or  recipients 

10,  15,  1 9 , 2 1, 
31,  36, 37 

Producing and using indicators 

Box 37  -

Proposals for  key indicators: Results 

Interest 

Indicator 

See  example 

Satisfaction  of addressees  or  recipients 

Satisfaction  rate (%  of  addressees  or  recipients 
that  are satisfied  or  highly  satisfied) 

15,  19,25,27 
36, 37, 44 

Benefits  gained  by addressees  or recipients 

Average  speed  between  principal  economic 
centres 

1,2,32 

Investments  facilitated  for  addressees 
or  recipients 

Leverage effect  (private sector  spending 
occurring  as a counterpart  of the financial 
support  received) 

Box 38  -

Proposals for  key indicators:  Impacts 

Interest 

Indicator 

Sustainable  success 

Rate of  placement  (e.g.: %  of  individual 
addressees or recipients who  are at work  after 
12 months, incl. %  still employed after 24 months) 

16,21,22,24, 
39, 46, 47 

See  example 

10,36,48 

Rate of  survival (e.g.:  %  of assisted  economic 
units that  are still active after  24  /  36  months) 

23,37 

Impact  perceived  by addressees  or  recipients 

Value added  generated  (e.g.: after  24  months 
in terms of  euros 

/  year  /  employee) 

Employment  created  or  safeguarded 
(e.g.: after  12 months  Full Time  Equivalent) 

Impact  globally perceived  in the  area 

Residential attractiveness  (e.g.:  % of 
inhabitants  wishing to  remain  in the area) 

Indirect  impact 

Regional  knock-on  effects  (e.g.:  %  of  regional 
firms within the suppliers  of  assisted firms  after 
12 months) 

7 , 8 , 9,  13,14,  16, 
1 8 , 2 1 , 2 4 , 2 6 , 3 1, 
38, 39, 40, 42, 
46,47 

7 , 8 , 9 , 1 3,  14, 16, 
18,20,23,24,35, 
37, 38, 39, 42, 49 

25,51 

24,39 

* 

** 

** 

** 

* 

*** 

**# 

M 

* 

Indicators for monitoring and evaluation 

Measuring  performance  for  better  management 

More  and  more  systems  of  indicators  are  created  for  the  purposes  of 
"performance  management".  These  systems  are  a  form  of  New  Public 
Management  which  favours  results  and  impacts  obtained,  as  opposed  to 
older  forms  of  management  based  on  the  allocation  of  resources  and  the 
control  of  outputs. 

Managing  performance  rather  than  resources 

In a spirit  of  performance  management,  operators find themselves  endowed 
with  greater  autonomy  in the  use  of  their  resources.  In  return, they  commit 
themselves  to  clearer  objectives  as  regards  the  results  and  impacts  to  be 
obtained.  They  have  to  measure  their  performance  in  order  to  evaluate 
themselves  and  submit  periodic  reports.  This  new  balance  between 
decentralisation  and  performance  measurement  is  at  the  base  of  many 
developments  in  public  administration.  Thus,  in the  U.S.,  the  "Government 
Performance  and  Results  Act  (GPRA)"  of  1993  made  the  use  of  new 
indicators  compulsory  throughout  the  entire government  administration. 

Box  39  describes  the  system  of  indicators  of  the  British  Employment 
Service,  designed  in  the  same  spirit.  It  emphasises  the  measurement  of 
results  and  impacts,  rather  than that  of  resources  and  outputs. 

Box  39  -  Decentralisation  and  development  of  indicators 

The  mission  of  the  British  Employment  Service  is  to  develop  a  competitive, 
effective  and  flexible  job  market.  In  particular,  it  must  promote  the  integration  of 
the  most  disadvantaged  unemployed,  through  a  system  of  specific  funding.  The 
national  network  of  the  Employment  Service  has  over  1,000  agencies. 

Since 1994, the degree of autonomy  of the agencies has increased. In return, each 
agency  must  calculate  a  series  of  results  and  impact  indicators  and  compare  its 
performance  with that of  neighbouring agencies. The indicators are aggregated  by 
region and  at the  national  level, in an  Employment  Service annual  report. 

For  example,  in  Northern  Ireland, the  running  costs  of  the  agencies  in  relation  to 
the  number  of  unemployed  recruited, has evolved  as  follows: 

1992-93 
Baseline 

1993-94 
-  6% 

1994-95 
-  7% 

1995-96 
+  6% 

In many  European  regions, the administrative  culture  remains  impervious  to 
decentralisation  and  to  performance  management.  It  is  therefore  hardly 

Producing and using  indicators 

surprising that the development  of  result and  impact  indicators  is generally 
considered  to  be  difficult.  Programme  managers  are  more  familiar  with 
resource  and  output  indicators.  Cultural  changes  are, however,  slowly  but 
surely taking  place in certain  countries,  under  pressure from  administrative 
reforms initiated by national governments. The monitoring and evaluation of 
programmes  co-financed  by  the  European  Union  is  a  factor  favouring 
performance  management  in terms  of  results  and  impacts. 

In a perspective of performance management, indicators play an increasingly 
important  role for  operators and  must therefore  be used with caution. 

Interpreting  and  comparing  indicators 

Situations  exist  in which  indicators  require  little  interpretation,  but  these  are 
exceptions.  In  general,  indicators  have  to  be  interpreted  by  means  of  the 
relevant comparison or breakdown. In the following example, the comparison 
of  three  indicators  showed  that  the  training  financed  with  European 
assistance  did  not  reach the long-term  unemployed  as it should have. 

Box 40  -  Comparing several indicators for revealing a  phenomenon 

Within the framework  of  its intermediate  evaluation  of the  European Social  Funds 
in the  Italian Objective  1 regions, the  ISFOL gathered  and  interpreted  a series  of 
monitoring  indicators.  In  certain  cases,  the  comparisons  made  between  these 
indicators  led  to  strong  and  clear  conclusions.  For  example,  the  part  of  the 
programme  benefiting  the  long-term  unemployed  can  be  measured  by  means  of 
three  indicators: 

planned  expenditure  on the  long-term  unemployed: 

18%  of the total  planned  expenditure 

funds committed  to the  long-term  unemployed: 

11 %  of the total funds  committed 

funds  actually  spent  for the  long-term  unemployed: 

8%  of the total  funds  spent 

The  comparison  of  these  figures  clearly  shows  that  the 
long-term 
unemployed  are  disqualified  from  the  programme  at  each  step  of  its 
implementation. 

For  a  reader  who  glances  at  the  table  of  indicators  and  sees  these  three 
figures  in  isolation,  the  information  would  be  meaningless.  To  reveal  the 
problem  of  reaching  the  priority  public,  it  was  necessary  to  break  down 

Indicators for monitoring and evaluation 

these three  indicators  in terms  of  the  length  of  unemployment  of  the  direct 
addressees  or  recipients,  and for the  ISFOL to think  of  comparing  the three 
pieces  of  information. 

Whenever  possible,  it  is interesting  to  compare  programme  indicators  (e.g. 
number of  unemployed trained)  with the appropriate context  indicators (e.g. 
number  of  unemployed  in  the  region).  This  type  of  information  can  be 
presented  in a concise way  by  means  of derived  indicators that  express  the 
programme  outputs  or  effects  as a percentage  of the  context  indicator,  like 
those  described  in  Box 4 1. 

Box  41  -  C SF  (Objective 
and  context  ind 

1)  1 9 8 9 - 93  -  C o m p a r i s on  of  p r o g r a m me 
cators  -  Spain 

Number  of  digital 
telephone  lines 

Recorded 
data 

239  800 

Funded  universities 

20 

Data  as  a  percentage 

6  %  of  the digital  lines  installed 
in  Spain during  the  period 

91  %  of the  Universities  in  the 
Objective  1 regions 

Creation  and  improvement 
of  irrigated  land  (in  ha) 

34  236 

1,5  %  of  the  surfaces  irrigated  in 
the Objective  1  regions 

Number  of  modernised 
farms 

Number  of  young  farmers 
subsidised 

107  000 

4,8  %  of  the  farms  in Spain  (1992) 

11  000 

0,5  %  of  the  farms  in Spain  (1992) 

Number  of  persons 
trained 

2  693  000 

17,7  %  of  the  working  population 
in  Spain 

Most  evaluation  professionals  recognise 
indicators  have  to  be 
interpreted  by also  relying on qualitative  information. To interpret  indicators, 
it  is  therefore  necessary  to  consider  the  context  as  a  whole,  the  factors 
which  help  to  facilitate  or  hinder  the  performance  of  the  programme,  the 
raison  d'être  of  the  programme,  and  the  process  of  implementation  of  all 
sorts  of  qualitative  elements  (Burt  Perrin Associates,  1997). 

that 

An  original  technique  consists  of  asking  an  expert  panel,  capable  of 
combining  the  quantitative  and  qualitative  elements  of  the  situation,  to 
interpret  the  performance  measures.  This  technique  has  been  used  for 
about  ten  years  to  evaluate the  performance  of  university  research  centres 
is  monitored 
in  the  UK.  The  work  of  all  the  UK  research  centres 

Producing and using  indicators 

quantitatively  and  periodically  evaluated  by  expert  panels.  In  their 
indicators  (e.g.  number  of 
evaluation,  the  experts  take  into  account 
publications)  but  also  purely  qualitative  elements  (e.g.  quality  of  each 
researcher's 
in  a 
four  best  publications).  The  evaluation 
classification  of the  research teams  (on  a scale of  1 to 5). The allocation  of 
public  funds  is directly  related  to this classification  (Bone & Carter,  1997). 

results 

The  interpretation  of  indicators  often  makes  use  of  comparisons  between 
the  performance  of  two  interventions.  This  type  of  comparison  mainly 
concerns efficiency  indicators, for example: cost of a trainee's training; cost 
of  a job  created; cost  of  a  home  connected  to  the  sewerage  system;  etc. 
However, comparisons are not always acceptable for programme  operators 
and  managers.  In  particular,  performance  comparisons  will  seem  unfair  to 
managers  who  work  in  the  most  difficult  contexts.  To  make  comparisons 
acceptable, the comparison  procedures  must  be considerably  refined  with 
a view  to  putting  those  interventions  functioning  in  a difficult  environment, 
on an equal footing. 

An  interesting  technique  was  tested  in  the  domain  of  education.  The 
technique consists  of  reconstituting, for each  intervention  to  be compared, 
a  group  of  several  similar  interventions  implemented  in other  regions  in an 
environment  with  the  same  degree  of  difficulty.  The  comparison  is  made 
with the  average of the  interventions  of this  "benchmark  group". 

Box 42 - Example  of  comparison  of performance  accepted 

by the  operators 

The  comparison  of  performance  between  training  organisations  often  stumbles 
over the fact  that  the  different  publics trained  do  not  have the same difficulties. A 
few  years  ago  an  evaluation  team  had  to  compare  133  training  organisations 
(school districts)  in different  states in the U.S. The team decided to compare  each 
organisation to a "benchmark  group"  of  14 other organisations with similarities as 
regards  the  characteristics  of  the  public  enrolled  (poverty;  home  language; 
parents'  level  of  education;  belonging  to  a  minority  group;  etc.).  To  form  the 
standard groups, the evaluation combined  a factor  analysis  and expert  panels. At 
the  end  of  the  evaluation, the  performance  comparisons  were  accepted  without 
question. 
Henry  & ai,  1992 

This  approach  assumes  that  a  performance  indicator,  measured  in  a 
comparable  way,  is  available  for  a  large  number  of  interventions.  The 
method  is difficult  to  apply  if the  performance  is judged  on  several  criteria, 

Indicators for monitoring and evaluation 

indicators  but  is  not 

then  be  adopted,  in  which 

and  even  more  so  if  the  performance  is  not  completely  measurable.  A 
the  comparison 
global  approach  must 
encompasses 
limited  to  them.  The  appropriate 
technique,  in  this  situation,  is  that  of  benchmarking  which  consists  of 
comparing  an  intervention  to  a  model  outside  the  programme,  recognised 
for  its good  performance.  Use of this  technique  implies that  a reference  for 
comparison  can  be  found,  for  example  in  a  data  bank  of  the  "best 
practices".  Benchmarking  is a tool which  produces  acceptable  conclusions 
because  it  can  take  into  account  multiple,  and  not  only  quantitative, 
elements  of  comparison.  One  of the  advantages  of  benchmarking  is that  it 
orientates  the  comparisons  towards  the  best  practices  and that  it  therefore 
avoids  convergence  towards  average  performance.  The  benchmarking 
technique  is described  in Volume  3 of  the  MEANS  Collection. 

Avoiding  adverse  effects  of  indicators 

Use of  indicators  is often  hindered  by the  fear of  provoking  adverse  effects. 
An  example  of  adverse  effects  is  the  skimming-off  effect  which  appears 
when  the  performance  of  training  organisations 
is  measured  by  the 
placement  rate  of  trainees.  To  obtain  a  better  placement  rate  for  their 
trainees,  it  is  in  the  organisations'  interests  to  recruit  people  in  the  best 
possible  situation  who  also  meet  the  eligibility  criteria.  The  operators 
therefore  tend  to  "skim  off"  potential  direct  addressees  or  recipients  by 
favouring  those  whose  employability  is  higher.  This  effect  is  undesirable 
because it helps to focus assistance on those who are relatively  less in need. 

to 

result 

indicators 

Another  example  of  an  adverse  effect  appeared  after  a  few  years  of 
implementation  of the  British Audit  Commission's  system  of  indicators  (see 
Appendix  2). This system organises the quantification of about two  hundred 
output  and 
the  services  offered  by 
relating 
municipalities.  The  indicators  are  comparable  from  one  town  to  the  next 
and  are  published  annually  in  the  local  press.  This  system  creates  a  very 
strong  impression  of the town  when  one of the  services  performs  badly.  As 
a  result,  many  municipalities  increased  the  budgets  of  their  services  with 
the  lowest  performance,  in the hope of  raising the standard. In these cases, 
financial resources were sometimes  drawn from the most  effective  services. 
Use  of  indicators  thus  caused  a  reduction  in  differences  by  convergence 
towards  the  average.  This  was  an  adverse  effect  since  the  measurement 
towards 
exercise  would  have  been  expected 
excellence. 

to  push  performance 

Producing and using  indicators 

An adverse effect appears when an indicator encourages  behaviour  leading 
to  sub-standard  performance.  This  occurs  when  the  indicator  rewards 
undesired  results or when the system  causes the operators  to work  for  the 
indicator  rather  than  for  the  result.  Adverse  effects  inevitably  appear  after 
two  or three years of  functioning  of  a system  of  indicators,  no  matter  how 
well  it is designed. These  undesirable effects  are generally  not foreseeable. 

The  probable  appearance  of  adverse  effects  must  not  be an argument  for 
refusing to measure performance. It is relatively easy to guard against them, 
either  by  amending  the  indicator  causing  the  problem,  or  by  creating  a 
procedure  for  interpretation  of  the  indicator  by  expert  panels.  It  is  then 
important to watch out for the appearance of adverse effects and to  correct 
the  system when these effects  appear. 

Creating  performance  incentives 

Box 43 -  Selection of  projects  in terms  of their  effectiveness 

Within the framework  of the Objective 2 programme for the period  1989-93 in the 
Pays-de-la-Loire  region  (F),  the  ESF  actions  were  managed  by  the  Regional 
Council.  For  the  management  of  the  ESF, the  services  of  the  Regional  Council 
applied  a  project  selection  method  used  for  their  own  training  activities.  This 
method  is  based  on  performance  management  and  functions  in  the  following 
way. 

The  training  organisations  filing  their  first  application  for  funds  are  subjected  to 
less severe selection  procedures, on the basis of the quality of their projects. This 
selection  mechanism  is exceptional  since  "primary  applications"  are rare. 
Organisations  already  receiving  assistance  and  which  apply  for  a  new  subsidy 
(secondary  applications)  enter  into the  mechanism  of  normal  project  selection. In 
this  normal  procedure,  funds  are  granted  first  to  organisations  with  the  best 
placement  rate.  The  monitoring  system  obliges  all  organisations  that  offer 
subsidised  training  to  provide  the  Regional  Council  with  an  evaluation  of  the 
placement  rate of  its trainees  after  six  months. 

The  effect  of  this  mechanism  is  to  favour  those  organisations  which  are  most 
tuned  in to the  needs of  business  and which  have the  best  impacts. 

There  are  several  ways  of  using  indicators  to  promote  an  improvement  in 
operators'  performance. These include: 

•  Operators with poor performance  receive specific technical  assistance to 
help  them  progress.  If  the  situation  does  not  improve,  the  budget  is 

Indicators for monitoring and evaluation 

restricted. This method functions on the principle that  it is not the  mistake 
that  must  be  punished, but  rather  the  inability  to  correct  mistakes. 

•  Operators  with  the  best  performance  are  granted  greater  autonomy  and 

are controlled  less. 

•  Operators  with the  best  performance  receive support  for  presenting  their 

outputs  and  results  to  the  general  public. 

•  Operators  who  did  not  perform  well  enough  are  disqualified  from  the 
selection  procedures  for  future  projects,  as  in the  example  in Box  43. 

•  Operators  with  the  best  performance  are offered  additional  funds. 

Conclusion 

Box  44  summarises  the  main  message  conveyed  by  this  Volume.  It  is 
presented  in the form of a set of assumptions that are made, often  implicitly, 
when  a  system  of  indicators  is  created  with  the  intention  of  improving  a 
programme's  performance. 

Box 44  - The usefulness of a system of indicators 

The system of  indicators 
is constructed 

\^ 

\^ 

It reflects the objectives 
and makes them 
more legible 

Information 
is disseminated 
to the public 

\. 
/ 

/ 

It  produces 
the desired  information 

* 
The information is 
correctly  interpreted 

V 

1 Γ 

The information is used 
to  provide an incentive 
for better  performance 

N. 

\ 

τ 
A watch is kept for 
perverse  effects 

' 

Managers 
make better 
decisions 

/ 

/ 

' 
The performance of the 
programme  improves 

' 

/ 

! 
1 
1 

Citizens 
are in a position 
to form an opinion 

The  advantage  of  this  diagram  is  that  it  shows  that  the  construction  of  a 
system  of  indicators  is not  really  a first  step. At  least  six  other  steps  occur 
before the system  is actually  useful. The first step must therefore  be treated 
pragmatically, always  keeping  in view the prospect  of maintaining the effort 
until the end  of  all the other  steps. 

Appendixes 

Appendix  1 -  General  bibliography 

The  bibliography  concerning  indicators  is  immense.  The  following  titles 
have been  selected  because they  are general  and  recent. 

Auditor  General  of  Canada  (1997),  Reporting  Performance 

the 
Expenditure  Management  System,  Report  of  the Auditor  General  of 
Canada to  the  House of  Commons,  Chapter  5. 

in 

Auditor  General  of  Canada  (1997), Moving  towards  Managing  for  Results, 
Report  of the Auditor  General of Canada to the House of Commons, 
Chapter  11. 

Bone, J., Carter,  N. (1997), On the Performance  of  Performance 

Indicators, 
l'évaluation  des 
Communication  au  colloque 
politiques  publiques  de  l'école  normale  supérieure  de  Cachan, 5  / 
6  February  1997. 

International  sur 

Boyle,  R.  (1996),  Designing  and  using  performance 

indicators,  Dublin 

Institute  of  Public  Administration. 

Burt  Perrin  Associates  (1997),  'Evaluation  in  Partnership:  The  May,  1997 
Canadian Evaluation Society Conference' in: Evaluation 3(3), 377-379. 

Carter,  N.,  Klein,  R.  and  Day,  P.  (1992)  How  Organisations  Measure 
in  Government. 

the  Use  of  Performance 

Indicators 

Success: 
London:  Routledge. 

Commission  européenne  (1994)  Le  rapport  européen  sur  les  indicateurs 
scientifiques  et  technologiques.  Série:  Sciences  et  technologies 
nucléaires.  Luxembourg:  Office  des  publications  officielles  des 
Communautés  européennes. 

Commission  européenne (1995)  Orientations  communes  pour  le suivi et les 
évaluations  intermédiaires.  Luxembourg:  Office  des  publications 
officielles  des Communautés  européennes. 

Conseil du Trésor du Canada  (1997), Rendre  compte  des résultats,  Rapport 
annuel  au  Parlement,  Direction  des  affaires  publiques,  Secrétariat 
du  Conseil du Trésor du  Canada. 

Derosières  A.  (1997)  'Refléter  ou  instituer  :  l'invention  des  indicateurs 
statistiques'  in  Dupoirier  E.  et  Parado  J.L.  Les  indicateurs  socio-
économiques  aujourd'hui.  Paris:  L'Harmattan,  15-34. 

Indicators for monitoring and evaluation 

Halachmi, Α., Bouckaert G. (Eds) (1995), Public Productivity through Quality 

and Strategic Management, Amsterdam: IOS press. 

Hatry, H.P., Fall, M., Singer, TO., Liner, E.B. (1990), Monitoring the 

outcomes of economic development programs. A manual, 
Washington, D.C.: The Urban Institute Press. 

Henry G., McTaggart M., Macmillan J. (1992) 'Establishing benchmarks for 
outcome indicators, a statistical approach to developing 
performance standards'. Evaluation review, 16(2), 131-150 

Innés De Neufville J.E. (1994), Knowledge and Policy, the Search for 

Meaningful Indicators. New Brunswick: Transaction Publishers. 

Kumar, K. (1995) Measuring the Performance of Agricultural and Rural 

Development Programs in Evaluation and Development. The 
proceedings of the 1994 World Bank Conference. Washington, D.C.: 
The International Bank for Reconstruction and Development. 

OCDE (1994) 'Mesure des performances et gestion axée sur les résultats', 

Gestion publique: La gestion des performances dans 
l'administration, Collection Etudes Hors Série, 3. Paris: OCDE. 

OCDE (1994) Créer des indicateurs ruraux pour étayer la politique 

territoriale, Paris: OCDE. 

OCDE, (1994), Making Education Count: Developing and Using International 
Indicators. Washington, D.C.: Centre for Educational Research and 
Innovation. 

Pollitt, C. (1986), 'Performance Measurement in the Public Services: Some 

Political Implications', in: Parliamentary Affairs, 39 (3) pp 315-329. 
Oxford University Press. 

Pollitt, C, Summa, H. (1995), Performance Audit: Travellers' Tales, paper 
presented at The Canadian Evaluation Society and the American 
Evaluation Society Conference, October 1995. 

Newcomer K.E. (1997) 'Using Performance Measurement to Improve 
Programs', in: New Directions for Evaluation, (75), 5. 

Salzer, M., Nixon, CT., Schut, L.J.A., Karver, M.S, Bickman, L (1997), 

Validating quality indicators. Quality as relationship between 
structure, process, and outcome, Evaluation Review, 21(3). London: 
Sage Publications, 292-309. 

Appendixes 

Sorber, Α. (1993), 'Performance measurement in the Central Government 
Departments of the Netherlands', Public Productivity and 
Management Review, autumn, vol. 17(1). 

United Nations Development Programme (1997), Results-oriented 

Monitoring and Evaluation, New York: UNDP, OESP Handbook 
Series. 

Vakkuri, J., Meklin, P. (1997), Risks in value for money-related performance 

measurement, Leuven: Conférence annuelle du Groupe Européen 
d'Administration Publique. 

Wholey, J.S., Newcomer K.E. (1997) 'Clarifying Goals, Reporting Results', 

New Directions for Evaluation, (75), 91. 

Wishlade, F., Yuill, D. (1997), Measuring disparities for area designation 

purposes: issues for the European Union, Regional and Industrial 
Research Paper Series, N°24. Glasgow: European Policies Centre, 
University of Strathclyde. 

World Bank - Operations Evaluation Department (1997) Evaluating 

Development Operations: Methods for Judging Outcomes and 
Impacts, Washington: World Bank, OED, Lessons & Practices, 
Number 10. 

Appendix 2 - Lessons from international experience 

In  the  1970s  this  success  gave  rise  to 

International  experience  concerning  indicators  is vast and  has existed for a 
long  time.  At  the  origin  of  all  developments  in  this  domain,  we  find  the 
success  of  a  few  major  macro-economic  indicators  (GDP,  price  index, 
unemployment  rate). 
the 
development  of  social  indicators,  an activity  which  mobilised  considerable 
energy  and  was  the  source  equally  great  disappointment.  Numerous 
lessons  were  drawn  from  this  experience  and  were  soundly  established. 
They  could  almost  be  summarised  in  the  titles  of  the  three  noteworthy 
publications  by  Judith  Innes  de  Neufville:  Social  indicators  and  public 
policy: 
(1975); 
Disappointments  and  Legacies  of  Social  Indicators  (1989); Knowledge  and 
Policy:  the Search  for Meaningful  Indicators  (1994). 

of  design  and  application 

processes 

interactive 

Whereas  more  recent  developments  called  "performance  measures"  are 
appearing,  some  public 
invest,  sometimes 
massively, in indicators.  For example, the UN and the World  Bank  launched 
a  2  M  euros  project  to  develop  a  system  of  indicators  for  the  urban 
environment. 

institutions  continue  to 

The  countries  involved  can  give  only  a  very  partial  view  of  the  experience 
which  has  been  capitalised  world-wide.  The  examples  have  been  chosen 
for their  similarities  with  the  main  subject  of this Volume. 

Science  and  Technology  Indicators  at the  Commission 

Massive 

investment 

Initially,  from  1988  to  1992,  the  European  Commission  (DG  XII)  formed  a 
network  of experts on the  basis of  one specialist  per country. This  network 
held two  meetings  per year for four years, under the aegis of the  MONITOR 
programme.  Two  large  conferences  were  organised  in  Paris  and  Leiden 
(about  100  participants).  The  Commission's  investment  in  this  network 
represented  about  4  persons  χ  month  per  year.  Part  of  the  network 
leadership  was  sub-contracted  to  an outside  consultant.  At  the  end  of  the 
MONITOR  programme  the  Commission  stopped  funding  the  network, 
although  it supported  a new  meeting  held  recently  in Antwerp. 

After  this  first  investment,  DG  XII  formed  an  "Indicators"  unit,  representing 
an investment  of 30 to 70 persons χ month, depending on the year. This  unit 

Indicators for monitoring and evaluation 

sub-contracts  specific  tasks  to  numerous  consultants.  The  unit  issues  a 
400-page  report  every  two  years  (the  last  editions  were  in  1994  and  1996). 
This  report  presents  research  context  indicators  in 50  countries,  as  well  as 
resource,  output,  results  and 
indicators  for  European  policy 
impact 
(framework  programme).  It  is  partly  based  on  the  work  of  Eurostat,  which 
devotes  a  large  proportion  of  its  resources  to  science  and  technology 
indicators  (e.g. REDIS News,  Research and  Development:  annual  statistics). 

Highly  favourable  international 

environment 

There  is a strong  convergence  of  the  Commission's  interests  and those  of 
certain  Member  States.  France  has  created  a  Science  and  Technology 
Observatory  and  publishes  a report  similar  to  that  of  the  Commission.  The 
Netherlands  also  invests  substantially,  with  the  MERIT  programme,  and 
Germany  has  developed  systematic  surveys  on  research  and  innovation. 

The  United  States  has,  in  many  respects,  been  a  forerunner  in  this  field, 
with  significant  investments  in  S&T  indicators  by  both  public  and  private 
organisations  (NSF and  the  Science  Citation  Index,  respectively). 

Thus,  the  Commission's  efforts  concerning  science  and 
technology 
indicators have been supported and encouraged  by previous  developments 
of at  least equal  importance, as in the  USA and certain  Member  States,  and 
the OECD. 

Problems  in  regionalising 

indicators 

In  1993  and  1994  an  interservices  commission  drew  up  a  complete 
inventory  of the  possibilities  of  regionalising  R&D indicators. This work  was 
presented  in a report  and  in a chapter  on  regional  R&D indicators  in DG XII 
publications. 

The  interservices  commission  then  carefully  selected  all the  indicators  that 
could  be  quantified  on  a  regional  level  and  which  had  good  information 
potential  for  regional  policy.  It  turned  out  that  certain  indicators  could  be 
regionalised:  R&D  Spending;  Employment 
in  R&D;  and  patents.  The 
quantification  was  rarely  broken  down  below  the  NUTS  2  level.  These 
limited  results  reflect  the  difficulty  of  regionalising  indicators. 

International  OECD  indicators 

Twenty  years  later  in  the  domain  of  education 

the 
The  OECD 
intermediary  of  its  Centre  for  Educational  Research  and  Innovation.  The 

invested  massively 

in  education 

indicators 

through 

Appendixes 

effort  can  be  traced  back  to  the  1970s  when  it  was  part  of  the  social 
indicators  movement. 

Initial  ambitions  were  too  great  and  the  disappointments  of  the  late 
seventies  nearly  cost  the  programme  its  life.  The  current  analysis  of  this 
setback  is  as  follows:  research  was  the  predominant  focus,  the  indicators 
were expected to show too  much, and they were not  linked closely  enough 
to the  main  political  issues. 

The  work  was  revived  at  the  initiative  of  the  American  government  which, 
from  1983,  became  heavily  involved  in  improving  the  effectiveness  of 
education.  Towards  the  end  of  the  1980s,  the  OECD 
its 
programme  for  developing  indicators,  through  a  series  of  international 
conferences  supported  by  volunteer  countries  (e.g.  USA,  France).  The 
programme  produced  numerous  results  in  4  years  (at  least  7  significant 
publications,  including  "Education  at a Glance, OECD  indicators"). 

launched 

Contrary  to  its initial ambitions, current  work  is considered  by the OECD  to 
be  highly  pragmatic.  The  system  of  indicators  has  been  constructed 
without  any coherent  theoretical  model. The theory  remains  to  be done -  if 
ever  it can  be done  in such  a highly  sensitive  field. 

The  other  areas  in  which  the  OECD  has  invested  heavily  in  terms  of 
indicators  are:  Science  and  Technology;  Health;  Development  aid;  the 
Environment;  Tourism;  Fisheries  and  Agriculture.  The  working  method  is 
roughly  the same as in the  domain  of  education,  i.e.: 

•  the initiative  is supported  by one  or  more countries  highly  involved  in the 

development  of  a system  of  indicators  for  their  own  use; 

•  an  international  task  force  is  formed  for  several  years,  to  negotiate  a 

common  language  and  set  pragmatic  objectives; 

•  a detailed  questionnaire  is sent  to  the  Member  States  which  provide  the 
best  quantification  possible for the  indicators  selected  by the task  force; 

•  the  first  version  of  a  directory  is  published  with  information  partially, or 

totally  lacking  for  several  countries; 

•  all the countries  progressively  complete their  statistics  in order to join the 

project. 

The development  of systems of  indicators is very progressive. For example, 
the  first  directories  for  the  environment  contained  indicators  which  were 
new  for  many  countries.  Thus,  the  quality  of  their  quantification  varied 
widely.  However,  these  directories  act  as  models  and  the  Member  States 

Indicators for monitoring and evaluation 

are  gradually  equipping  themselves  so  as  to  be  able  to  answer  OECD 
questionnaires  and thus  obtain  references  for  international  comparison. 

Régionalisation  and  territorialisation  of  statistics 

is  potentially 
The  OECD  has  a  territorial  development  service  which 
interested  in  indicators  developed  for  the  Structural  Funds.  However,  the 
only  progress  made  by  the  OECD  in  this  field  has  concerned  rural 
development. 

The  OECD  Rural  Development  Programme  was  launched  in  1991.  It 
developed  a  set  of  indicators  concerning  population  (density;  evolution; 
structure;  households),  the  economy  (working  population;  employment; 
sectoral  structure;  productivity;  and  investment),  social  welfare  (income; 
housing;  education;  health; and  security)  and the  environment  (topography 
and climate;  land  use; habitats  and  species; ground  and water;  air  quality). 
A rural indicators data base gives  internationally  comparable  information  for 
categories  of  territory  corresponding  essentially  to  degrees  of  population 
density. 

The  reason  for  the  OECD  not  developing  other  territorial  indicators  is 
probably  that the Organisation  is made up of countries  and  is not  supposed 
to  concern  itself  with  regional  affairs.  The  régionalisation  of  systems  of 
indicators  could  be  justified  in  the  framework  of  co-operation  between 
member  states,  in  the  field  of  regional  policies,  but  such  co-operation  is 
very  limited.  This  is  because  the  United  States  does  not  have  significant 
regional  policies  comparable  to  those  of  Europe  and  Canada. 

World  Bank  monitoring  and  evaluation  indicators 

Operations  financed  by  the  World  Bank  account  for  an  annual  expenditure 
virtually  equivalent  to  that  of  the  Structural  Funds.  Self-evaluation  reports 
are  systematically  written  at  the  end  of  each  operation,  by  the  operations 
departments.  These  reports  are  submitted  to  the  borrower  countries  for 
their  opinions. About  280  self-evaluation  reports  are  produced  annually. 

The  Bank  has an Operations  Evaluation  Department  (OED)  primarily  for  the 
purposes  of  "counter-evaluation"  (about  a quarter  of  the  reports  submitted 
by  operational  departments  are verified  in the  field),  "returning  to  the  field" 
after  five  years  (about  10%  of  the  operations),  more  in-depth  analysis  of 
certain  themes  (e.g.  women's  participation),  evaluations  by  country,  and 
periodic  checks  of  the  monitoring  and evaluation  procedures. 

Appendixes 

For  a  long  time  the  Bank's  information  system  focused  on  volumes  of 
financial  commitments  and  on  physical  output  indicators. The  Wapenhans 
report  in  1992,  recommending  that  "results  in  the  field"  be  given  more 
attention, was a turning  point. The Bank then undertook to establish  menus 
of  results  and  impact  indicators  for  each  domain.  In  1993  and  1994  each 
sector  service  published  its  menu  of  indicators,  to  which  a  list  of  generic 
indicators was added. 

This  work  was  consulted  and  used  in the  present  study. The  quality  of  the 
work was found to vary widely, with  many  indicators oriented  more  towards 
a  description  of  context  than  towards  results  and  indicators  for  specific 
domains. Moreover, these  menus are generally  abundant  but  hardly  related 
to the  Bank's  objectives. 

The  OED  recently  audited  the  Bank's  monitoring  /  evaluation  practices.  It 
showed that the use of  indicators  is way below the stated  ambitions. Yet, in 
a  few  years,  the  proportion  of  operations  using  results  and 
impact 
indicators  has  risen from  12% to  36%. 

Without  abandoning  its  project  to  enhance  these  menus  of  sectoral 
indicators, the Bank now places the emphasis on the use of a limited number 
of  indicators  peculiar  to  each  operation.  These  indicators  are  to  be 
constructed  in  relation to  the  specific  logic  of  each  operation,  by  means  of 
the  logical  framework  method.  Efforts  point  towards  a  high  degree  of 
decentralisation  in  the  development  of  indicators.  Rather  than  unifying 
systems  of  indicators,  the  idea  is  rather  to  unify  the  operations  managers' 
procedures  for  constructing 
indicators.  In  this  bottom-up  approach, 
comparability  is a secondary concern. The OED's instructions are as follows: 

•  use  a  very  small  number  of  indicators  for  each  operation  (between  ten 
and twenty); these  indicators  must  cover  all the  logical  steps:  resources, 
outputs,  results and  impacts. 

•  favour  risk  indicators,  likely  to  sound  the  alarm  in  cases  of  failure  of  the 

operation. 

•  Performance  indicators  at  the Audit  Commission 

Development  combining  obligation  and  partnership 

In  July  1991  the  British  government  launched  the  "Citizens'  Charter",  an 
initiative aimed at improving the quality of public services. One of the key points 
in this initiative was the development  and use of output  and result indicators. 

Indicators for monitoring and evaluation 

The  Audit  Commission  is  responsible  for  developing  and  managing  a 
system  of  indicators  applied  to  about  one  thousand  local  authorities  in 
England and Wales. The role of the Commission and local authorities varies, 
depending  on the  case: 

•  Initially,  the  local  authorities  are  always  encouraged  to  set  their  own 
output  and  results  objectives  and  to  choose  appropriate  indicators  for 
monitoring  them. 

•  In  certain  cases,  the  Commission  sets  the  indicators  and  the  way  of 

quantifying  them, but  does  not  define  objectives. 

The Commission  may  also  act  in a more  authoritarian  way  by  deciding  that 
certain  indicators  must  be quantified  and that  the  local  authorities  must  set 
their  own  objectives  for  these  indicators. 

Lastly,  the  Commission  may  decide  on  both  the  indicators  and  the 
objectives. 

Each  year,  the  Audit  Commission  publishes  a  manual  with  the  detailed  list 
of  about  200  indicators  that  must  be  measured  by  the  local  authorities 
during the following  year.  No authority  is obliged  to  quantify  more than  130 
indicators.  The 
their 
performance  measurements  in  the  local  newspapers.  The  Commission 
regularly  gives  itself  the  means  to  ensure  that  the  required  systems  are  in 
place  and  produce  accurate  information. 

local  authorities  are  legally  obliged  to  publish 

relating 

The  Commission  produces  a  report  in  three  volumes  (Local  Authority 
Performance)  which  presents  a  synthesis  of  all  data 
to 
performance,  with  national  averages.  Performance  indicators  are  also 
available  on  computer  disk.  A  limited  report  concentrates  on  about  forty 
comparable  indicators  aggregated  at  national  level.  This  report  includes 
between  one  and  five  indicators  for  each  of  the  following  fifteen  fields: 
Education;  Social  affairs;  Libraries;  Expenditure  management;  Social 
housing;  Recycling;  Planning;  Tax  collection;  Inspection  of  food  outlets; 
Handing  of  complaints,  calls  to  the  police  and  emergency  services;  Crime 
and  investigations;  Resources  of  the  police  and  fire  brigade.  For  instance, 
the  percentage  of  recycled  household  waste  is one  of the  indicators  of  the 
"Recycling"  domain. 

Schedule  for  implementation  of  new 

indicators 

To  update  and  improve  the  system,  the  Commission  maintains  in-depth 
interaction  with  local  authorities,  consumer  groups,  government  services, 

Appendixes 

professional  bodies  and other  stakeholders.  It also  employs  consultants  to 
carry  out  national  surveys  to  determine  what  must  be  measured,  what  is 
understood  by good  performance,  what  should  be  published, etc. 

The introduction  of a new indicator takes about four years, according to the 
following  schedule: 

Yearl 

September 

Consultation  and  survey 

December 

Guidelines  published  by the  Commission 

Year  2 

March 

Beginning  of  performance  measurement  by 
the  Local  Authorities 

Year  3 

March 

First  performance  measurement 

December 

Year  4 

March 

Publication  of  performance  measurement  in 
local  newspapers 

Publication  of  performance  measurement  in 
the  national  annual  report 

Eligibility  indicators for  German  regional  policy 

It  seems  that  Germany  is the  most  advanced  European  country  in the  use 
of  statistics  for  regional  policy  purposes.  For  several  years,  local  and 
regional statistics  have  been the object  of  long-term  investments  and  have 
been  used  effectively  in  policy  planning.  The  following  lines  attempt  to 
summarise  an old and  varied  practice. 

Use  of indicators  for  eligibility 

is  partly  responsible  for  the 
The  German  regional  planning  ministry 
"Common Task"  (Gemeinschaftsaufgabe), which makes up German  regional 
policy. This policy  is managed intergovernmentally,  by a planning  committee 
consisting  of  ministers from  both  levels (Federal and the  Länder). 

Traditionally,  the  areas  eligible  for  regional  support  are  determined 
rigorously  on  the  basis  of  a  system  of  indicators.  This  system  is  revised 
periodically,  under  the  responsibility  of the  planning  committee  and  a sub­
committee  of  experts.  The  last  revision  mobilised  several  institutes: 
Bundesforschungsanstalt  für  Landeskunde  und  Raumordnung  (BFLR)  -
Bonn, 
für 
Arbeitsmarkt  und  Berufsforschung  -  Nürnberg. 

-  Dortmund,  and 

für  Raumplanung 

Institut 

Institut 

the 

The  current  system  consists  of  four  key  indicators:  unemployment  rate; 
wage  level;  infrastructure  index  (see  below);  and  manpower  requirement 

Indicators for monitoring and evaluation 

forecasts.  The  infrastructure  index  has  been  one  of  the  key  indicators  for 
regional  policy  since  the  1980s.  In  its  original  form, the  index  consisted  of 
22  sub-indicators.  The  new  index  does  not  fully  warrant  its  initial  name.  It 
consists  of  10  sub-indicators:  four  in  the  transport  domain,  five  in  the 
human  resources  domain, and  one  population  density  indicator. 

For  transport,  the  four  indicators  are  as  follows:  accessibility  by  car  of  the 
closest  regional  centre;  accessibility  by train  of  the  closest  regional  centre; 
accessibility  by  any  means  of  transport  of  the  main  European  cities  (see 
following  paragraph);  travelling  time  to  the  closest  train/bus  station.  For 
human  resources,  the  five  indicators  are  as  follows:  number  of  young 
people  in  occupational  training  exceeding  the  capacities  of  the  local 
industry;  number  of  places  in  occupational  training  in  regional  public 
institutions;  percentage  of  the  regional  working  population  benefiting  from 
occupational 
training;  percentage  of  the  regional  working  population 
benefiting  from technical training. 

Accessibility 

indicators 

The  BFLR  calculates  an  accessibility  indicator  for  the  34  largest  European 
cities,  by  the  best  means  of  transport.  Calculation  of  the  indicators  makes 
use  of  a  model  associated  with  a  European  data  base.  It  is applied  to  the 
main towns  in all the  NUTS 3 areas  in all the  Member  States. This  indicator 
appeared  in  a  study  for  DG XVI  in  1992. The  last  quantification  was  made 
in  1995. 

the  gain 

in  accessibility 

Accessibility  indicators  are used for determining  eligibility. They are also very 
useful  in  ex  ante  evaluations.  For  example,  a  recent  study  by  the  BFLR 
estimated 
these  two  basic 
infrastructure programmes in the new Länder. The first programme  concerned 
11  new  railway  links  between  the  western  and  the  eastern  Länder.  The 
second  programme  was  composed  of  motorway  projects.  The  evaluation 
showed an increase in accessibility that was far greater in the case of the first 
programme  than  in  that  of  the  second  one.  There  is  no  example  of  use  of 
accessibility  indicators for  intermediate  or ex  post  evaluations. 

resulting 

from 

•  An  original  approach  to  the  régionalisation  of  statistics 

Over and above its responsibilities as regards  Regional  Policy indicators, the 
BFLR also runs a regional statistical data base, similar to the Eurostat  REGIO 
data base. These regional statistics came into existence in 1975. At the time, 

Appendixes 

it  was  discovered  that  the  German  statistics  institutes  were  organised  in 
such  as way  as to  make  any  comparison  on  a  level  lower  than  the  regions 
impossible.  In fact, the federal office was  only  responsible for  comparability 
at  the  Länder  level.  Only  the  offices  of  the  Länder  were  in  a  position  to 
harmonise  definitions  at  the  sub-regional  level. Today,  owing  partly  to  the 
efforts  of the  BFLR, 95%  of all local data are comparable. The BFLR  orders 
special temporal series from the statistics  institutes (e.g. migratory flows  into 
and out of the territory), amounting to about  100 K  euros  /  year. The  BFLR 
also supports data collection at the sub-regional level. 

Regional  data  are  disseminated  in  various  ways,  such  as  publication  of  a 
300-page  statistical  report  every  two  years  and  provision  of  data  to  the 
different  ministries  and  to  private  companies  and  academic  institutions 
which  can  procure  updated  disks  at  a  reasonable  price  (approx.  100 
customers  /  year). 

The data  base relies  partly on a regional  observation  system  which  gathers 
data  by  means  of  regular  surveys  on  samples  of  about  4,000  households. 
Questions  concern  population,  mobility,  housing,  social  environment, 
foreigners, etc. Because of the  method  used to gather the data, the  system 
supplies  statistics  not  on  particular  areas  but  on  only  a  few  categories  of 
area.  The  main  categories  (type  of  housing)  combine  the  degree  of 
centrality  of  the  region with  the  local  population  density. 

Still  close  to  the  field  of  regional  policy,  the  BFLR  is  currently  undertaking 
interesting  work  on  intra-urban  statistics. The aim  of this work  is to  provide 
data  on  population,  housing,  land  use,  and  mobility  in  relation  to  the 
different  categories of  urban area (city centre, suburbs, etc.). The  BFLR  has 
created a network  of 60 volunteer  towns and has developed  an initial list of 
statistical  series to  collect. After  a few years of working  together,  35  towns 
supply  all  the  required  data.  This  project  has  produced  comparable  data 
and  national  averages.  It  has  attracted  more  and  more  volunteer  towns. A 
similar  process  has  been  launched  in the  same field,  on  a  European  scale, 
by  Eurostat. 

Appendix 3 - Programme indicators for the 1994-99 period: 
a state of the art 

The  elements  presented  in this  appendix  were  drawn  from  an  inventory  of 
the  indicators  cited  most  frequently  in  Objective  1  and  Objective  5b 
programming documents for the period 1994-99. This inventory is limited to 
thirteen typical  interventions  covering  the sectors  with the  largest  budgets. 
The  list  drawn  up  on  this  basis  was  submitted  to  about  fifty  professionals 
who  are thoroughly  familiar  with  the Structural  Funds. They  were  asked  to 
select  those  indicators  considered  to  be  realistic  and  relevant.  This 
selection  affords  a view  of the state of the art on the  question  of  indicators 
during  the  1990s. 

Basic  infrastructure 

Indicator 

Unit and  definition 

Application 

Length of 
road/rail 

Length of 
journey 

Hours of 
transport 

Population 
served 

Km of  road/rail  built/improved, 
broken down  by type: road, 
motorway,  standard  railway, 
high-speed  railway, etc. 

Transport 
Road, motorway,  railway 

Hours of travelling between  two 
points. Shortening of time in % 

All types of  transport 

Total number  of  hours saved 
per year  by  users 

Number of  households  served 

All types  of  transport 

Water  and  environment 
Supply  networks for water/ 
gas/electricity 

Water & environment 
Water supply  network 

Water 
consumption 

m3 (thousands)  of  water 
consumed  per year  per 
household. 

Waste 
processing 
facilities 

Volume of 
waste 

Number of facilities  installed. 

Water & environment 
Waste/sewerage  treatment 

Tonnes of waste  treated  per year 
broken down into type of 
treatment  (incineration,  recycling) 

Water & environment 
Waste treatment 

Indicators  for monitoring  and evaluation 

Human  resources 

Indicator 

Unit and  definition 

Application 

Hours of 
training 
proposed 

Trainees 
having 
completed 
the  course 

Placement 
rate 

Duration  (hours) of  training 
financed  broken  down  by  type 
of  training  programme 

Training 

Number  of  trainees 
broken down  by  status  of  the 
trainee  in the job  market 
(unemployed,  long-term 
unemployed,  employees, 
employers) 

Proportion  of trainees who  found 
a job  broken  down  by type of 
beneficiary  (man/woman, age, 
etc.) 

Training /  R&D 

Training 

Training 

Rate of 
coverage 

Proportion  of  eligible  public 
(e.g. long-term  unemployed 
recruited  by the  programme) 

Appendixes 

Productive  environment 

Indicator 

Unit  and  definition 

Application 

Partnerships 
between 
research  and 
enterprise 

Surface area 
of  developed 
business 
areas 
Occupation 
rate of 
business 
areas 
Survival  rate 
of  businesses 

Investments 
generated 

Value added 
/  wages 
Export 
turnover 

Assisted 
economic 
units 

Operations 
to  promote 
quality 
Surface of 
protected 
sites 
Visitors to 
protected 
sites 
Jobs  created 

Number  of  businesses 
participating  in joint  research 
projects, in partnership with an 
R&D centre 

Hectares 
Broken down by type of  area 
(new development  or 
rehabilitation) 

Measures for  promoting 
research and  innovation 

Measures for  developing 
business  areas 

Surface  built  in percentage of 
building  land 

Measures for  developing 
business areas 

Number of  businesses  which 
remain active, as a share of all 
assisted  businesses  broken 
down  in terms of the period of 
observation  (2 years, 5 years) 

In euro, amount  of  investments 
generated  by the  assisted 
economic  units 

In euro, value added  generated 
by the assisted economic  units 
In euro, turnover  generated  by 
exports  in the assisted  economic 
units  broken down  by size (SME, 
other) and by sector of  activity) 

Measures for assisting the 
creation of  new  businesses 

All measures 

All measures 

Measures to  promote  exports 

Number  broken down by size 
(SMEs, others) 

All measures that  include 
assistance 

Number 

Measures to  promote quality 

Surface  in hectares 

Number  of visitors  per year 

Measures to  protect the rural 
environment 

Measures to  protect the rural 
environment 

Number  of jobs created owing to 
aid granted  to the assisted 
economic  units 

All measures that  include 
assistance 

Appendix  4 
Grid for assessing the quality  of a system of  indicators 

This appendix  proposes  a procedure for analysing the system  of  indicators 
of  a  programme  and  for  assessing  its  quality.  The  procedure  consists  of 
seven steps. 

Defining the  system to  be assessed  (1) 

Specify  the  document  to  be analysed. This  may be: 

•  a single  programming  document  (SDP) and  its  appendixes 

•  a data  base used  by the  programme  managers 

•  one or more  reports submitted  to the  Monitoring  Committee 

•  a report  published  by  an  observatory 

Dividing the  system  into  homogeneous  components  (2) 

A component  is homogeneous  if: 

•  it is possible to  apply the same output  indicator  to  all the  projects; 

•  all the  projects  have the same target  public. 

The homogeneous  components  are generally  measures or actions. 

For  each  homogeneous  component,  note  is taken  of  the  budget  allocated 
to  it and  its budgetary  weight  as a percentage. 

The  following  steps  from  3  to  6  are  taken  for  each  homogeneous 
component. 

Identifying  and  counting  indicators  (3) 

In  the  analysed  documents  all  the  information  that  can  be  qualified  as 
indicators is looked for. All quantitative information is likely to be an indicator. 

After  this  step,  the  ratio  of  "density  of  indicators"  is  calculated  for  each 
component  (number  of  indicators /  total  planned  expenditure,  in  hundreds 
of  millions  of  euro). This  ratio  is an  element  for  assessing  the  relevance  of 
the system. The density of indicators must be homogeneous. It may also be 
more important  for the  most  strategic  measures. 

Indicators for monitoring and evaluation 

How to  count the  indicators  of a  system ? 

•  The following conventions are adopted: 

•  an indicator broken down into parts is counted as a single indicator (e.g. table 
with the number of applications broken down per year and per target public) 
•  if an indicator is broken down a second time, it is counted twice (e.g. 2nd table 

of the number of applications broken down per year and per territory) 

•  a compound indicator is counted as a single indicator if its components are not 

quantified separately and as several indicators if they are. 

Classifying  the  indicators  by  category  (4) 

When an indicator  is identified, it should  be placed  into one of the  following 
four categories:  resources, outputs,  results, and  impacts, as defined  in this 
Volume. 

At the  end  of  this  step, the  percentage  of  components  which  have at  least 
one  result  or  impact  indicator  must  be counted, for the  entire  programme. 

Attributing  a  quality  score  to  each  indicator  (5) 

Each  indicator  is  given  a  score,  from  0  to  3,  for  the  applicable  quality 
reliability,  comparability, 
criteria:  availability, 
normativity,  and  meaning. 

freshness,  sensitivity; 

Multicriteria  scoring  for  each  component  (6) 

For  each  component,  the  average  scores  are calculated  for  each  criterion 
(availability,  freshness,  sensitivity,  reliability,  comparability,  normativity, 
meaning)  and  for  each  category  of  indicator  (resources,  outputs,  results, 
impacts). Then the average score per category  is calculated. The result  may 
be  represented  in the form  of the following  table: 

Appendixes 

A v e r a ge  scor 

e  of  t he  in 

d i c a t o rs  of  a  c o m p o n e nt 

Category 

Number 
of  indie. 

Average  score  per  criterion 
and  per  category 

Score  per 
category 

Resources 

Outputs 

Results 

Impacts 

1 

10 

3 

0 

crit.1 

c r i t2 

3,0 

3,0 

2,1 
/ 

3,0 

2,5 

1.1 
/ 

crit  7 

3,0 

2 

0,9 
/ 
Average 

3,0 

2,2 

1,2 

0,0 

1,6 

With this  procedure  a maximum  score can  be given to a component  which 
has  only  one  indicator  in  each  of  the  four  categories,  provided  that  these 
indicators  are  of  a  high  quality.  The  average  score  decreases  if  more 
indicators  are  added  and  if  they  are of  not  of  the  same  quality.  Thus, this 
scoring  procedure favours the quality of the system  and not the quantity  of 
the  indicators. 

Moreover, the average score declines  slightly  if there  is no result  or  impact 
indicator. 

Attributing  a  global  score to the  entire  system  (7) 

The global  score  of the  entire  system  is the  average weighted  score  of the 
indicators  of  all  the  components.  These  are  scores  obtained  in  step  6, 
weighted  by  the  budgetary  weight  of  the  corresponding  components  (see 
step 2). 

The  assessment  procedure  described  above  has  integrated,  in  its  very 
construction, 
the  global  assessment  criteria:  coverage,  balance, 
selectiveness  and  relevance (see Box 33). 

Appendix  5 -  List of examples 

Example  1:  New section of  motorway  connecting A and  Β 

Example 2: 

Improvement  of  a railway  line between A and  B, 

bringing  it  up to the high speed  range 

Example 3: 

Expansion  of  a regional airport 

Example 4: 

Improvement  of a container terminal  in a sea port 

Example 5:  Upgrading  of a railway  line for freight operations 

Example 6:  Building  an intermodal  (rail  /  road)  centre 

Example  7:  New electricity  plant  using fossil fuels 

Example 8:  Constructing  or  upgrading  an electric  power 

distribution  network 

Example 9:  Upgrading  a gas distribution  network 

Example  10:  Financing  of training for  new  trades 

for the  long-term  unemployed 

Example  11:  Skills  improvement  programme for young  people 

with few  qualifications 

Example  12:  Assistance to  businesses  for  adapting 
their employees'  skills 

Example  13:  Support  for the creation and  improvement 

of accommodation 

Example  14:  Support  for the creation of tourist  facilities 

(museums, amusement  parks, etc.) 

Example  15:  Funding for  language training 

Example  16:  Grants for aquaculture farms 

Example  17:  Subsidies for  laying up fishing vessels 

Example  18:  Modernisation  of a fishing  port 

Example  19:  Support  for  retraining  of fishermen 

Example 20:  Support for a science and technology  park for  SMEs 

53 

54 

55 

56 

57 

57 

63 

64 

65 

72 

73 

74 

80 

81 

82 

87 

88 

89 

90 

97 

Indicators for monitoring and evaluation 

Example 21:  Financial  and  informational  support  for  technology 

acquisition  by SMEs 

Example 22:  Support  for  post-graduate  research 

Example 23:  Financial support to  assist the setting up of  young 

farmers 

Example 24:  Supported  investment  in agricultural  product 

processing  plants 

Example 25:  Village renewal -  public  and  private  projects 

Example 26:  Support  for  establishing tourist  accommodation 

on farms 

Example 27. 

Establishment  or  regeneration  of community  centres 

Example 28 

Improvement  of water  supply  system 

Example 29. 

Construction  of  wastewater  treatment  plants 

Example 30. 

Improvement  of solid waste management  facilities 

98 

99 

105 

106 

106 

107 

112 

118 

119 

120 

Example 31 

Support  for  new SMEs  providing  environmental  goods 
& services 

121 

Example 32:  Construction  of  an inter-urban  light  railway  between A 

and  Β 

Example 33:  Deployment  of a broadband  Hybrid  Fibre Coax 

(HFC) network 

Example 34:  Upgrading  of an existing  narrowband  network 

with ADSL technology 

Example 35:  Setting  up of  a suburban telework  centre 

Example 36:  On-line  education  project  in professional training 

Example 37:  Support  for  entrepreneurs  creating  new  businesses 

122 

128 

129 

130 

131 

138 

Example 38: 

Informational  support  for SMEs to  promote  exports 

139 

Example 39:  Subsidised  infrastructure for setting  up of firms 

Example 40:  Support  for  projects  in regional  research centres 

Example 41:  Creation of  new organisations  providing  debt 

counselling  and local credit  schemes 

140 

141 

147 

Appendixes 

Example 42:  Renovation  of a rundown  urban area 

Example 43:  Support for  community  organisations 

Example 44:  Training  programme for  long-term  unemployed 

Example 45:  Support  for the construction  of a conference 

and exhibition  centre 

Example 46:  Direct  support  for  capital  investment  in enterprises 

Example 47:  Venture capital  scheme for  small  business 

development 

148 

149 

150 

161 

161 

162 

Example 48:  Return-to-job  programme for  long-term  unemployed 

168 

Example 49:  Assistance for the  setting  up of an equestrian 

event  centre 

Example 50:  Construction  of  a new  hospital 

Example 51:  Upgrading  a bus system  in neglected  urban areas 

169 

175 

176 

Index 

Index 

accountability  197 

actor  29 

addressee  or  recipient  30 

adverse  effect  115,  206 

aggregation  2 1,  197 

Agriculture  and  rural  development 

103 

availability  193 

Β 

balance  195 

baseline  26 

benchmark  group  205 

benchmarking  152,  206 

displacement  effect  34,167,169,171 

double  counting  188 

Economic  development  155 

effect  34 

effectiveness  35,  207 

efficiency  35,  205 

elementary  indicator  19 

ELIZE  data  base  192 

Employment  165 

endowment  26,  27,  5 1,  116 

Energy  infrastructure  61 

Environment  115 

Equal  opportunities  111 

best  correspondence  approach  192 

evaluation  indicator  37 

best  practice  206 

breakdown  20 

expert  panel  204 

externality  33 

comparability  194,  196 

Competitiveness  of  SMEs  135,  161 

compound  indicator  20 

confounding  factor  26,  34,  167 

context  indicator  24,  26,  184,  204 

coverage  195 

field  43 

Fisheries  85 

freshness  194 

deadweight  34,  167,  168,  169 

derived  indicator  20,  204 

descriptor  18 

direct  addressee  25,  29,  30,  32,  36, 

40,  206 

disaggregation  20 

disparity  26,  157 

generic  indicator  2 1,  186,  198 

global  impact  28,  29,  43,  184 

global  objective  24,  28,  155 

Gross  Domestic  Product  156 

gross  effect  34,  168 

Η 

harmonisation  189 

I 

immaterial  services  40 

impact  33 

indicator  of  structural  composition 

193 

indirect  addressee  29,  30,  40 

information  feedback  182,  189 

Information  society  125 

information  system  (GISCO)  193 

interpolation  method  192 

interpretation  203 

key  indicator  2 1,  198 

knock-on  effect  137 

logical  framework  28 

M 

mainstreaming  111,135,158,159 

management  by  results  37 

meaning  194 

monitoring  indicator  35 

N 

need  26, .34 

net  effect  34,  168 

normativity  194 

Indicators for monitoring and evaluation 

performance  35,  202,  207 

physical  outputs  39 

priority  43 

programme  funder  29 

programme  indicator  24,  184,  204 

qualitative  information  204 

Quality  of  life  173 

QUID  data  base  192 

REGIO  data  base  191 

relevance  34,  195 

reliability  194 

Research  and  Technological 

Development  93 

resource  28,  29,  31 

responsibility  37 

result  28,  29,  32 

sector  43 

selectivity  195 

sensitivity  194 

skimming-off  effect  206 

specific  impact  28,  29,  33 

specific  indicator  21 

spin-off  33 

operating  report  18,  182,  187,  189 

standard  indicator  197,  198 

operator  29,  188,  207 

output  28,  29,  32 

substitution  effect  34,  167,  169,  171 

sustainable  development  115 

Index 

τ 
temporary employment  52,  166 
theme  See field 

Tourism  77,  161,  169 
Training  69,  168 
Transport  infrastructure  49,  176 

υ 
unexpected impact  33 
Urban development  145 

" 
value added  156 

Acknowledgements 

The preparation of this Volume took  place over several years from  1996 to  1998. The 
preparatory  work  included the following  steps  : 

•  Several workshops  and meetings  allowed experts and  European  desks officers  to 

provide their  points  of view  on indicators  for  evaluation. 

•  A  survey  was conducted  of  several  institutions  which  have  implemented  systems 

of  indicators. 

•  A  questionnaire  survey  was  undertaken  on  the  subject  of  indicators  on  the 

occasion  of the second  MEANS conference  in Berlin. 

The  production  of  this  Volume  was  coordinated  by  Andrea  Mairate  (D.G.  XVI.G.2  : 
Coordination  of  Evaluation)  and  involved  several  Directorate-Generals  of  the 
European  Commission,  notably  D.G.  II, V, VI, XIV  and  XVI and  in  particular  Philippe 
Goybet,  Anne  Bucher,  Anastassios  Bougas,  Miguel  Benito  Alonso,  Sylvie  Finne, 
Olivier  Rouland  and Patrick  Salez. 

This  Volume  has  been  written  by  the  Centre  for  European  Evaluation  Expertise, 
notably  with  the  participation  of,  Cristina  Diaz  Gomez  and  Kevin  Williams.  It  was 
translated  into  English  by  Liz  Libbrecht. 

The  authors  wish  to  thank  all  those  who  contributed,  either  directly  or  indirectly,  to 
the  production  of this Volume, and  in particular  the following  experts: 

Transport  infrastructure: José  Viegas, Cesur,  Lisboa 

Energy  infrastructure:  Maurizio  di  Palma, Ecoter,  Rome 

Training:  Patrick  Werquin, Cereq,  Marseille 

Tourism:  Daniel  Poulenard, Parcourir,  Toulouse 

Agriculture  and  rural development:  Gunter Tissen, Fal,  Braunschweig 

Information  society:  Paolo  Manacorda,  Reseau, Rome 

Competitiveness  of  SMEs  and  enterprises  in  general:  Ivan  Turock,  University  of 
Glasgow 

Urban  development: Tony  Bovaird, Aston  University,  Birmingham 

Economic  development:  Henri Capron, Université Libre  de  Bnjxelles 

Quality  of  life:  Sanjiv  Lingayah  et  Alex  Macgillivray,  New  Economics  Foundation, 
London. 

European  Commission 

Evaluating socio-economic  programmes 
Selection and  use of  indicators for monitoring and  evaluation 

Luxembourg: Office for  Official  Publications  of the  European Communities,  1999 

1999 —  252  pp. —  14.8 χ 21  cm 

ISBN  92-828-6626-2 

Venta  ·  Salg  ·  Verkauf  ·  Πωλήσεις  ·  Sales  ·  Vente  ·  Vendita  ·  Verkoop  *  Venda  ·  Myynti  · Försäljning 

BELGIQUEÆELGIE 

OSTERREICH 

J«an D· Laiinoy 
Avenue du Roi 202.Koningslaan 202 
B­1190 BnjxellesHrussel 
Tél. (32­2) S38 43 08 
Fax (32­2) 538 08 41 
E­mail: («an de lannoyOmfoboard.be 
URL: http. 'www jean­de­lannoy be 

La librairie européenne/ 
De Europese Boekhandel 
Rue de la Loi 244/Wetstraat 244 
B­1040 Brux alles/Brussel 
Tel. (32-2)295 26 39 
Fax (32­2) 735 08 60 
E­mail. mailOlibeurop.be 
URL http/Zwww libeurop.be 

Moniteur belge/Belgi »c h Staatsblad 
Rue de Louvain 40­42/Leuvenseweg 40­42 
B­1000 Bruxelles'Brussel 
Tél. (32­2)552 22 11 
Fax (32­2) 511 01 84 

DANMARK 

J. H. Schultz Information A/S 
Herstedvang 10­12 
DK­2620 Albertsiund 
TU (45) 43 63 23 00 
Fax (45) 43 63 19 69 
E­mail 5chulU0schulU.dk 
URL: http://www.schuiu dk 

DEUTSCHLAND 

Bundesanzeiger Verlag GmbH 
V e ft neb sa b le ι lung 
Amsterdamer Straße 192 
D­50735 Koin 
Tel. (49­221)97 66 80 
Fax (49­221) 97 66 B2 78 
E­Mail: vertrieb O bundesanzeiger de 
URL: http://www bundosanzeiger.de 

ΕΛΛΛΛΑ GREECE 

G. C. Eletiheroudakls SA 
International Bookstore 
Panepistimiou 17 
GR­10564 Athina 
Tel (30­1) 331 41 8071/2/3/4/5 
Fax (30­1) 323 98 21 
E­mail eiebooksOnetor.gr 

ESPANA 

Boletín Oficial del Estado 
Trafalgar. 27 
E­2B071 Madrid 
Tel (34)915 38 21 11 (Libros). 
913 84 1 7 l 5 ( S u s c n p) 
Fax (34) 915 38 21 21 (Libros). 
913 B4 17 14(Suscnp) 

E­mail. dieniesO com.boe.es 
URL: http//www boe es 
Mundi Prensa Libros, SA 
Castella, 37 
E­28001 Madrid 
Tel (34)914 36 37 00 
Fax (34) 915 75 39 98 
E­mail litweriaOmundiprensa es 
URL: http//www mundi prensa com 

FRANCE 

Journal officiel 
Service des publications des CE 
26. rue De san 
F­75727 Paris Cedex 15 
Tel (33) 140 58 77 31 
Fax (33) 140 58 77 00 
URL: http //www (oumal­officiel gouv ft 

IRELAND 

Government Supplies Agency 
Publications Section 
4­5 Harcourt Road 
Dublin 2 
Tel (353­1)661 31 11 
Fax (353­1)475 27 60 

ITALIA 

Licosa SpA 
Via Duca di Calabria. 1/1 
Casella postate 552 
1­50125 Firerne 
Tel (39)055 64 83 1 
Fax (391055 64 12 57 
Ε­mari iKXisaOtTbcc n 
URL: http ­www ftbcc itflicosa 

LUXEMBOURG 

Mani sehe Verlags­ und 
Unlversltåtsbuchhandlung GmbH 
Kohlmarkt 16 
A­1014 Wien 
Tel (43­1)53 1611 00 
Fax (43­1) 53 16 11 67 
E­Mail bestellen Omani co at 
URL. htlp //www mani, a L'index htm 

PORTUGAL 

Distribuidora de Livros Bertrand Ld.' 
Grupo Benrand, SA 
Rua das Tenas dos Vaies, 4­A 
Apartado 60037 
P­2700 Amadora 
Tel. (351-1)495 90 50 
Fax (351­1)496 02 55 

Imprensa Nacional­Casa da Moeda, EP 
Rua Marquês Sa da Bandeira. 16­A 
P­1050LtsDoa Codex 
Tel (351­1)35303 99 
Fax (351­1) 353 02 94 
E­mail: dehncmemail.telepac.pt 
URL: httpJ/www inem pt 

SUOMI FINLAND 

Akateeminen Kirjakauppa/ 
Akademiska Bokhandeln 
Keskuskatu 1/Centralgatan 1 
PL/PB 12B 
FIN­00101 HelsinkLrHelsrngfors 
Ρ ./tfn (358­9) 121 44 18 
F/lax (356­9)121 44 35 
Sah k o po s 11. akali la us β aka 1 ee minen o 
URL: http//www akateeminen com 

SVERIGE 

B T J AB 
Traktorvägen 11 
S­221 82 Lund 
T!n(4S­46) 1800 00 
Fax (46­46) 30 79 47 
E­post bt|eu­pubObt|.s< 
URL: http //www.btj.se 

UNITED KINGDOM 

The Stationery Office Ltd 
International Sales Agency 
51 Nine Elms Lano 
London SWa 5DR 
Tel (44­171)873 90 90 
Fax (44­171)873 84 63 
E­mail: ipa encuinesStheso.co uk 
URL: htlp//www ihe­stationery­officeci 

ISLAND 

Bokabud Larusar Blónda I 
Skulavordustig. 2 
IS­101 Reykjavik 
Tel (354)551 56 50 
Fax (354) 552 55 60 

NORGE 

Swets Norge AS 
Oston joveicn 18 
Boks 6512 Etierslad 
H-0606OSIO 
Tel (47-22)97 45 00 
Fax (47-22) 97 45 45 

SCHWEIZ SUISSE'SVIZZERA 

Euro Info Center Schweiz 
c/o OSEC 
Stampfenbachstrafle θ 5 
PF 492 
CH­8035 Zürich 
Tel (41­1)365 5315 
Fax (41­1)365 54 11 
E­mail eesOosecch 
URL: hnp//wvfwosec ch/etes 

BALGARUA 

Europress Euromedia Ltd 
59. blvd Vitosha 
BG­1000 Sofia 
Tel (359­2) 980 37 66 
Fax (359­2) 980 42 30 
E­mail: MilenaOmbox en bg 

CESKAREPUBLIKA 

USIS 
NlS­prodojna 
Havelkova 22 
CZ­l3OO0Praha3 
Tet (420­2)24 23 14 66 
Fax (420­2) 24 23 11 14 
E­mail nkpospOdec ms cz 
URL http7/usiscr cz 

s du livre SARL 

CYPRUS 

HRVATSKA 

Medlatrade Ltd 
Pavia Haua ι 
HR­10000 Zagreb 
Tel (385­1)481 94 π 
Fax (385­1)481 94 11 

MAGYAROHSZAG 

Euro Into Service 
Europa Haz 
Margitsziget 
PO Box 475 
H­1396 Budapest 62 
Tel (36­1)350 80 25 
Fax (36­11350 90 32 
E­mail: euroinfoOmail matav hu 
URL: http Λν/ww. eu rointo.hu/index. htm 

ROY International 

41, Mishmar Hayarden Streel 
PO Box 13056 
61130 Tel Aviv 
Tel ¡972­3)649 94 69 
Fax (972­3) 648 60 39 
E­mail. royilOnetvision nel il 
URL: http://www.royint co il 

Sub­agent for the Palestinian Authonty: 

Index Information Services 

PO Box 19502 
Jerusalem 
Tol (972­2)627 16 34 
Fax (972­2) 627 12 19 

MALTA 

Miller Distributors Ltd 
Malta Inte mal lonal Airport 
PO Box 25 
Luqa LOA 05 
Tel.(356)66 44 88 
Fax (356) 67 67 99 
E­mail: gwirthOusa net 

POLSKA 

Ara Polona 
Krakowskw Przedmiescie 7 
Skr poeztowa 1001 
PL­00­950 Warszawa 
Tel. (48­22)826 12 01 
Fax (48­22) 826 62 40 
E­mail ars_potebevy.hsn com.pl 

ROMANIA 

Euromedia 
Sir G­ral Bermelo! Nr 4 
RO­70749 Bucuresti 
Tel (40­1)31544 03 
Fax (40­1 ) 314 22 S6 

RQSSIYA 

CCEC 
60­lelrya Oktyabrya Av. 9 
117312 Moscow 
Tel. (7­095) 135 52 27 
Fax (7­095) 135 52 27 

SLOVAKIA 

Centrum VTI SR 
Nam Slobody. 19 
SK­81223 Bratislava 
Tel (421­7)54 41 83 64 
Fax (421­7) 54 41 83 64 
E­mail. europOtbbl s\\Y stuba ι 
URL http /www sitk.stuba sk 

SLOVENUA 

Gospodarskl Vestnlk 
Duna|ska cesta 5 
S LO­1000 Ljubljana 
Tel (386)613 09 16 40 
Fax (386) 613 09 16 45 
E­mail eurooíígve .tr k :.· 
URL: http ,Vwww gvestnik.si 

TURKIYE 

Du nya Infot el AS 
100. Yil Mahallossi 34440 
TR­80050 Bagcilar­lslanbul 
Tel. (90­212)629 46 89 
Fax (90­212)629 46 27 
E­mail infoici O duny a­gaze te c 

AUSTRALIA 

Hunter Publications 
PO Box 404 
3067 Abbotsford, Victoria 
Tel (61­3)94 17 53 61 
Fax (61­3) 94 19 71 54 
Email | pria vie s β oz email com au 

CANADA 

Les éditions La Liberté Inc. 

Tel (1­418)658 37 63 
Fax (1­800)567 54 49 
E­mail hberloOmediom pc ca 

Renouf Publishing Co. Ltd 
5369 Chemm Canotek Road Unit 1 
KU 9J3 Ottawa. Ontano 
Tel (1­613)745 26 65 
Fax (1­613) 745 76 60 
E­mail: order.deptOrerKiufbooks.com 
URL: h rtp//www renouf book s com 

EGYPT 

The Middle East Observer 
41 Sherif Streel 

Tel (20­2)392 6919 
Fax (20­2) 393 97 32 
E­mail: matoudaOmeobservar.com eg 
URL: http//www meob server com eg 

PSI­Japan 

Asahi Sanbancfio Plaza »206 
7­1 Sanbancho. Chryoda­ku 
Tokyo 102 
Tel. (81­3)32 34 69 21 
Fax (B1­3)32 34 69 15 
E­mail: booksOpsi­japan cojp 
URL: ht!p://www psi­|apan com 

MALAYSIA 

EBIC Malaysia 

Level 7, Wisma Hong Leong 
18 Jalan Perak 
50450 Kuala Lumpur 
Tel (60­3)262 62 98 
Fax (60­3$ 262 61 98 
E­mail ebic­kiemol.net.my 

Mundi Prensa Mexico, SA do CV 

Río Panuco No 141 
Colonia Cuauhlomoc 
MX­06500 Mexico, DF 
Toi. (52-5)533 56 58 
Fax (52­5) 514 67 99 
E­mail: 101545 2361 Ocompuservo com 

PHILIPPINES 

EBIC Philippines 

191h Floor, PS Bank Towor 
Son. Gil J. Puyat Ave. cor Tindalo St. 
Makali City 
Metro Manilla 
Tei (63­2) 759 66 80 
Fax (63­2) 759 66 90 
E­mail ocepcom β globe com ph 
URL http//www eccp com 

EBIC Sri Lank.] 

Trans Asia Hotel 
115 Sir chi It amp a I am 
A Gardiner Mawolfia 
Colombo 2 
Tel. (94­1)074 71 50 78 
Fax (94­1) 44 87 79 
E­mail: ebicslOitmin com 

EBIC Thailand 

29 Van issa Building, 8 th Floor 
Soi Chidlom 
Ploenchil 
10330 Bangkok 
Tel (66-21655 06 27 
Fax (66­2) 655 06 28 
E­mail 0biCbkkOksc15 thcom 
URL: hnp/www.ebicbkk org 

UNITED STATES OF AMERICA 

Berna π Associates 

4611 ­F Assembly Dnve 
Lanham MD20706 
Tel (1­800) 274 44 47 (loll free telephor 
Fax (1­800) 865 34 50 (loti free tax) 
E­mail queryObeman com 
URL. http//www bernan com 

Bitte wenden Sie sich an ein Büro Ihrer 
Wahl/ Please contact the satas office 
ol your choice/ Veuillez vous adresser 
au bureau de vente de voire choix 

2, rue Morc»or 
L­2985 Luxembourg 
Tel (352)29 29­42455 
Fax (352) 29 29­42758 
E­mail inlomfoOopocececbe 
URL: http//euf­op eu.inf 

5, rue Rartfeisen 
L­2411 Luxembourg 
Tel (352)4010 20 
Fax [352)49 06 61 
E­mad marlOmdilu 
URL: hrtp://www mol. lu 

NEDERLAND 

SDU Servicecentrum Ulloevers 
Christoffei Pianti]nsiraat 2 
Postbus 20014 
2500 EA Den Haag 
Tel (31­70)376 9880 
Fax (31­70) 378 97 83 
E­mail eduOsdunl 
URL: http//Www sdu.nl 

Cyprus Chamber of Commerce and Industry 
PO Box 1455 
CY­1509 Nicosia 
Tel (357-2)66 95 00 
Fax (357­2) 66 10 44 
E­mail de met rap O ceci org cy 

EESTI . 

INDIA 

Eesti Ka uba ndus­Toostu skoda (Estonian 
Chamber of Commerce and Industry) 
Toom­Kooli 17 
EE­0001 Tallinn 
Tel (372)646 02 44 
Fax (372) 646 02 45 
E­mail einfo 0 koda ee 
URL httpi/www.kodaee 

EBIC India 
3rd Floor, Y. B Chavan Centre 
Gen J Bnosale Marg 
400 021 Mumbai 
Tel (91­22)282 60 64 
Fax (91­22) 285 45 64 
E­mail ebicOgtasbmOI vsnl.net in 
URL. http / www ebicindia com 

The  MEANS  collection: 
"Evaluating  socio-economic  programmes" 

This  six-volume  set  gives  wider  public  access  to  the  lessons  of  the  MEANS 
programme which the  European  Commission  introduced  with a view to  impro­
ving  and  promoting  evaluation  methods.  Its  publication  meets  a  growing 
need  for  reliable  benchmarks  to  deal  with the  problems  of  evaluation  encoun­
tered when implementing  public  measures  in general and Structural Fund ope­
rations  in  particular. 

In  this  context,  the  set  constitutes  a  unique  methodological  guide  that  can 
solve  the  technical  and  organisational  problems  arising  in  the  course  of 
evaluation  work  and  for  which  the  existing  literature  provides  little  help. 

The  collection  is  aimed  not  only  at  professional  evaluators,  for  whom  it  will 
provide  a valuable  handbook,  but  also at  non-specialists,  in particular  manag­
ing  authorities  and  structural  programme  Monitoring  Committees. 

Evaluation  design  and  management 

Each  volume  is  designed  to  be  independent  of  the  others,  yet  they  are  also 
complementary.  Together, they  deal with  all aspects  of the evaluation of socio­
economic  programmes: 
I. 
II.  Selection  and  use of  indicators  for  monitoring  and  evaluation 
III.  Principal  evaluation  techniques  and  tools 
IV.  Technical  solutions  for  evaluation  within  a partnership  framework 
V.  Transversal  evaluation  of  impacts  on  the  environment,  employment  and 

other  intervention  priorities 

VI.  Glossary  of  300  concepts  and  technical  terms 

OFFICE FOR OFFICIAL PUBLICATIONS 
OF THE EUROPEAN COMMUNITIES 

_-2985 Luxembourg 

EC Structural  Funds 

Evaluating socio-economic programmes 

Principal evaluation techniques and tools 

1 

MEANS 
Collection 

Volume 3 

II  an Commission 

EC Structural Funds 

Evaluating socio-economic programmes 

Principal  evaluation 
techniques  and  tools 

Volume 3 

European Commission 

MEANS  is a programme  of  the European  Commission. 
It is aimed  at  improving  Methods  of Evaluating  Stmctural  Policies. 
The MEANS  Programme  and  Collection  was supervised  by 
Phillipe  Goybet  and  Miguel  Angel  Benito  Alonso, 
Unit  for  the Coordination  of  Evaluation  in the Directorate  for  Coordination 
and  Evaluation  of  Operations  of  DG XVI 
Coordination-evaluation@fmb.  dg 16. cec. be 

The realisation  of  the Programme  was entrusted  to  the 
Centre  for  European  Evaluation  Expertise (C3E) 
under  the  direction  of  Eric Monnier  and Jacques  Toulemonde. 
Responsibility  for  the  opinions  expressed  in this 
book  lies solely  with  the authors  of  the text: 
C3E  -  13 bis  Place Jules  Ferry  -  F  -  69006  Lyon  -  France  -  c3e@c3e.fr 

A great deal of additional information on the European Union is available 
on the Internet. 
It can be accessed through the Europa server (http://europa.eu.int) and 
on the Inforegio Website (http://inforegio.cec.eu.int). 

Cataloguing data can be found at the end of this  publication. 

Luxembourg: Office for Official Publications of the European 
Communities,  1999 

ISBN 92-828-6626-2 

©  European Communities, 1999 
Reproduction  is authorized, except for commercial  purposes,  provided 
the source is  acknowledged. 

PRINTED  ON WHITE CHLORINE-FREE  PAPER 

Printed in  Italy 

Foreword 

is  directly 

Developing  good  evaluation  practices 
has  become  a  priority  area  for 
the 
relevant 
Commission  and 
to 
the  effective  use  of  Community 
resources. 
Indeed, the  sound  financial  management 
principles  which  the  Commission  seeks 
to  promote  via  its  SEM2000  (Sound  and 
Efficient  Management)  Programme  rely, 
specifically,  on  the  regular  evaluation  of 
the  Community  activities. 
The  Directorates  General  responsible  for 
structural  policy  have considerable  expe­
rience  in this area.  Since  1988 they  have 
in  the  process  of 
integral  part  of  each  stage 
made  evaluation  an 
Community  Structural  intervention.  Thus,  the  programme  development 
stage  includes  a  prospective  evaluation;  the  implementation  stage  a 
strengthening  of the monitoring systems  and, at the end of the  intervention, 
evaluation  establishes  what  has  been  achieved  in  terms  of  programme 
impacts. 

However,  the  capacity  of  evaluation  to  contribute  to  the  effectiveness  and 
transparency  of  structural  interventions  rests  ultimately  on  the  integrity  of 
the  methods  adopted  to  verify  the  credibility  of  results. 
This  is why,  four  years  ago,  on  the  initiative  for  the  Directorate  General  for 
Regional  Policy  and  Cohesion  (DG  XVI),  the  MEANS  programme  was 
initiated  as a response  to  the demand  for  developing  evaluation  methodo­
include  practices  combining  both  general  scientific 
logy.  This  also 
requirements  and  the  specific  requirements  of  Community  interventions 
whilst,  at the  same  time,  remaining  accessible  to  a  large  audience. 
One  of  the  more  tangible  results  of  the  MEANS  programme  has  been  the 
publication  of this  six volume  set  by a group of  independent  experts,  focu­
sing  on  the  principal  elements  of  socio-economic  programme  evaluation. 
A  short  summary  of this work  can  be found  at the  beginning  of the  present 
volume. This collection  covers the various  methodological  approaches  and 
innovations  that  have  taken  place  in  the  context  of  structural  funds  eva­
luation.  It  will  enable  those  responsible  for  evaluation  to  develop  their 
approach  in  a  flexible  and  independent  manner  and  for  the  various 

programme  partners  to  exercise  greater  control  over  the  evaluation  pro­
cess.  Attending  the  conferences  of  evaluation  specialists  and  "clients" 
convened  in  Brussels  (1995),  Berlin  (1996)  and  Seville  (1998),  I  gained 
some  impression  of the  interest  that  new  evaluation  techniques  can gene­
rate.  I would  stress  that  the  development  of  evaluation  needs  to  progress 
in  tandem  with  efforts  to  incorporate  evaluation  findings  in  the  decision 
making  process. This  implies  evaluation  as  a  means  of  establishing  whe­
ther  policies  are equitable,  effective  and  relevant  to the  identified  needs. 
In  satisfying  these  requirements,  evaluation  should  be  seen  as  a  valuable 
instrument  affording  both  Member  State  and  Community  authorities  the 
necessary  guarantees  that  Community  funds  are  being  deployed  in  the 
best  interests  of the  EU citizen. This  is a cost worth  paying when the  goals 
of  Economic  and  Social  Cohesion  and  the  European  Union  enlargement 
are at  stake. 

Finally,  I wish  to  extend  my  thanks  to  the  members  of  the  Committee  of 
Independent  Experts  established  at  the  launch  of  the  programme.  Their 
expertise and  standing  have contributed  greatly to the quality  and relevan­
ce  of  the  MEANS  programme.  I therefore  thank:  E.  Chelimsky  (USA),  ex-
President  of  the  "American  Evaluation  Association";  K.  Kennedy  (IRL), 
Director  of  the  "Economic  and  Social  Research  Institute"  (ESRI);  J.  R. 
Cuadrado  Roura  (SP),  President  of  the  "European  Regional  Science 
Association"; C. Seibel (F), Director of the  "Statistiques démographiques  et 
sociales de l'Institut  Nat onal de la Statistique et des Etudes  Economiques" 
(INSEE);  L.  Senn  (I),  Director  of  the  Department  of  Economie  Regional  of 
the  University  Bocconi;  E. Stern  (U.K.), Director  of the  unit  of evaluation  of 
the  Tavistock  Institute;  A.  Sorber  (t)  (NL),  Head  of  the  "Policy  Analysis 
Department  of the  Ministry of  Finance"; L. I. Strom  (SV), Statens  Institut  för 
Regionalforsking  (SIR);  L.  Tsoukalis  (GR.),  professor  at  the  University  of 
Athens;  H. Wollmann  (D), professor  at  the  University  of  Humboldt  -  Berlin 
and founder  member  of the  "European  Evaluation  Society". 

E.  Landaburu 

The  MEANS  Collection: 

"Evaluating socio-economic programmes" 

of  European  Union 

The  evaluation 
structural 
interventions  has  entered  into  a  phase  of  maturity, 
attested by the extension of work carried out  by both the 
Member States  and the Commission. Regulations  based 
on  decisions  by  the  Council  have  largely  contributed  to 
the  development  of  evaluation  practice  and  have 
stimulated  methodological  research. 

initiated  by  the  Commission 

Over  the  past  ten  years, the  Commission  has  striven  to 
promote  and  improve  methods  for  evaluating  Structural 
Funds. The main  methodological  conclusions  have been 
debated  and  disseminated  as  part  of  the  MEANS 
in  1991. 
programme 
Achievements  to  date  provide  enough  material  for  a 
complete  and  accessible  publication  intended  for  a 
broad public: a six volume Collection covering all aspects 
of the evaluation  of  socio-economic  programmes. These 
Volumes  are  written  with  a  perspective  that  is  broader 
than  the  current  administrative  framework,  for  this  is 
likely to evolve  more rapidly than professional  standards. 

information  on 

In general, the Collection is aimed at a readership of non-
specialists  seeking  essential 
the 
evaluation of socio-economic  development  programmes. 
These  programmes  have  multiple  objectives,  a 
partnership  dimension  and  a  multi-sectoral  content  -  all 
characteristics  which call for suitable evaluation  methods 
and  solutions  not  yet  provided  by  the 
international 
technical literature. The complete Collection consists of 6 
volumes  designed  to  be  complementary  and to  provide 
essential 
for 
professionals  wanting  to  independently  develop  their 
own evaluation  practices.  Each volume can, however,  be 
read and  understood  without  having to  refer to the  other 
volumes  in the Collection. 

recommendations 

information  and 

Principal evaluation techniques and tools 

The  first  Volume  in  the  MEANS  Collection  entitled 
"Evaluation  design  and  management"  serves  as  an 
introduction  both  to the  Collection  and  to the  evaluation 
of  socio-economic  programmes.  It  enables  readers  to 
understand  the  implications  of  evaluation  and  to  know 
what they  can and cannot  expect  from an evaluation  at a 
given  time  in  a  particular  context.  It  covers  all  the 
elements  needed to steer  an evaluation  project from A to 
Z:  sharing  of  responsibilities,  defining  of  the  aims  and 
method  of  an  evaluation,  establishing  a  schedule  and 
budget, assessing the quality of the work  performed and, 
finally,  managing the  use of  conclusions. 

The second Volume focuses on the  "Selection  and use of 
indicators  for  monitoring  and  evaluation".  It  deals  with 
the  subject  of 
indicators  used  for  monitoring  and 
evaluation.  It  shows  how  indicators  can  be  used  to 
support  policy  decisions,  improve  management  and 
measure  the  overall  performance  of  programmes.  The 
volume  proposes  a  methodological  framework  and  a 
Dractical  guide for the use of  indicators  in these  different 
situations. 

The  third  Volume  presents  a  selection  of  "Principal 
evaluation  techniques  and  tools"  applicable  to  the 
evaluation  of  socio-economic  programmes.  These  tools 
are taken from  relevant  scientific  disciplines:  economics, 
sociology,  management, geography, etc.  Each one of the 
23 tools  is described  in the  form  of  an  information  sheet 
featuring  a  general  description,  the  principles  of  its 
implementation, a list of its strengths and weaknesses for 
the  evaluation  of  Structural  Funds,  an  example  of  its 
application  and  a list of  references. 

A number of methods adapted to the evaluation of socio­
economic  programmes  are the  object  of  a more  detailed 
description 
in  the  fourth  Volume  entitled  "Technical 
solutions  for  evaluation  within  a partnership  framework". 
Several  pilot  evaluations  were  implemented  within  the 
MEANS  programme to design and test the following  four 
innovations,  the  application  of  which  is  described  in 

The MEANS Collection 

detail  in  this  Volume:  impact  mapping,  cross-impact 
matrix, score cards  and  multi-criteria  analysis. 

The  fifth  Volume  deals  with  "Transversal  evaluation  of 
impacts  on  the  environment,  employment  and  other 
intervention  priorities"  of  the  European  Union  and 
Member  States.  Among  the  questions  addressed  are 
impacts  in  terms  of  equal  opportunities  between  men 
and  women  and  the  competitiveness  of  SMEs.  The 
Volume provides  the elements  required for an analysis of 
those  impacts  which  are  not  necessarily  part  of  the 
explicit  objectives  of all the actions  evaluated. 

Finally,  the  last  Volume  is  a  "Glossary  of  concepts  and 
technical  terms".  The  compilation  of  the  Collection  has 
necessitated  the  rigorous  standardisation  and  definition 
of  the  terms  used.  Each  of  the  first  five  volumes  has  an 
index  of terms which, along  with  hundreds of  definitions, 
are brought  together  in the  glossary. 

Contents 

Introduction 9 

I Presentation of evaluation tools 15 

■ Tool, function, method 17 

■ Properties of evaluation tools 19 

■ Advantages and risks of technical complexity 22 

■ Steps and functions of an evaluation 24 

■ Evaluation situations 28 

II Tools for the overall evaluation of a programme 33 

Examples of the evaluation of multi­sectoral programmes 35 

Tools for structuring an evaluation 39 
H SWOT Analysis (Strengths­Weaknesses­Opportunities­Threats) 41 
■ Concept mapping of impacts 47 
■ Colour vote 55 

Tools for observing changes in the field 59 
■ Individual interview 61 
■ Focus Group 67 
■ Case study 73 

Tools for analysing data 81 
■ Geographical Information Systems 83 
■ Shift­Share analysis 89 
■ Input­Output Model 95 
■ Macro­economic model 103 

Tools for making a judgement 115 
■ Expert panels 117 
■ Multicriteria analysis 123 

III Tools for in­depth treatment of an evaluative question 129 

Examples of evaluations with in­depth treatment of a question 131 

Tools for structuring an evaluation 135 
■ Logical framework 137 

METAPLAN® 

145 

Principal evaluation techniques and tools 

Tools for observing changes in the field 151 
■ Questionnaire survey 153 
■ Ethnographic observation 161 

Tools for analysing data 167 
■ Factor analysis 169 
■ Delphi Survey 175 
■ Comparison groups 179 
■ Regression analysis 187 

Tools for making a judgement 193 
■ Cost-Effectiveness Analysis 195 
■ Benchmarking 201 
■ Cost-benefit analysis 207 

IV Recommendations for selecting and assembling tools 217 

Guide for identifying the most appropriate tools 219 

Recommendations for assembling several tools 225 

Conclusion 231 

Index 233 

Introduction 

A  professional  who  wants  to  know  what  the  evaluation  toolbox  contains, 
has  to  consult  numerous  texts  and  manuals  devoted  to  particular 
techniques  in  the  economic  and  social  sciences.  Fortunately,  some 
reference  documents  present  a  range  of  tools  for  evaluation  purposes. 
Among  the  classics  are  Rossi  &  Freeman  (1993),  as  well  as  the  Program 
Evaluation  Kit  collection1.  These  books  feature  "technical  information 
sheets"  illustrated  by  means  of  examples  and  all  are  organised  according 
to  each  author's  own  typology.  The  most  frequent  typological  distinctions 
separate  tools  in  terms  of  their  nature,  i.e.  quantitative  or  qualitative, 
deductive  or  inductive, etc. A particularly  interesting  typology  is  presented 
by C.  Pollitt  (Box 1). 

Is there any need for a new book  on evaluation tools and, if so, what  might 
its contribution  be? The main  need that this Volume has to fulfil  is to  make 
existing knowledge readily available to readers working  in the field of socio­
economic  programmes.  In  fact,  most  documents  on  evaluation  tools  are 
illustrated  with  examples  of  social  programmes  or  development  projects. 
The  present  document  provides  a  link  between  general  methodological 
works  and the context  of  European Structural  Funds. 

is 

It is secondly necessary to adapt available knowledge to specific  difficulties 
in the evaluation of socio-economic  programmes.  Most texts  on evaluation 
tools refer to simple programmes, aimed at a well-defined target  group and 
at producing  precise change within that group. However, a socio-economic 
development  programme 
typically  multi-sectoral,  with  multiple 
objectives.  Its  purpose  is to  produce  economic  and social  transformations 
on  the  scale  of  an  entire  territory.  To  achieve  this  goal,  an  integrated 
strategy  is adopted, oriented  towards  different  publics  (firms, workers,  the 
unemployed, communities, etc.). Its global impact  is produced  by means of 
a  combination  of  multiple  specific  impacts  on  various  targets.  In  this 
context,  evaluation  encounters  complex  technical  problems  for  which  the 
existing 
first 
methodological  guide to record and present a number  of relevant  solutions. 

few  solutions.  This  book 

literature  provides 

the 

is 

1  Rossi RH, Freeman  H.E. (1993) Evaluation: a Systematic  Approach (5th edition) 

Newbury Park CA: Sage. 
Program Evaluation Kit, Center for the Study of Evaluation, University of California. 
Newbury Park CA: Sage 

Principal evaluation techniques and tools 

Box  1-  Four  approaches  to  evaluation 

Experimental  approaches  treat  phenomena  of  causality  without  analysing  them 
directly. The  effects  of  programmes  are demonstrated  by  comparing  a treatment 
group  with  a control  group.  Evaluations  carried  out  in this  framework  are  typical 
of  American  practice  in  the  1960s  and  70s.  These  evaluations  are  long, 
cumbersome, and are not always conclusive. They are often criticised, sometimes 
severely,  by  their  commissioners  who  want  operational  conclusions  that  can  be 
used  rapidly  for  improving  programmes. 

Economic  approaches  are based on an  individualistic  view  of society,  inspired  by 
welfare economics. From this point of view, the value of a public  action is the sum 
of  the  benefits  it  provides  to  individuals.  All  individuals  are  supposed  to  have  a 
system  of  references  and  it  is  assumed  that  these  systems  of  reference  can  be 
aggregated.  These  hypotheses  are  used  to  give  an  objective  character  to 
techniques.  Despite  their  elegance, 
evaluations  carried  out  by  economic 
economic 
techniques  have  been  strongly  criticised,  in  both  practical  and 
theoretical terms. 

Naturalistic  or  pluralistic  approaches  are  based  on  the  idea that  the  political  and 
social  world  is  a  collective  construction.  This  construction  results  from  the 
interaction  of  differing  social  groups  that  have  their  own  interpretations  of  the 
same  phenomena  and  issues.  Consequently  public  programmes  are  seen  as 
temporary  compromises  between  groups  of  actors.  By  means  of  appropriate 
techniques,  based to a large extent  on group work, evaluation  acts as a mediator 
between the different  points of view. It is a tool for promoting conciliation  between 
the  stakeholders,  and  its  conclusions  are  all  the  more  robust  when  they  are  a 
product  of  consensus. 

Pragmatic  approaches  adopt  simplified  views  of  the  processes  that  they  have  to 
describe.  Theoretical  references  are  less  pure  and  more  eclectic.  Evaluation 
objectives are of an essentially managerial rather than scientific  nature. Evaluations 
carried  out  in  this  spirit,  using  techniques  inspired  by  management  approaches, 
are  oriented  more  towards  the  implementation  and  improvement  of  efficiency. 
Their  cognitive  dimension  is  less important  than  in other  forms  of  evaluation. 

Some  of  the  technical  suggestions  in  this  Volume  would  not  exist  without 

the  numerous  studies  and  experiments  carried  out  by 

the  European 

Commission.  To  adapt  available  knowledge  to  the  difficult  field  of  socio­

economic  programmes,  substantial  methodological 

investments  were 

required,  including  the  precise  definition  of  the  basic  concepts  (e.g.  tools, 

methods),  the  definition  of  functions  likely  to  be  fulfilled  by  the  tools,  the 

construction  of  new 

tools 

(e.g.  specialised  macro-economic  models, 

impact  mapping)  or  the  proposal  of  new  ways  of  using  classical  tools. 

Introduction 

These investments have benefited the entire professional evaluation 
community. This is the third and last reason for the existence of this manual. 

The manual contains a detailed presentation of twenty­three evaluation 
tools, in the form of technical information sheets. The choice of tools is a 
compromise between the most frequently used tools and others that, 
although rarer, have significant potential for the evaluation of socio­
economic programmes. Since all compromises are to some extent arbitrary, 
certain frequently used tools are only mentioned in passing, without having 
technical information sheets devoted to them (e.g. variance analysis). 

Box 2 ­ For a 

rapid pre­selection of 

appropriate tools 

S Θ 

High 
technical 
complexity 

Medium Low 
technical technical 
complexity complexity 

O  O 

Retrospective Prospective 

use use 

Φ 

Low 
cost 

(è) @% 

V^~"»// νΛ*·»*JJ) 
Medium High 
cost cost 

Θ 

0 

Stabilised 
use 

Innovative 
use 

O 

Overall 
evaluation 

A 

In­depth 
evaluation 

Each information sheet provides data on the functions of the tool, the 
conditions of its use, and the main steps in its implementation. Possibilities 
for application in the framework of Structural Fund evaluations are 
suggested, and the strengths and weaknesses of the tool in practice are 
highlighted. An example taken from a Structural Fund evaluation is 
described, if the tool has been used in that context. Each sheet also 
contains a commented bibliography of books and articles that refer to the 
tool. The information sheets are accompanied by pictograms that enable 
readers to rapidly select the tools applicable to a given situation. The 
meaning of these pictograms is given in Box 2. 

Principal evaluation techniques and tools 

This  Volume  is divided  into the following  four  main  sections: 

•  Presentation  of  evaluation  tools 

•  Tools for  the overall  evaluation  of  a programme  (12 tools) 

•  Tools for  more  in-depth  treatment  of  an evaluative question  (11 tools) 

•  Recommendations  for  selecting  and assembling  the  tools 

I 
Presentation of evaluation tools 

In  this  Volume  the  terms  "tools"  and  "techniques"  are  taken  to  be 
synonymous  and  applicable 
to  standardised  procedures  which  any 
evaluator can learn to use and  apply  in  carrying  out  her/his  missions. 

The  evaluation  "toolbox"  contains  over  23  techniques,  presented  in  this 
Volume. The evaluator's  art  therefore  involves  the choice  of an  appropriate 
tool and the rejection  of inappropriate  ones, depending  on the function  they 
are meant  to fulfil in a particular  situation. 

Presentation of evaluation tools 

Tool, function,  method 

Evaluation  practitioners  use various  words  to  describe  their  instruments  or 

procedures:  technique,  tool,  approach,  procedure,  method,  design,  etc. It 

is  essential  to  agree  on  certain  definitions  before  reading  this  Volume. The 

following  example  will  be used  to  establish  these  definitions. 

Box  3-  Example  of  a  tool(s)-based  evaluation 

In  1989 the British  National  Audit  Office  evaluated  eight  regional  development 
programmes  in  Wales.  The evaluation  calculated  the  cost  per job  created,  in 
other  words,  it was based on cost-effectiveness  analysis, a standard  evaluation 
tool. To evaluate the eight programmes, the NAO adopted an approach in several 
stages: 

1 -  Selection of a single evaluation criterion: net jobs created. 

2 -  Construction  of an indicator for measuring  net jobs,  in terms of the following 
definition:  net jobs  = gross  jobs  -  deadweight  effects  -  displacement  effects 
+ supplier  effects. 

3 -  Selection of a sample of about  300 assisted  businesses. 

4 -  Questionnaire  survey  by 
displacement  coefficients. 

telephone,  estimation  of  deadweight  and 

5 -  Estimation  of  total  jobs  created  by  extrapolation  from  the  coefficients 

obtained from all jobs  reported  by all the assisted  businesses. 

6 -  Estimation  of  supplier  effects  by  extrapolation  from  studies  of  similar 

programmes. 

7 -  Recapitulation  of the public  funds  mobilised  for the programmes,  and cost-

effectiveness  analysis  in terms  of cost  per job created. 

8 -  Comparison  with  the  cost  per  job  created  in  the  framework  of  similar 

programmes  - here, a regional development  programme in Scotland. 

For further  information  on this  evaluation, see: S. Roberts  and Ch. Pollitt,  (1994) 
Audit or Evaluation?  a National  Audit  Office  VFM Study,  Public  Administration 

In this  example,  there  are at  least  t wo tools  commonly  used  in  evaluation: 

•  the questionnaire  survey  (4th step) 

•  cost-effectiveness  analysis  (7th step) 

It  should  also  be  noted  that  the  evaluation  work  involved  far  more  than 

simply  the  use  of  t wo  tools.  At  each  of  the  eight  stages,  a  decision  was 

made  on  how  to  carry  out  the  work.  Each  decision  depended  on  the 

'specific  context  of  this  evaluation.  The  NAO thus  constructed  an  ad  hoc 

evaluation  method  for fulfilling  its  mission. 

Principal evaluation techniques and tools 

This example will  be used to  define three of the  basic  terms  in this Volume: 
"tool",  "method" and  "function". 

A  tool  is used  to  carry  out  standard  treatment  and  may  be  applied  to  any 
evaluation  requiring  such treatment.  For example: 

•  a questionnaire  survey  is  used  to  gather  the  opinions  of  a group  on  one 

or several  points  defined  in advance; 

•  a cost-effectiveness  analysis  calculates the  public  funds  spent  to  obtain 
a given effect; the result  is presented  in a form, which allows  programmes 
and  projects  to  be compared. 

A technique  is so similar to a tool that  in this Volume they  are often  used as 
synonyms.  However,  a  tool  can  be  considered  to  be  an  object  (e.g.  a 
shovel)  and a technique the way  in which this object  is used (e.g. the mani­
pulation  of the shovel). 

A  method  is  an  ad  hoc  procedure,  specially  constructed  for  a  given 
evaluation.  It  may  include  one  or  more  tools,  or  none.  In  its  evaluation  of 
regional  development  programmes  in  Wales,  the  NAO  constructed  a 
method  adapted  to  the questions  raised  (Box  3). By  contrast,  for  the  prior 
appraisal of programmes  proposed by the Member States, the  Commission 
requests  appraisals  that  are carried  out  within  a few  weeks.  In general, the 
methods  applied to  these evaluations  comprise  no tools. 

Thus  defined,  the  term  "method"  is  similar  to  "design".  In  the  American 
literature,  "evaluation  design"  is  applied  to  the  way  in  which  a  given 
evaluation  is conducted.  However, the term  is often  applied  to  the  analysis 
of  causality,  that  is to  say,  to  only  one  of  the  functions  that  an  evaluation 
method  must fulfil. The terms  "approach"  and  "procedure"  are also  similar. 
They  apply  to  a  given  method  and  to  the  spirit  in  which  this  method  has 
been constructed. 

The method  constructed  for carrying out  a particular  evaluation  can  always 
be  broken down  into  several  steps.  Each of  these  steps  consists  of  a task 
and  fulfils  an  elementary  function.  The  same  basic  elementary  functions 
can be found in most evaluations. A function  may be fulfilled with or without 
a tool.  For example: 

•  in the  NAO example  (Box 3) the data collection function  is carried  out  by 

means of  a questionnaire  survey; 

•  in  certain  evaluations  of  the  Commission,  the  estimation  of  effects  is 

carried out  by  means  of a macro-economic  model. 

Presentation of evaluation  tools 

In this Volume, methodology  is defined as the  "science" of the  construction 
of  methods. The aim of this  Volume  is to  promote  the development  of  this 
science. In reality, however,  professional  practice  is still closer to  expertise 
than to a science. The word  methodology  is therefore  somewhat  ambitious 
at this stage. 

Properties  of  evaluation  tools 

A  tool  is  used  to  process  raw  materials  (inputs)  to  transform  them  into 
products  (outputs).  Inputs  and  outputs  are  information,  data,  facts, 
opinions,  judgements,  etc.  Inputs  and  outputs  can  be  different  in  nature. 
For  example,  a  questionnaire  survey  collects  opinions  and  produces 
quantitative data about these opinions. All types of  information feed  into an 
expert  panel to produce a judgement, etc. The tool can be defined  because 
the  mode  of  processing  is  stable,  that  is  to  say,  inputs  have  a  standard 
form,  outputs  are  always  of  the  same  nature,  and  transformations  occur 
according  to  the  same  principles.  Yet  a  tool  always  has  a  degree  of 
flexibility  so  that  it  can  be  adapted  to  use  in  specific  situations.  These 
"adjustments"  are limited  so that the tool  remains  identifiable. 

Box 4 - The road worker's shovel and  cost-effectiveness  analysis 

The  road  worker's  shovel  is  used  to  fulfil  a standardised  function,  consisting  of 
moving  a  loose  material  from  one  place  to  another,  for  example,  by  filling  a 
wheelbarrow  with  earth.  In  this  example,  the  "inputs"  are  the  user's  physical 
strength  and the  earth to  be  moved, and the  "output"  is the earth  that  has  been 
moved in the wheelbarrow. A priori, there are not many types of shovel or different 
ways of using them. One could say that this tool performs a stable transformation. 
The  main  variants  are the  size  of  the  shovel, the  distance  of  the  movement  and 
the nature of the material to  be moved (i.e. how  loose it is). The case of the shovel 
may  be transposed, term for term, to the field  of evaluation, for  example to  cost-
effectiveness  analysis. 

The  inputs  in  cost-effectiveness  analysis  are  twofold.  It  is  necessary  first  to 
measure  the  impact  obtained  (in  a  single  dimension,  e.g.  net  jobs  created)  and 
then the amount spent. The output  is simple; it is the cost  ratio per unit of  impact. 
The  transformation  is  stable;  it  consists  of  a  simple  division.  The  degree  of 
freedom concerns, above all, the way of taking cost  Into account:  budgetary  cost 
for  all the public  partners,  budgetary  cost  for  the only  public  actor  that  commis­
sions the evaluation, opportunity  cost of public funds, cost  limited to the  duration 
of the programme  or cost  extended  to  all public  support,  etc. 

Principal evaluation techniques and tools 

A tool may be used freely by anyone who needs it. Most of the tools used 
in evaluation are free of charge but some have to be bought. For example, 
a tool used for conducting meetings, called METAPLAN®, presented in 
this Volume, is a registered trademark. A tool has instructions for use, 
described in one or more manuals. Training courses on tools and their use 
are offered for beginners. Quality standards ensure that tools are used 
professionally. 

To summarise, a tool can be recognised from the following characteristics, 
featured in Box 5, it can be identified by a name, it is used to carry out a 
process with standard inputs and outputs, it can be adapted but in a very 
limited number of ways, it has a manual and quality standards. All these 
properties make the tool transferable from one evaluation to another and 
from one professional to another. In particular, its use is public and not 
limited to its inventor. 

Box 5 ­ Features of an evaluation tool 

Standards 

inputs  O 

Name 

O O O 

V '/ 

Adjustments 

■=>¡ 

Standard 
outputs 

Manual 

Quality 
standard 

This diagram corresponds perfectly to some of the tools presented in this 
Volume, as the example of regression analysis developed below shows. In 
this example, the basic characteristics of a tool can be recognised, i.e.: 

• standard inputs (quantitative data on apparent effects, on participation in 
the programme and on the other explanatory variables since these data 
are available for a large number of individuals); 

• some adjustments (number and nature of variables, nature of relations, 

thresholds of validity); 

Presentation of evaluation  tools 

•  standard  outputs  (distribution  of  apparent  or  gross  effects  between  the 
net  effects  attributable  to  the  programme  and  the  effects  of  the  other 
explanatory  variables). 

Moreover,  the  tool  is  identified  by  means  of  a  name  (regression  analysis), 
irrespective  of  the  variants  corresponding  to  the  different  adjustments 
(linear  regression, logistic  regression,  etc.).  Finally, the  tool  is described  in 
numerous  manuals and  in all  European  languages. 

Box 6 -  Regression  analysis  as an evaluation tool 

Data for 
numerous 
individuals : 

Apparent  effect 

Participation  in 
the  programme 

0 

Other  explanatory 
variables 

Regression 
analysis 

0 

%  of  apparent 
effects  which  are 
attributable  to 
the  programme 

Several  of  the  23  information  sheets  presented  in  this  Volume  have  only 
some  of  the  characteristics  of  a  real  "tool".  For  example,  some  have  not 
been fully  standardised  (e.g. SWOT analysis), others  do  not  have  a  proper 
manual (e.g. colour vote, METAPLAN) and others have no stabilised  quality 
standards  (e.g. multicriteria  analysis). 

Principal evaluation techniques and tools 

Apart  from  those  tools  developed  specifically  for  evaluation  (e.g.  cost 
benefit  analysis,  multicriteria  analysis),  most  tools  are  borrowed  from  the 
main  scientific  disciplines.  The  questionnaire  survey 
from 
sociology, regression analysis was created by statisticians, the  input-output 
model  comes  from  economics,  the  expert  panel  draws  largely  upon  legal 
information  system  and  ethnographic 
practices,  and  the  geographic 
observation  emerged  from their  corresponding  sciences. 

is  taken 

When  a technique  is  "borrowed"  from  a discipline for  use as an  evaluation 
tool, it is often adapted and simplified. From a purely scientific point of view, 
there  are  always  regrets  about  the  loss  of  the  "canonical"  form  of  the 
original  technique.  However,  evaluation  is  above  all  a  pragmatic  activity. 
The  technical  information  sheets  presented  in  this  Volume  should  not  be 
judged with the academic  rigour  of their  original disciplines,  but  in terms of 
their  usefulness  for  evaluation  practitioners. 

Advantages  and  risks  of  technical  complexity 

The use of a tool has numerous advantages in terms of guaranteeing  results, 
evaluation  content,  comparability  of  results, and quality  and cost  control. 

It also  provides  a guarantee  as to the form  of  results, on  condition that  the 
use  of  the  tool  complies  with  the  established  procedures.  Thus,  a 
questionnaire  survey  conducted  properly  (e.g. confidentially,  by  telephone) 
can serve to  accurately  estimate  net  effects. 

The  choice  of  certain  tools  makes  it  necessary  to  structure  the  entire 
evaluation.  For  example,  cost-effectiveness  analysis  means  that  the 
evaluation  will  favour  a  principal  effect  (e.g. employment).  The  tool  will,  in 
this  case,  offer  the  commissioner  a  guarantee  of  this  main  effect  being 
prioritised,  if that  is what  is desired. 

Use  of  a  known  tool  facilitates  the  estimation  of  the  time  and  budget 
required  for  an  evaluation  because  practitioners  have  references  for 
grounding their  estimations.  Moreover, the  fact  that  an evaluation  is  based 
on standardised  tools  facilitates  control  of the conditions  of  its execution. 

Several evaluations based on the same  tools  produce results that are easier 
to  compare.  In the example  in  Box  3, it was  possible to  compare the  costs 
per  job  created  in  several  regions  because  all  the  evaluations  had  used 
cost-effectiveness  analysis. 

Presentation of evaluation tools 

In short, a more technical  evaluation  is often simpler to commission, easier 
to control, more credible  for  readers of the report  and  more  predictable  for 
the  commissioner.  All  these  advantages  obviously  make  evaluation  tools 
highly  attractive,  but they are no without  certain  risks. 

The risks  of  "tool(s)-based"  evaluation  can  be summed  up  in the  following 
two  sentences: 

•  "With  a  hammer  in  hand,  I see  nails  everywhere"  -  risk  of  choosing  the 

wrong  tool 

•  "Because  I have  a hammer,  I can  build the  entire  house"  -  risks  of  over­

estimating  the use of  a tool. 

A tool  must always  be adapted  to  its context  of  use and to the functions  it 
is  to  fulfil.  The  risk  is  that  someone  with  a  hammer  might  try  to  use  it  to 
hammer  in a screw. In an example studied  in the framework  of the  MEANS 
programme,  the  evaluation  commissioner  and  team  decided  to  use 
regression analysis. After  a few weeks, they  realised that the available  data 
were  insufficient  and that  they  were  unable to  apply  the chosen tool.  They 
were forced to quickly  reorganise the evaluation  work. To know  whether  an 
evaluation  tool  is appropriate  in the context,  it  is necessary to  examine  the 
object  to  be  evaluated, the  available  data,  the  evaluation  budget,  and  the 
deadline  for  presentation  of  the  results.  Choosing  a tool  without  taking  all 
these  factors  into  account  could  have  serious  consequences:  absence  of 
conclusions  or  irrelevant  conclusions,  deadlines  that  are  not  met,  conclu­
sions that  are not  credible, etc. 

The weaknesses  of the tools  used  must  also  be  recognised  and accepted. 
It is better to carry out a less technical evaluation  by specifying the  limits of 
the work, than to  base conclusions on the  use of sophisticated tools  which 
were not  skilfully  used. 

Since  technical  complexity  is tempting,  there  is  often  a tendency  to  over­
value  tools  used  in  evaluation.  Thus,  the  NAO  evaluation  (Box  3)  was 
presented  as  an  exercise  in  "value  for  money",  that  is,  cost-effectiveness 
analysis.  The  entire  evaluation  was  given  the  name  of  one  of  the  tools 
implemented  and  this  tool  became  emblematic.  This  phenomenon  has 
become  widespread  and  applies 
to  numerous  evaluations  which, 
depending on the case, may be called cost-benefit  evaluation, econometric 
evaluation, quality/price  evaluation, etc. 

Equating  an  evaluation  to  its  emblematic  tool  is  always  a  mistake.  An 
evaluation tool is used to fulfil a function and, most of the time, one function 

Principal evaluation techniques and tools 

only.  Favouring  one function  amounts  to  neglecting  all the  other  functions 
required by evaluation, and thus undermining the overall quality of the work. 

Steps  and  functions  of  an  evaluation 

In general, a tool fulfils the function for which it was designed, and does not 
fulfil the others. An example of  a specific  tool  is a macro-economic  model, 
which  can  fulfil  no  other  function  than  that  of  estimating  impacts  on  the 
scale of a country  or region. 

Yet tools do exist which are capable of fulfilling several evaluation functions. 
These  are  generic  tools.  It will  be  seen  in this  Volume  that  the  case  study 
technique  can  be  used  for  collecting  data  but  also  for  other  functions. 
Similarly,  an  expert  panel  can  be  set  up  for  making  judgements  and  for 
estimating  impacts. 

All  evaluations  have  several  steps  with  which  one  or  more  functions  are 
associated. In the example of the evaluation  carried out  by the  NAO, it  can 
be seen that there were at  least five functions. 

Box 7 - Several steps  in an  evaluation 

(Evaluation of  economic  development  programmes  in Wales) 

Choosing  the  effects  to  be 
evaluated  (employment) 

Defining  an observation  tool 
(indicator  of  net jobs  created) 

Collecting  data 
(300 telephone  interviews) 

Estimating  effects 

(cost-effectiveness  analysis) 

ι 
\  1 

1 

ι 

STRUCTURING 

OBSERVING 

ANALYSING 

JUDGING 

By  analysing  a  large  number  of  evaluations  in  the  same  way,  the  most 
common  functions  can  be  identified.  These  are  presented  below  in  eight 
basic steps (Box 8), grouped together  into four main stages: structuring  the 

Presentation of evaluation tools 

evaluation, observing changes in the field, analysing data, and judging the 

programme. 

Box 8 ­ Standard steps of an evaluation 

Structuring 

; Clarifying and grading the effects 
; to _be_e_va[ua_ted,_defi_njng_ enterra . 

I Choosing observation 
l_ instruments ___.. 

Observing 

; Defining the observation field %~», 
r 

Analysing 

; Collecting data 

I Comparing data 

Judging 

J Estimating effects ,«·»' 
| Judging in terms of the """■ 
' different criteria 

ι Rirmulating a synthetic judgement 

The diamond­shaped diagram is borrowed from Scriven M. (1980) 
The Logic of Evaluation, Inverness CA: Edgepress 

The structuring steps normally occur before the evaluation work as such 
(before the terms of reference are drawn up). Their place in the present 
Volume may therefore be questioned. However, the evaluation team 
frequently has to revert to these steps at the start of its work, for example 
for explaining the judgement criteria in more detail. It will also be seen that 
tools exist which may facilitate structuring work. It is therefore logical for 
this Volume to cover evaluation as a whole, including the very first steps. 

Clarifying and grading the effects to be evaluated, defining criteria 

The first step consists of listing the questions to which the evaluation must 
provide answers. For example: Are the taxpayers getting value for their 
money? What are the most effective parts of the programme? Is the pro­
gramme justified? Does the programme have negative unexpected effects? 
Each question can be subdivided into the effects to be evaluated, e.g. job 
creation, investments induced, market share gained, increase in 
employability of a category of the population or increase in the 
attractiveness of a region. Each question also refers to a judgement 
criterion, e.g. is the effect sought and/or obtained relevant, consistent with 
the objectives, efficient, lasting and fair? 

Principal evaluation techniques and tools 

The  tools  applicable  in  this  step  are  used  to  list  questions,  clarify  them, 
grade them and select those for which the evaluation  must provide answers 
(see  the  information  sheets  relating  to  the  following  tools: 
logical 
framework,  impact  mapping, SWOT analysis, METAPLAN®,  colour vote). 

Defining  observation  instruments 

This  second  step  consists  of  the  translation  of  evaluation  questions  into 
indicators  or descriptors  observable  in the field.  In the  example  of  regional 
development  programmes  in Wales (Box 3), the question was as follows: "Is 
the  employment  effect  of  the  programmes  sufficient  compared  to  the 
money  spent?"  It  was  translated  in the  form  of  an  efficiency  indicator,  as 
follows:  "cost  to the national  budget  of a net job created". 

This  Volume  proposes  no  tools  for  this  step.  The  reader  is  referred  to 
Volume  2  devoted  to  indicators,  and  to  Volume  4 for  more  information  on 
the construction  and use of  descriptors. 

Defining  the field  of  observation 

This  third  step  consists  of  defining  a  field  of  observation  that  is  limited 
enough  not to  generate  excessive  costs, large enough to  produce  credible 
information, and contrasted  enough to  reveal the effects  to  be evaluated. It 
is therefore  a compromise  involving  choices  such  as: Should  all  categories 
of  addressees  be  observed?  Should  one  or  more  populations  which  are 
equivalent  but  which  have  not  participated  in  the  programme  also  be 
observed?  How can cases be selected to obtain the fullest picture  possible 
of  the  programme?  Should  cases  of  best  practice  be  selected  in  order  to 
understand the  key factors of  success? 

The  field  of  observation,  that  is  to  say,  the  definition  of  "groups"  to  be 
observed  (businesses,  trainees,  inhabitants,  farmers,  etc.),  should  not  be 
confused  with  the  field  of  evaluation  (definition  of  the  intervention  to  be 
evaluated). 

Collecting  data 

The fourth  step  consists  of  collecting  primary  data  in the  field.  Numerous 
survey  tools  exist  for  this  purpose.  They  are  borrowed  from  various 
disciplines in the social sciences that collect  data: sociology, political scien­
ce, ethnography,  etc. 

Presentation of evaluation  tools 

This step is essential if the evaluation cannot rely entirely on secondary data 
such as the evaluation of  previous  research, statistical  data and  monitoring 
data. 

The tools  applicable  to  this  step  allow  the  collection  of facts  and  opinions 
and  to  reconstitute  them  in  a  qualitative  or  quantitative  form  (see  the 
information  sheets  relating to the following tools: individual  interview,  focus 
group, questionnaire  survey,  ethnographic  observation). 

Comparing  observations 

The fifth step consists  of the presentation of  primary and/or secondary  data 
in  such  a  way  as  to  suggest  conclusions  and  comparisons  that  can  be 
made  intuitively  or  inductively. 

If  the  example  of  the  evaluation  of  a  training  course  for  the  long-term 
unemployed  is  taken,  the  comparison  of  observations  will  consist  of 
breaking  down  the  rate of  return  to work  in terms  of  occupational  sectors, 
to see whether there are significant differences and to better understand the 
factors  of  success. 

The  basic  tools  used  for  this  function  are  graphs  and  comparative  tables. 
This  Volume  also  proposes  several  more  sophisticated  tools  (see  the 
information  sheets  relating to the  following  tools: geographical  information 
systems, factor  analysis). 

Estimating  effects 

The  sixth  step  is  the  one  in  which  the  net  effects  of  the  programme  are 
estimated. This step  requires an analysis  of  causality  which  almost  always 
follows a deductive  approach, i.e. an approach  based  on the verification  of 
hypotheses of cause and effect. The technical  literature on evaluation  often 
favours  quantitative  analysis  tools  used  to  fulfil  this  function  (e.g. 
comparative  analysis, regression  analysis). 

to  qualitative  and  quantitative  analysis 

In this  respect,  this  Volume  differs  from  the  existing  literature.  It  attributes 
tools  by 
the  same  value 
demonstrating  the strengths  and weaknesses  of  each  one.  Furthermore,  it 
does not over-value the analysis of causality, often considered  as being the 
single  role  of  evaluation.  The  main  assumption  is  that  the  functions  of 
observation  and  judgement  are  at  least  as  important  as  the  function  of 
estimating  net  effects. 

Principal evaluation techniques and tools 

Tools  applicable  to  this  step  allow  the  net  effects  of  programmes  to  be 
estimated,  both  in  the  past  and  the  future,  and  at  both  the  micro  and 
macro-economic  level  (see the  information  sheets  relating  to  the  following 
tools:  shift-share  analysis,  input-output  models,  macro-economic  models, 
Delphi survey, comparison  groups,  regression  analysis). 

Judging  effects  in terms  of  each  criterion 

The evaluation must  be able to judge whether the effects that the programme 
has  produced  or  will  produce  are sufficient  or  insufficient.  In the  example of 
regional development  programmes  in Wales (Box 3), the evaluation team had 
to  judge job  creation  in terms  of  the  efficiency  criterion. The tool  used  was 
cost-efficiency  analysis.  It  was  thus  possible  to  conclude  that  programmes 
were more satisfactory  when the cost  per job created was low. 

For the  evaluation  to  be complete,  it was  also  necessary  to  be able to  say 
whether the cost  per job created was  sufficiently  low. The standard  chosen 
was  the  average  cost  per  net  jobs  created  by  comparable  programmes  in 
Scotland.  By  comparing  Wales  and  Scotland  it  was  possible  to  conclude 
that the  evaluated  programmes  were sufficiently  efficient. 

This  Volume  presents  several  tools  which  can  be  used  in  the  judgement 
step  (see  the  information  sheets  relating  to  the  following  tools:  cost-
effectiveness  analysis,  benchmarking). 

Formulating  a synthetic  judgement 

The eighth and last step in an evaluation consists of formulating  conclusions 
by making a final synthesis of the partial judgements on the different criteria. 
This  function  is  necessary  when  several  criteria  have  been  selected.  To 
make  a  synthetic  judgement  of  a  programme,  it  is  necessary  to  combine, 
weight  or grade the different  effects that the evaluation  has assessed. In the 
framework  of  socio-economic  programmes,  this  function  is  particularly 
difficult  since  there  are  always  several  partners  who  do  not  attribute  the 
same value to the  criterion. There are several tools  which  totally  or  partially 
fulfil this function  (see the  information  sheets  relating to  the following  tools: 
expert  panels, multicriteria  analysis, cost-benefit  analysis). 

Evaluation  situations 

A  tool  is  designed  to  fulfil  a  given  function.  There  is  therefore  a  strong 
connection  between  tool  and  function.  However,  several  tools  are  often 

Presentation of evaluation tools 

able  to  fulfil  the  same  function  (see  Box  9). When  several  evaluation  tools 
are available, the  most  suitable tool for the situation  has to  be selected. 

Box 9-  Significance  of the  situation for the  choice  of a tool 

The  shovel  is  a tool  of  which  the  function  is  to  move  a  loose  material  from  one 
place to  another. To fill a 50-litre wheelbarrow, an ordinary  shovel  is used. To fill a 
25-tonne truck,  it would  definitely  be better  to  use a mechanical shovel. 
In  both  examples,  the  function  of  the  tool  is  the  same,  but  the  situation  is  very 
different. The tool  must  be adapted to  both its function  and the situation  in which 
it  is to  be used. 

For example,  in a situation  of  massive  public  intervention  on the scale  of  a 
large territory, the function  of  estimating  effects  will  be fulfilled  by  a macro-
economic  model.  Thus,  the  HERMIN  model  was  constructed  and 
successfully  used  to  estimate  the  macro-economic  impacts  of  Structural 
Funds  in  Ireland, Portugal  and Spain.  In a different  situation, for  example  if 
the  amounts  involved  are  small  compared  to  the  economy  of  the  territory 
under  consideration, or  if the territory  is small, the  macro-economic  model 
is not  suited to the estimation  of  effects. 

As the example of the  HERMIN  model suggests, one of the first  differences 
is that  of the scale of effects to be evaluated, which  may be micro (a small 
group  of  potential  addressees)  or  macro  (a large  region  or  a country).  The 
evaluation  of  an  impact  on  the  national  economy  requires  the  use  of 
specific  tools,  particularly  macro-economic  models, and disqualifies  others 
such as case studies. 

Another  important  difference  is  that  of  partnership.  In  an  evaluation 
involving several decision-makers, the choice of effects to be evaluated and 
of evaluation criteria is of  major  importance and has to respect the  differing 
points  of  view.  Evaluations  involving  a  multiplicity  of  decision-makers 
require  the  use  of  open  tools,  such  as  group-work 
techniques  or 
multicriteria  analysis.  On  the  other  hand,  they  are  ill  suited  to  closed 
techniques  such cost-effectiveness  analysis. 

is  that  of  the  use  of  evaluation:  formative  or 
The  third  difference 
recapitulative.  A  case  study,  which  makes  it  easier  to  understand  and  to 
learn quickly  from an experience,  is a suitable tool for formative evaluation. 
By  contrast,  this  tool  produces  conclusions  that  are difficult  to  synthesise 
for  decision-makers  and  the  public.  As  a  result,  it  is  less  appropriate  for 
recapitulative evaluation. 

Principal evaluation techniques and tools 

Box  10 - The three  phases  of  evaluation 

In the framework of  European Structural  Fund programmes for the period 1994-
99, evaluation is carried out in three stages. 
Ex ante evaluation focuses on the planned programme. It concerns the coherence 
and relevance of the project, and the realism of the expected effects. 
Intermediate  evaluations  focus  on  the  effects  of  the  very  first  outputs  of  the 
programme.  They  are  used  by  decision-makers  at  all  levels.  They  are 
complementary  to  monitoring and are intended to reorient the programme mid­
way. 
Ex post evaluations are intended primarily to report on the effects of programmes. 

The  fourth  difference  is that  of  the  schedule  (see  Box  10). The  evaluation 
cannot  use  the  same  tools  to  evaluate  those  effects  which  have  already 
been  obtained  and those  which  are still to  be  produced. 

The  fifth  difference  is one  of  the  most  important  for  the  choice  of  tools.  It 
concerns  the  complexity  of  the  programme.  The  majority  of  socio­
economic  programmes  are  multi-sectoral,  that  is to  say,  they  intervene  in 
several  different  forms,  are  aimed  at  several  different  publics  and  produce 
multiple  kinds  of  impact.  However,  most  evaluation  tools  have  traditionally 
been  used  in  straightforward  situations  (a  single  sector  of  intervention,  a 
single  public,  and  a predominant  expected  impact). 

The  latter  difference  is  very  important  for  the  choice  of  tools.  It  is  also 
peculiar  to  socio-economic  programmes,  which  is why  it  is a focal  point  in 
this Volume. The presentation  of  tools  is therefore  set out to  correspond  to 
two  main  situations: 

•  Global  evaluation  of  an  entire  programme.  In  this  situation  a  regional 
multi-sectoral  programme  has  to  be  evaluated  globally,  in  its  entire 
complexity. The evaluation  of  impacts  at a macro  level is required. 

•  More  in-depth  consideration  of  an  evaluative  question. This  situation  is 
similar  to  the  evaluation  of  a  simple  programme,  described  in  most 
evaluation  manuals. 

In general, the  greater  the  complexity,  the  more  attention  must  be  paid  to 
the functions of clarification and structuring, the higher the cost of  empirical 
observation, and the more difficult the analysis of causality. Normal  practice 
consists of  breaking  down the programme  into categories of  homogeneous 
interventions.  Each  intervention  is  evaluated  in  terms  of  its  relevance, 
effectiveness  or  efficiency.  The  evaluation  then  makes  a  synthesis  at  the 

Presentation of evaluation  tools 

programme  level.  The  difficulty  of  this  approach  is  that  the  different 
components of the programme (measures, actions, etc.) must  be evaluated 
in terms  of criteria common to the entire programme.  If all the  criteria  were 
specific the synthesis would be impossible. A variant  of this situation  is one 
in which  macro-economic  effects  on  a regional  or countrywide  scale  must 
be estimated. This  requires  first  an  estimation  of  the  micro  effects  of  each 
component  of the  programme  according to  a common analytical  grid, then 
the  introduction  of  these  effects  into  all  the  macro-economic  mechanisms 
in order, finally, to deduce the global effect which is not equal to the sum of 
all the micro-economic  effects. 

Various  issues  may  need to  be considered  in greater  depth: 

•  Evaluations focused  on an elementary  component  of the programme, for 
example  a priority  axe, a measure  or an action.  In this case, the  outputs 
are  homogeneous  and  the  addressees  may  easily  be  identified  and 
questioned. The  collection  and  analysis  of  primary  data  are  easy,  as  is 
judgement  by comparison  with  similar  interventions  in other  regions. 

•  Evaluations focused  on a specific  public  such  as SMEs or the  long-term 
unemployed.  This  case  is  very  similar  to  the  preceding  one.  The  only 
difference  is that several elementary  components  of the programme  may 
affect  the  same  public. 

•  Evaluations  focused  on  a  priority  theme.  In  this  case,  the  relevance  or 
impact  of  all  the  components  of  the  programme  on  a  particular  theme, 
such  as  employment,  the  environment  or  equal  opportunities,  is 
evaluated. The  difficulty  in this  type  of  situation  is that  the  interventions 
are  multi-sectoral,  as  are  the  results  and  impact  mechanisms.  This 
situation  therefore  involves  almost  as  many  difficulties  as  the  overall 
evaluation  of  a complex  programme.  It  is considered  in Volume 5 of  this 
Collection. 

•  Evaluations  focused  on a specific  territory.  In this  case, the  relevance  or 
effectiveness  of  all  the  measures  of  the  programme  on  a  limited  part  of 
the  area  covered  by  the  programme  (e.g. problem  neighbourhoods,  the 
most  underprivileged  areas)  is evaluated. This  situation  presents  exactly 
the  same difficulties  as the  overall evaluation  of  a complex  programme. 

Throughout  this Volume, the term  "in-depth evaluation" will refer to only the 
first  two  sub-divisions  that  produce  simple  situations,  in  which  traditional 
evaluation  tools are fully  appropriate. 

Tools for the overall  evaluation 
of a programme 

Texts concerning  evaluation  tools usually refer to simple programmes  aimed 
at a well-defined  public  and  at producing  limited  change  within  that  public. 
By  contrast,  a socio-economic  development  programme  is  typically  multi-
sectoral  and  multi-objective. 
Its  purpose  is  to  create  economic  or  social 
transformations  on  a  scale  that  covers  an  entire  territory.  To achieve  this 
goal,  an  integrated  strategy  is  implemented,  aimed  at  various  publics 
(businesses, employees,  the jobless,  communities,  etc.). It has to produce  a 
global  impact  by  means  of  the  right  combination  of  multiple  specific  im 
pacts  on  its  different 
In  this  context,  evaluation  encounters 
technical  problems  for  which  the  existing  literature  provides  few  solutions. 
This section  presents  twelve  tools  adapted  to  this  situation.  Many  of  them 
are innovative  or are employed  in an innovative  way. 

targets. 

Examples of the evaluation of multi-sectoral programmes 

The following  three  examples  describe  situations  in which  the  programme 
to  be  evaluated  was  clearly  multi-sectoral.  The  evaluations  used  one  or 
more tools very  successfully. 

In the first  example, the  programme  provided  substantial  funds to  a  region 
that  was  lagging  in  its  development.  The  programme  financed  basic 
infrastructure  as  well  as  assistance  to  businesses.  The  objectives  were, 
however, numerous and not always precise. One of the questions  answered 
by  the  team  responsible  for  intermediate  evaluation  was  the following:  "Is 
the regional  economic  context  developing  in such  a way that the goals will 
be  achieved?"  The  evaluation  made  it  possible  to  ascertain  whether 
regional economic  problems were  improving  or worsening. 

Box  11 -  Intermediate  evaluation of a regional programme  in Spain 

In  1996,  the  Generalität  de  Valencia  launched  an  intermediate  evaluation  of  a 
regional  socio-economic  development  programme  (Objective  1  Operational 
Programme)  co-financed  by  the  European  Structural  Funds.  The  programme 
comprised  25  measures  intended  to  act,  together  or  separately,  on  several 
categories  of target  population. The numerous programme  objectives  were  rarely 
graded and quantified. 

The  evaluation  started  with  a  series  of  meetings  during  which  the  partners 
established  a  list  of  the  ten  main  expected  impacts.  This  list  was  obtained  by 
means  of  the  concept  mapping  of  impacts  technique.  Each  expected  impact 
was given  a name (e.g. "diversification  and efficiency  of  energy  resources"). 

Each of the ten  impacts was associated with one or more context  indicators (e.g. 
"number  of  regional  firms  that  use  natural  gas").  The  evaluation  team  then 
gathered  statistical  data to establish the  overall  evolution of these indicators  at a 
regional level. The underlying evolution  of each indicator  was analysed  by  means 
of the before-after  comparison technique. 

In this  example, the  concept  mapping  of  impacts technique  was  used  to 
"clarify  and  grade  the  effects  to  be  evaluated"  (first  evaluative  function), 
while  the  before-after  comparison  technique  was  used to  compare  data 
in order  to show  any interesting  developments  (fifth  function). 

In the second example, the situation was that of a programme financing the 
development  of  a  French  region.  Here too,  interventions  are  multi-sectoral 

Principal evaluation techniques and tools 

and  multi-objective.  The  team  responsible  for  the  intermediate  evaluation 

wanted  to  show  the  spatial  dimension  of  the  outputs  and  results. 

Box  12  -  Intermediate  evaluation  of  a  regional  programme  in  France 

The decision to launch the Integrated Mediterranean  Programmes (IMP) was taken 
in the  1980s. The aim was to  help those  regions  facing strong competition  due to 
the  entry  of  Spain  and  Portugal  into  the  European  Community.  The  IMP  of  the 
Languedoc-Roussillon  region  in  France  consisted  of  several  sub-programmes, 
oriented  towards  the  different  sectors:  agriculture  and  fishing,  industry,  crafts, 
services, forestry and the environment, and tourism. Over half the funds  mobilised 
were allocated  to  agriculture  and fishing, since these  sectors  were  hit  hardest  by 
the integration  of Spain  and Portugal  into the  European  Community. 

An  intermediate  evaluation  was  carried  out  on  the  Languedoc-Roussillon  IMP. 
Since  the  IMPs  were  designed  to  be  complementary  to  regional  policies,  the 
evaluation  had  primarily  to  analyse the  synergy  between  these two  policy  levels. 
This  evaluation  focused  above  all  on  the  location  and  spatial  integration  of  the 
different  programme  measures.  The  evaluation  team  gathered  together  all  the 
monitoring data as well as certain statistical  indicators  in the database, and  noted 
their location. A geographical Information system was used on this occasion for 
the first time for evaluative purposes. The evaluation team presented its results in 
the form of  maps, some of  which  revealed  numerous  discrepancies  between the 
location  of funds  and the  most  threatened  rural areas. 

In  this  example,  the  geographical  information  system  technique  was 

used 

to  compare  data, 

thus  producing 

interesting  conclusions 

(fifth 

function). 

The  third  example  concerns 

the  global  evaluation  of  Structural  Fund 

assistance  in  the  Republic  of  Ireland.  The  situation  is  that  of  a  set  of 

programmes  covering  numerous  sectors  and  providing  massive  funding 

countrywide.  The  global  objectives  of  the  programme  are  of  a  macro-

economic  nature.  There  are  few  of  them  and  they  are  very  clearly  defined. 

The  team  responsible  for  ex  ante  evaluation  has  to  estimate  likely  macro-

economic  impacts. 

In  this  example,  the  evaluation  team  used  a  macro-economic  model  to 

estimate  effects  (sixth  function).  It  should  be  noted,  however,  that  the  tool 

used  includes  a  series  of  options  concerning  the  variables  to  be  modelled 

or  excluded. Thus,  the  tool  implicitly  served  to  clarify  and  grade  the  effects 

to  be  evaluated  (first  function). 

Examples of the evaluation 

Box  13 -  Ex ante  evaluation  of a set  of  programmes  in the 

Republic  of  Ireland 

For the period  1989-93, all European development  assistance for Ireland was co­
ordinated  within  a  "Community  Support  Framework"  (CSF).  The  European 
Commission  and the  national  government  jointly  commissioned  an  ex  ante 
evaluation  of this  highly  complex  set of aid. The evaluation  had to estimate the 
potential  impacts of the CSF on the Irish economy. An important  question  was to 
know  whether  a  massive  intervention  (about  5%  of  the  Irish  GDP) would  not 
indirectly  cause  a  worsening  of  the  state  of  public  finances  because  of co-
financing  requirements. 

The  evaluation  was carried  out by the Economic  and Social  Research  Institute 
(ESRI)  in Dublin. The evaluation  team  estimated the micro-economic  impacts of 
support  by  drawing  on  research  work  and the  evaluations  of  similar  support 
mechanisms.  These  primary  impacts  were  introduced  into  a  macro-economic 
model,  making  it  possible  to  simulate  the functioning  of the national  economy. 
The  ESRI  had  previously  created  this  model  for  the  European  programme 
HERMES. The model first  had to be adapted to the specific  evaluation situation, 
particularly  by breaking  down  its initial  sectoral  structure to more effectively  take 
into  account  the effects  of  European  support.  The model  made  it  possible  to 
estimate  impacts  on the scale  of the national  economy,  by taking  into  account 
numerous  macro-economic  mechanisms. The risk of an excessive increase in the 
public  debt  was confirmed. 

These  three  examples  illustrate  the  use  of  several  tools  in  situations  in 

which  evaluation  covered  an  entire  multi-sectoral  programme.  The 

following  section  will  show  that  the  evaluation  toolbox  is  sufficiently  well 

stocked  for  most  of  the  functions  of  this  type  of  evaluation  to  be  tool(s)-

based. 

Tools for structuring  an evaluation 

When  a  socio-economic  programme  is  complex,  the  structuring  of  the 
evaluation  is  a  particularly  important  phase  because  it  conditions  the 
comprehensive  and  comprehensible  nature  of  the  results.  Moreover,  if  the 
evaluation  is  being  undertaken  in  a  partnership  context,  this  phase  is 
essential for clarifying the evaluation criteria suited to the different  partners' 
needs.  It strongly  determines  the acceptability  of the  conclusions. 

SWOT  analysis  is  generally  used  ex  ante  for  defining  a  programme 
strategy.  This  tool  is  helpful  in  validating  objectives  and  defining  more 
precisely the effects to be achieved and the most appropriate  interventions. 
It may also be used to evaluate a strategy that has already been established 
or even  implemented. 

Concept  mapping  of  impacts  is used to  define the  effects  that  are to  be 
evaluated  and,  in  cases  where  there  are  multiple  objectives  and  where 
these  have not  yet  been firmly  established, the  associated  indicators. This 
tool  may  also  be  very  useful  if  the  set  of  objectives  lack  precision.  It  is 
suited  to  evaluation  in  a  partnership  context  because  it  is  based  on  the 
aggregation  of  individual  points  of  view  for  the  purpose  of  reaching 
consensus  between  the  partners. 

The colour vote is particularly well suited to the clarification  and grading of 
evaluative  questions  in  an  evaluation  involving  several  decision-makers.  It 
also  provides  a  fast  and  robust  means  for  choosing  evaluation  criteria, 
validating  conclusions  and  formulating  recommendations.  This  tool  is 
useful  when there are many  stakeholders  and when their  differing  points  of 
view  have to  be taken  into  account. 

together 

It  basically  consists  of  grouping 

The impact  matrix  is a tool that  provides  a simplified  overview  of  a complex 
programme. 
the  different 
components of the programme into between ten and twenty sub-sets (the lines 
of the matrix) and of clarifying the main expected  impacts (the columns of the 
matrix). Each cell  in the matrix  is given a score  representing the probable im­
portance  of  a given  impact  for  a given  component.  The tool  helps to  ensure 
that  all the  essential  effects  of  the  programme  have  been  considered  before 
selecting  those  that  are  to  be  evaluated  more  thoroughly.  This  tool  is  not 
described  in  the  following  pages  and  for  further  information  the  reader  is 
referred to Volume 4 of the Collection devoted to  innovative techniques. 

Principal evaluation techniques and tools 

Some classical  structuring tools are not suited to the overall evaluation  of a 
complex  programme.  For  example,  the  objectives  tree  and  the  logical 
framework  do  not  allow  the  multiplicity  of  actions  and  their  expected 
impacts  to  be adequately  taken  into  account. 

Tools for structuring  an evaluation 

SWOT  Analysis 
(Strengths-Weaknesses-
Opportunities-Threats) 

in  a 

fluctuating 

SWOT  analysis  was  developed  50 
years  ago  to  help  firms  define  their 
strategy 
and 
restrictive  competitive  environment. 
This  decision-making  tool  owes  its 
name  to  the  fact  that  it  examines 
the  strengths  and  weaknesses 
the 
within 
the 
the 
opportunities  and 

firm,  as  well  as 
threats  of 

O 

Overall 
evaluation 

Φ 

Low 
technical 
complexity 

Φ 

Low 
cost 

G 

Θ 

Prospective 
use 

Use 
stabilised 

market.  It is one of the  classical  tools  of strategic  analysis,  like the  BCG  (Sos-
fon  Consulting  Group)  matrix  that  is  similar  in  many  ways. 

In the  public  sector,  cities  were  the  first  to  use  SWOT  analysis  in  the  1980s 
as  a  framework  for  reflection  on  different  development  scenarios.  The  use 
of  this  tool  has  recently  spread  to  territorial  diagnoses  and  an  ex  ante 
evaluation  of  regional  programmes. 

Example:  Prior  appraisal  of the  Swedish  Objective  6  plan 
SWOT analysis was used in drawing up the Objective 6 regional development  plan in 
Sweden.  The  results  produced 
the  main 
environmental,  structural  and  demographic  characteristics  of  the  context  of  the 
chosen  region.  The  identification  of  strategies  and  priorities,  as  well  as  the 
formulation of the action plan, were subsequently  made on the basis of major factors 
identified through SWOT analysis. These can  be summarised  as follows: 

this  method  highlighted 

through 

•  Strengths:  Large  nature  reserves, considerable  natural  resources. 
•  Weaknesses: Isolated region with a low population density; vulnerability to  economic 
fluctuations  and to fluctuations of  public  budgets; structural weaknesses of firms. 
•  Opportunities:  Tourism,  quality  of  forest  resources,  fresh-water  piscicultural 

production;  mineral  prospecting. 

•  Constraints:  Population  loss; lack of jobs. 
The  recommendations  from  this  analysis  based  on  regional  socio-economic  data 
were  judged  by  the  managers  as  being  relevant  for  their  strategic  planning.  Their 
usefulness  for  justifying  the  choices  made  in  the  framework  of  the  Regional 
Development  Plan seemed to  be recognised. Since this first  attempt  at the  concrete 
use  of  SWOT  analysis  in  the  framework  of  programmes  financed  by  the  European 
Commission, this  tool  has  been  used in many  other  regions. 

Source: Prior appraisal of the regional development  plan for the Objective 6 region of 
Sweden,  Nordland  Research  Institute,  Bodo, June 95 

Principal evaluation techniques and tools 

What  is the  purpose  of  the  tool? 

The  purpose  of  SWOT  analysis  is  to  incorporate  into  the  reflections  on  a 
programme,  both the intrinsic  characteristics of the territory concerned  and 
the  determining  factors  in the  environment  in which  it takes  place. The tool 
is intended to reduce the areas of uncertainty  related to the  implementation 
of  a  project  or  programme  applicable  to  the  relevant  territory.  It  thus 
enables the definition of a more relevant strategy  in the context  in which  the 
action  is to take  place. The aim  of the tool  is therefore: 

•  to  highlight  the  dominant  and  determining  factors,  both  within  and 

outside  of the territory,  likely to  influence  the success  of  the  project; 

•  to  produce  relevant  strategic  guidelines  by  linking  the  project  to  its 

environment. 

SWOT analysis may be expanded  by  means of tools  similar to those  known 
as  "portfolio  management",  such as the  BCG matrix, to examine the validity 
of a strategy that has been proposed or is being applied, and to  recommend 
changes where relevant. The classification  of the different  possibilities takes 
into  account  their  feasibility  (assets  available  "in  house"  =  strengths  and 
weaknesses),  as  well  as their  potential  (attractive  features  in  relation  to  the 
outside environment  = opportunities  and threats). 

The  usefulness  of  such  tools  in  evaluative  processes  lies  in their  capacity 
for  drawing  a systematic  ¡mage  of  the  relationship  between  the  evaluated 
programme  and  its direct  environment. 

• 

In which  cases  should  it be  used? 

Developed  about  twenty  years  ago  by  specialists  of  private  management, 
strategic  decision-making  tools are, once  again, being  used  increasingly  in 
public  policy  strategic  reflection.  Originally  constructed 
in  terms  of 
products,  customers,  markets  and  competitive  advantages,  their  use  has 
now  spread  to  public  collectivities,  in  so  far  as  the  policies  of  these 
collectivities  are aimed  at  creating  competitive  advantages  on the scale  of 
the  territory  for  which  they  are  responsible.  The  notions  of  strengths, 
weaknesses, opportunities  and threats  may apply to a regional  economy  in 
the  framework  of  European  competition  or  to  a  firm  competing  in  the 
market. 

SWOT  analysis  serves  to  identify  the  most  relevant  strategic  guidelines  in 
relation  to  a  global  objective  of  economic  development.  Use  of  the tool  is 

Tools for structuring  an evaluation 

therefore  more  particularly  recommended  in the  planning  of  a  programme 
and  during  its  ex  ante evaluation,  helping  to  improve the  integration  of  the 
programme  into  its  context. 

How  should  it  be  implemented? 

The  implementation  of  a  strategic  approach  such  as  SWOT  analysis 
involves  six  distinct  steps: 

Step  1.  "Scan"  of  the  environment  of  the  programme:  This  operation 
enables the  detection  of  the  major trends  and  problems  likely  to  affect  the 
future of the territory  under consideration,  by means of  socio-demographic, 
economic, political and physical indicators. Indicators of regional  disparities 
are  particularly  useful  for  revealing  opportunities  and  threats.  This  phase 
should  not  be  exhaustive,  for  the  aim  is  limited  to  obtaining  an  overall 
picture  to  illustrate  the  key  issues  the  community  in  question  will  have  to 
face. 

Step  2.  Inventory  of  possible  actions:  Possible  actions  are formulated  in 
general terms, and constructed  in relation to the  main  problems  identified. 

Step 3. External analysis  of opportunities and threats: This consists of a 
list  of  the  parameters  of  the  environment  which  are  not  under  the  direct 
control  of  the  public  authorities  and  which,  it  is  assumed,  will  strongly 
influence  socio-economic  development. 

Step  4.  Internal  analysis  of  strengths  and  weaknesses:  This  step 
completes the external analysis by making an inventory of the factors  which 
are at  least  partly  under  the  control  of  the  public  authority,  and  which  may 
either  promote  or  hinder  development. 

Step  5.  Classification  of  possible  actions: This  is  aimed  at  highlighting 
those  actions  (strategic  guidelines)  most  likely  to  reduce  development 
problems by focusing on the strengths  and reducing or even eliminating  the 
weaknesses,  with  a  view  to  maximising  opportunities  and  minimising 
threats. 

Step  6.  Evaluation  of  a  strategy: This  optional  stage  may  be  included  if  it 
is appropriate forjudging  the relevance of a strategy already  implemented or 
being  planned.  It  is  designed  on  the  basis  of  a  "portfolio  of  activities" 
analysis.  Like  a  firm  with  its  products  and  markets,  a  socio-economic 
programme  contains  a  set  of  interventions  some  of  which  combine 
strengths  and opportunities  while others try  to  compensate  for  weaknesses 

Principal evaluation techniques and tools 

or to  warn  of  threats. The  evaluation  team  places  interventions  on  a  plane 
with  two  axes:  (1)  internal  feasibility,  strengths  and  weaknesses,  and  (2) 
external  environment,  opportunities  and threats. The discussion  of the  map 
thus produced can be used to judge the relevance of the evaluated strategy. 

Application  to  the  Structural  Funds 

Strategic analysis is suited to the clarification of  issues relating to the socio­
economic  development  of  those  regions  covered  by the  Structural  Funds. 
It is recommended  for the planning  of  regional development  strategies  that 
have to  be centred on strengths  and weaknesses, while trying to  overcome 
weaknesses and threats. A tool such as SWOT analysis proposes a starting 
point  for  defining  the  strategic  guidelines  to  be  applied  in  a  specific 
environment. 

SWOT  analysis  may  therefore  serve  as  a  management  tool  for  assessing 
the  relevance  of  a strategy  in the  planning  or  implementation  stage.  In the 
framework  of  ongoing  evaluation, this  tool  may  be  used to  check  whether 
a strategy  remains  relevant to the  needs of  the regional  economy. 

Strengths  and  limits  of  use 

•  SWOT analysis applied to public action is oriented towards the search for 

an effective  strategy. 

•  The matrix tool  is derived from  management  theories that  are already  old 
and are often  criticised for being simplistic. This limitation should  be kept 
in  mind  in  its  transposition  to  public  management,  so  as  to  avoid 
restricting the analysis to a simplistic  framework. 

•  SWOT  analysis,  as  an  evaluation  tool,  serves  to  make  a  simple 
classification  of  activities  in terms  of  their  relevance.  Its  main  weakness 
derives from the often-subjective  procedure  used  by the evaluation  team 
to  classify  the activities. The association  of  partners  in this  classification 
is a way  of  enhancing the  credibility  and  usefulness  of the analysis. 

•  SWOT analysis requires a deliberate intention, on the  part  of the  different 
actors  participating  in  its  application, to  reach  a  stable  consensus.  The 
process  of  formulating  strategic  guidelines  is  only  of  value  under  this 
condition.  Otherwise,  this  model  may  tend  to  produce  an  erroneous 
and/or  inapplicable  diagnosis. 

Tools for structuring  an evaluation 

Further  reading 

Bryson  Ü.M.  and  Roaring  W.D.  (1987)  "Applying  Private  Sector  Strategic 
Planning  in  the  Public  Sector",  Journal  of  the  American  Planning 
Association,  n°53, pp.  9-22. 
Examination  of  the  application  of  corporate  strategic  planning 
methods to  contingencies  in the  public  sector. 

Quinn,  J.B.  &  al  (1988)  The  Strategy  Process,  New  Jersey:  Prentice  hall. 

Critical  study  of  strategic  planning. 

Tools for structuring an evaluation 

C o n c e pt m a p p i ng 
of impacts 

The concept mapping of impacts is 

an adaptation of the technique 

known as "concept mapping", 

used over the past ten years for 

programme evaluation. It was first 

used to help to define strategies or 

the reorganisation of firms, before 

being applied to public policy 

making, in particular, as regards 

social programmes. 

O 

Overall 
evaluation 

O 

0 

Medium 
technical 
complexity 

O 

Φ 

Medium 
cost 

© 

Retrospective 
use 

Prospective 
use 

Innovative 
use 

Data processing software is used to obtain a graphic ¡mage of the actors' 

representations of a socio­economic reality, in general, and of expected 

impacts, in particular. The MEANS programme developed an application 

specially designed for the clarification of the impacts of a regional 

development programme and to find associated indicators within a 

partnership framework (see Volume 4 in this Collection, devoted to 

innovative techniques). 

Example: Clarification of expected impacts of the ERDF 1994­99 Operational 

Programme of the Generalität de Valencia. 

This example corresponds with that of Box 11. 

The concept mapping of impacts used in the intermediate evaluation of the ERDF 

1994­99 O.P. of the Generalität de Valencia had two objectives: 

• clarification of specific and global impacts of the programme, in order to determine 

criteria for judging the effectiveness of interventions for intermediate evaluation; 

• proposal of intermediate and global impact indicators in order to complete the 

battery of programme indicators. 

The evaluation took place in two phases focusing first on intermediate impacts and 
then on global impacts, with the assistance of the managers of programme measures 
and of the overall programme managers, respectively. 

Officials of the Generalität in charge of the implementation of measures within the 
programme were divided into three groups corresponding to the fields of intervention 
(direct support to businesses, indirect support to businesses, infrastructure). At the 
end of a series of three meetings, each of the groups determined specific evaluation 
criteria for actions in the field of intervention concerned. The same groups then 
determined the intermediate impact indicators most likely to provide the information 

Principal evaluation techniques and tools 

required  to  make  a judgement  in terms  of the  specific  evaluation  criteria  selected. In 
a  second  phase  and  by  means  of  the  same  process,  the  overall  programme 
managers  (representatives  of  the  European  Commission,  of  the  central  Spanish 
administration  and  of  the  regional  authorities)  constructed  global  evaluation  criteria 
for  the  programme.  They  drew  upon  the  results  of  the  groups'  work.  The  process 
ended with the selection  of the corresponding  global  impact  indicators. 
Source:  Programme  MEANS  1996  -  Instituto  Valenciano  de 
Económicas,  "Projet  pilote  de  cartographie  conceptuelle  des  impacts" 

Investigationes 

What  is the  purpose  of the  tool? 

The  concept  mapping  of  impacts  is  used  to  assist  in  the  clarification  of 
impacts  expected  from  a  public  intervention.  The  clarification/grading  is 
obtained  by taking  into account  the representations  of all the  stakeholders. 
The  group's  work  is  facilitated  by  the  systematic  nature  of  the  approach 
that  avoids  fruitless digressions, and  by the visualisation  of the  similarity  or 
difference  between  impacts.  It  is  thus  possible  to  conduct  collective 
reflections  in  large  heterogeneous  groups  (up  to  200  persons)  and  to 
structure  and  grade  the  main  impacts  expected  from  an  intervention. The 
tool  makes  it  possible,  in an ex ante  context, to  determine the  main  objec­
tives and  priorities.  In an ex  post context, the impact  mapping technique  is 
used to draw  up the evaluation  reference  system  (effectiveness  criteria  and 
impact  indicators). 

In which  cases  can  it be  used? 

This  tool  is valuable for structuring the content  of  a complex  programme.  It 
was  used, for  example,  in a neighbourhood  in  Northern  Ireland  to  define a 
social  programme 
in  partnership  with  the  Protestant  and  Catholic 
communities  concerned.  It is useful  in evaluation when the evaluability  of a 
programme has to be enhanced, that  is to say, when it is necessary to make 
the objectives  of  a programme  more  explicit. 

This  technique  is particularly  relevant  when the evaluability  of a programme 
has to  be enhanced  in the partnership framework  involving several decision 
makers/funding  bodies. It is thus possible to produce a common  evaluation 
reference system, despite  differences  in objectives  that  may exist  between 
these  partners. 

Two characteristics  frequently  found  in  programmes  can thus  motivate  the 
use of  impact  mapping: the  lack  of  clarity  and  precision  of  the  programme 
objectives  and  of  its  expected  impacts,  and  the  lack  of  impact  indicators 

Tools  for structuring  an evaluation 

in  the  programme.  Since 

planned 
is  relatively 
cumbersome,  it  is advisable to  limit the  use of  impact  mapping  to  complex 
programmes  with  many  expected  impacts. 

implementation 

its 

How  should  it  be  implemented? 

Step 1. Constitution of a group of stakeholders  in the  programme. This 
step  is  essential  because  it  will  determine  the  framework  in  which  the 
subsequent  steps  in  the  process  are  to  take  place.  The  group  must  be 
balanced (in terms of the different actors in the programme),  knowledgeable 
(in-depth  knowledge  of  the  intervention  by the  members  of  the  group)  and 
stable  (full  participation  continues throughout  the  process  as a whole). 

Step  2.  Inventory  of  expected  programme  impacts  in  the  form  of 
descriptors.  This  step  consists  of  producing  an  exhaustive  list  of  all  the 
impacts  expected  from  the  programme,  in  the  form  of  short  qualitative 
statements  called  "impact  descriptors"  (e.g.  "actors  in  tourism  became 
more  professional"). 

This  list  must  include  any  unfavourable  impacts  of  the  programme  which 
were  not  expected  but  could  have  been foreseen. 

This  list draws  upon two  sources: 
•  identification  by  the  evaluators  of  all  the  impact  descriptors  featured  in 

the  programming  documents; 

•  impact  descriptors  formulated  by  members  of  the  group  in  relation  to 
their  knowledge  of  interventions.  These  concern  the  impacts  of  the 
measures  or  actions  of  programmes,  which  each  member  of  the  group 
expects  to  observe  in  the  short,  medium  or  long  term.  Formulation  of 
these descriptors  may take  place  during  interviews  or  in the  context  of  a 
first  group  meeting. 

A group  meeting  is essential for validating  the descriptors  and  determining 
the final  list  together. 

Step 3. Weighting  of descriptors and constitution of piles. In the  context 
of a second group meeting, each member  is given a recapitulative  list of the 
descriptors,  as  well  as  a  set  of  cards  (with  a  descriptor  marked  on  each 
one). Each  member  of the  group  is asked,  individually: 

•  to  weight  the  different  descriptors,  that  is to  say,  to  give  a  score  to  the 
descriptors  in terms of their strategic  importance for the programme. The 
more  important  the  role  of  the  described  impact  in  the  success  of  the 
programme, the  higher the  score. 

Principal evaluation techniques and tools 

•  to  constitute  piles  of  descriptors  in terms  of  their  conceptual  proximity; 
each  member  of  the  group  will  thus  group  together  conceptually  similar 
descriptors, that  is to  say, those  in the same  "family"  of  impacts. 

Step  4.  Calculation  of  the  impact  map.  The  data  produced  by  each 
member  of the group  (weightings  and piles) will be analysed statistically  by 
an appropriate  software  package. 

At  the  end  of  a  non-metric  positioning,  multidimensional  analysis  the 
descriptors  are  represented  in  the  form  of  a  cloud  of  points  in  a  two-
dimensional  space.  In this  cloud  of  points, the  descriptors  that  were  often 
placed in the same pile by the participants will be positioned close together. 
Those descriptors which were never put in the same pile by the  participants 
will  be furthest  apart. 

At the  end of a second  analysis consisting  of  a hierarchical classification  in 
clusters, the software  will  identify  groups  of  descriptors. Those  descriptors 
that  were  most  often  grouped  together  by  the  different  members  of  the 
group  will  be  put  into  the  same  cluster.  Groups  of  descriptors  that  are 
similar from  a conceptual  point  of view  are thus  constituted. 

Tools for structuring an evaluation 

Finally, the average weighting of descriptors  included  in each of the groups is 
calculated. The final  result  is a map showing  weighted  groups  of  descriptors 
(in the diagram in Box  14 the weight  is represented  by a thicker line). 

Step 5.  Definition  of  effectiveness  criteria. The  impact  map  is examined 
in  a  third  meeting  when  the  composition  of  the  different  groups  of 
descriptors  is  discussed.  Some  descriptors  can  be  moved  to  make  its 
composition  more  coherent.  Thus,  the  map  calculated  by  the  software  is 
not  a final  result  but  rather  a basis for  discussion. 

Once the  composition  of  the families  has  been validated, they  are named. 
The aim is for the members of the group to agree on a name for each family, 
which  is  representative  of  all  the  descriptors  in  the  family.  This  name 
corresponds  to  an  evaluation  criterion.  Thus,  there  will  be  as  many 
evaluation  criteria  as families  of  impact  descriptors.  Since the  families  are 
weighted, the  criteria obtained are also weighted. This allows these  criteria 
to  be used  directly  for a multicriteria  judgement. 

Group  reflection  may  be continued  with the  aim  of  specifying  a time-scale 
for the impact families (impacts which are expected in the short, medium or 
long  term),  and  the  presumed  causal  links  between  these  impact  families 
(e.g. a particular impact family constitutes an intermediate impact which will 
help to achieve  a more global  objective). 

This  process  may  finally  be  taken  further  with  the  aim  of  proposing  impact 
indicators  directly  related  to  the  different  evaluation  criteria. The  descriptors 
included  in each of the families  are helpful  in defining  these  indicators. Thus, 
some of these descriptors, and particularly those to which the most substantial 
weightings were given, may directly  inspire the definition of  an indicator. 

Discussion  by the group on each of the proposed  indicators  will relate  both 
to  the  relevance  of  the  indicator  in  terms  of  its  relation  to  the  evaluation 
criterion, and the availability  of sources  of  information. 

Application to the  Structural  Funds 

The concept  mapping of impacts  has proved to be particularly well adapted to 
the context of structural  programmes. The aims of this type of  programme are 
defined  by the  European  Union  and  adapted  to  suit  to  the  characteristics  of 
each Member State or beneficiary  region. Since the Single Programming Docu­
ments are the product  of  negotiation, the programme objectives are frequently 
not  specific  enough  for  constructing  a reliable  evaluation.  Impact  mapping  is 

Principal evaluation techniques and tools 

consequently useful in the first stage of clarification. It helps to clarify evaluation 
questions, grade effects and define evaluation criteria in a partnership context, 
which facilitates the appropriation of the evaluation results. 

The impact map is useful in the preparation of intermediate programme 
evaluation. It makes it possible to systematically establish impact families 
cutting across several measures, and thus to construct relevant criteria for 
the evaluation of the programme as a whole. The procedure can be divided 
into two phases, as in the case of Valencia, which are summarised in the 
following diagram. 

Box 15- Partnership process of impact mapping 

Operators 
Group A 

Operators 
Group Β 

Operators 
Group C 

PHASE 1 

■ 

' 1 

Map of 
specific 
impacts A 

' 

Map of 
specific 
impacts Β 

Map of 
specific 
impacts C 

PHASE 2 

■ 

' 
Group of 
managers 

1 

Map of global 
impacts 

Strengths and limits in practice 

• Using a computer programme to mediate is a valuable support for 

reaching agreement, despite the heterogeneity of initial points of view. 

Tools for structuring  an evaluation 

•  The group dynamics  generated  by the technique  has proved to  be useful 
and  positive  for  increasing  mutual  knowledge  of  different  management 
actions,  and  may thus favour  adjustments  to the  implementation  of  pro­
grammes.  By  mobilising  the  stakeholders  in  the  preparatory  phase  of 
evaluation work, it facilitates the subsequent  stages of the evaluation and 
makes the results far  more  readily  acceptable. 

•  Impact  mapping  is a technique that  is used to enhance the evaluability of 
programmes.  It  makes  it  possible  to  produce  a  common  evaluation 
reference  system, taking  into  account  the  points  of  view  of  the  different 
partners.  The  successive  group  discussions,  which  are  part  of  the 
implementation  of  this  tool,  guarantee  the  legitimacy  of  the  reference 
system obtained. 

•  The  results  may  be  produced  in  a  few  days.  The  technique  is  thus 

compatible  with the evaluation  cycle. 

•  The  resources  required  (even  if  they  are  limited)  and  the  mobilisation  of 
numerous  stakeholders  during  successive  meetings  can  be justified  only 
when the evaluation  stakes  warrant  such an  investment. 

•  The constitution  of the  group  of  partners  can  be difficult  and care has to 
be  taken  to  ensure  that  there  is  a  sound  balance  between  the 
stakeholders.  Considerable  efforts  may  be required  in cases  of  conflict. 

•  Although the  technique  of drawing  up impact  maps  limits  risks related  to 
the leadership of a few members of the group, this risk remains during the 
phases  of  production  of  descriptors  and exploitation  of the  impact  map. 

•  This tool  can  be  used for  all the functions  of the  structuring  stage. 

Further  reading 

Knox, C. (1995)  'Concept  Mapping  in Policy  Evaluation: A Research  Review 
of  Community  Relations  in  Northern  Ireland',  in: Evaluation,  Vol. 1, 
Number 1. 
An  interesting  example  of  use of the tool  in a European  context. 

Trochim,  W.M.K.  (1989)  An  introduction  to  concept  mapping  for  planning 
and  evaluation', in: Evaluation and  Program  Planning,  12 (1):  1-16. 

Presentation  of the tool  and its advantages for evaluation. 

In the  MEANS  Collection: Volume 4 devoted  to  innovative  techniques. 

Tools for structuring an evaluation 

Colour vote 

In the field of evaluation, group 

work is often essential, particularly 

when several partners are involved 

in commissioning and steering an 

evaluation. The different voting 

techniques can be used to 

structure collective reflection, when 

the aim is to improve effectiveness, 

stimulate creativeness or define a 

common standpoint. While there is 

a large number of these 

techniques, this information sheet 

O 

Overall 
evaluation 

A 

Θ φ 

Low Low 
technical cost 
complexity 

O o 

In­depth 
evaluation 

Retrospective Prospective 

use use 

© 

Innovative 
use 

focuses more particularly on the "Abaque 

de Régnier", or colour vote. 

Example: Evaluation of an economic development programme in Switzerland 

As part of a study of the measurement of the effects of public actions in Switzerland, 
the CEAT (research community for national development) evaluated a canton policy 
for economic development. Two work groups were formed with the actors involved 
in the implementation of the policy. The first group was to focus on the "objectives 
and strategies" of the policy and the second group on its "implementation and co­
ordination". 

The evaluation team used the Abaque de Régnier as a technique for conducting 
group sessions. The groups based their work on assertions formulated by the 
evaluation team. These were often provocative, the intention being to encourage the 
participants to take a stand. By asking the participants to vote on these assertions, 
the technique created a starting point from which to express their positions and to 
become aware of the other opinions present in the group. The debate helped to 
define a common position on the objectives of the policy of economic development. 
Source: Alberton S., Rey M. et L. Vodoz (1993) La politique valaisanne de promotion 
économique: une évaluation privilégiant le point de vue des acteurs. Berne: 
Programme National de Recherche 27. 

W h at is the purpose of t he tool? 

Voting techniques facilitate the expression of opinions on a given subject by 

all the participants. It is the way in which these opinions are obtained that 

varies. The colour vote is based on the idea of making participants aware 

Principal evaluation  techniques and tools 

of  all opinions  in the  group,  by  means  of  a coloured  visual  presentation  of 
the  results  of  individual  votes.  In  this  way  the  strong  personality  of  some 
does  not  conceal  the  opinions  of  other,  more  reserved,  participants.  With 
the colour  vote, issues on which  there  is consensus  are soon  distinguished 
from  conflictual  issues.  In  this  way  time  is  saved  by  allowing  the  group's 
reflections  to  be concentrated  on  most  problematical  issues. 

Voting techniques  also  help to avoid certain  drawbacks  of group work  such 
as  pressure  to  conform,  convergence  of  opinions  around  core  values, 
predominance  of  certain  actors,  etc. 

In which  cases  should  it be  used? 

A work  group  may  use these techniques  for any type  of  evaluation: ex  ante 
to  define  challenges  or  needs to  be  met, ongoing  to clarify  objectives  as in 
the  example  above  and,  where  relevant,  ex  post  to  validate  evaluation 
conclusions. 

Very often  the objectives  and judgement  criteria of a programme  cannot  be 
legitimately  clarified  by  a single  person.  In this  case  the  voting  techniques 
facilitate  the  work  of  a group  composed  of  people  with  very  different  skills 
and  levels  of  responsibility.  In  other  words,  meetings  for  managing  the 
performance  of  an  evaluation  are  an  ideal  opportunity  to  apply  voting 
techniques.  From this  point  of view, the colour  vote  is particularly  useful for 
grading  programme  objectives,  defining  evaluation  criteria  and  selecting 
the  most  important  criteria. 

How  should  it  be  implemented? 

Step  1.  Formulation  of  statements.  The  statements  are  assertions 
expressed  concisely,  generally  in  a  single  sentence.  They  are  formulated 
beforehand  by the evaluation team, on the basis of  personal  interviews  and 
of  an  initial  group  discussion, during  breaks  in the  meeting. 

In the  above  example the  items were formulated  as follows: 
•  "For  industrial  enterprises, financial  support  is  derisory" 
•  "For  industry,  economic  promotion  should  be aimed  at  'high-tech'  firms" 
•  "Economic  promotion  should  benefit  all areas  of  the canton  equally" 
•  etc. 

At  this  stage,  it  is  useful  to  collectively  check  the  understanding  of  each 
statement. 

Tools for structuring  an evaluation 

Step 2. Individual vote: the  participants  give a colour  to each  statement. 

The signification  of the  colours  is simple: 

Red 

Pink 

Yellow 

Total 
disagree­
ment 

General 
disagree­
ment 

No clear-
cut opinion 
on the 
subject 

Light 
green 

Dark 
green 

General 
agreement 

Total 
agreement 

White 

Black 

No 
opinion 
on the 
subject 

Does 
not want 
to 
participate 
¡n the vote 

Step  3. Transcription  of  votes  on a  magnetic  board  or  on  a  computer 
screen. The  result  is thus  immediately  presented  to  the  participants  in the 
form  of a table filled  with  coloured  disks. Tendencies are clearly visible  and 
this  enables  each  person  to  know  whether  consensus  exists  or  not  in  the 
group. 

Step  4.  Group  discussion.  The  group  leader  selects  statements  for 
discussion.  Most  often,  he  or  she  will  choose  those  on  which  there  was 
disagreement. 

Step  5.  Reformulation  of  statements  with  the  group.  This  final  step  is 
used to define  a collective  position  on the  subject. 

Application  to the  Structural  Funds 

The  Structural  Funds  are  managed  within  a  partnership  framework,  which 
means that  numerous working  meetings  are organised. Apart from those of 
an administrative nature, all meetings with a "creative"  aim can be based on 
voting techniques  to structure  and motivate the work  (meetings  devoted  to 
the  selection  of  projects,  the  evaluation,  reorganisation  or  renewal  of 
programmes, etc.). 

The  colour  vote  is  particularly  well  suited  to  situations  in  which  a  common 
standpoint  or  consensus  are  required.  It  is  also  used  to  grade  proposals 
made in a group, which can make it easier to reach conclusions  in a meeting. 

Strengths  and  limits  in  practice 

•  Groups appreciate the colour vote when they want to reach consensus in 
a  relatively  conflictual  domain.  The  transparency  and  simplicity  of  the 
technique  ensure  a high  level of  confidence  in the  process. 

Principal evaluation  techniques and tools 

•  The  results  of  this  vote  are  immediately  visible  and,  thanks  to  the 
technique,  tendencies  are easily perceived. The possibility of voting  "whi­
te"  or  "black"  helps  to  highlight  a  lack  of  information  or  points  of 
disagreement  on the  subject  under  discussion. 

•  The  facilitator  has  a  high  level  of  responsibility;  he  or  she  therefore 

requires  extensive  experience. 

•  The  fact  that  the  tool  is  not  of  a  highly  technical  nature  enhances  the 
participants'  confidence  (NB: the  most  recent  version  of  the  Abaque  de 
Régnier  is computerised.)  The  game-like  nature  of  the  method  also  has 
the  advantage  of  relaxing  the  atmosphere  during  meetings.  There  is, 
however,  a  risk  of  rejection  if  the  participants  find  the  technique  too 
scholastic. 

•  Potential  predominance  of one actor  in the debates  is limited by the  non­

verbal  nature of  the  tool. 

Further  reading 

Anzieu  D. et Martin J.-Y.  (1994) La dynamique  des groupes  restreints.  Paris: 

PUF. 395  p. 
Reference  manual  for  the  functioning  of  small  groups.  Annex 
containing  various  methods  of  "discussion-meetings". 

Mucchielli  R.  (1992) La conduite  des  réunions.  Paris: Librairies  techniques. 

120  p. 
Popularised  presentation  containing  practical  exercises. 

Viveret  P.  (1992)  'Les  outils  d'intelligence  collective  et  l'évaluation',  in: 
L'évaluation  en développement.  Paris: La Documentation  Française, 
pp.  179-183. 
Summarising  group  techniques. 

Tools for observing  changes in the field 

When  a programme  is complex  and  has to  be  evaluated  in  its  entirety, the 
field  of  observation  to  be covered  is very  broad. Exhaustive  observation  is 
therefore  extremely  costly.  Tools  do,  however,  exist  for  collecting  the  data 
needed for evaluation. 

Observation  in  the  field  can  be  carried  out  by  organising  a  series  of 
individual  interviews  with  operators  (indirect  observation)  or  with  the 
addressees  of  the  programme  (direct  observation).  This  technique  is 
particularly  appropriate  when the information  to  be collected  is factual. 

Focus groups  , with  operators  or  addressees  as  participants,  are  used  to 
collect  both  opinions  and  data.  A  well-conducted  focus  group  allows 
reliable, high quality  data to  be collected. 

Systematic  field  observation  can  be  carried  out  by  means  of  carefully 
selected case studies. In the evaluation of a complex  programme, the case 
study  seems to  be the only  appropriate tool for  observing  effects  on  direct 
or  indirect  addressees,  while  respecting  the  complexity  of  the  different 
programme  components. 

Some  classical  observation  techniques  are  difficult  to  apply  in the  overall 
evaluation  of  complex  programmes.  For  example,  the  questionnaire 
survey  of  direct  addressees  of  the  programme  is  too  expensive  if,  as  is 
often the case, there are up to ten  different  categories  of  addressees. 

Tools for observing  changes in the field 

Individual  interview 

The  interview  technique  is  used  to 

gather  qualitative  information  and 

the  opinions  of 

those  persons 

affected  by a particular  programme 

or  project,  its  context,  implemen­

tation,  results  and  impact.  Several 

f o r ms 

of 

interview 

can  be 

distinguished,  each  of  which  fulfils 

a  different  purpose:  the  informal 

conversation 

interview;  the  semi-

structured,  guide-based 

interview; 

and  the  structured 

interview  (the 

most  rigid  approach). 

O 

Overall 
evaluation 

A 

£3 <D 

Medium 
technical 
complexity 

Medium 
cost 

O  G 

In-depth 
evaluation 

Retrospective  Prospective 

use 

use 

Θ 

Use 
stabilised 

The  following  presentation  is  based  on  a  semi-structured  interview,  the 

technique  that  is used  most  frequently  in the evaluation  of Structural  Funds. 

Example:  Evaluation  of the North Jutland  Technology  (NordTek)  programme 

The goal of the  NordTek  programme  was the development  of North  Jutland  through 
the  introduction  of  information  technologies  into  the  regional  industrial  fabric. The 
programme  had two aims: job creation and the improvement  of a negative  image as 
a  peripheral  region.  To  this  end, the  development  strategy  implemented  by  this 
programme  was based  on four  objectives  that  were  to  be achieved  through  three 
types of action. 

lastly,  developing  attractiveness  for  foreign 

The objectives were: reinforcing  local networks  of actors; improving the qualification 
of workers in new technologies; orienting regional  production towards  local demand; 
and, 
investments.  The  actions 
implemented for that purpose were measures favouring the knowledge and  utilisation 
of  new technologies,  measures  for training  people  in the use of  new technologies, 
and measures  providing  investment  support for SMEs. 

The  programme  was evaluated  by  means  of  semi-structured  interviews  among 70 
beneficiary  firms. The goal of these  surveys was to understand  how firms  used the 
services  received  and what  they  got out of them.  Those  firms  in which  interviews 
were  held  were  selected  from  250 beneficiary  firms  in five  of the  most  important 
projects in the NordTek programme  (the different  projects comprising the programme 
concerned  specific  subjects  such  as  consultancy  or  training  and  networking, for 
example).  The firms  selected  had to  have  received  a  relatively  large  amount  of 
assistance during at least six months, and to be representative of the different  forms 
of assistance  that  had  been granted. 

Principal evaluation  techniques and tools 

The approach  chosen was a method  using an  interview  guide, so as to gather all the 
information  concerning  the firms. The interview  guide also  made  the approach  more 
rigorous, thus facilitating  the comparison  of the  interview  results obtained  by  several 
different  interviewers. After  the  interview,  the  survey  continued  with  a  more  informal 
conversation  during  a  visit  to  the  firm.  It  lasted  a  total  of  about  two  hours.  The 
interviewee  then  received  the  summary  of  the  interview  in  the  form  of  a  4-8-page 
document,  for  validation. 

The themes  addressed during the interview were, for example: the reasons for joining 
the  project,  the  expected  benefits  from  the  assistance  and  the  benefits  actually 
received, and  the  possibilities  of  obtaining  assistance  from  a different  source. 
The  results  of  the  interviews  made  it  possible  to  identify  the  reasons  behind  the 
firms'  decision  to  participate  in the  NordTek  projects.  On  the  whole,  the  interviews 
revealed  that  the  assistance  provided  by  NordTek  was  unique  in certain  cases  and 
could  not  be  found  elsewhere,  whereas  for  certain  aspects  it  competed  with  local 
companies  and  even  other  NordTek  projects.  The  NordTek  networking  and 
consultancy  services  were  developed  over  a  period  of  several  months,  and  even 
several  years;  personal  knowledge  and  geographical  proximity  were  then  seen  as 
important  factors  for  the  firms.  It  nevertheless  remained  difficult  to  quantify  the 
benefits  that  the  NordTek  projects  were  able  to  provide  to  firms,  due  to  the 
numerous  external  factors  capable  of  influencing  the  process  in  which  the  services 
were  provided. 
Source: Olsen, Leif  and  Olaf  Rieper.  NordTek-evaluieringen,  AKF  Forlaget,  1991 

W h at  is  t he  p u r p o se  of  t he  tool? 

This  type  of  survey  is  a  way  of  learning  about  and  examining  the  views  of 

the  actors  (addressees  and  other  stakeholders)  on  a  project  or  programme, 

e.g.  how  far  the  project  or  programme  meets  needs,  or  its  results  as 

c o m p a r ed  to  expectations.  Interviews  are  also  used  in  social  science  as  a 

tool  for  investigating  users'  reasoning. 

The  individual  interview  is  an  exploratory 

technique 

that  serves  to  obtain 

relevant  information  on  the  reasoning,  conceptions  and  representations  of 

the  persons  questioned  on  a  project  or  programme.  Apart  from  subjective 

representations,  it  also  serves  to  gather  information  on  individual  practices. 

It  is  particularly  valuable  in  exploring  the  ways  in  which  an  intervention  has 

been  implemented  and  for  identifying  g o od  practice.  It  is  therefore  often 

used  as  part  of  a  formative  evaluation  that  is  designed  to  test  a  theory  of 

action  and/or  to  provide  guidance  about  fine-tuning  a  policy  and  program­

me. 

It  may  also  be  helpful 

in  providing  a  summative  evaluation  of  a 

Tools for observing changes in the field 

programme  which  is  intended  to  achieve  changes  in  actors'  behaviour  or 
perceptions  (such  as  technology  transfer  or  training  programmes)  as 
opposed  to  the  more  tangible,  'harder'  outcomes  (such  as  creating  new 
jobs  or constructing  new facilities). 

In which  cases  can  it be  used? 

The  interview  is  used  in  an  exploratory  context,  in other  words,  when  one 
does  not  have a priori  hypotheses  or  adequate  knowledge  on  a  project  or 
a target  public  to  make a questionnaire  survey  possible. 

It is also  a relevant  technique when the  public  concerned  by the  evaluated 
programme  or  project  is  too  small  to  be  the  subject  of  a  statistically 
representative  survey. 

Semi-structured,  guide-based 
relatively 
standardised  information is required about the programme being studied. A 
certain  amount  of  prior  knowledge about the subject  is needed when  using 
this  technique. 

interviews  are  used  when 

The  individual  interview  technique  may  be  used  at  any  stage  of  the 
evaluation  process:  to  establish  a theory  on  the  evaluated  programme  or 
identify the  problems  and  needs of  a region and the  improvements  needed 
during  the  implementation  of  the  programme.  It  may  be  particularly 
powerful  when  used  to  gain  the  views  and  understandings  of  key  actors 
such  as senior  policy  makers  or  programme  managers. 

•  How  should  it be  implemented? 

In  order  to  conduct  an  interview  well,  the  following  steps  have  to  be 
adhered to: 

Step  1. Selection  of  interviewees:  The  samples  needed  for  carrying  out 
interviews  are  smaller  than  for  questionnaire  surveys.  The  information 
obtained  is validated  by the context  and not  by the probability  of  occurrence 
peculiar to  questionnaires. The number  of  interviews  depends  on the subject 
of the study, on the variety  of  reactions  to the  subject, and on the  resources 
available  (in general  20 to  60  interviews). The  selected  sample  is  most  often 
based  on  the  selection  of  components  said  to  be  characteristic  of  the 
population  (diversified  sample). The sample  may  be selected  either  by  direct 
access (e.g. via administrative  lists) or through the intervention of a third party, 
which  allows for  more targeted selection  but  presents risks of distortion. 

Principal evaluation  techniques and tools 

Step  2.  Planning  the  interview:  Planning  the  interview  includes  the 
drawing  up of an interview  guide. This consists  of specifying  the topics  that 
the  interviewer  wants  to  address.  It  is  not  essential  to  follow  the  interview 
guide  in  any  precise  order.  This  guide  is  more  a sort  of  checklist  enabling 
the  interviewer  to  check  that  he  or  she  is  dealing  with  the  essential 
questions. The  interviewer  may  modulate  her/his  intervention  in  relation  to 
the  interviewee, and formulate  new questions. The first  interviews  may  give 
rise to  adjustments  or  amendments  in the  definition  of  the  questions  if  the 
interviewees  have  a  problem  with  them.  In many  cases  it  is also  helpful  to 
gather  basic  information  from  interviewees  in  advance  to  save  time  during 
the  interview  itself.  It  may  also  be  useful  to  provide  the  interviewee  with  a 
guide  to  the  issues  that  are  to  be  covered  so  that  he  or  she  can  gather 
together  any  necessary  factual  information  prior to the meeting. Certainly  it 
is usually worth  establishing  a 'contract'  with the  interviewee explaining  the 
purpose of the interview,  how  long  it will  last, the  level of  confidentiality,  the 
use made  of  findings  etc. 

Step 3. Selection and training of interviewers: If they are to be conducted 
properly,  interviews  require  a  high  degree  of  professionalism 
in  the 
interviewer.  He or she must  have skills in communicating, listening and note 
taking. To facilitate the smooth  running  of the  interview  and ensure that  the 
interviewee  feels  at  ease,  it  may  be  useful  to  ensure  that  there  is  social 
proximity  between  interviewer  and  interviewee.  The  least  structured  inter­
views also require the interviewer to have substantial  knowledge of the field. 

Step  4.  Course  of  the  interview:  An  interviewer  must  have  a  "respectful" 
attitude  vis-à-vis  the  interviewee  and  the  information  gathered,  but  also  be 
able to  convey  a good  understanding  of the subject  matter  and context.  The 
initial  contact  is  very  important,  for  it  is  the  basis  of  communication.  The 
interviewer  must  be  careful  not  to  influence the  interviewee  by  approving  or 
orienting  her/his  answers. The  interview  may  be  recorded, to  ensure that  the 
interviewee's  assertions  are  not  distorted  and  that  the  most 
important 
remarks are not omitted from the report. However, in some cases  interviewees 
may  feel  less  able  to  give  full  and  frank  answers  on  tape,  and  the  costs  of 
transcribing  interviews  should  be  taken  in  to  account  in  designing  the 
evaluation  budget  and timetable. The  interview  transcript  or summary  should 
be checked  with  the  interviewees  and  it may  be useful  at this  stage to  follow 
up  any  unresolved  questions.  It  may  also  be  useful to thank  interviewees  by 
letter  for  their  contributions  (their  co-operation  may  be  needed  in  future 
studies  and  it  is therefore worthwhile  ensuring their goodwill). 

Tools for observing changes in the field 

Step  5.  Analysis  of  results:  This  final  phase  consists  of  analysing  the 
conversations,  interpreting  and  comparing  the  information  given  by  the 
interviewees,  and finding  common  and divergent  viewpoints  so  as to  draw 
up a review of the evaluation. A summary focused on the coherence of each 
interview,  or  a transversal  thematic  analysis  more  suited  to  the  search  for 
models  capable  of  explaining  individual  practices,  may  be drawn  up. 

In  order  to  produce  results  and  then  discuss  them,  the  interviewer 
establishes  an  analysis  grid  based  on  the  reading  of  the  interviews  and  of 
descriptive  hypotheses.  This  grid  is  an  explanatory  tool  applied  to  each 
interview  report. 

Application  to the  Structural  Funds 

Interview  techniques  are  used  extensively  in  the  evaluation  of  structural 
interventions  and,  in  particular,  for  the  evaluation  of  programmes. 
Interviews  with  programme  managers  and  addressees  remain  one  of  the 
most  commonly  used  tools  in  the  intermediate  evaluation  of  structural 
programmes.  The  principle  consists  of  selecting  several  addressees  or 
managers, depending on the characteristics  of the action implemented and 
the  public  concerned. 

Used  as  a  tool  for  analysis,  the  interview  is  a  technique  which  makes  it 
possible  to  make  a  brief  overview  of  the  programmes.  Very  loosely 
structured  interviews  - informal conversations  with the  managers  and other 
persons concerned  by the programme - may be used to identify those  parts 
of  the  programme  which  need to  be  considered  in  more  depth.  Interviews 
with  policy  makers  may  also  form  an  important  part  of  the  early  stage  of 
focusing  an  evaluation  and  determining  the  key  outputs  which  policy 
makers require. 

The interviews may also prove relevant and provide useful information in the 
framework  of  programmes  of  a  social  nature  where  the  addressees  often 
lack  motivation  for filling  in the  questionnaires,  such as measures  aimed at 
giving  jobs  to  the  long-term  unemployed  who  have  been  on  a  training 
course. 

Strengths  and  limits  in  practice 

•  The  advantage  of  this  type  of  survey  is  that  it  provides 

in-depth 
information  on  the  values,  facts  and  behaviour  of  the  interviewees;  it 

Principal evaluation techniques and tools 

makes it possible to link up a group of elements, thus producing a 
relatively exhaustive study on a given subject. 

• An individual interview takes into account situational and individual 

factors. It is therefore difficult to draw general conclusions. Individual 
interviews may allow for an exhaustive identification of effects and 
possible causes, but cannot be used to measure impacts or grade 
causes. 

• A well conducted interview may provide insight into the mechanisms of 
implementation and the causal links peculiar to a programme, and help 
to identify success stories or obvious shortcomings. In that case, the 
technique produces information that can easily be communicated in the 
form of boxed examples in a text. 

• If it is to be effective, this practice requires a lot of time and the 

contribution of professionals. Specific skills are needed to plan, conduct 
and interpret an interview; inadequate skills of this nature will produce 
information of no value. Interviews are not always conducted with the 
professionalism required to produce effective results. 

• The benefit of these surveys depends on the knowledge of the 

interviewees and on their co-operation in answering the questions. 
Information may also be distorted due to the choice of interviewees. 

Further reading 

Blanchet Α., Gotman Α. (1992) L'enquête et ses méthodes: l'entretien. 

Nathan Université 
This book is entirely devoted to interview surveys, from an educative 
point of view. 

Casley D., Kumar K. (1991) Collecte, analyse et emploi des données de 

suivi et d'évaluation. Banque Mondiale 
Clear and accessible presentation of interview techniques - Ch. 2 
and 3, p p . 1 2 - 47 

Patton M.Q. (1987) How to use qualitative methods in Evaluation, Sage 

Publications 
This book devoted to all the qualitative techniques gives a glimpse 
of their application in the field of evaluation. 

Tools for  observing  changes  in  the  field 

F o c us  G r o up 

A 

focus  group 

involves  several 

people  with 

the  same  characte­

ristics,  and  provides  qualitative 

information  during  a 

targeted 

discussion. 

The 

focus  group 

technique  was 

devised  by  marketing  specialists  to 

analyse 

the  potential 

impact  of 

advertising 

strategies 

and 

advertisements. The  method  solicits 

the 

participants' 

creativity 

carrying  out  simulations. 

It 

in 

is 

O 

Overall 
evaluation 

e m 

High 
technical 
complexity 

Medium 
cost 

A 

O  o 

In-depth 
evaluation 

Retrospective  Prospective 

use 

use 

© 

Innovative 
use 

particularly  valuable  for  analysing  themes  or fields which  give  rise to  divergent 

opinions  or  which  involve  complex  issues  that  need  to  be  explored  in  depth. 

Example: Thematic  evaluation  of the  Objective  3 SPD, 1994-99,  UK 

In  the  framework  of  an  evaluation  of  the  Objective  3  SPD  in the  UK,  a focus  group 
technique was  used to  complement  other techniques. The general objectives  of  this 
evaluation  were  to  check:  a)  whether  the  actions  implemented  within  the  SPD  had 
effectively  reached the targeted addressees  and  b) whether the structure  of the SPD 
favoured  the  emergence  of  effective  projects 
in  terms  of  the  integration  of 
addressees  into the job  market. 

The  method  devised  for  this  evaluation  had  a  quantitative  section  based  on  the 
exploitation  of  monitoring data, and a qualitative  section based, in particular,  on four 
focus groups formed in four regions representative of the diversity  of situations  in the 
UK: Scotland, the  "West  Midlands",  London  and the  Southwest. 
These  focus  groups  consisted  of  representatives  of  project  promoters  (notably 
training  centres,  regional  councils,  higher  education  institutions,  non-governmental 
organisations,  and  Training  and  Enterprise  Councils).  The  subjects  and  questions 
addressed  were structured  as follows: 

•  examination  of  the  sources  of  information  available  for  planning  projects 
corresponding to the needs of targeted  publics  as well as for choosing the  means 
to  reach these  publics; 

•  highlighting the strengths and weaknesses of the modes of selection, approval and 
management  - especially financial - of  projects, and their effectiveness  in terms  of 
targeting; 

•  discussion  on  the  means  to  measure  and  assess  the  results  of  these  actions  in 
relation  to  the  specificity  of  the  publics  targeted  and  employers'  requests  which 
have been debated; 

Principal evaluation  techniques and tools 

•  finally,  debate  on  the  effectiveness  of  the  SPD  as  an  instrument  for  promoting 

projects  suited  to the addressees'  needs. 

These  discussions  served  to  reveal  new  ideas  and  considerations,  often  covering 
themes closely related to the questions  asked, such as the existence and  advantages 
of  "integrated  projects", the  modes  of  functioning  in  networks  of  project  promoters, 
the  "additionality"  of  projects,  etc.  The  results  drawn  from  these  focus  groups, 
although  not  representative  in  the  statistical  sense  of  the  term,  served  to  highlight 
"ideas  to  work  on"  in various  fields  such  as: access  to  problem  publics,  conditions 
facilitating  the  emergence  of  relevant  projects  in terms  of targeting, and  so on. 
Source:  Targeting  and  the  ESF  Objective  3  1994-1996  Single  Programming 
Document  (SPD),  final  report  to  the  Department  for  Education  and  Employment 
(DfEE)  ECOTEC, June  1996 

What  is the  purpose  of the  tool? 

Structured  discussions  in focus  groups  are starting  to  be  used  extensively 
today,  especially  in  the  social  sciences  and  particularly  for  evaluation.  By 
playing  on  the  interaction  and  confrontation  of  points  of  view,  this  techni­
que  serves  to  reveal  the  participants'  perceptions  on  a  given  theme, 
whether 
its 
implementation,  results  or  impacts. 

it  concerns  the  relevance  of  the  evaluated  programme, 

The  focus  group  technique  is  a  particular  application  of  techniques  for 
chairing  meetings. The tool is standardised, with a view to creating  optimum 
interaction for the collection of information. Other group  techniques  are used 
in  the  social  sciences,  such  as  the  Delphi  technique  or  the  nominal  group 
(recording  of  all  individual  points  of  view  which  are  then  submitted  to  the 
group  and  discussed), that  do  not  feature truly  interactive  discussion. 

The  focus  group  makes  it  possible  to  bring  together,  simultaneously  or 
sequentially,  several  stakeholders  in  a  programme  (managers,  addressees 
and/or  operators)  and to  collect  a large amount  of qualitative  information  in 
a  relatively  short  space  of  time,  by  means  of  the  "shared  intelligence" 
technique.  The  comparison  of  the  experiences  and  representations  of  the 
participants  in  the  group  make  it  easier  to  understand  the  phenomenon 
under  observation. 

By  involving  the  actors  or  addressees  of  a  programme  in  the  evaluation 
work,  the  conclusions  of  the  study  will  be  more  credible  and  more  easily 
accepted.  The focus  group  technique  may  also  be  used  for  the  validation 
of  data  collection, or for  completing  quantitative  data. 

Tools for observing changes in the field 

In which  cases  can  it be  used? 

Focus groups are adapted to those cases where the topics to be addressed 
provoke  divergent  opinions  but  where  discussion  remains  possible.  The 
subjects  dealt  with will  be  in the  "public  interest",  so as to  be debatable  in 
a group. 

How  should  it be  implemented? 

For  the  same  reasons  as  those  applying  to  individual  interviews,  several 
steps are required to  prepare and conduct  focus  groups  properly: 

Step  1. Selection  of  participants: The  choice  of  participants  in the  group 
depends on the characteristics  of the population and the subject addressed. 
It  is  preferable  to  form  several  types  of  groups  (between  3  and 5)  in  the 
selected  categories. Limiting the work to a single group  may  undermine  the 
validity of the study. The group dynamic could, for example, be distorted due 
to  the  heterogeneous  composition  of  the  sample.  The  participants  will  be 
selected  so as to ensure that there is a degree of homogeneity  in the group. 
The number of  participants  generally  ranges from 6 to  10 per group, so that 
each  person  has  a turn  to  speak,  and  so  that  sub-groups  are  not  formed. 
Usually the  participants  do  not  know  one  another. 

The participants  are sometimes  remunerated. However,  rather than  offering 
an  incentive  remuneration,  it  is  preferable  to  offer  to  refund  their  travel 
expenses,  where  relevant,  and  to  serve  refreshments  at  the  end  of  the 
session. 

Step  2.  Designing  the  interview  guide:  Given  the  time  limits  and  the 
number  of  persons,  it  is  important  to  carefully  define  and  limit  the  topics 
addressed. The guides for each group  will be different. The group  interview 
guide must  be short because more time is necessary when several  persons 
participate  in the  discussion  and  answer  the  same  questions.  Most  of  the 
questions asked are open; they must also be carefully defined and arranged 
in a series with the  most  general ones  first. 

Step  3.  Choice  and  training  of  facilitators:  The  facilitator  must  prove 
her/his  creativeness  and  motivation, and  have a good  relationship  with  the 
participants.  In their  training  facilitators  must  learn which  attitudes to  have 
vis-à-vis the group  (e.g. respect  for  people's  ideas) and what  knowledge  to 
the  objectives  of  the  study  and  data  analysis 
absorb  concerning 
techniques.  Facilitators  must  also  have  communication  skills,  note-taking 

Principal evaluation  techniques and tools 

skills,  etc.  It  is  useful  to  make  provision  for  a  second  person  per  group, 
particularly  if the session  is not tape  recorded, so that  one person  can  take 
notes  while  the  other  leads  and  facilitates  the  discussion.  Co-facilitators 
may  observe  the  discussion  and  then  make  recommendations  to  the 
facilitator  on  the  way  the  meeting  was  conducted.  When  this  technique  is 
being  used  to  test  advertising  or  products,  the  co-facilitators  usually 
observe  the  group  from  another  room  by  means  of  a  two-way  mirror  or 
video  link.  In the  case  of  scientific  study, the  use of  this  approach  without 
the  knowledge  of  the  participants  poses  a number  of  ethical  problems  and 
is not therefore  recommended. 

Step  4.  Course  of the  discussion: The discussion  may  be  launched  fairly 
openly,  by  introducing  the  subject  of  the  session  and  asking  a  simple 
question  of  general  Interest.  This  will  enable  each  participant  to  give  an 
initial  opinion  or  a  remark  on  the  subject.  The  interview  guide  serves  to 
remind  the  facilitator  of  all the  essential  points  that  need  to  be  addressed. 
If all the  advantages  of the  interview  are to  be  maintained, there  must  be a 
written  report.  It  is  strongly  recommended  to  record  or  even  to  film  the 
interview,  although  the  latter  alternative  is  often  seen  as  an  "intrusion"  by 
the  participant.  Some  rooms  are  designed  in  such  a  way  that  the 
participants  do  not  notice they  are  being  recorded  or filmed. 

Step  5.  Analysis  and  report  on  results:  This  final  phase  consists  of 
interpreting  and  comparing  the  information  given  by  the  participants,  and 
looking  for  shared  and  divergent  opinions  within  each  group. The  informa­
tion  collected  is  codified  so  as  to  organise  the  results  in  relation  to  the 
objectives  of  the  survey. The  interpretation  of  data  must  take  into  account 
and  distinguish  two  major  aspects  of the  discussion: what  the  participants 
consider  as  interesting,  and  what  they  judge  as  being  important.  The 
analysis will depend on the  number  of focus groups questioned, and on the 
nature  of  the  interviews  (for example,  did  the focus  group  discussion  have 
a  structured  approach  or  not?).  The  results  from  the  different  groups  are 
compared  so as to  identify  any convergence  there may  be. The report  may 
quote  the  most  noteworthy  statements  made  by the  participants,  together 
with  a summary  of the  discussions. 

Application  to  the  Structural  Funds 

The focus  group  technique  may  be  used to  test  an  innovative  measure  (ex 
ante  evaluation),  clarify  the  objectives  of  a  project,  establish  a  theory  of 

Tools for observing changes in the field 

region  and 

action  for  the  programme  being  evaluated, and  identify  the  problems  and 
the 
needs  of  a 
implementation  of the programme. This technique  may also  be used at  the 
end of the  programme  in the framework  of an ex post  evaluation, to  collect 
information for  identifying  and/or  interpreting the effects  and impacts of the 
programme  concerned, and to fix  new  priorities  and  orientations. 

required  during 

improvements 

the 

Strengths  and  limits  in  practice 

•  This  type  of  survey  provides  in-depth  information  on  the  values  and 

opinions  of the  interviewees; 

•  The fact  of  bringing  several  persons  together  provides  a certain  balance 
in the answers given and makes it easier for the evaluation team to  define 
the general opinion on a particular  programme. Owing to the  participation 
of  several  persons,  the  focus  groups  provide  a  certain  "quality  control" 
over  data  collection,  by  judging  the  "pros  and  cons"  of  each  person's 
arguments  and thus  avoiding  extreme opinions. 

•  In a short  space of time (from  one  and  a half to two  hours),  it  is  possible 

to  collect  a large amount  of  qualitative  information. 

•  Specific skills are required for managing the group dynamic and  obtaining 
a  balanced  discussion  while  avoiding  the  dominant  influence  of  opinion 
"facilitators",  for  example  and  the  technique  of  time  and  resource 
intensive. There  is also  a risk  of  "polarisation", that  is to  say, that  at  the 
end  of  each  group  discussion the  participants  may  be  influenced  by  the 
majority  opinion  expressed  at the beginning  of the discussion. 

•  The  discussions  may  sometimes  be  biased,  due  to  the  fact  that  the 
participants  (addressees)  of  public  policies  are  subject  to  an  effect  of 
dependency  and will  produce  a positive judgement  a  priori. 

•  The  number  of  people does, however,  limit the  number  of  questions  due 
to  the  time  required  to  answer  them.  Moreover,  conflicts  may  emerge 
between  the  different  persons,  which  may  undermine  the  quality  and 
effectiveness  of the  interview. 

•  It  is  possible  that  participation  in  a  focus  group  changes  peoples' 
perceptions  -  either  because  of  the  'Hawthorne  effect'  (the fact  that  the 
behaviour  of  persons  who  know  themselves  to  be  under  observation 
changes)  or  because  their  interaction  with  other  participants  gives  them 
new  insights  and  perspectives.  Thus  for  example  programme  managers 

Principal  evaluation  techniques and tools 

may  actually  improve  their  performance  as  a  result  of  participating  in  a 
focus  group.  In  this  way  the  focus  group  methodology  may  therefore 
have  an  impact  on  the  programme  being  evaluated  or  on  successor 
programmes. 

Further  reading 

Anzieu  D,  Martin  J.Y.,  (1994)  "La  dynamique  des  groupes  restreints",  PUF 

Le  psychologue 
Complete  presentation,  by  psychosociologists,  of  the  concept  of  a 
limited  group,  the  main  group  phenomena  and  some  fields  of 
application. 

Aubel  J.,  (1992)  "Guide  pour  des  études  utilisant  les  discussions  de 

groupe",  BIT 
Practical  guide  to  conducting  group  interviews. 

Greenbaum,  TL.,  (1998) 

'The  Handbook 

for  Focus  Group  Research' 

(Second  edition).  London  : Sage  Publications  ; 

Krueger,  R.A.,  (1988),  'Focus  Group  :  A  Practical  Guide  for  Applied 

Research'.  London  : Sage  Publications. 

Moore  M. C,  (1994)  "Group  techniques  for  idea  building",  Sage 

A  more  detailed  approach  on  group  techniques,  presenting  specific 
techniques  (Nominal  Group  Technique, 
Interpretive 
Structural  Modeling,  etc.) 

Ideawriting, 

Morgan,  D.L.,  (1997)  'Focus  Groups  as  Qualitative  Research'  in  Qualitative 
Research  Methods  Series,  16. London  : Sage  Publications. 

Tools for observing changes in the field 

Case  study 

This  tool  is  based  on  an  in-depth 

study  of 

the  data  collected  on 

specific  cases. 

It  may  c o n c e rn 

individuals,  programmes,  organi­

sations,  projects,  groups  of  people 

or  decision-making  processes.  The 

a p p r o a ch 

uses 

a 

range 

of 

qualitative  and  quantitative  data 

collection 

techniques.  The  case 

study 

is 

r e c o m m e n d ed 

for 

O 

Overall 
evaluation 

A 

In-depth 
evaluation 

0 

Medium 
technical 
complexity 

O 

φ 

Medium 
cost 

Θ 

Retrospective 
use 

Use 
stabilised 

evaluations  aimed  at  a  detailed  analysis  of  c o m p l ex  processes  of 

implementation  of  programmes  and  the  representation  of  their  impacts.  The 

case  study  generally  applies  to  situations  that  require  an  exploratory  field 

study  due  to  the  lack  of  available  information. 

Example  : Evaluation  of  the  NordTek  programme,  Denmark 

This programme, intended for small and medium-sized firms, consisted  of about  100 
projects  offering  services  to  firms  in  the  areas  of  training,  consultancy,  access  to 
modern  computer  equipment,  etc.  It  was  therefore  a  programme  that  could  be 
qualified as  "soft", as opposed  to  "hard"  programmes  aimed at improving  infrastruc­
tures. The results of such programmes are difficult to evaluate precisely because they 
are not  tangible. 

Six  large  projects  were  selected  as  cases  to  be  studied.  In  terms  of  the  selection 
criteria, the  projects  had to  be  representative  of  the diversity  existing  in  intervention 
strategies  and  locality.  The  selected  projects  also  had  to  have  benefited  from 
significant  funding  in  the  framework  of  the  NordTek  Programme  and  be  oriented 
exclusively  towards  firms  (rather  than  research  institutions  or  administrations).  The 
six  projects  evaluated  represented  such  a  large  portion  of  the  total  Programme  that 
the  analysis  of  the  impacts  of  these  interventions  could  be  used  to  cover  all  the 
important  phenomena.  It  was  recognised  without  ambiguity  that  the  selected 
projects  represented  the  "best  cases"  of  the  programme. 

The data  were collected  at two  operational  levels: at the level of  the project  as such, 
and  at that  of the firms. As far  as the  project  was  concerned, the evaluators  studied 
each of  the cases  in the field four  or five times  over  a period  of two  years. Each time 
the managers were questioned  and each  project  had to  keep a record, in a  "register" 
designed  especially  for  the  purpose,  of  the  services  proposed  and  the  firms 
concerned. The analysis of the documents  thus  obtained  was  handed to  the  project 
leaders  and discussed  during  visits. As for  the firms, the  data  were collected  during 

Principal evaluation techniques and tools 

interviews with a sample of 20 SMEs that  had used the services offered. Finally, 40 
SMEs who had not used the proposed services were questioned in the final stage of 
the evaluation. 
Source: Olsen, Leif and Olaf Rieper. Nordtek-evaluieringen, AKF Forlaget,  1991. 

What  is the  purpose  of the  tool? 

The  case study  may  have the following  aims: 

•  illustration: the case study  is a tool that  may  be used to add  realism to an 
evaluation  if  it  is  presented  in  a  narrative  form. The  case  must,  however, 
be chosen  carefully  because  it  must  be  representative  of the  programme 
as  a  whole  or  illustrate  a  specific  point  -  for  example  a  particularly 
effective  action  or  an  approach  which  was  found  to  have  serious 
deficiencies  and which  should therefore  be  avoided  in future. 

•  exploration:  putting 

forward  hypotheses 

for  future 

investigations, 

identifying  the  various  points  of  view  of  the  stakeholders. 

•  critical analysis: verify and validate a statement  concerning  a programme, 

project  or  strategy. 

•  analysis  of  implementation:  examine  the  diffusion  of  services  and  its 

mechanisms,  often  in different  places. 

•  analysis  of  the  impacts  of  programmes:  understanding  the  nature  of  the 

processes  producing  impacts. 

The  results  of  a case  study  are always  presented  in  a  narrative  form,  as  a 
story,  thus  giving  the  reader  an  "inside  view"  of  the  case  studied  and  an 
impression  of  authenticity.  The  case  study  therefore  has  an  analytical  and 
communicative  aim.  Readers  are  more  likely  to  relate  to  cases  where  the 
programme(s)  and  personnel  involved  are  identified.  However,  particularly 
when  a case study  is  being  used to  illustrate  and  learn from  failures  it  may 
be  necessary  to  anonymise  some  or  all  of  the  material  in  order  to  secure 
access  to  data  and  personnel. 

Case  studies  can  often  be  designed  in  a  cumulative  way  to  help  answer 
evaluation  questions. The same case  programme  may also be studied  over 
time  to  provide  an  analysis  that  is  updated  iteratively.  Cases  may  be 
descriptive,  normative  or  designed  to  show  causality.  They  may  be 
particularly  useful  in  pedagogic/training  situations  for  example  being  used 
to  enable  officials  to  evaluate  alternative  evaluation  methodologies. 

Tools for observing changes in the field 

In which  cases  should  they  be  used? 

Case  studies  have  been  used  extensively  in  evaluation  during  the  past 
decade. Today this approach  is known to  provide valid  information for  both 
the  evaluation  of  programmes  and  the  diffusion  of  new  knowledge.  Case 
studies  which  use  sophisticated  selection  procedures  (e.g.  "multiple  case 
studies  with  replication  design")  tend  to  replace  large-scale  quantitative 
surveys  carried out  in diverse  cultural  contexts. 

its  use  is  appropriate  for  the 

The  case  study  is  a  method  of  holistic  analysis  applied  to  complex 
in-depth 
situations.  This  means  that 
understanding  of  behaviours  and  social  phenomena,  by  using the  persons 
and  organisations  analysed  as  a  frame  of  reference.  Case  studies  are 
valuable 
inductively,  by 
developing  assumptions  concerning  the  phenomena  linking  cause  and 
effect.  These  assumptions  must  then  be  supported  by  information  drawn 
from the different  case studies and tested through the search for alternative 
explanations. 

the  effects  of  programmes 

identifying 

for 

This  may  prove useful for observing  expected  results and impacts, but  also 
for  revealing  unexpected  ones.  The  approach 
is  less  suited  to  the 
identification  of  causal  links, although  it  may  be  used  to  demonstrate  that 
they  are likely  to  exist. 

The case study  is intended to  be the most  complete  illustration  possible  of 
a given  situation, so  as to  give  a precise  image of  current  phenomena  and 
to  understand  their  causes.  This  is  obtained  by  means  of  the  description 
and  then  the  analysis  of  examples  situated  in their  context.  It  follows  that 
this  type  of  analysis  must  be  based  on  multiple  data  sources,  such  as 
interviews, observations  over time, statistics, physical  information, etc. The 
data  must  also  be  cross-checked  in  order  to  ensure  its  coherence.  The 
notion  of  "context"  encompasses  all the  factors  that  could  affect  the  case 
studied.  Thus,  for  example,  the  impacts  of  a  specific  project  on  the 
addressees  are  influenced  by  a large  number  of  external  factors. 

How  should  it  be  implemented? 

The  quantity  of  work  required  by  a case  study  may  vary  widely.  One  must 
bear  in mind that the case study  must  be sufficiently  rich to  give the  reader 
an impression of what actually  occurred. However, the case study  is part of 
the  least  standardised  methods  and  may  encompass  a  range  of  different 

Principal evaluation techniques and tools 

approaches  in  different  situations.  Carrying  out  a  case  study  involves  the 
following  steps: 

Step  1.  Selection  of  cases  to  study:  There  are  at  least  three  criteria  for 
selecting cases: convenience/access, the purpose to which they are to be put 
and  the  extent  to  which  they  can  be  considered  to  provide  wider  insights 
beyond the particular  case in question. The selection of cases is a critical step 
for  generalising  and  answering  evaluation  questions.  It  is difficult  to  justify  a 
selection  based  only  on  convenience  (easy  access  to  data)  and  probabilistic 
surveys are sometimes difficult  to carry out. Thus, a choice based on  purpose 
is appropriate  in most  cases, as shown  in the following table: 

Which  questions  can  it  answer? 

Basis  of  selection 

What  happens  at  the  extremes? 
What  explains  these  differences? 

contrasting  cases 

What  explains  the effectiveness  of  a  project? 

the  best  cases 

Why  does  a project  not  function? 

How  can the  different  types  of  project  be 
compared? 

the worst  cases 

by  sub-sets 

Among  the examples  chosen  to  represent 
significant  variations,  what  happens  and  why? 

representative  cases 

On  a typical  site, what  happens  and  why? 

typical  cases 

In these  specific  circumstances,  what  happens 
and  why? 

particular  cases 

(Adapted from GAO, 1990, p.23) 

Step 2.  Data  collection and  process: Theoretically,  data collection  covers 
all  available  information  about  a  case  including  that  derived  from  project 
documents,  project  meeting  reports,  and  collected  at  the  various 
operational  levels:  interviews  with  project  leaders  and  staff;  observation  of 
the  site  of  the  project;  surveys  among  the  addressees  of  the  services 
provided  by  the  project.  These  data  must  be  collected,  recorded 
(compilation  of  a  "register")  and  pieced  together  so  that  they  can  be  used 
in the final  report. 

Step  3.  Case  report:  Drawing  up  the  report  on  the  case  involves  the 
organisation  of  all  the  raw  data  on  the  case  into  a  body  of  exploitable 
information.  This  is  then  edited,  redundant  information  is  eliminated,  and 
the  different  parts  are combined. The  report  is organised  in such  a way  as 

Tools for observing changes in  the field 

to be easy to consult, either chronologically orthematically. The report  must 
include  all the  information  required  for  subsequent  analysis, that  is to  say, 
for  constructing  an account  of the  case study. 

Step  4.  Account:  The case  monograph  should  give  the  reader  immediate 
access  to  relevant  information  and  to  the  particular  situation  of  the  case  -
the situation  of a project  - and provide an understanding  of the project  as a 
whole.  Each case  study,  in an evaluation  report,  must  be  isolated  (the  size 
may vary between one and five pages). Nevertheless, in the last steps of the 
analysis  the  cases  may  be  used  as  contrasts  or  comparisons,  depending 
on the evaluation  objectives. 

Application  to  Structural  Funds 

The multiple case study  method  is particularly well suited to analyses of the 
various  member  States  and  regions,  but  also  to  thematic  evaluations.  The 
flexibility  of  each  case  study  makes  it  possible  to  draw  up  an  adequate 
description  of the peculiarities of a given  place or a project. The formulation 
of  a  common  set  of  questions,  relative  to  the  evaluation,  facilitates  the 
analysis  of  the  results  obtained  from  multiple  case  studies.  In  fact  the 
results  prove  to  be  more  sound  when  they  are  produced  in  relation  to  a 
variety of  places (through  re-using case studies). Similarly, the specificity  of 
success  stories  or  failures  will  then  seem  more  obvious.  It  should  be 
recalled  that the transversal  analysis  of  cases  consists  of  cross-referenced 
qualitative examinations  and  a description  based  on the frame of  reference 
established  by  the  evaluation  questions.  A  cumulative  process  may  be 
sought  when  the  evaluation  is  focused,  for  example,  on  operational 
programmes  in  Objective  1 regions  and  when  the  conclusions  have  to  be 
synthesised  on the scale  of several  member  States. 

The presentation of the  results of  several  case studies could  be a barrier  to 
more generalised  use. This  difficulty  may  be  solved  by  means  of  a graphic 
summary  providing  a  brief  report  of  the  case  history  and  a  graphic 
presentation  of the  results, in relation to  each of the questions.  In this form, 
the answer to each of the evaluation  questions, for each case, is set out  on 
a  single  page: a graphic  presentation  at the top,  a short  but  rich  summary 
of  the  case  history,  with  the  main  results,  and  a  concise  conclusion.  The 
summary  of  the  transversal  case  could  be  sketched  in  the  same  way, 
followed  by conclusions  and  recommendations. 

Principal  evaluation  techniques and tools 

Another  type  of  case  study  that  may  be  applied  to  the  Structural  Funds  is 
the  "integrated  approach". This  approach  takes  into  account,  for  example, 
the  study  of  results  and  impacts  in the context  of  a specific  programme.  In 
the  NordTek  example  mentioned  above, for each  of the  projects,  a  number 
of  addressees  (SMEs)  were  selected  for  a  mini  case  study.  In that  way  the 
report  by the SMEs on their  use of the  results obtained could  be used in the 
context  of  the  project.  Thus,  variations  in  the  results  among  the  different 
SMEs  can  be  explained  by  differences  in  the  situation  of  the  SMEs 
concerned. 

Strengths  and  limits  in  practice 

•  The  case  study 

is  relevant 

for  giving  a  view  of  processes  and 
complexities  that  are  impossible  to  see  in  any  other  way.  It  may  even 
make  outside  persons,  such  as  European  managers  who  are  hardly 
involved  in this field,  aware of the  reality of daily  actions. It provides  them 
with a clearer view of the way  in which the programme  is put  into  practice 
once  the  decision  has  been taken  with  the  national  authorities. 

•  Depending  on the  selection  strategy  opted  for,  if several case studies  are 
carried  out,  it  may  prove  impossible  to  extrapolate  beyond  the  results  of 
those  case  studies.  It  is  however  impossible  to  generalise,  in  the 
statistical  sense,  the  results  to  a  global  population;  to  go  from  a  few 
projects  to  a programme,  or from  a few addressees to  all the  addressees 
of  a certain  category. 

•  This approach  is less appropriate for  measuring the amplitude  of  impacts 

or  for  inferring the  causality. 

•  Due  to  the  cost  of  setting  up  a  good  case  study  (requiring  sources  of 
multiple  data  and  competent  evaluators),  it  is  necessary  to  limit  the 
number  of  observations.  The  case  study  may,  however,  be  re-used  and 
applied  to  other  contexts,  thus  providing  economies  of  scale. 

•  The  credibility  of  the  results  of  the  case  study  is likely  to  be  undermined 
if  the  method  is  not  implemented  correctly:  incompleteness,  arbitrary 
selection of information, comments  cut  short, distortion of  results, etc. To 
enhance  the  reliability  of  a  case  study,  several  precautions  are 
recommended,  e.g.:  rereading  of  the  case  studies  by  the  persons 
concerned  in order to verify the  precision  and the veracity of the data and 
their  interpretation;  or  having  two  different  evaluators  write  down  their 

Tools  for observing changes in the field 

comments  on  the  same  case;  involving  outside  professionals  (such  as 
journalists)  in the writing  of the  comments. 

Further  reading 

Dufour  S.,  Fortin  D.,  Hamel  J.  (1994),  'L'enquête  de  terrain  en  sciences 
sociales:  l'approche  monographique  et  les  méthodes  qualitatives', 
Montréal:  Saint-Martin 

GAO (1990), 'Case Study  Evaluations.'  Washington  DC: General  Accounting 

Office,  133  pp 
Sound  general  approach to  the  method. 

Yin  R.K.  (1994),  'Case  Study  Research  -  Design  and  Methods',  2nd  ed. 

Newbury  Park, Sage  Publications 
Complete  guide to  the case study  technique. 

Tools for  analysing  data 

An  important  function  of  evaluation  is  the  comparison  of  primary  and 
secondary  data.  In  the  situation  concerned,  these  data  are  generally 
numerous  and  heterogeneous,  making  the  use  of  sophisticated  tools 
difficult.  Most evaluations use classical tools such as tables or graphs (see, 
for example:  Henry G.T. ed., 1997, "Creating  Effective  Graphs: Solutions for 
a Variety  of  Evaluation  Data", New  Directions  for  Evaluation,  no.  77). 

An  innovative  and  particularly  useful  technique  in  this  situation  is  the 
geographical  information  system which  can be used to compare data  by 
considering  their  location.  The  maps  produced  by  means  of  this  tool  are 
used to  make visual  comparisons  that  are often  highly  informative. 

Owing to  its flexibility, the  case  study technique,  mentioned  above, is  one 
of  the  rare techniques  allowing  for  an  exhaustive,  although  qualitative  and 
non-graded,  analysis  of  all the effects  of  a complex  programme. 

the 
Most  quantitative  analysis  tools  are 
heterogeneity  of  data  and  the  multiplicity  of  publics  and  processes  of 
production  of  impacts.  For example, it seems very difficult,  in this situation, 
to  use  tools  such  as  factor  analysis,  regression  analysis  or  comparison 
groups. 

inappropriate  because  of 

On the  other  hand,  macro-economic  tools  are  appropriate  in  cases  of  vast 
programmes  covering  an  entire  territory,  provided  that  the  situation  meets 
the  following  criteria.  The  funds  spent  in the  framework  of  the  programme 
must  be  significant  compared  to  the  economy  of  the  territory  concerned. 
This  territory  must  be  large  enough  (a country  or  a  large  region),  or  closed 
enough (an island) for the global functioning of its economy to be considered 
in  isolation.  When  these  conditions  are  met,  the  evaluation  team  can  use 
several tools  borrowed  from  regional economics  or  macroeconomics. 

Shift-share  analysis  consists  of  projecting  national  economic  trends  onto 
the economy  of a region. This tool is used to estimate a policy-off  situation. 
By  comparing  this  with  the  policy-on  situation,  the  global  impact  of  the 
programme  can  be evaluated. 

The  input-output  model  and  the  macro-economic  model  are  used  to 
simulate the economic  development  of the region. These tools are generally 
applied  ex ante to two  types  of assumption  (with /  without  the  programme) 
and  provide  an overall  estimation  of  probable  macro-economic  impacts. 

Tools for analysing data 

Geographical  Information 
Systems 

O 

Φ  © 

High 
cost 

Overall 
evaluation 

High 
technical 
complexity 

Geographical  information  systems 
(GIS)  were  originally  developed  in 
the  1970s  to  stock  and  process 
information  related  to  the  physical 
environment  (natural  and  man-
made).  More  recently,  the  interest 
of  GIS  users  has  spread  to  socio­
economic 
information.  GIS  have 
thus  become  a way  of  assembling 
any  data  that  can  be  referred  directly  or  indirectly  to  a  geographical 
location. They are now used in the field of spatial planning to collect, gather, 
accumulate,  analyse,  exploit,  display  and  update  all  spatially  referenced 
data and  information. 

©  G 

© 

Retrospective  Prospective 

Innovative 
use 

use 

use 

in  computer 

technology,  particularly  computer-assisted 
Advances 
cartography,  have led to the development  of  high-performance  information 
systems. The management  of  public  interventions  is thus able to  cope  with 
a considerable  number  of  interdependent  factors,  so as to  better  deal  with 
the  complexities  of  the  real  world. Thanks  to  GIS, these  complexities  can 
be  described  and  analysed  by  combining  information  from  the  thematic, 
spatial and temporal  dimensions. 

Example: Evaluation of the Languedoc-Roussillon  IMP in France 
This example corresponds with that of Box 12. 

The GIS technique was used in an evaluation of the Languedoc-Roussillon Integrated 
Mediterranean  Programme  (IMP),  primarily  to  analyse  its  spatial  effects.  This 
particular tool was chosen because the evaluation was  required to account for the 
territorial  synergy  of  a  programme  composed  of  multiple  actions  consisting  of 
subsidies granted in selected areas. 

A wide range of fields was affected by this programme. These concerned problems 
of regional development such as the maintenance, diversification and adaptation of 
agriculture, the creation of new jobs, the development of tourism, and the protection 
of  the  environment,  within  spatial  (in  defined  geographical  zones),  thematic  (by 
economic activity) and temporal (reference to the initial state) perspectives. 

To assess the impact of the IMP measures and the effects of synergy, it was necessary 
to observe the changes brought about by the programme and then to explain them. 

Principal evaluation techniques and tools 

For this  purpose  an  "initial  state"  was  constituted  to describe the reference situation. 
This  step  required  a  substantial  effort  gathering  all  the  data  and  indicators  of  the 
socio-economic  structure  in  the  territory  under  consideration  -  an  effort  that  was 
essential  in order  to  be  able  to  make  a comparison  with  the  situation  observed  after 
the  allocation  of  the  subsidies.  Once  the  data  base  had  been  compiled,  the 
evaluators,  using  mainly  geographical  representations,  attempted  to  verify  if  the 
actions  undertaken  respected  the  logic  underpinning  geographical  location,  spatial 
integration  and spatio-temporal  integration. 

The  use  of  cartographic  representations  associating  material  and  relational  data 
(social,  political and economic)  is relevant when the  programme  itself  is based on the 
identification  of  sensitive  areas within  a given  territory.  Thus, for  example,  actions  to 
prevent  forest  fires  (reforestation,  protection)  were  correlated  cartographically  with 
the  areas  prone  to  fires  (frequency)  in  order  to  assess  the  logic  behind  their  spatial 
integration. 

While representation  on a map  is relevant  in this case, it should  be recalled that  very 
often  the  understanding  of  a  map  is  not  immediate  and  the  results  require 
interpretation.  Initially, only the actions  of the  programme  in question  and the  current 
year  were  registered.  Actions  of  the  same  type  undertaken  by  other  organisations 
between  the  time  of  the  "reference  situation"  and the  "observed  situation"  were  not 
recorded. This  led to  a biased  interpretation  of the  real situation. 

Source:  Commission  européenne  DG  XVI  (1993)  L'évaluation  du  PIM  Languedoc-
Roussillon, MEANS  pilote. 

W h at  is  t he  p u r p o se  of  t he  tool? 

In  general,  GIS  are  used  to  gather  all  sorts  of  information  of  a  spatial  nature 

and  to  present  it  in  a  form  that  makes  it  easy  to  read.  GIS  is  able  to 

incorporate  s o c i o - e c o n o m ic  data,  data 

from  censuses,  surveys  and 

inquiries,  and  monitoring  data.  The  spatial  representation  may  be  direct 

(cartographic  co-ordinates,  precise  postal  address)  or  indirect  (postal  code, 

area  of  census). 

In  terms  of  evaluation,  the  main  a d d ed  value  of  a  GIS  is  that  it  allows  the 

cross-referencing  of  data  that  other  techniques  do  not  allow.  In  particular, 

the  tool  can  c o m p a re  information  on  which  the  geographical  references 

differ,  e.g.  address  of  firms  assisted  and  urban  areas  in  difficulty,  or  areas 

eligible  for  assistance  and  environmentally  sensitive  areas.  A  GIS  also 

makes  it  possible  to  estimate  the  value  of  an  indicator  for  a  given  area, 

when  its  vasue  is  registered  on  different  statistical  scales.  (This  point  is 

discussed  in  detail  in  Volume  2  of  this  Collection,  devoted  to  indicators.) 

Tools for analysing data 

In which  cases  can  it  be  used? 

Constructing  a GIS is a heavy investment, especially  in terms of data  input. 
It  is therefore  difficult  to  make  such  an  investment  in the  framework  of  an 
evaluation  with  a  limited  budget  and  duration.  Such  an  investment  is, 
however, justified  in the context  of  monitoring. 

If a  GIS  already  exists,  an  evaluation  would  be  well  advised  to  use  it. The 
collection  of  evaluation  data  has  to  include  systematic  geographical 
references,  so  that  cross-referencing  with  other 
information  already 
contained  in the  system  can  be performed. 

How  should  it be  implemented? 

GIS  use a  basic  structure  of  special  data. All the  information  is  linked  to  a 
system  of  geographical  co-ordinates  characterised  by  nodes,  lines  and 
areas  (and therefore  also of  points,  arcs,  polygons, etc.). 

Box  16 -  Principle  of a Geographic  Information  System 

Topography 

Properties 

Protected  areas, 
forests, etc. 
Density  of  population, 
Constructed  areas, etc. 
Fire protection  projects 

Resulting  map 
(selecting  and 
combining  variables) 

The different  steps  in a GIS are as follows: 

Step  1.  Collection  of  basic  data:  The  first  phase  is  the  collection  of 
information.  The  basic  data  are  taken  from  a  wide  variety  of  sources: 

Principal evaluation  techniques and tools 

géomorphologie  maps,  aerial  photographs,  existing  socio-demographic 
data,  etc.  In  some  cases,  when  the  raw  data  cannot  be  saved  directly  on 
computer  (e.g. a sample  bank),  it  is recorded  by  means  of  indicator  cards. 
This  step  includes  the  digitisation  and  inputting  of  geographical  data  and 
associated  alphanumerical  data. 

Step  2.  Storage  and  management  of  geographical  data  base:  This 
function  includes  the  storage  of  data  in  an  alphanumerical  form  and  the 
programming  of  recall  mechanisms  to  meet  the  needs  of  the  subsequent 
steps, such as users'  requests. The idea is to organise the  data (creation  of 
an  index)  to  facilitate 
thematic, 
topological, etc.  criteria. 

its  access  based  on  geographical, 

Step  3.  Exploitation  and  analysis  of  data: A  wide  range  of  techniques  is 
needed for the transformation  and use of data in the framework of a specific 
problem. The user proceeds  either by "questioning", that  is to say, he or she 
asks  the  GIS  a question  directly,  or  by  "extraction"  when  he  or  she  wants 
to  compile  a partial file for a specific  purpose.  Moreover, the GIS allows  the 
combination  of  data  and  the  identification  of  correlation  between  certain 
variables  with  a  view  to  highlighting  the  characteristics  of  a  geographical 
area. 

Step  4.  Rendering  results:  A  GIS  comprises  different  graphical  systems 
allowing  for  the  organisation  of  data  in  models,  such  as  maps,  graphs, 
pictures, projections  of sequences, etc. These facilitate communication  and 
allow  for  user-friendly  exploitation  of the  data. 

Application  to the  Structural  Funds 

This  tool  has  considerable  potential  for  integrating  and  synthesising 
information. Its ability to  integrate the  territorial  dimension  is relevant for the 
Structural  Funds.  It  is  used  to  centralise  results  from  statistics  offices, 
monitoring, and  evaluation  surveys  carried  out  in the  eligible  territory. 

information  relating  to  the 
A  GIS  also  allows  easy  visualisation  of 
the 
programme.  This  characteristic  can  be  extremely  useful  during 
presentation  of  the  results  to  the  various  committees  and  work  groups 
concerned. 

This potential for  synthesis facilitates the  illustration of the coherence of the 
measures  implemented  in  the  framework  of  the  programmes,  something 
that  is an asset  for  the  publication  of  synthesis  documents. 

Tools  for analysing data 

Strengths  and  limits  in  practice 

•  Above  all, GIS have a didactic  faculty:  results  can  be rendered  in a  user-
friendly  way, which  helps  to  enhance  the  awareness  of  actors.  However, 
cartographic  language  is  not  univocal;  signs  and  keys  are  codes  that 
vary, depending  on the  designers,  and  need  to  be explained. 

•  Apart from the fact that the creation of a complete geographical  database 
often  proves  to  be costly  in  both  time  and  money,  the  use of  a GIS  may 
be complex.  Existing  data  are often  collected  on different  spatial  entities 
(administrative  partitioning  into communes,  cantons, etc.) which  requires 
time  consuming  transformations  or  sometimes  even  a  new  collection  of 
data.  Existing  data  may  also  be stored  on  information  systems  that,  due 
to their  incompatibility,  make  it  impossible  to  use them. 

•  Before  being  integrated  in a GIS, the  data  produced  by  monitoring  must 
that  may  be 

techniques 

be  processed  by  means  of  statistical 
cumbersome. 

•  In practice, GIS facilitate the  processing  of  a substantial  number  of data. 
It  also  has the  advantage  of  allowing  the  transmission  of  data  from  one 
user to another. The centralising  nature of the tool facilitates the  updating 
of  data  and  allows  for  the  manipulation  of  geographical  information  on 
varying  scales  (e.g. from  a small area of  land to  an entire  region). 

•  GIS  are  also  useful  when  they  are  combined  with  other  analysis 

techniques  such  as factor  analysis. 

•  Experience  tends  to  show  that  the  tool  may  generate  conflict  between 

territorial  officials  due to  their  biased  reading  of  the maps  produced. 

Further  reading 

Collet  C. (1992), Système  d'information  géographique  en mode  image,  coll. 
"Gérer  l'environnement",  Lausanne:  Presses  polytechniques  et 
universitaires  romandes.  186  p. 
Simple  but  very  effective  for  an  introduction  to  GISs. 

Fisher  M.  and  Nijkamp  P. (1993),  Geographic  Information  Systems,  Spatial 

Modelling  and  Policy  Evaluation, Berlin: Springer-Verlag. 280  p. 
Volume  comprising  background  articles  with  a  specific  focus  on 
applications  to  evaluation. 

Principal  evaluation  techniques and tools 

Martin  D.  (1996),  Geographic 

Information  Systems,  socioeconomic 

applications,  London:  Routledge. 
Introduction  to  GISs  with  a  specific  focus  on  socio-economic 
applications. 

Tools for analysing data 

O 

Overall 
evaluation 

S h i f t - S h a re  a n a l y s is 

Shift-share  analysis  has  been  used 

since 

the  1960s 

for  evaluating 

regional  policies.  It  is  a  t o p - d o wn 

evaluation  technique  in  so  far  as  it 

projects  observable  trends  at  the 

level  of  a  country,  o n to 

the 

economy  of  a  region.  The  starting 

point  of  a  shift-share  analysis  is  not 

the  study  of  the  consequences  of  a 

p r o g r a m m e,  but 

the 

o b s e r v ed 

Θ  Φ 

Low 
technical 
complexity 

Low 
cost 

Ο  Θ 

Retrospective  Stabilised 

use 

use 

trends  at  the  national  and  regional  levels.  Shift-share  analysis  focuses  only 

on  macro-economic  dimensions,  to  construct  a  basis  for  comparison  of 

what  would  have  happened  without  the  programme. 

Example:  Employment  impact  of  regional development  policies  in  Scotland 

100 

-200 

This  graph 
represents  a 
difference  in 
thousands of  jobs 
between the  policy-
on situation  and the 
policy-off  situation 

A positive value corresponds to a regional 
creation of jobs greater than  forecasts 
based on national trends. The  difference 
(residue)  is attributed  to the effects of 
regional  policy. 

The case  of  Scotland  is  a classical  example  of  shift-share  analysis.  It  is  particularly 
meaningful  because  it  presents  two  clearly  distinct  phases  in  regional  policy:  a 
passive phase during the  1950s, during which there was  no regional  policy,  followed 

Principal evaluation techniques and tools 

by  a  more  active  phase  during  the  1960s  when  the  British  government  provided 
abundant  support  for  the  development  of  Scotland. The application  of  shift-share 
analysis to the evaluation of British regional policy in Scotland was used to draw the 
graph above. 
It  can  be seen that,  in the  1970s, jobs  in  Scotland tended  to  increase  faster than 
forecasts  based  on  national  trends  suggested. This  leads  to  the  conclusion  that 
regional policies had a positive impact on employment  in this region. 
Source: Diamond D., Spence N. (1983), Regional Policy Evaluation. A Methodological 
Review and the Scottish Example, Gower House: Ashcroft 

What  is the  purpose  of the  tool? 

Shift-share  analysis  is used to  give an indication of the effects  of a policy  in 
a given  region. The effect  of the  policy  is determined  by comparing the  real 
situation  in  the  region,  i.e.  the  policy-on  situation,  with  a  reconstruction  of 
what  the  regional  situation  would  have  been  like  had  it  followed  national 
trends,  i.e.  a  policy-off  situation. 

In which  cases  can  it  be  used? 

Shift-share  analysis  is designed  to  evaluate the  effects  of  regional  policy  at 
a  macro-economic  level.  Used  for  retrospective  analysis,  it  is  based  on 
statistical  indicators  and  requires  no  survey  on  the  target  groups  of  the 
programme. 

How  should  it  be  implemented? 

Shift-share  analysis  consists  of  two  steps:  the  construction  of  a  policy-off 
situation,  and  the  comparison.  Both  steps  are  illustrated  by  means  of  an 
analysis  in  terms  of  employment  (the  most  frequent  use  of  shift-share 
analysis). 

Step  1. Construction  of  the  policy-off  situation: this  is  based  on  simple 
modelling.  The  national  employment  trends  are applied  to  the  structure  of 
regional  employment.  The formulation  of  this  procedure  is as  follows: 

IM, =  IM0 +  NS 
IM  characterises  the  sectoral  distribution  of  employment  in  the  region  (i.e. 
the  industry  mix)  and  NS  is the  variation  of  the  employment  that  would  be 
produced  if  each  regional  sector  developed  at  the  pace  observed  for  this 
sector  nation-wide  (i.e. the  national  share  component). 

Tools  for analysing data 

Step 2. Comparison for  determining  the  impact  of the  action: This  step 
consists  of  the  comparison  between  the  policy-off  situation,  obtained 
during  the  preceding  step  (IMt) and  the  policy-on  situation. The  difference 
between  these two  situations  (the residue)  is used to  highlight  the  impacts 
of the  political  action. 
RS =  E, - IM, 
Here,  the  programme  impact  is  equivalent  to  the  residue  (i.e. the  regional 
shift  component). 

Box  17 -  Example  of shift-share  analysis  of  regional  employment 

The  number  of  jobs  in  a  region  in which  the  public  authorities  provide  e m p l o y m e nt 

incentives  is  300,000  ( I MJ  in  which  100,000  are  industrial  jobs,  150,000  are  in  the 

service  sector  and  50,000  are  in  agriculture  and  the  f o od  sector,  during  the  period 

t  =  0. 

Nationally,  employment  trends  ¡n  the  industrial,  tertiary  and  agricultural  sectors 

are 

- 1 %,  3%  and  - 2%  respectively  between  t  =  0  and  t  =  1.  Assuming 

that 

regional  employment  developed  at  the  same  pace  as  national  employment,  the 

increase  in  regional  employment  w o u ld 

increase  by  2,500  (NS).  The  level  of 

employment  in  the  policy-off  situation  w o u ld  then  be  302,500  (IM,). 

By  contrast,  during  the  period  t  =  0  to  t  =  1,  employment  in  the  assisted  region 

progressed  by  5 %,  i.e.  15,000  jobs.  The  number  of  jobs  w as  then  315,000  (E,). 

According 

to 

this  analysis, 

the  number  of  jobs  created  as  a  result  of 

the 

intervention  is  12,500  (RS). 

Application of national trends 
to the  region 

I M0  = 
300  000 
jobs 

IM-,  = 
302 500 
jobs 

315 000 
jobs 

RS  = 
12  500 

Residue 

Impact 

Distribution  per 
sector  of 
employment  in 
the  region 

Service 
sector 
+  3% 

Industrial 
sector 
-  1% 

Agricultural 
sector 
- 2% 

Before the 
programme 

Policy-off  situation  Policy-on  situation 

After  the programme 

Principal  evaluation  techniques and tools 

Application  to  the  Structural  Funds 

Because  shift-share  analysis  was  designed  to  evaluate  the  effects  of 
regional  policies  it  is,  theoretically,  directly  applicable  to  the  Structural 
Funds.  It  can  therefore  be  used  in  ex  post  evaluations  of  regional 
development  programmes,  whenever  the  emphasis  is  on  the  study  of  a 
particular  impact  such  as employment  or value added. Shift-share  analysis 
is appropriate  for  use  in programmes  implemented  in the framework  of  the 
ERDF, due to  their  regionalised  nature. 

Because this technique  has few requirements, it has the advantage of  being 
easy to  use. The statistical  data  it  requires  are available  for  most  European 
regions,  particularly  in  the  form  of  employment  data  and  value  added  for 
NUTS  1 and  2,  provided  by  EUROSTAT. 

Strengths  and  limits  in  practice 

•  Shift-share  analysis  is  used  to  obtain  quantified  conclusions  on  the 

effects  of  policies. 

•  The  implementation  of a shift-share  analysis  is not standardised  (dates of 
start  and conclusion, sectoral  breakdown, national  reference); as a result 
it  is difficult  to  compare  the  results  of  analyses  of  different  programmes. 

•  It  is  an  inexpensive  tool  because  it  can  use  widely  available  statistical 
data.  All  that  is  needed  is  an  impact  indicator  (employment  or  value 
added), with  a sectoral  breakdown  at the  regional  and  national  levels. 

•  The  logic  of  the  implementation  of  a  shift-share  analysis 

is  often 
simplified  to  the  extreme.  Usually,  the  variations  observed  in  regional 
employment  can be explained  by factors other than the simple  projection 
of  national  trends.  Among  such  factors  are  the  growth  of  the  region's 
exports,  productivity,  quantification  of  manpower,  population-growth 
pressure,  immigration,  and  the  intensity  of  research  and  development.  It 
is not  credible to  attribute  the effect  of  all these factors to  regional  policy 
only,  unless  this  policy  had  a  very  strong  hold  over  all  the  regional 
economic  agents. 

•  In  shift-share  analysis  the  impact  consists  of  the  residue,  which  is  the 
difference  between  the  policy-on  and the  policy-off  situations.  However, 
the  residue  may  be  attributed  both  to  the  programme  under  evaluation 
and  to  all other  national  and  European  political  actions,  as  well  as to  the 
former  trajectory  of  regional  policy.  In theory,  shift-share  analysis  can  be 

Tools  for analysing data 

applied  more  satisfactorily 
if  a  very  far-reaching  regional  policy  is 
evaluated at the end of  a period characterised  by the absence of  regional 
policy. 

Further  reading 

Bartels C.P.A. (1982), Measuring  Effects of  Regional Policy. An introduction, 

Regional  Science  and  Urban  Economics  12, 41 p. 
A  review  of  all  techniques  used  to  analyse 
impact  of 
employment  effects. Shift-share  analysis is addressed from a critical 
point  of view.  Highly  accessible. 

the 

Steevens  B.H.,  Moore  C.L.  (1980),  A  Critical  Review  of  the  Literature  on 
Shift-Share  as  a  Forecasting  Technique,  Journal  of  Regional 
Science,  Vol. 20, N°4, pp.  420-435 
A  highly  formalised  article  presenting  the  main  shift-share  models 
and their  possible  uses  in forecasting. 

Tervo  H.,  Okko  P. (1993),  A  Note  on  Shift-Share  Analysis  as  a  method  of 
estimating  the  employment  effects  of  regional  economic  policy, 
Journal  of Regional  Science,  Vol. 23, N°1, pp.  115-121 
A critical article on the use of the shift-share technique for  evaluating 
regional  employment  effects.  Based on formalised  demonstrations. 

Tools  for analysing data 

Input-Output  Model 

I n p u t - o u t p ut 

matrices 

were 

originally  d e v e l o p ed 

to  analyse 

interaction  between 

the  different 

sectors  within 

relatively  c l o s ed 

economies 

(low 

level  of  exports 

and 

imports).  They  describe 

the 

way 

in  w h i ch 

the 

productive 

system 

satisfies 

final 

d e m a nd 

(consisting 

of 

c o n s u m p t i o n, 

investments  and  exports).  An 

O 

Overall 
evaluation 

0 

High 
technical 
complexity 

O 

φ) 

High 
cost 

Θ 

Prospective 
use 

Use 
stabilised 

input-output  matrix  represents  the  links  between  an  economy's  resources 

and  their  consumption.  The  matrix  may  vary  from  the  simple  (three  sectors: 

industry,  services  and  agriculture)  to  the  complex  (over  500  branches).  It  is 

one  of  the  only  techniques  applicable  to  the  evaluation  of  the  sectoral 

impacts  of  structural  interventions,  because  it  makes  it  possible  to  divide 

up  the  productive  structure  of  an  economy  in  great  detail.  An  input-output 

matrix  can  be  compared 

to  a  macro-economic  model 

that 

is  highly 

simplified  as  regards  the  economic  mechanisms  represented,  but  extremely 

detailed  from  the  sectoral  point  of  view. 

Example: The  impact  of  Structural  Funds  on  Objective  1 countries  and  regions 

A  team  of  German  academics  adapted  input-output  analysis  to  the  study  of  the 
impact  of  Structural  Funds  on  Objective  1  countries  and  regions.  It  used  matrices 
provided by EUROSTAT for the reference year  1985 and forecasts  for the years 1991, 
1994  and  1999.  In  this  example,  the  team  tried  to  estimate  the  impact  on  overall 
economic  growth  and  on  sectoral  distribution.  The  following  results  are  the  most 
representative  ones from this  study: 

Without  the  Structural  Funds,  average  economic  growth  in Objective  1 areas  would 
have  been less than 0.7%  in absolute terms. 

In  1999, the Structural  Funds  are expected  to  induce a  1.9%  growth  in  employment 
in the Objective  1 areas. 

The analysis was  differentiated  for  the  different  countries  and  effects  on  growth  and 
employment  were shown to  be greater  in Greece  and Portugal. 

For  the  period  between  1994  and  1999,  changes  in  the  sectoral  structure  of  the 
production  (in terms  of  GDP) of  these  regions  can be  represented  as  follows: 

Principal  evaluation  techniques  and tools 

19.0% 
119.9% 

19.0% 
19.1% 

Public  sector 

Services 

Building  and 
civil engineering 

l9.6% 
_J9.1 % 

Manufactured 
products 

Energy 

Agriculture 

I5.594 
|5.7% 

u 

.2% 
5.2% 

ZJ42.8 
_ | 4 1 . 0% 

D  1999 
D  1994 

0% 

5% 

10%  15%  20%  25%  30%  35%  40%  4 5% 

Source:  J.  Beutel,  Konstanz  University 

What  is the  purpose  of  the  tool? 

Input-output  matrices  are  used  primarily  in  economic  forecasting,  where 
they  serve  to  verify  policy  scenarios,  based  on  the  technological  structure 
of  the  economy  of  the  country  and  on  the  state  of  final  demand.  In  an 
evaluation  they  can  be  used  with  or  without  policy  interventions,  in  the 
same way that a macro-economic  model  is used (see corresponding  sheet). 
The tool is used to estimate impacts  differentiated  according to the  different 
branches of an economy. Among the numerous applications of  input-output 
matrices  to  the  evaluation  of  development  programmes,  the  above-
mentioned  example  will  be  considered  because  it  has  the  advantage  of 
presenting  harmonised  findings  on  a  European  scale. 

The  model  was  produced  by  Konstanz  University  and  EUROSTAT.  It  was 
used for estimating the net  effects  of the main  programmes financed  by the 
Structural  Funds. 

In which  cases  can  it  be  used? 

In general, this  tool  is  used  on  a  national  scale,  because  this  is  where  the 
statistical  data  needed  for  the  construction  of  matrices  are  collected. 

Tools  for analysing data 

Regional  matrices  do  exist  and  also  lend  themselves  to  input-output 
analysis.  However,  their  statistical  value  is  not  as  great  because  of  the 
difficulty  of  monitoring  the  movement  of  goods  that  are  not  subject  to 
customs  duty. 

How  should  it be  implemented? 

The  following  lines  describe  the  implementation  of  the  tool  from  its 
construction  to  effective  simulations  in  the  context  of  the  evaluation.  In 
reality,  an  input-output  matrix  is  rarely  constructed  for  an  evaluation.  The 
implementation  of the tool  starts, in  practice, with  step 4. 

Step  1. Construction  of  a  transaction  matrix:  the  connections  between 
the different  branches of an economy  are described. The lines of this  matrix 
consist  of the outputs, which concern  resources supplied  by a given  sector 
to  each  of  the  sectors  of  activity  and  to  the  final  demand.  The  columns 
consist  of  the  inputs  from  the  different  sectors,  i.e.  the  consumption  per 
sector  required for  production. 

Outputs 
Inputs 

Agri­
culture 

Industry 

Services 

Final 
demand 

Inter 
mediate 
consump­
tion 

250 
400 
400 

100 
320 
600 

Total 
utilisa­
tion 

350 
720 
1000 

1050 

1020 

2070 

50 
100 
200 

350 

650 

1020 

1000 

2070 

Agriculture 
Industry 
Services 

Intermediate 
products 
Value 
added 

50 
150 
100 

300 

50 

Production 

350 

150 
150 
100 

400 

320 

720 

The  values  taken  in  this  example  are  monetary.  Thus,  for  example,  the 
agricultural  branch  provides  a value  of  150  to  the  industrial  branch  for  its 
production,  whereas  a  value  of  100  is  intended  for  the  final  demand.  The 
industrial  branch  consumes  a  value  of  150  in  intermediate  products  from 
the  agricultural  branch  for  its  production,  leading  to  the  creation  of  value 
added  of  320.  By  adding  the  intermediate  consumption  of  agricultural 
products  by  the  different  sectors,  to  the  final  demand  for  the  agricultural 

Principal  evaluation  techniques and tools 

branch, one  obtains the total  utilisation  of  agricultural  production. The  final 
demand may itself  be broken down into final consumption, investments  and 
exports.  The  sum  of  the  intermediate  inputs  consumed  by  the  agricultural 
branch  for  its  production,  and  the  agricultural  value  added,  equals  the 
agricultural production. The totals of the lines and the columns are therefore 
equal. 

Step 2. Construction  of a  matrix  of technical coefficients: This is  based 
on  the  matrix  of  transactions.  The  technical  coefficients  are  nothing  other 
than  values  taken  from  the  matrix  of  transactions  divided  by 
total 
production. The  relative part of the value  created  by each  branch for  all the 
other  branches  can  thus  be  seen.  The  technical  coefficients  serve  to 
provide  indications  on  the  technical  structure  of  the  modelled  economy; 
they  show  which  sectors  have a high  level of value added, productivity  and 
exports. These  coefficients  are artificially  fixed  for the following  steps. 

Step  3.  Construction  of  an  inverted  matrix: This  is the  result  of  a  matrix 
transformation  through  which  multiplier  coefficients  can  be  calculated. 
These  coefficients  summarise  all  indirect  effects. 

Step 4.  Hypotheses of primary  impacts: A quantitative  estimation  is  made 
of  the  main  impact  of  the  programme,  in  the  form  of  an  increase  in  final 
demand  (increase  in  public  spending,  increase  in consumption,  increase  in 
investment,  etc.).  The  assumption  of  increased  demand 
infrastructure 
includes a breakdown  per sector, so that it can be introduced  into the matrix. 

Step 5. Estimation of total impact: The calculation of total impact  is based 
on the assumptions  of  primary  impact  and  of the  inverted  matrix.  Based  on 
the  primary  impact  on  a  sector,  the  tool  is  used  to  estimate  all  indirect 
effects  on  all  sectors,  particularly 
in  terms  of  consumption,  imports, 
production  and  employment. 

Application  to  the  Structural  Funds 

Among the different  applications  of the input-output  models, the focus  here 
will  be on  the  model  mentioned  above,  known  as the  "Beutel  model". This 
example  is  particularly  interesting  because  the  model  was  constructed 
especially  for  evaluation  and  has  the  advantage  of  providing  harmonised 
estimations  on a  European  scale. 

The  Beutel  model  makes  use,  above  all,  of  a  series  of  harmonised  input-
output  tables  for  the  year  1985, drawn  up  by  EUROSTAT for  the  economies 

Tools  for analysing data 

of the  European  Union. These  input-output  tables are completed  by  regional 
tables,  particularly  in the  case  of  Spain and  Italy. The countries  covered  are 
Portugal,  Greece,  Ireland,  Spain,  Italy  and  the  UK  (Northern  Ireland). 
Germany  is, moreover,  included  in the  planning  period  1994-1999. Since the 
1985 database  is obviously  outdated for  any  practical  purposes,  it has to  be 
updated. For this purpose  Beutel developed a forecasting technique  adopted 
by EUROSTAT (see more detailed  information  at the end of this section). 

The  main  impact  of  Community  funding  is  estimated  by  dividing  total 
expenditure  into  three  categories:  property  investments,  investments  in 
capital  goods  and  salaries.  A  comparison  is  then  drawn  between  the 
development  of  the  economy  with  and  without  Community  funding. As far 
as  the  evaluation  of  interventions  in  the  framework  of  Objective  1  is 
concerned,  the  Beutel  approach  provides  estimations  focused  on  the 
following  key macro-economic  variables: 

•  Employment:  comparison  between  cases with and without  intervention  is 
established  not  only  in terms  of  total  employment,  but  with  a  distinction 
being  also  made  between  employees  on the  one  hand and  independent 
workers  on the  other. 

•  The GDP and its growth rate: It is possible to draw a comparison  between 
growth 
interesting 
comparison  is  made  between  the  annual  growth  rate  with  intervention 
and  cases  where  interventions  are  interrupted  during  a specific  year. 

rates  with  and  without 

intervention.  Another 

•  Gross  capital  formation:  This  variable  captures  changes  induced  in 

investments  by the  interventions  and  its annual growth  rate. 

•  Value  added  by  sector:  The  sectoral  distribution  of  changes  induced  in 
25 economic  sectors  by the  interventions  is calculated  each year. A stan­
dard  presentation  exists  of  the  evolution  of  the  6  sectors  undergoing 
structural  change:  agriculture,  forestry  and  fishing,  fuel  and  energy, 
building  and  civil  engineering, private  services,  and  public  services. 

•  Exports and  imports: The impact  of  interventions  on exports  and  imports 
is calculated  each year  and  sub-divided  into  exports  and  imports  to  and 
from  countries  belonging  to  the  European  Union,  and  in  exports  and 
imports to  and from  non-EU  countries. 

Strengths  and  limits  in  use 

•  The  most  interesting  contribution  of  input-output  matrices  concerns 
impacts  on  sectoral  distribution  and  trade.  Leakages  due  to  imports  by 

Principal  evaluation  techniques and tools 

the  beneficiary  country  from  Member  States  of  the  European  Union  and 
third countries  may  prove to be important  for  EU policy-making.  Similarly, 
knowledge  of the  impact  of  sectoral  demand  may  prove to  be  important, 
particularly  if  it  is  considered  that  some  sectors  must  be  stimulated  to 
accelerate  the  growth  rate. 

•  The value of this approach  lies above all in the broadness of its scope and 
in  its  coherent  treatment  of  the  main  economies  of  the  European  Union 
benefiting  from  Objective  1 support. 

•  The  advantage  of  the  tool  is  limited  to  the  estimation  of  effects  on 
demand.  Therefore,  it  does  not  take  into  account  one  of  the  most  im­
portant  effects  aimed  at  by  Objective  1 interventions,  i.e.  lasting  effects 
on  productive  potential.  Most  effects  on  supply,  which  are  likely  to  lead 
to  a  sustainable  increase  in  the  growth  rate  of  assisted  regions  and  to 
enable  them  to  catch  up  with  more  developed  areas,  are  totally 
overlooked  (creation  of  new  productive  capacity,  improvement  of  the 
training  and  education  of  the  workforce,  construction  of  infrastructure, 
productivity  gains  throughout  the  economy,  spread  of  technological 
progress  and  intensity  of  high-tech  activities  in the  productive  sector).  It 
is  all  these  effects  on  supply  which  transform  productive  capacity  in  a 
lasting  and  irreversible  manner. 

•  A  tendency  to  over-estimate  the  impact  of  Structural  Funds  on  demand 
has  been  observed,  because 
is  not 
the 
moderated  by  feedback  effects,  particularly  as  regards  price  changes 
originating  in  product  markets, financial  markets  and the  labour  market. 

interventions 

impact  of 

Further  reading 

W.  K.  Brauers  (1995),  Prévisions  économiques  à  l'aide  de  la  méthode 

Entrées  -  Sorties,  Paris:  Economica,  111p 
An  introduction  to  input-output  analysis,  for  the  general  public. 
Highlights  the  main  techniques,  recounts  the  debates  that  took 
place  around  this  subject  and  provides  examples  of  application. 
Concentrates  on the  most  important  aspects. 

R.  E.  Miller  and  P. D.  Blair  (1985),  Input-Output  Analysis,  Foundations  and 

Extensions,  New-Jersey:Prentice-Hall,  464p 
Intended  for  the  initiated,  this  book  describes  the  basis  of  input-
outpu:  analysis  and  provides  a  wide  view  of  the  different  types  of 

Tools  for analysing data 

analyses  that  may  be  carried  out  by  means  of  this  technique. 
Readers  can  easily  select  those  articles  of  particular  interest  to 
them. 

H.  D.  Kurz,  E.  Dietzenbacher  and  C.  Lager  (1998),  Input-Output  Analysis, 

Cheltenham:  Edward  Elgar, 3 volumes,  1504p 
In this work  the evolution of  input-output  analysis  is studied  through 
a  collection  of  articles  published  since  the  origins  of  this  type  of 
analysis. 

More  in-depth: the  construction  of  Beutel  model  matrices 

In the  Beutel  model, the  EUROSTAT  input-output  matrices  are  updated  by 
means  of  economic  forecasts  by  the  European  Commission  Directorate 
General  for  Economic  and  Financial  Affairs.  These  forecasts  are  derived 
from the QUEST model (see sheet on macro-economic  models). Use is also 
made  of  the  forecasts  of  25  productive  activities,  prepared  by  ERECO 
(European Economic  Research and Advisory  Consortium, composed  of Ifo-
NEI/Rotterdam, 
Institut/Munich, 
WIFO/Vienna, and Cambridge  Econometrics/Cambridge). 

Prometeia/Bologna, 

BIPE/Paris, 

The data  used  for  updating  the  model  consist  of  the  input-output  table  for 
the  reference  year,  including  the  import  matrix,  growth  rate  forecasts  of 
value added  per sector  (agriculture,  industry, services)  and the forecasts  of 
components  of  final  demand  (consumption,  investments,  exports  and 
imports).  The  updated  input-output  table  is  estimated  by  a  procedure  of 
iteration  which  provides  estimations  on  the  intermediate  consumption  of 
imported goods and services, the structural composition of final demand by 
product,  domestic  production  by  product  and  imports  by  product. 

This  technique  tries  to  avoid  random  changes  of  input  coefficients  by 
treating all activities equally. Apart from its limited need for data and the fact 
that  it  uses  official  sources,  this  technique  has  the  further  advantage  of 
producing  sound  updated versions  at  a low  cost. 

Tools for analysing data 

Macro-economic  model 

The 

p u r p o se 

of 

the  m a c r o-

economic  model  is to  represent  the 

functioning  of  an  entire  economy 

a nd  of 

the  different  markets 

comprising  it  (e.g.  product  market, 

job  market,  financial  market,  etc.). 

A 

model 

schematises 

the 

functioning  of  an  e c o n o my  at 

equilibrium. 

It  also  simulates 

the 

evolution 

of 

this 

e c o n o my 

O 

Overall 
evaluation 

Β 

High 
technical 
complexity 

φ) 

High 
cost 

O 

Θ 

Prospective 
use 

Use 
stabilised 

following  an  exogenous  shock  (e.g. 

variation 

in  oil  prices,  variation 

in 

exchange  rates,  tax  imposition),  the 

adjustment  of  modelled  economic 

variables  and  the  attainment  of  a  new 

market  equilibrium. 

Macro-economic  models  are  used  by  administrations  for  the  preparation  of 

their  budgets  and  for  the  study  of  political  choices  when  the  economic 

stakes  are  high.  Some  public  and  private  forecasting  institutes  also  use 

models  for  short-  and  medium-range  economic  forecasts. 

The  use  of  this  type  of  model  for  evaluation  is  far  more  recent  and  still 

involves  methodological  innovation.  This  does  not,  however,  preclude  the 

existence  of  success  stories  such  as  the  following  case. 

Example:  Evaluation  of  the  probable  impact  of  European  economic  support  to 

the  Republic  of  Ireland 

The evaluation of the probable impact  of European economic  support to the  Republic 
of  Ireland (CSF 89-93) was carried out  by the Economic  and Social  Research  Institute 
in  Dublin  (ESRI),  using  a  macro-economic  model  constructed  by  this  institute  in  the 
framework  of the HERMES  programme. The evaluation was to estimate the  probable 
impact  of  European  support  on  the  Irish  economy  as  a  whole  and,  in  particular,  on 
variables  such  as the  per  capita  GDP  or the  public  debt. 

The HERMES model was constructed to simulate the functioning of the  Irish economy 
in the context of an energy crisis. It was modified slightly  during the evaluation in order 
to  take  into  account  the  impact  mechanisms  of  European  programmes  more 
satisfactorily.  The  model  used  for  the  evaluation  contains  over  600  equations 
representing assumptions on the behaviour of the economic agents concerned  directly 
or  indirectly  by  European assistance. The evaluation conclusions  were as follows: 
In the year  2000, the  impact  of the  Community  intervention  is  estimated  at  2.7%  of 
Irish GNP. 

Principal evaluation  techniques and tools 

However,  this  impact  alone  will  not  be  sufficient  to  allow  the  Irish economy  to  catch 
up  with the  European  average. 

It is interesting to  note that this  evaluation  was the subject  of political  debates  during 
the  Irish  referendum  on  the  Maastricht  Treaty,  and  that  it  served  as  an  argument 
during  the  renegotiations  of  European  assistance  for  the  period  1994-99.  It  can 
therefore  be  seen  that  despite  the  inherent  fragility  of  any  estimation,  macro-
economic  evaluation  served as a basis for  constructive  debate. 
(Source:  Bradley  J.  and  Fitzgerald  J.,  Medium-Term  Review  1989-1994,  Economic 
and  Social  Research  Institute,  Dublin,  1989) 

What  is the  purpose  of the  tool? 

A  macro-economic  model  provides  a  simplified  representation  of  the 
mechanisms  of  an economic  system. Used  in evaluation, the tool  makes  it 
possible  to  simulate  one  scenario  with  public  intervention  and  one  without 
it,  and  therefore  to  estimate  the  consequences  of  the  intervention  by 
observing  the  differences  between  the  two  scenarios.  The  models  are 
entirely  quantified  and  therefore  provide  an  estimation  of  the  extent  of  the 
impacts  on all the  macro-economic  variables  that  have  been  modelled. 

Obviously  a  model  cannot  predict  everything,  nor  even  guarantee  the  full 
reliability  of  these  simulations.  However,  the  tool  remains  relevant  if  an 
estimation  of  macro-economic  effects  is required, for  no technique  is  more 
reliable.  It  provides  reference  values  on  the  impact  of  specific  measures 
that  are  difficult  to  obtain  by  other  methods  due  to  the  complexity  of  the 
mechanisms  in action  (e.g. impact  on employment  and private  investment). 
These  reference  values  are  used  to  stimulate  discussions  and  to  provide 
solid information  about  the real extent  of the issues. Finally, the value of the 
tool  is  often  educational.  Use  of  the  model  and  the  examination  of  the 
results  obtained  make stakeholders  in the evaluation  pay  more attention  to 
understanding  macro-economic  mechanisms. 

In which  cases  can  it  be  used? 

Effects  can  be  estimated  ex  ante.  In  this  case,  two  forecasts  (with  and 
without)  are  compared.  This  is  the  most  frequent  use  in  evaluation. 
Although  it has  very  seldom  been  done, a model  can  also  be  used  in an ex 
post  situation.  In  this  case,  the  comparison  concerns  past  economic 
developments  as recalculated  by the  model, and  a scenario  without  public 
intervention, reconstructed  ex  post  by  the  model  (simulation  of  the  policy-

Tools for analysing data 

off  situation).  In the  framework  of  intermediate  evaluation, the tool  may  be 
used,  where  relevant,  to  guide  the  re-orientation  of  the  programme  (e.g. 
distribution  of 
infrastructure 
development  and  support  for  R&D). 

investment  support, 

funds  between 

How  should  it be  implemented? 

The  implementation  of  an  evaluation  using  a macro-economic  model  must 
be  considered  only  if  the  evaluators  have  adequate  knowledge,  skills  and 
time. This  is because  numerous  highly  technical  steps  are involved. 

Step  1. Adapting  the  model to evaluation: The construction  of  a model  is 
a  heavy  investment  that  may  take  up  to  a  few  years.  A  macro-economic 
model is therefore  never constructed for the specific  needs of an evaluation. 
It  may  happen  that  several  models  are  available  and  that  the  first  step 
consists  of  choosing  one,  but  this  is  exceptional.  Generally,  the  evaluation 
team's  choice is dictated  by the existence  and availability  of a single model. 
The  problem  is  then  to  decide  whether  the  model  can  be  applied  in  its 
present  form  or  whether  it  must  be adapted  to  better  simulate  the  impacts 
of the evaluated  intervention. In the first case, the adjustments  can generally 
only  be minor ones  because the equilibrium  conditions  of the  model  cannot 
be challenged without the  risk of  having to  reconstruct  it entirely.  In general, 
adaptation of the model is limited to a few adjustments that  make it  possible 
to  take  into  account  the  content  of  the  programme  better,  and to  integrate 
the  primary  impacts  in the form  of  exogenous  shocks. 

Step  2. Adjusting  the  model:  Before  being  used, and  particularly  if  it  has 
been  modified,  the  model  must  be  adjusted 
recent 
developments  observed  and to the  most  recent  statistics. A model  is ready 
for  use when  it  is capable  of explaining  correctly what  happened  in the five 
to  fifteen  preceding  years.  In  practice,  the  model  must  regularly  be  re-
estimated  to  take  into  account  revisions  in  basic  information  and  thus  to 
allow  for  it to  be  "adjusted"  to the  most  recent  data. 

the  most 

to 

Step  3.  Estimation  of  primary  impacts: The  diagram  (Box  18) shows  the 
the  model. 
taken 
macro-economic  mechanisms 
Employment depends on the production  of firms, on productivity and on the 
level  of  salaries. These three  variables  are  endogenous,  i.e.  internal  to  the 
model. To evaluate the  effect  of  a public  intervention, for  example  its  effect 
on  employment,  this  public  intervention  must  be  considered  to  introduce 
exogenous  changes  into  the  model  (the  term  exogenous  shock  is  used). 

into  account 

in 

Principal evaluation  techniques  and tools 

The evaluation team must then select a few variables or coefficients  that are 
part  of  the  model  and  which  will  be  directly  affected  by  the  public 
intervention.  These  values  constitute  "bridges"  between  the  evaluated 
intervention  and  the  model.  In  the  following  example  they  are  public 
spending  and  productivity.  The  primary  impacts  of  the  Structural  Fund 
interventions  may also take the form of  investments  in productive  structures 
or  in  human  capital.  The  evaluation  team  must  then  estimate  the  primary 
impacts  of  the  intervention  (micro-economic  impact)  and  introduce  them 
into  the  model. 

Box  18-  Primary  impact  and  macro-economic  mechanisms 

Primary impact 
Supply side 

CSF 

_^  Primary impact 
Demand side 

Salaries  «-»  Employment 

Public debt 
A 
Taxes 

Costs and 
productivity 

Import. 
Export. 

Production  ·* 

Income 

Public 
spending 
I 
*  Demand 

International 
prices 

Step  4.  Simulations:  In  an  evaluation  context,  the  model  will  always  be 
used at  least twice. The first time it will  be used to adjust the intervention  to 
the  situation  (case  of  an  ex  post  evaluation)  or  to  simulate  this  situation 
(case of ex ante evaluation). The second time  it will  be used to  simulate the 
situation without  intervention. The impacts of the intervention  are estimated 
by  establishing  the  difference  between  the two  simulations. 

Step  5.  Presentation  of  results: The  last  phase  is that  of  the  formulation 
and  presentation  of  results.  Given  the  complexity  of  the  tool  and  the 
numerous  variables  that  can  be  quantified,  it  is  necessary  to  take  specific 
steps  to  present  results  in  a simple  and  easily  accessible  way. This  phase 
in  the 
must  therefore  be  considered  as  one  of  the  most 
implementation  of  the  tool. 

important 

Tools for analysing data 

Application  to the  Structural  Funds 

Macro-economic  evaluation  of  a Structural  Funds  programme  depends  on 
the existence of a model for the eligible territory. Available  macro-economic 
models are generally  limited to  national models for  reasons of  availability  of 
statistical  data  and  the  cost  of  modelling.  The  models  constructed  on  a 
regional  or  local  scale  are  rare,  particularly  because  economic  flows  are 
diluted  as  soon  as  they  are  not  in  the  territory  concerned.  However, 
modelling  remains  meaningful  for  small  economies,  provided  that  they  are 
relatively  closed.  That  is  why  insular  economies  are  often  equipped  with 
macro-economic  models. 

Thus, the  use of  models  is generally  possible  for  big  programmes for  which 
the  eligible  zone corresponds  to  national  borders  (Portugal,  Ireland, Greece) 
or to vast regional  blocs (South of Spain or of  Italy). Almost all the models are 
ill suited to  programmes  for  which the eligible zone  is a small, open region. 

Furthermore,  the  use of  a  model  has  meaning  mainly  if the  intervention  of 
the programme attains sufficient  mass in relation to the modelled  economy. 
It  is  considered  that  in  order  to  attain  this  critical  mass,  the  economic 
importance  of  the  programme  must  be  at  least  around  1% of  the  GDP  of 
the  economy  of  the  territory. 

In  the  framework  of  European  structural  interventions,  macro-economic 
evaluation  is limited to a few experiments  of which the two  most well  known 
used the QUEST and  HERMIN  models  presented  below. 

In the case of the QUEST  II model, Structural  Fund spending  is considered 
to  be an increase  in both  public  spending and the  national  stock  of capital. 
From  this  point  of  view,  the  expenses  incurred  are  considered  as  an 
exogenous  increase of overall  public  consumption  (demand  side effect), as 
well  as  an  exogenous  increase  in  public  investment,  leading  to  an  overall 
increase  in production  (supply  side  effect). 

This  type  of  approach  makes  a very  broad  estimate  of  supply  side  effects 
since all kinds of  investments  (infrastructure, R&D and training, both  private 
and  public)  are assumed  to  have similar  impacts  on  productive  potential. 

Further  refinements  of  the  model  might  consider  differentiated  impacts 
stemming  from  various  publicly  funded  interventions.  This  would  have  the 
advantage  of  fully  taking  into  account  the  supply  and  demand  effect  of 
Structural  Fund  interventions  in  a  clear  and  fairly  direct  way.  The  supply 
effects  would  be  simulated  separately  and  directly  via  the  impact  on 

Principal  evaluation  techniques and tools 

potential  production. The combination  of supply  and demand effects  would 
indicate the extent to which  the structure of the economy  is likely to  benefit 
from the  increase  in  productive  potential 

The  HERMIN  model  was  used in evaluations  in Spain, Ireland  and  Portugal. 
As  in  the  experiment  with  the  QUEST  model,  the  demand  effects  of 
spending  in the framework  of Structural  Fund interventions were  introduced 
in  the  form  of  exogenous  increases  in  public  spending.  Yet  evaluations 
carried  out  with  HERMIN  introduced  an  interesting  innovation  by  explicitly 
simulating  the  supply  effects  of  public  expenses,  trying  to  make  separate 
estimates  of  the  effects  of  three  categories  of 
interventions:  basic 
infrastructures,  human  capital  and  investments  in  the  productive  sector. 
The  primary  impacts  of  Structural  Funds  in  terms  of  supply  are  introduced 
into  the  model  in  several  ways.  They  first  take  the  form  of  an  increase  in 
productivity that affects  one or more parameters  of the production  function. 
In  the  Spanish  and  Irish  versions,  productivity  is  "embodied"  in  labour  (a 
single  parameter).  In  the  Portuguese  model,  it  is  total  factor  productivity 
which  is  increased  (all  the  parameters).  The  increase  in  productivity  is  a 
function  of  investment  generated  by  European  support  in  the  form  of 
private  capital,  human  capital  and  public 
infrastructure  capital.  It  is 
assumed that there  is a constant  relationship  (called elasticity)  between  the 
increase  in  investments  and  the  increase  in  productivity.  A  real  difficulty  in 
evaluation  relates  to  the  choice  of  the  value  of  this  relationship.  Empirical 
observations  which  might  allow  its  estimation  are  barely  reliable  and,  to 
avoid  totally  random  values,  similar  studies  based  on  American  or 
international  data  are often  used. The  credibility  of the  basis of  this form  of 
estimation  of  supply  effects  is somewhat  limited. 

A  second  hypothesis  on  primary  impacts  is  that  the  action  of  Structural 
Funds  helps to draw  more foreign  investments  and therefore to increase the 
their 
competitiveness  of  exports.  To  estimate  this  chain  of  effects, 
equivalent  in  terms  of  increase  in  international  demand  is  estimated.  Here 
again,  it  is  assumed  that  there  is  a  constant  relationship  between  the 
increase  in investments  generated  by the  European funds  and the  increase 
in  exports.  The  choice  of  a  value  for  this  relationship  is  even  more 
problematical  here.  First, arbitrariness  is inevitable,  given the total  absence 
of applicable empirical studies not only for the countries  concerned  but also 
at an international  level. Worse still, the final result seems to depend  heavily 
on  the  value  chosen  for  this  relationship.  This  is  particularly  true  for  the 
long-term  impact  on GDP. 

Tools for analysing data 

To  assess  the  potential  of  HERMIN  for  the  evaluation  of  macro-economic 
effects,  it  must  be  kept  in  mind  that  this  work  is  still  underway  and  that 
many  current  difficulties  are  likely  to  be  solved  as  the  project  progresses. 
Various teams'  estimations  of  the  impact  of  the  CSF should  help to  refine 
the assumptions  of  impact, the parameters of the models, the quality  of the 
data,  and the comparability  of  results. 

Strengths  and  limits  in  practice 

•  The  aim  of  the  Structural  Funds  is  economic  and  social  cohesion.  This 
objective  is essentially  defined  at the  macro-economic  level. The  macro-
economic  model  is practically  the  only  tool that  can  be  used  to  formally 
ascertain  whether  European  policy  has  achieved  its aim. 

•  In the evaluation  of  socio-economic  programmes, one of  the  major  limits  of 
models  is  that  they  do  not  give  a  credible  explanation  for  "supply-side 
effects".  For  example,  if  a description  of  the effect  of  new  infrastructure  on 
the productivity of firms is required, it will be difficult to find an equation which 
reflects this  effect  and which is convincingly  adjusted to  past statistics. 

•  One  of  the  main  limits  of  the  tool  relates to  the  construction  of  models. 
This  represents  a  substantial  investment  and  always  takes  more  than  a 
year.  Moreover,  the  parameters  of  a  model  have  to  be  continuously 
updated,  under  the  supervision  of  specialists  with  considerable  know-
how.  Experience  shows  that  statistics  and  forecasting  institutes  are 
obliged to  "readjust" their models several times a year in accordance  with 
recent  statistics. 

•  Only  exceptionally  is  a  model  constructed  specifically  for  an  evaluation. 
Use of the tool therefore depends on the existence of a model. The existing 
model  must  often  be adapted  to the questions  posed  by the evaluation. 

•  Each  macro-economic  model  is  based  on  its  own  specific  set  of 
assumptions.  When  two  models  are  constructed  for  the  same  territory, 
they  never  produce  exactly  the  same  simulations. 

•  The weakness of many  macro-economic  evaluations  relates to the way in 
which  the  primary  micro-economic  effects  are  evaluated  before  being 
introduced  into the model. In this sense, the  macro-economic  evaluation 
encounters  the  same  difficulty  as  the  other  evaluations  of  multi-sectoral 
programmes:  the  effects  of  a  multitude  of  heterogeneous  interventions 
have to  be summarised  in the form  of a few  standardised  and  quantified 
micro-economic  impacts.  This  task, which  is very  difficult  to  accomplish 

fi 

Principal  evaluation  techniques and tools 

for  a  normal  evaluation,  is  only  the  first  step  in  a  macro-economic 
evaluation.  It  is  the  step  that  was  defined  above  as  the  estimation  of 
primary  impacts.  Irrespective of the accuracy  of  its  internal  assumptions, 
the  model  only  transforms  estimations  of  primary  impacts  into  macro-
economic  estimations.  Thus,  it  accumulates  the  difficulties  of  the  two 
levels of  estimation. 

Further  reading 

In the  interests  of clarity, the  preceding  paragraphs  deliberately  omitted  the 
technical aspects of the construction of models. For readers who would  like 
more  detailed  information,  a  list  of  publications  on  macroeconomics  and 
modelling  is  provided  below. 

Since  the  use  of  models  is still  at  its  experimental  stage  for  the  evaluation 
of socio-economic  programmes,  it is difficult to find manuals with a detailed 
explanation  of  techniques  for  estimating  macro-economic  impacts.  This 
information  sheet  has  therefore  been  extended  to  cover  the  QUEST  and 
HERMIN  models  in greater  depth. 

Andersen,  K.V.  (1995) 

Information 

systems 

the  political  world: 
Implications  of  Economic  Modeling 

in 

Implementation,  Use  and 
Amsterdam:  IOS  Press.  , 283  p. 
This  book  investigates  the  use  of  econometric  models  for  policy 
analysis  and  evalution  in  Europe  and  the  USA,  at  central  and 
regional  level. It focuses on the social and political dimensions of the 
processes  by  which  models  are  built  and  used. 

Daloz  J.P.  and  Goux  C.  (1985),  Macro-économie  appliquée.  Du  simple  au 

complexe,  Paris:  Cujas,  156p. 
Very clear,  basic  approach  which  introduces  econometrics  in a very 
simple  way. 

Gray A.W. (1995), EU Structural  Funds  and  other  public  sector  investments. 

A guide  to  evaluation  methods,  Gill  & Macmillan,  106  pp 
Considers  a  set  of  models  adapted  to  the  evaluation  of  Structural 
Funds,  and  explains  problems  in this  field.  Highly  accessible. 

Kebabdjian  G.  (1994),  Les  modèles 
Paris:  Dunod,  194  pp 
A  macroeconomics  manual  which  addresses  all  current  trends 
without  any  bias. A demanding,  synthetic  and  useful  tool. 

théoriques  de  la  macro-économie, 

Tools for analysing data 

More  in-depth: the  QUEST  and  HERMIN  models 

QUEST: a multinational  model 

The QUEST  model  (Quarterly  European Simulation Tool) was  developed  by 
DG  II of  the  European  Commission.  Its  main  purpose  is to  make  medium-
term  and  long-term  forecasts.  The  model  is  used  to  quantify  scenarios  in 
various  fields:  monetary  policy,  budgetary  policy,  and  the  economic 
incidence  of  changes  on  the  international  scene,  such  as  fluctuations  in 
exchange  rates, interest  rates, oil prices  and other world  prices. The way in 
which the  multinational dimension  is modelled  is an essential  characteristic 
of  the model. 

The  QUEST  model  uses  national  sub-models  to  cover  all  the  Member 
States  of the  European  Union  (with  Luxembourg  associated  with  Belgium), 
the  United  States  and  Japan.  For  the  rest  of  the  world,  only  international 
trade  is modelled.  It is divided  up  into  seven  main OECD  countries  and  six 
other  regions. The equations  of  bilateral  exports for the  commodity  market 
are calculated, on the basis of  import  volumes and export  prices, for the 26 
countries/regions.  On  this  basis,  the  models  of  international  commodity 
trade  can  be  represented  in a series  of  equations  on  commercial  relations. 
This  bloc  of  commercial  relations  can  be  used  to  monitor  an  economic 
shock  arising  from  international  trade.  It  is  separated  from  national  sub­
models.  It  is  thus  possible  to  study  an  impact  on  a  country,  either  taken 
alone or  embedded  in  international  trade  as a whole. 

The  national  sub-models  all  have  the  same  basic  structure.  They  have  a 
relatively  low  number  of  behavioural  equations,  for  which  the  parameters 
have been adjusted for each country. The basic QUEST structure  combines 
three types  of  relationship: the  supply  bloc,  the demand  components,  and 
the wage/price  ratio. 

(a)  Supply 

The  main  purpose  of  the  supply  bloc  is to  determine  productive  capacity. 
The basic  assumption  is that  productive  capacity  is roughly  limited  by  the 
stock  of  national  capital.  It  should  be  noted  that  QUEST  is a single  sector 
model  that  establishes  no differences  between  the  products  of the  various 
sectors.  Productive  capacity  serves,  together  with  actual  production,  to 
define  the  degree  of  capacity  utilisation.  The  latter  variable  plays  an 
important  part  in  the  model  and  partly  determines  demand  in  terms  of 
investments  and  imports, and therefore  global demand  and  price  levels. 

Principal  evaluation  techniques and tools 

Productive  capacity  depends  on  private  and  public  capital,  employment 
and  energy,  plus an exogenous  increasing  productivity  trend  applying  to all 
factors.  Full effects  on  potential  output  take  about  six  years to  materialise. 

(b)  Demand 

Private  consumption  is  determined  by  the  present  discounted  income  of 
current  and  expected  income  and  wealth  of  households.  Also  current 
disposable  income  plays a role since  a proportion  of  households  is  subject 
to  liquidity  constraints. 

Investment  is  positively  related  to  the  present  value  of  profits  and  is  a 
negative  function  of  the  cost  of  capital. 

Imports  depend  on  domestic  final  demand  and their  relative  price  vis-à-vis 
domestic  prices. 

(c)  Salaries  and  prices 

It  is  assumed  that  firms  fix  their  prices  on  the  basis  of  a  mark-up  over 
marginal  cost.  Exports  depend  both  on  domestic  production  costs  and 
prices  of  international  competitors.  There  is  also  short  run  price  rigidity, 
modelled  using  an  adjustment  costs  constraint.  Nominal  wages  principally 
depend  on four factors: (1) the disincentives  that  arise from the  provision  of 
social  protection  and  the  value  of  leisure  time,  (2)  labour  productivity,  (3) 
labour  market  pressures  and  (4)  the  general  price  level.  Wages  do  not 
immediately  adjust  to  changes 
in  economic  conditions  since  wage 
agreements  are signed  for  a period  of  several  years. 

Interest  rates and exchange  rates are also treated as endogenous  variables 
in the  QUEST  model  (contrary to  many  other  models), that  allows  crowding 
out  effects  to  be estimated  (the fact  that the funding  of  public  expenditures 
generates  an  increase  in  real  interest  rate,  and  a  consequent  decrease  of 
private  investment). 

HERMIN:  a  four-sector  model 

The  HERMIN  models  (HER  from  the  macro-sectoral  model  HERMES  from 
which  it  was  inspired  and  MIN  to  indicate  the  limited  dimensions  of  this 
model)  cover 
Ireland,  where  the  prototype  was  originally  developed, 
Portugal,  Spain  and  Greece.  The  HERMIN  prototype  was  taken  from 
HERMES,  a  system  of  complex  models  (eleven  sectors,  over  600 
equations), spawned  by trans-European  collaboration  conducted  by  DG XII 

Tools  for analysing data 

and covering, with various degrees of success, all the Member States of the 
European  Union. The  HERMES  models  were  developed  to  analyse  energy 
policies  and  therefore  focused  on  the  role  of  energy  as  a  factor  of 
production. They  consequently  included  a careful simulation of  production-
related  processes. 

The  HERMIN  models  are  small-  and  medium-scale  (4  sectors  with  100  to 
150 equations each). They were developed  by  national  laboratories  in each 
of  the  four  peripheral  countries  in  the  European  Union.  One  of  the  main 
objectives  of  these  models  is  the  comparative  study  of  the  impact  of 
Structural  Funds  on  the  periphery  of  the  Union.  The  basic  structure  of 
HERMIN  combines  three types  of  relations:  the components  of  supply,  the 
components  of  demand  and the  wage/price  ratio. 

(a)  Supply 

The  HERMIN  model  comprises  four  sectors:  the  exchangeable  goods 
sector,  the  non-exchangeable  goods  sector, the  agricultural  sector  and  the 
public  sector.  These sectors  have the following  distinct  characteristics: 

The exchangeable  goods  sector  is subject to  international  competition  and 
its  production  is  determined  primarily  by  international  demand  (expressed 
in  terms  of  world  production)  and  competitiveness  (measured  by  the  ratio 
of  the  unit  cost  of  labour  to  world  unit  costs).  Prices  are  determined 
externally  in international  markets. 

The  non-exchangeable  goods  sector  covers  services,  building  and  civil 
engineering, and community facilities. It is subject  to domestic  demand  and 
prices  are determined  by the  profit  margin on  labour  costs. 

The  production  of  the  exchangeable  goods  and  non-exchangeable  goods 
sectors  is  modelled  in  the  form  of  a  function  where  the  value  added 
depends  on  capital  and  labour.  As  a  result,  the  levels  of  employment  and 
investments  depend  on  the  level  of  production.  The  demand  of  the  two 
factors  of  production  also  takes  into  account  their  respective  cost  (under 
the  assumption  of  the  elasticity  of  constant  substitution).  Technological 
progress  is  considered  to  be  an  exogenous,  time-related  trend  that  differs 
from  one  sector  to  the  next. 

The  agricultural  sector  covers  agriculture,  forestry  and  fishing.  It  is 
considered  to  be essentially  exogenous, with  prices  and production  quotas 
defined  in  the  framework  of  the  European  Union  Common  Agricultural 
Policy. 

Principal  evaluation  techniques and tools 

In  the  public  sector,  which  includes  the  administration,  education  and 
health, the  level  of  activity  depends  on government  decisions.  Spending  is 
limited  by the  rules  of  budget  equilibrium  and debt  repayment. 

(b)  Demand 

Consumption  is  determined  by  real  available  personal  income  and  by  real 
financial  wealth.  Private  non-residential  investments  are  derived  from  the 
production  function  as  a demand  for  fixed  capital.  Residential  investments 
are  a  function  of  available  income  per  capita.  Finally,  public  investment  is 
an  exogenous  variable.  The  financial  sector  is  not  modelled  and  the 
exchange  rate,  as  well  as  the  domestic  interest  rate,  are  exogenous 
variables. 

the 
Agricultural  exports  are  estimated  exogenously.  Exports 
exchangeable  goods  sector  are  a function  of  the  production  of  this  sector 
and of the  GNP growth  rate  (a reflection  of the  economic  cycle)  in the  case 
of  Spain. Tourism  and  service  exports  are  a  function  of  world  production. 
Imports  are  determined  in  a  residual  fashion  to  differentiate  between  the 
total  production  of the four  sectors and final demand  (private consumption, 
public  consumption,  investments  and  exports). 

from 

(c)  Wages  and  prices 

Wages  are  fixed  by  an  equation  that  simulates  the  process  of  wage 
negotiations  between  employers  and  unions.  Negotiations  on  nominal 
wage  rates are influenced  by the situation of the labour  market, as  reflected 
in  the  current  unemployment  rate  or  its  rate  of  variation.  They  are  also 
subject  to  the  influence  of  "fiscal  leakages",  i.e.  the  difference  between 
gross  salary  before  tax,  expressed  in  terms  of  production  costs,  and  net 
salary  after  all  deductions,  expressed  in terms  of  consumer  prices.  Finally, 
the  level of  production  costs  and that  of  labour  productivity  also  affect  the 
process  of  wage  negotiations.  In  the  case  of  Ireland,  the  job  market  is 
assumed  to  be  homogeneous  and  wages  in  the  exchangeable  goods 
sector  dominate  all  the  other  sectors.  In  Spain,  salaries  in  the  non-
exchangeable  goods  sector  are  evaluated  separately,  but  the  public  and 
agricultural  sectors  are also  dominated  by the exchangeable  goods  sector. 
The  supply  of  manpower  is determined  differently  in the  various  countries. 
Migration  plays  an  important  part  in the case  of  Ireland,  particular  into  and 
from  the  UK.  In  the  case  of  Spain,  a  distinction  is  made  between  the 
employment  rate  of  men  and  of  women. 

Tools for making a judgement 

The overall evaluation of a complex  programme  is characterised  by the fact 
that there are numerous expected effects  and that these do not  all have the 
same  degree  of  importance  for  the  different  partners.  Some  tools  for 
making judgements  are particularly  well  suited to  this  situation. 

For  example,  the  expert  panel  constructs  a  synthetic  judgement  of  the 
programme  being  evaluated.  This  tool,  when  implemented  with  optimum 
efficiency,  enhances  the  credibility  and  acceptability  of  the  evaluation 
conclusions  because  differences  between  points of view are respected  and 
consensus  is reached. 

the 
Multicriteria  analysis  has  significant  advantages 
acceptance  of  a  multiplicity  of  points  of  view.  The  tool  includes  a 
mechanism  for  the  identification  and  explanation  of  points  on  which  there 
is  consensus. 

in  terms  of 

The  most  well  known  tools  for  making  a  judgement  are  cost-benefit 
analysis  and  cost-effectiveness  analysis.  For  various  reasons,  these 
tools  do  not  have  the  flexibility  required  for  taking  into  account  the 
complexity  of  a  judgement  on  an  entire  socio-economic  programme. 
Benchmarking  is a more flexible  tool for  making judgements,  which  could 
theoretically  be applied to the  overall  evaluation  of  a complex  programme. 
However,  to  date  this  tool  has  been  used  only  for  simple  evaluations. 
Consequently,  it  is presented further  on  in this  document. 

Some  tools  presented  above,  such  as those  for  structuring  an  evaluation, 
may also  be used  to formulate  a judgement.  For example,  SWOT  analysis 
may  be  used  to  judge  the  relevance  of  a strategy,  just  as the  colour  vote 
may help to validate  key  points  in the final evaluation  synthesis  in a context 
of  partnership. 

Tools  for  making  a 

judgement 

E x p e rt  p a n e ls 

Expert 

panels 

bring 

together 

several 

independent 

specialists 

recognised  in  the  fields  covered  by 

the  evaluated  p r o g r a m m e.  The 

experts  collectively  p r o d u ce  a 

value judgement  on the  programme 

and  its  effects.  This  low  cost  tool 

rapidly 

provides 

an 

overall 

judgement  incorporating  the  main 

information 

available 

on 

the 

programme,  as  well  as  numerous 

previous  and  external  experiences. 

O 

Overall 
evaluation 

A 

In-depth 
evaluation 

Θ  © 

Medium 
technical 
complexity 

Medium 
cost 

O  o 

Retrospective  Prospective 

use 

use 

Θ 

Use 
stabilised 

The  panel  may  be  considered  as  an  evaluation  tool  in  so  far  as  there  is  a 

standard  and  reproducible  procedure  for  forming  it,  bringing  it together  and 

leading  it  to  produce  its  conclusions.  Inspiration  for  the  tool  was  based  on 

university  juries  -  which  explains  why  it  appeared  in  the  early  1970s  -  in  the 

field  of  Research  and  Development  programme  evaluation. 

An  "expert  panel"  is  a  specially  constituted  work  group  that  meets  for 

evaluation.  A  similar  tool,  the  Delphi  survey,  is  described  elsewhere  in  this 

Volume.  It  also  relies  on  experts  but  differs  in  several  other  respects. 

Example:  Evaluation  of  the  quality  and  relevance  of  development  programmes 

in  Sweden 

For  several  years  the  Swedish  National  Council  for  Technical  Development  has  be 
financing  programmes  for  the  development  of  basic  skills  and  knowledge.  Since 
1981 the quality  of these  programmes  has  been regularly  evaluated. 

The first  step consists  of  selecting  five  international  experts  who are recognised  and 
representative  of  the  diverse  points  of  view  concerning  the  programme.  These 
experts  make  an undertaking  to  participate  in all the  panel  meetings.  One of them  is 
elected  as the chairperson. 

In the second  stage the  panel  examines  the  programme  documents  concerning  the 
resources  mobilised  (staff,  equipment,  finances),  the  research  themes,  results, 
publications  and co-operation. The  panel  interviews  several  addressees. 

The following  stage consists  of visits  in the field  by each expert, either  individually  or 
by groups  of two. 

In the final  stage the  panel draws  up  a joint  report.  If the  points  of  view  diverge, the 
panel discusses  them  until  a consensus  is  reached. Experience  has  shown  that  the 

Principal evaluation techniques and tools 

search for compromise inevitably softens the conclusions. However, the value of the 
evaluation  is  enhanced  by  the  legitimacy  of  the  panel's  conclusions.  Programme 
managers are invited to express themselves and the final report is then published. 
This tool is considered to be effective and inexpensive. It is applied extensively in all 
the  Scandinavian  countries  and  more  generally  in  all  countries  that  evaluate  their 
science policies. The OECD also applies this tool to numerous other political fields. 
Source: Christensen, H.C (1987) "Evaluation of research Programmes", in E. Ormala 
(ed.)  Evaluation  of  Technical  Research  and  Development  ,  pp.  88-108,  Espoo, 
Nordforsk 

What  is the  purpose  of  the  tool? 

The expert  panel  is used  mainly  to  assess  a programme,  but  it  is a  generic 
tool. The  terms  of reference  given to the  panel  may  include a wide  range of 
questions,  from  the  relevance  of  programme  objectives  to  an estimation  of 
real or  probable  effects. 

In which  cases  can  it  be  used? 

Expert  panels  are  used  to  reach  consensus  on  complex  and  ill-structured 
questions for which other tools do  not  provide univocal or credible answers. 

The  use  of  groups  of  experts  makes  it  possible,  within  a  few  months,  to 
gather  the  main  points  of  view  and  knowledge  relevant  to  the  evaluation. 
The  tool  is therefore  recommended  when  sufficient  expertise  exists  in  the 
field  and  when the  evaluation  is  complex. 

It  is also  well  suited  to  small, simple  programmes,  the  evaluation  of  which 
does  not  warrant  the  mobilisation  of  many  resources. 

How  should  it  be  implemented? 

Step  1. Identification  of  a  list  of  potential  experts: The  members  of  the 
panel must  be specialists  recognised  in at  least one of the fields  concerned 
by  the  programme.  They  must  have  extensive  experience  in the  field  and 
be  independent  vis-à-vis  the  commissioners  of  the  evaluation. They  must 
also  have  the  required  availability  and  wish  to  become  involved  in  the 
evaluation. 

Step  2.  Selection  and  mandating  of  the  experts: The  panel  is  generally 
composed  of between six and twelve members  belonging to different  "fields 
of  expertise". The current  tendency  is to  broaden the  range  of  interests  and 

Tools for making  a judgement 

to  seek  the  greatest  possible  diversity  of  points  of  view  in the  panel.  The 
experts are nominated intuitu personae  and do not represent their institution. 
Each  expert  signs  a contract  which, depending  on  the  case,  does  or  does 
not  provide  for  remuneration.  The  president  of  the  panel  is  chosen  by  the 
commissioner  or  elected  by  her  or  his  peers.  It  is  essential  that  the 
secretariat  of  the  panel  be entrusted  to  a  person  with  sufficient  availability, 
which  is generally  not the case with the experts  themselves. 

Step  3.  Investigations: The  experts  meet  between  three  and  six  times, at 
one-month  intervals. All the dates of their  meetings  must  be planned  at the 
outset.  The  panel's  internal  debates  are  under  the  seal  of  secrecy.  The 
members  of  the  panel  consult  the  programme  or  project  documents 
(reports,  preliminary  studies,  inquiries)  and  interview  the  programme 
leaders  and  several  typical  addressees.  They  may  also  make  field  visits, 
generally  in groups  of two  to  limit  the risks of  bias. 

Step  4.  Synthesis:  The  panel  produces  a  report  and 
formulates 
conclusions  and  recommendations  that  are  collectively  accepted.  In  case 
of disagreement,  It may  be useful to express the  majority conclusion  and to 
attach  a  comment  by  the  minority  expert.  The  panel  must  hold  a  final 
meeting after reception of the commissioner's  comments on its draft  report. 
Part  of  the  panel's  work,  particularly  in  its  final  phase,  may  make  use  of 
distance  communication  techniques  (Internet  or  videoconferences). 

Application  to  the  Structural  Funds 

The  tool  may  be  used  to  carry  out  an  entire  ex  ante  evaluation,  provided 
that at least three months are available. For example, an expert  panel  could 
be  asked  to  estimate  the  probable  impact  of  a  programme  in  terms  of 
employment,  or to assess the merits of the programme  in terms of  potential 
synergy. 

The tool  may  also  be  useful  every  time  the  structuring  or judgement  stage 
needs to  be  reinforced. In that  case, the  expert  panel  may  intervene  at the 
beginning  and  end  of  the  evaluation,  in combination  with  other  tools  used 
for the collection  or  analysis of  the data  (see example  in  Box 39). 

The  expert  panel  may  be  used  to  formulate  an  independent,  authoritative 
judgement,  which  is  particularly  useful  in a  partnership  context,  especially 
if there  are differences  in the  partners'  views.  In that  context  it  is  possible 
to  ask each  partner  to choose some  of the experts,  based on the  similarity 
of  their  points  of  view.  A  list  of  experts  exceeding  the  needs  of  the  panel 

Principal evaluation  techniques and tools 

may also be proposed, and each partner  be given the  right to delete  one or 
two  names  from  the  list. 

Strengths  and  limits  in  practice 

•  The  expert  panel  is  a  relatively  inexpensive,  rapid  and  flexible  tool.  Its 
conclusions  enjoy  a  high  degree  of  credibility  when  recognised  experts 
are used. 

•  Expert  panels  may  fulfil  numerous  functions,  it  is  preferable  however  to 
limit  its work to  only  a part  of the  evaluation: the structuring  of  objectives 
and  estimations  of  effects  or  judgements.  The  more  clearly  the  panels' 
work  is  defined,  the  more  its  significance  will  be  recognised.  The 
reliability  of  the  tool  may  be  undermined  if  the  questions  put  to  the 
experts  are too  broad. 

•  The  risk  of  bias  from  empathy  is  significant  in  so  far  as  panels  are  too 
often  limited  to  the  specialists  in the  fields  covered  by  the  programmes 
(i.e.  peers)  who  are  hardly  inclined  to  criticise  the  relevance  of  the 
objectives  or to  focus  on  any  undesirable  effects. 

•  The  comparison  of  opinions  often  leads  to  the  under-evaluation  of 
minority  points of view. The consensual  mode of functioning  on which the 
dynamics  of  the  panel  is  based,  produces  a  convergence  of  opinions 
around  majority  values  which  are not  necessarily  the  most  relevant. 

•  The  tool 

is  more 

recommendations  if  it  is  combined  with  suitable  leadership 
such  as those  presented  in this Volume  (METAPLAN®,  colour  vote). 

likely  to  produce  creative  conclusions  and  rich 
techniques 

Further  reading 

Callón  M.,  Laredo  P.  and  P.  Mustar  (1995),  La  gestion  stratégique  de  la 
recherche  et de  la  technologie.  Paris:  Economica,  pp.  31-88. 
Presentation  in the first two chapters of this  book of the expert  panel 
model  used  by the  Commission  of  European  Communities. 

Cozzens  S.E. (1987),  'Expert  Review  in  Evaluating  Programs',  Science  and 

Public  Policy,  14(2),  71-81 
Inventory  and  analysis  of  the  American  experience  of  the  use  of 
expert  panels  for  evaluation.  The  interest  of  the  conclusions 
transcends  the  context  of  science  policy. 

Tools for making a judgement 

Nadeau  M-A.  (1988), L'évaluation  de programme.  Laval,  Québec:  Presses 

de  l'université  de  Laval, pp.  349-352 
This  manual  presents  the  methods  and  tools  of  programme 
evaluation;  a few  pages  are devoted  to  expert  panels. 

Witkin  B.R  and  J.W  Altschuld  (1995).  Planning  Conducting  Needs 

Assessments,  Thousand  Oaks: Sage.  pp.  193-203. 
This  work  presents  a series  of  tools  applicable  to  the  evaluation  of 
needs; a section  is devoted to  expert  panels. 

Tools for making a judgement 

Multicriteria  analysis 

Multicriteria  analysis 

is  used 

to 

make a comparative  assessment  of 

alternative 

projects 

or 

heterogeneous  measures. With  this 

technique  several  criteria  can  be 

taken  into  account  simultaneously 

in  a  complex  situation.  This  helps 

"decision-makers"  to  integrate  the 

different  options,  reflecting 

the 

opinions  of  the  actors  concerned 

O 

Overall 
evaluation 

0  φ 

Medium 
technical 
complexity 

Medium 
cost 

© 

O  G 

Innovative 
use 

Retrospective  Prospective 

use 

use 

by  a  decision-making  process, 

into 

a  prospective  or 

retrospective 

framework.  It  implies  the  participation  of 

the decision-makers  and  results  in 

operational  advice  or  recommendations. 

Example:  Evaluation  of  the regional  development  programme  in the  Hainaut 

region 

Multicriteria  analysis  was  used  in  the  evaluation  of  a  regional  development 
programme co-financed by the three European Structural  Funds and the government 
of the Walloon region. The programme  applied to the Hainaut province. It covered the 
period  1994 to 1996 and  its budget  amounted  to over  one  billion  ECU, almost 50% 
of  which  was financed  by the European  Union. The multicriteria  analysis  tool was 
chosen for its "multi-judge"  feature that  makes it possible to use different  weighting 
systems  particular  to  each  partner.  Eight  evaluation  criteria  were  defined  in 
agreement  with the representatives  of both  levels  of government:  (1) viability  of the 
trained;  (3)  sectoral 
assisted  businesses;  (2)  employability  of  the  persons 
diversification  of  the  economy;  (4)  openness  of  the  economy;  (5)  environmental 
integration  of activities;  (6) territorial  distribution of activities;  (7) services to firms; (8) 
synergy  between the components  of the  programme. 

The  effectiveness  of  the  measures  within  the  programme  was  assessed  by the 
managers of the measures.  During the interviews  they  gave scores for effectiveness 
to each measure, in terms of these eight criteria. The results showed vast  differences 
in  the scores  given  for effectiveness,  depending  on the measures  and the criteria. 
Then, by means of a formalised  interview  procedure, six "judges"  (2 members of the 
European  Commission,  2  from  the  Walloon  government  and 2  from  the  Hainaut 
Region)  established  a weighting  of the criteria  used,  based  on the importance  they 
attributed  to each of these  criteria. Once  again, the results  varied  widely.  One of the 
judges  gave a score of  "3" to the viability  criterion of businesses,  while  another one 
gave the same criterion a score of "26". This demonstrates the differences  in centres 
of  interest. 

Principal  evaluation  techniques and tools 

After  being  processed  by  computer,  the  measures  were  sorted  by  order  of 
effectiveness,  taking  into  account  the  fact  that  there  was  a category  for  each  judge 
(since  each  one  had  her  or  his  own  weighting  system  for  the  criteria).  Despite  the 
differences  in  opinion  -  among  managers,  on  the  effectiveness  scores,  and  among 
the  judges,  on  the  weighting  of  the  criteria  -  the  various  classifications  converged 
towards  a virtually  identical  result  on the  relative  value of  the  measures  in relation  to 
each  other.  This  new  light  shed  on  the  subject  was  of  unquestionable  use  to 
decision-makers, particularly  in a context  where the intention was to reallocate a part 
of  the  budget  during  the course  of  the  programme. 

Source:  MEANS  Handbook  n°4  (1995)  Applying  the  Multicriteria  Method  to  the 
Evaluation  of  Structural  Programmes.  Brussels:  European  Union. 

What  is the  purpose  of the  tool? 

The purpose of the tool is to structure and combine assessments taken  into 
account  in  decision-making.  Decision-making  is  understood  here  to  be  a 
process  made  up  of  multiple  choices.  Decisions  relative  to  these  choices 
condition  the  final  decision  to  a  large  extent. 

Multicriteria  analysis  is  intended  above  all  to  highlight  the  reasoning  and 
subjective  convictions  of  the  actors  on  a  particular  question.  It  also  serves 
to  synthesise  the  opinions  expressed  by  these  same  actors,  in  order  to 
determine  the  priority  structures,  to  analyse  conflictual  situations,  or  to 
formulate  recommendations  or operational  advice. 

In which  cases  can  it  be  used? 

Multicriteria  analysis  is  similar  to  the  techniques  adopted  in  the  field  of 
organisational  development  or  information  systems  management,  such  as 
the  Critical  Factors  of  Success  method  developed  in  the  1970s.  It  also 
resembles cost-benefit  analysis although it does  not reduce  highly  disparate 
phenomena  to  a  common  unitary  (monetary)  base.  This  improvement  is 
essential  because  it allows  criteria to  be taken  into account  even when  their 
cost  is not  known  in advance  (scenery,  noise, satisfaction, etc.). 

Multicriteria  analysis  is  particularly  useful  during  the  formulation  of  a 
judgement  on  complex  problems.  That  is  the  case  with  contradictory 
judgement  criteria  (e.g.  jobs  as  opposed  to  the  environment)  or  when  a 
choice  between  the  criteria  is  difficult. 

In  general, this  technique  is  used  in  ex  ante  evaluations  of  public  projects 
and  of  their  variations  (the  layout  of  a  highway,  the  construction  of  a  new 

Tools for making a judgement 

infrastructure,  etc.).  More  rarely,  multicriteria  analysis  is  applied  to  the 
intermediate  or ex  post  evaluations  of  programmes. 

How  should  it be  implemented? 

Multicriteria analysis is a tool for comparison  in which several  points of view 
are  taken  into  account.  It  can  be  broken  down  into  several  phases 
described  chronologically  below.  It  is  possible  to  repeat  the  phases  and 
thus to  make  corrections. 

Step  1. Definition  of  the  projects  or  actions to  be judged:  Inventory  of 
the  planned  or  implemented  actions,  or  the  elements  on  which  the 
comparative judgement  will  be made. 

Step 2. Definition of a battery of judgement  criteria: The evaluator or the 
commissioners of the evaluation will specify the criteria. Particular  attention 
must  be  given  to  the  definition  of  criteria,  in  order  to  be  as  exhaustive  as 
possible  and  to  define  the  question  properly.  The  criteria  must  reflect  the 
preferences  of  the decision-makers  or the different  points  of  view,  so as  to 
summarise  and  group  together  diverse  characteristic  dimensions  used  to 
evaluate  an action. 

Step  3.  Analysis  of  the  impacts  of  the  actions:  When  the  projects  and 
criteria  have  been  defined,  a  quantitative  estimation  or  a  qualitative 
description  of the impact of each project,  in terms of these criteria, must  be 
made.  For  this  purpose  short  statements  describing  the  different  levels  of 
impact  may  be used:  "impact  descriptors". 

Step 4. Judgement  of the effects  of the actions in terms of each  of the 
selected  criteria:  Evaluating  impacts  by  giving  them  scores  may  be  done 
by  the  evaluator,  based  on  quantitative  data,  or,  more  subjectively,  by 
experts  or the  addressees  of the evaluation  themselves. 

Step  5.  Aggregation  of judgements: This  combination  may  be  based  on 
three  principles: 

•  Personal judgements: the different  judgement  criteria are not  synthesised 
in any way. Each of the addressees of the evaluation constructs  her or his 
own  personal  judgement  based  on the  analysis  and  uses  it to  argue  her 
or  his  point  of  view. 

•  Assisting  coalition: a computerised technique  sorts the actions  in relation 
to  each  other.  An  action  will  be  classified  above  another  one  if  it  has  a 
better  score for  the  majority  of  criteria  (maximum  number  of  allies)  and if 

Principal  evaluation  techniques and tools 

it  has  less  "eliminatory  scores"  compared  to  the  other  criteria  (minimum 
number  of  opponents). 

•  Assisting  compromise:  a  weighting  of  the  criteria  is  proposed  by  the 
evaluator  or negotiated  by the  addressees  of the  evaluation. The  result  is 
a  classification  of  actions  in  terms  of their  weighted  score. 

The  main option  in multicriteria  analysis  is that  of the  involvement  or  not  of 
the  evaluator  in  the  definition  of  criteria,  in  their  evaluation  and  in  their 
weighting.  If the  evaluator  is actively  involved  in the  analysis, the  credibility 
of the results is undermined. On the other hand, when the addressees of the 
evaluation  participate  in the  definition  of  the  criteria, each  partner  prolongs 
the discussion  until at least one judgement  criterion  is found that  places  her 
or  his favourite  action  in first  position. 

•  Application  to  the  Structural  Funds 

As  seen  in  the  above  example,  multicriteria  analysis  is  well  suited  to 
managing structural  programmes  in partnership  and to their evaluation. The 
opinions  of  national  and  supranational  members  may  be  expressed  jointly 
without 
losing  any  of  their  specificity  or  having  to  make  too  many 
concessions  regarding  their  value  scales. 

The Hainaut  experiment  was seen as a possible use of the technique  by the 
Commission,  in  the  evaluation  of  the  Structural  Funds. 
It  produced 
encouraging  conclusions  as to  its  possible  implementation.  In particular,  it 
served to develop a variation of the method called  "multicriteria-multijudge" 
analysis,  which  enables  each  partner  to  construct  her  or  his  judgement 
based  on the  criteria  and weights  of  her  or  his own  choice. 

Strengths  and  limits  in  practice 

•  Multicriteria  analysis  provides  a framework  in  which  the  actors  take  part 
in  decision-making  and  in  problem  solving.  Its  negotiated  character  and 
explicit 
to  an 
treatment  of  judgement  criteria  help  to  give  form 
unstructured  reality.  It  thus  allows  the  values  and  individual  opinions  of 
several  actors  to  be  taken  into  consideration,  and  the  processing  of 
functional  relations  within  a complex  network,  in a quantitative  way. 

•  The intervention  of  an expert, the  leeway  left  to the  decision-makers  and 
similarities  with  vote  based  methods  make  this  technique  a suitable  tool 
for  a  partnership  approach. 

Tools for making a judgement 

•  The technique may help to define a coalition or a compromise,  but it does 

not  dictate the  individual  or collective judgement  of the  partners. 

•  Multicriteria  analyses  are  often  preferred  by  decision-makers  because 
they  are  readily  involved  in  the  process  through  a  relatively  simple 
technical framework,  unlike cost-benefit  analyses that artificially  measure 
all the effects  in monetary  terms. 

•  Specific  problems  of  implementation  may  limit  the  use  of  multicriteria 

analysis, or  require the  presence  of  experts. 

•  This technique  is  not  always  used  in an  interactive  way,  as  it  should  be, 

and tends to fix  criteria that  are, in reality, fluid. 

Further  reading 

Roy  B. and  Buyssou  D. (1993), Aide  Multicritère  à la décision:  Méthodes  et 

Cas, Paris:  Economica,  695  p. 
Very  complete  work.  The  most  essential  elements  are  marked, 
facilitating  a reading  at several  levels. 

Saaty  T.L.  (1984), Décider  face  à la  complexité,  Paris:  Entreprise  Moderne 

d'Edition,  231  p. 
Book  describing  the  technique,  based  on  varied  examples,  with 
reflection  on the decision-making  process. 

Schärlig  A.  (1990,  2ème  éd.),  Décider  sur  plusieurs  critères,  panorama  de 
l'aide  à  la  décision  multicritère,  Lausanne:  Presses  polytechniques 
et  universitaires  romandes, 303  p. 
Complete  and  highly  accessible  manual. 

Toulemonde,  J.,  Fontaine,  C,  Laudren,  E.,  Vincke,  P. (1998),  Evaluation  in 
in 

partnership.  Practical  suggestions 
Evaluation, 4(2). London  : Sage publication. 
The  article  describes  the  evalution  of  an  urban  development 
programme  using a multicriteria-multijudge  analysis. 

for  improving 

their  quality 

Vincke, P. (1992), Multicriteria  Decision-aid,  New-York  : John Willey & Sons. 

Comprehensive  presentation 

Tools for in-depth treatment 
of an evaluative question 

An  in-depth 
treatment  of  an  evaluative  question  is  possible  when  this 
question  concerns  a  single  intervention  or  a  small  number  of  similar 
interventions,  when  the public  concerned  is homogeneous  and  when  there 
are  few  effects  to  evaluate.  Thus,  if  an  evaluative  question  is  sufficiently 
simple,  it is possible  to consider  it in depth,  that  is to say,  to collect  primary 
data  from  direct  addressees,  to  make  an in-depth  analysis  of  impacts  and 
to seek comparable  interventions  to serve  as a reference  for judgement.  In 
this  situation,  described  in  most  evaluation  manuals,  almost  all  evaluation 
tools  are applicable  without  any need  for  innovation.  This section  presents 
eleven tools, emphasising  the conditions  in which  they can be applied  to the 
evaluation  of socio-economic  programmes. 

Examples of evaluations with in-depth treatment 
of a question 

The examples  presented  below  describe  situations  which  are simple,  either 
because  the  evaluation  focuses  on  a  single  action  or  an  homogeneous 
group of actions, or because  it focuses  on a single theme such as obtaining 
a result or a particular  impact  (for more information on thematic  evaluations, 
the reader  is referred to Volume 5 of the Collection  devoted  to that  subject). 
When the object of the evaluation is simple enough, the collection of  primary 
data,  the  analysis  of  impacts  and  the  formulation  of  judgements  can  be 
carried out  in greater detail. Almost all evaluation tools are applicable to this 
situation. The following  three  examples  provide an illustration. 

In  the  first  example,  officials  in  a  French  ministry  wanted  a  comparative 
estimation  of  the  employment  effects  of  several  interventions.  The 
evaluative question was as follows:  "Are some employment  measures  more 
effective  than others?" 

Box 19 - Evaluation of the impact of recruitment assistance in France 

In France, the  Employment  Ministry  frequently  conducts telephone  questionnaire 
surveys  on  samples  of  addressees  of  the  different  employment  assistance 
measures.  Each  of  these  surveys  is  used  to  estimate  a  deadweight  coefficient 
and,  where  relevant,  substitution  or attraction  coefficients. These coefficients  are 
used  to  extrapolate  monitoring  data  (gross  effects)  in  order  to  estimate  net 
effects. An expert  panel  met on several  occasions  to compare  all the  coefficients 
resulting  from  the  various  surveys,  and  to  propose  general  rules  for  estimating 
deadweight  effects,  in relation to  the  nature of  the  assistance. 

In  this  example,  the  questionnaire  survey  technique  was  used  for 
observation.  The  expert  panel  technique  helped  to  consolidate 
the 
estimation  of  effects. 

The  following  example  is  drawn  from  an  experiment  carried  out  by  the 
MEANS  programme  in  Catalonia.  An  evaluation  of  assistance  to  co­
operatives  was  requested  after  an  audit  of  the  implementation  of  this 
assistance,  which  expressed  doubts  as to  its  effectiveness  in terms  of  the 
creation  and  safeguarding  of jobs. 

In this example, the observation tool was the questionnaire survey.  Factor 
analysis  made  it  possible  to  fulfil  the  function  of  comparing  observations 
very  successfully. 

Principal  evaluation  techniques and tools 

Box 20  -  Ex  post  evaluation  of  assistance 

to  Catalonian  co-operatives 

In  1992 the  Catalonian  authorities  developed  a  programme  for  assisting  co­
operatives.  They  considered  that  they  could  have  an  important  impact  on 
employment,  due to the fact  that  the demand  for  profitability  was less  than  in 
privately  owned enterprises.  However, doubts  remained as to the effectiveness of 
this assistance  and the Department  of Labour  commissioned  an evaluation of the 
programme. 

Because of the number of co-operatives, their diversity  and the lack of knowledge 
about the addressees, it was  decided to carry out a questionnaire  survey on the 
entire  targeted  population, i.e. about  3,000  co-operatives. The answers  obtained 
were treated by factor analysis in order to constitute a typology of co-operatives. 
This  analysis  helped to clarify  the characteristics  of those  co-operatives  that had 
created  jobs,  and  of  those  that  had  not. The  analysis  showed  that  the co­
operatives  which  were  most  likely  to transform  the assistance  received  into the 
creation  and maintenance  of jobs,  were  those  which  were  already  established, 
were  members  of a federation  of co-operatives,  were  not situated  in the coastal 
area,  were  medium-sized,  and so on. On the other  hand,  it  also  appeared  that 
these  co-operatives'  investment  was unrelated to the assistance. 

The  third  example  describes  work  carried  out  to  evaluate  the  effectiveness 

of  a consultancy  programme  for  SMEs  in the  UK. 

This  evaluation  used  a  large  number  of  tools  in  the  framework  of  a  very 

complete  procedure  of  observation  in the  field.  The  collection  of  data  was 

carried  out  by  means  of  two  questionnaire  surveys,  a series  of  individual 

interviews  and  case  studies.  The  function  of  estimating  effects  was 

treated  by  means  of  the  comparison  groups  technique  (by  matching 

pairs). 

Examples of evaluations 

Box 21  -  Evaluation  of consultancy  to  businesses  in the  UK 

A  programme  financed  by  the  British  ministry  of  commerce  and  industry  offered 
SMEs management  consultancy, training, and information services. The impact of 
this  programme  could  not  be  expressed  in  jobs  created,  since  it  was  aimed  at 
enhancing  productivity  which,  in the  short  term, led to  the  destruction  of jobs.  It 
was therefore necessary to evaluate the programme in terms of other criteria such 
as productivity,  profitability  or  investments  induced. 
The method  developed for  the evaluation  was  as follows: 

1.An  initial  questionnaire  was  sent  to  the  75  programme  operators  (service 
providers). This questionnaire  concerned the  consultancy  services  provided  to 
the addressees, and the categories  of  businesses  targeted. 

2. Individual interviews  with 20 operators  were used to describe  in greater  detail 

the strategies  developed  by the  operators. 

3. Case  studies  were  carried  out  for  six  operators.  In  these  case  studies,  the 

particulars  of the  addressees  were noted. 

4. A sample of 300 beneficiary firms was constituted. Another sample of 200 non-
beneficiary  firms  was  also constituted, in  keeping with the comparison  group 
technique  (matching  pairs). 

5. A second  questionnaire  was  addressed  to  the  beneficiary  firms  and  to  those 
in  the  matching  group.  A  statistical  comparison  between  the  firms  in  the  two 
groups  provided  quantitative  indications  on  the  effects  of  the  programme,  in 
terms  of  turnover,  productivity  and  investments.  In  order  to  avoid  "biased" 
statements,  the  questionnaires  comprised  numerous  and  partially  redundant 
questions  (triangulated  questionnaire).  The  survey  incorporated  a  search  for 
deadweight  effects. 

To  recap, these  three  examples  show  that  the  evaluation  tools  were  used 
mainly  for  collecting  and  analysing  data.  The  following  pages  show  that 
tools  can also  be chosen  for  structuring  and judgement  functions. 

Tools for  structuring  an  evaluation 

In  a  situation  of  in-depth  evaluation,  the  following  tools  can  be  used 
effectively  for clarifying the effects to  be evaluated  and selecting  evaluation 
criteria and  indicators. 

The  logical  framework  is  a  tool  designed  for  the  ex  ante  evaluation  of 
development  aid projects. It is used to  rapidly clarify  the logic  underpinning 
an action, define expected  impacts  and  construct  impact  indicators. 

METAPLAN®  is  a  tool  which  is  used  extensively  for  group  work.  In 
evaluation,  this  tool  is  particularly  well  suited  to  clarifying  and  grading 
expected  effects. 

The  colour  vote  (described  above)  can  also  be  applied,  particularly  for 
choosing  evaluative  questions. 

By  contrast,  tools  such  as  the  impact  matrix  or  concept  mapping  are  too 
cumbersome  for  situations  characterised  by the  simplicity  of  the  object  to 
be evaluated. 

Tools  for structuring  an evaluation 

Logical  framework 

The  logical  framework  technique  is 

both  an  exercise  in the  structuring 

of  c o m p o n e nt  elements  of  a 

project  (or  single  programme)  and 

a 

t e c h n i q ue 

for  analysing 

t he 

internal  and external  coherence  of 

the  project.  The  product  of 

this 

technique,  the  logical  framework, 

is  a  formal  matrix  presentation  of 

the 

internal 

f u n c t i o n i ng  of  t he 

A 

In-depth 
evaluation 

Θ 

Low 
technical 
complexity 

G 

Φ 

Low 
cost 

Θ 

Prospective 
use 

Use 
stabilised 

project,  of the means  for verifying  the achievement  of the  goal, 

and  of the 

endogenous  and exogenous  factors  conditioning  its  success. 

Example: Logical  framework  of a local  fishing  project 

This  example  concerns  a  project  financed  by  the  European  Union  as  part  of  its 
development  aid policy. 

The  global  objectives  (or aims) of the project  are as follows:  raising the standard of 
living, guaranteeing a more stable food supply, increasing export earnings, increasing 
employment, and using  resources  in a sustainable way. 

The  specific  objective  of the project  (or goal)  is a  higher  income  for the  economic 
actors in the local fishing  sector. To evaluate the achievement  of this goal, a series of 
objectively  verifiable  indicators  can be  used:  evolution  of  income  from  fishing  or 
distribution  of  income  by  economic  actor  (fishermen,  industries,  transport  firms, 
wholesalers, fishmongers), gender  and  fishing  zone. 

Among the factors  conditioning the achievement  of these goals  are, in particular, the 
need  for  sufficient  quantities  of  consumer  goods  at  affordable  prices,  as  well  as 
sufficient  accessibility  of other  services  (health, education,  advisory  services, etc.). 

The  expected  results  are:  optimum  production,  optimum  processing  of  fish, 
satisfactory  commercialisation  of  fish  production,  and  adequately  defined  local 
fishing zones. An evaluation  of the achievement  of these  results may rely on a series 
of  objectively  verifiable  indicators:  number  of  catches,  number  of  products  sold, 
distance between the fishing zone and the shore, types of fish caught, duration of the 
fishing,  balance  in the distribution  of  catches  between  fishermen,  duration  of the 
sale,  time  between  delivery  and sale,  depth  of  resources,  and agreement  between 
fishermen  on fishing  zones. 

Among  the factors  conditioning  the achievement  of the results  is, in particular, the 
fact  that  demand  has to be great  enough, as does the level of prices for producers. 

Principal evaluation  techniques and tools 

The following  activities  are part  of the  project:  creation  of fishing and  fish  processing 
co-operatives  -  particularly  by  women  -  making  fishing  equipment  available,  drying 
sheds  and  warehouses,  organisation  of  transport,  information  on  demand  and  the 
market,  training  -  particularly  for  women  -  in  new  techniques,  training  in  new 
packaging  techniques,  adequate  information  on  potential  fishing  zones,  negotiation 
of  agreements  between  fishermen,  organisation  of  a  system  of  control  over  fishing 
activities,  etc. 
Source:  European  Commission  -  DG VIII 

What  is the  purpose  of  the  tool? 

In so far as  it  is a technique  supporting  the objective  definition  of a  project, 
as  well  as  its  formulation  in  operational  terms  and  its  implementation, 
monitoring  and  evaluation, the  logical  framework  facilitates: 

•  the  systematic,  synthetic  and  comprehensible  description  of  a  project; 

•  the  articulation  and  clear  grading  of  objectives; 

•  the  clarification  of the  conditions  for the  success  of the  project; 

•  the  clarification  of  the  assumptions  peculiar to  the  project; 

•  the  identification  of  the  causal  links  between  resources,  results  and 

objectives; 

•  and,  finally,  the  identification,  during  the  planning  of  the  project,  of 
success  criteria  and  means  for  verifying  the achievement  of  objectives. 

is  most  effective  when 

The  technique 
team 
interaction and applied as soon  as the  problem that the project  is  supposed 
to solve  has been  identified. It will continually  evolve and expand  in relation 
to  the  environment  of  the  project  and  thus,  by  extension  will  allow  for  the 
complete  organisation  of  all the  phases  of  the  "project  life  cycle". 

it  is  prepared  through 

The  logical  framework  is a tool  which  facilitates  the  formulation  of  the  key 
stages  of  a  project:  it  is  used  to  draw  up  a  detailed  budget,  to  allocate 
responsibilities,  to  plan  activities,  and  to  draw  up  the  monitoring  schedule 
and terms  of  reference  of  on-going  and  ex  post  evaluations. 

In which  cases  can  it  be  used? 

Conceived about thirty years ago in the United States, the logical  framework 
technique  has  progressively  become  the  preferred  methodological  tool  of 
development  project  planners.  The  approach  is  currently  used  in  most 

Tools  for structuring  an  evaluation 

national  and  supranational  cooperation  bodies,  including  the  European 
Union. 

Its  success  owes  much  to  its  capacity  to  describe,  in  an  ordered,  causal 
and  logical way, the internal functioning  of a project  in a given  environment. 
It  is this  faculty  which  has enabled  it  to  impose  itself  so  rapidly  in the  field 
of  development  co-operation. The logical framework  technique  has  proved 
to  be  ¡deal  in the  context  of  simple  programmes  where the  objectives  and 
main  actors  (from the sponsors  to the  operators)  are clearly  identified. 

How  should  it  be  implemented? 

The application  of a logical framework  to a project  necessitates  a  complete 
analysis  of  its context,  objectives  and  strategies. 

Step  1. Analysis  of  the  context:  The  context  is  the  general  framework  in 
which  the  project  is  situated.  By  analysing  this  context,  the  problems  that 
the  project  must  help to  solve  are identified. The following  are required: 

•  a definition of the political, institutional, social and economic  environment 
(government  policy, characteristics  and  prospects  of  the sector,  etc.); 

•  an analysis of the motives  and expectations  of the  beneficiaries and  main 

actors; 

•  an  identification  and  grading  of  the  problems  to  be  solved  and  their 

probable  causes. 

Step 2.  Analysis  of  objectives: The  problems  that  the  project  is  aimed  at 
solving  lead  to  the  definition  of  the  project's  objectives.  Formulating  or 
reformulating  them  consists  of focusing  on: 

•  the  prospects  and  the  description  of  the  future  situation  when  the 

problems  have  been solved; 

is  not  merely  discursive. 

•  the  grading  of  objectives  in  terms  of  goals  and  aims.  This  distinction 
between  goals  and  aims 
It  meets  a 
methodological  need  in so far as the construction  of a logical  framework, 
in  order  to  be  coherent,  requires  the  use  of  a  limited  number  of  main 
objectives  (or aims)  and specific  objectives  (or goals). The use of  specific 
tools  for  grading  objectives,  of  the  "objectives  tree"  type,  is  necessary 
here. 

•  the  visualisation  of  the  set  of  relations  between  resources  mobilised  and 

aims. 

Principal  evaluation  techniques and tools 

Step  3. Analysis  of the  logic  of the  project 

The  logic  of  a  project  generally  consists  of  four  different  "value  levels": 
resources or inputs  (e.g. human, financial, material  or regulatory  resources), 
products  (outputs),  main  goals  (specific  objectives)  and  aims  (global 
objectives).  These  levels  are  related  to  each  other  by  assumptions  of 
causality  (of the  "if...then..." kind). Whereas the first three  levels are  internal 
to  the  project,  the  fourth,  that  of  aims,  links  the  project  to  a  broader 
programme  or  policy  which  is frequently  defined  outside  of  the  sphere  of 
management. 

Thus,  the  vertical  logic,  or  intervention  logic,  is  intended  to  clarify  the 
intentions  of  the  project  while  specifying  the  uncertainties  related  to  their 
fulfilment.  It serves to describe the  logical articulation  of the different  "value 
levels"  as  well  as the  critical  conditions  influencing  their  realisation.  These 
critical  conditions,  whether  internal  or  external  to  the  project,  are  the 
expression  of  the  constraints  needed  to  move  from  one  "value  level"  to 
another  in  the  logical  framework.  These  moves,  of  which  the  causal  links 
are expressed  in the form  of  assumptions,  have to  be formulated. 

The  horizontal  logic  is  aimed  at  identifying  and  describing  precisely  that 
which  is  produced  at  each  of  the  four  "value  levels",  by  means  of  two 
concepts: 
Indicators  (OVI)  and  Sources  of 
Verification  (SoV) of  these  indicators. 

the  Objectively  Verifiable 

The OVI correspond to  all the criteria  likely to show that what was  expected 
was  also  produced.  They  are,  in  a  sense,  the  "operationalisation"  of  the 
declaration  of  intent  of  the  intervention  logic, thus  providing  tangible  proof 
of  success  at  each  of the  four  levels: inputs,  outputs,  goal  and  aim. 

The  SoV  of  these  indicators  are  described  by  detailing  the  means  and 
sources  of  information  needed  to  obtain the  required  data. 

The  inter-related  nature  of  the  elements  of  the  logical  framework  can  be 
presented  schematically  as  follows: 

Tools for structuring  an evaluation 

Box  22  -  Basic  elements  of  t he  logical  f r a m e w o rk 

Intervention 
logic 

AIM 
Global  impact 
Raison d'être  of 
the  programme  to 
which the  project 
belongs 

Objectively 
verifiable 
indicators 

Sources  of 
verification 

Critical 
conditions 

Measurement 
of  the  decrease 
in needs  that 
have to  be  met 

Data from 
outside 
the  project 

GOAL 
Specific  impact 
Raison d'être 
of the  project 

Elements  of 
the  success 
of  the  project 
from the  recipients' 
point  of view 

Data from 
outside 
the  project 

OUTPUTS 
All that  has 
to  be  produced 
in order  to 
achieve  the  goal 

Indicators  used 
to  quantifiy 
and  qualify 
the  outputs 
and  correlation 
with  planned 
production  dates 

Data from 
within 
the  project 

INPUTS 
All the  resources 
needed to  produce 
the outputs  and  to 
achieve the goal 

Monitored 
inputs 

Data from 
within 
the  project 

Causality 
assumptions 
"if...goal, 
then...aim" 

Causality 
assumptions 
"if...outputs, 
then...  goal" 

Causality 
assumptions 
"if...inputs, 
then... outputs" 

Preliminary 
conditions 
before  project 
launching 

Source:  adapted 
from  ACDI  (1981)  Guide  d'utilisation  de  la  Méthode  du  Cadre 
Logique  dans  la  gestion  et  l'évaluation  des  projets  de  coopération  de  l'ACDI, 
Toronto:  ACDI. 

Application  to  the  Structural  Funds 

The  advantage  of  this  tool  for  the  evaluation  of  Structural  Funds  is  its 
capacity  for  clarifying  and  structuring  public  action  and  its  intentions.  Its 

Principal  evaluation  techniques and tools 

contribution  is further  enhanced  when  the  public  action  can  be  organised 
around  a  single  main  objective,  taken  here  to  be  the  aim.  The  logical 
framework  technique  is  used  in  this  case  as  a  procedure  for  analysing 
overall  coherence. 

Within  a  complex  programme  the  tool  helps  to  structure  the  objectives  in 
the  form  of  a  tree.  The  trunk  is  the  aim  that  branches  out  into  sub-
programmes,  priorities,  measures  and  projects. The  advantage  of  this  type 
of  complete  interpretation  in steps  is its simplicity,  but  it often  proves  to  be 
ineffective  for  grasping  complex  realities. 

Strengths  and  limits  in  practice 

•  As  an  analytical  process  for  the  planning  and  monitoring  of  projects,  the 
logical  framework  technique  has  proved,  in  practice,  to  be  a  structuring 
frame  that  is  rated  highly  by  practitioners.  It  is  a  synthetic  tool  that 
clarifies links between means and ends, as well as exogenous  constraints 
and  the  solutions  that  can  be  used  to  guard  against  their  influence.  The 
tool  also facilitates the  establishment  of a monitoring  and  control  system 
from  the  planning  stage  of  the  project.  It  is,  moreover,  used  to  delimit 
precisely the responsibilities of each actor  involved in the execution of the 
project. 

•  This technique  provides a framework  for  examining the  logic of a  project, 
but those persons  responsible for the  project have to fill in the  framework. 
This technique  should therefore  be implemented  with the  participation  of 
the  partners  involved  in the  project. 

•  The  logical  framework  technique  is  more  difficult  to  apply  to  cases  of 

complex  projects  and  programmes  with  multiple  aims. 

•  Although  this tool clearly  lends  itself  more  readily to  use in the context  of 
ex ante analyses or monitoring. For ex post  evaluations  its contribution  is 
one of complementing  other, less systematic  approaches that focus  more 
directly  on  the  main  evaluation  questions. 

•  The  presence  of  several  decision-makers  (sometimes  with  differing 
interests)  may  cause them to design  a programme  with  a  "vague"  outline 
and  mechanisms.  In  such  cases  only  the  presence  of  intermediate 
phases  of  collective  reflection  and  the  regular  emergence  of  strong 
consensus  can  ensure the  tool's  potential  is fully  realised. 

Tools  for structuring  an evaluation 

•  If the  application  of  the  logical  framework  technique  is  seen  merely  as a 
compulsory  technocratic  exercise in the planning of the project, its ability 
to reflect the evolution of the state of the  project during  its application  will 
be  strongly  undermined.  This  could  be  harmful  to  all  the  documents 
obtained  by  extension, such  as the terms  of  reference  of  an evaluation. 

Further  reading 

Commission  of 

'Project  cycle 
the  European  Communities 
Framework'. 
management  :  'Integrated  approach  and  Logical 
Brussels  :  Commission  of  the  European  Communities,  Evaluation 
Unit  Methods  abnd  Instruments  for  Project  Cycle  Management. 
Brochure  presenting  the  project  cycle  management. 

(1993), 

Agence  canadienne  de  développement 

international 

(1981)  Guide 
d'utilisation  de  la  Méthode  du  Cadre  Logique  dans  la  gestion  et 
l'évaluation  des  projets  de  coopération  de  l'Agence  canadienne  de 
développement  international, Toronto:  ACDI. 
Reference  manual  of  the  Evaluation  Division  of  the  Canadian 
International  Development  Agency. 

Cracknell,  B.  (1989),  'The  Logical  Framework  :  A  project  Management 

technique'  in The Crown  Agents  Review,  N°3. 

Cracknell, B. (1989), 'Evaluating  the Effectiveness  of  the Logical  Framework 
System  in Practice'  in Project Appraisal, 4(3). Guildford  : Beech Tree 
Publishing. 
Evaluation  of  two  American  and  Canadian  experiences  dealing  with 
the  development  of the  Logical  Framework  technique. 

Swedish  International  Development  Cooperation  Agency,  Department  for 
Policy  and  Legal  Services  (1996)  Guidelines  for  the  Application  of 
LFA in  Project  Cycle  Management. 
Guidance  manual  on  how  to  use the  Logical  Framework  technique. 

Tools for structuring an evaluation 

M E T A P L A N® 

The METAPLAN® technique (a 

brand name registered by a 

German firm) is used for 

structuring group discussions. 

Originally designed to facilitate 

collective decision making in urban 

planning, it has spawned 

numerous variants. All are based 

on the same principles: asking 

participants in a meeting for their 

opinions, structuring and 

synthesising the information 

gathered, and grouping it together 

O 

Overall 
evaluation 

A 

In­depth 
evaluation 

Θ 

Medium 
technical 
complexity 

O 

<D 

Low 
cost 

O 

Retrospective Prospective 

use use 

Θ 

Use 
stabilised 

in a form on which consensus is reached by the participants, 

by weighting 

the ideas produced in the meeting. 

Example: 

In the context of the launching of intermediate evaluation of the Objective 2 
programme in Franche­Comté (France), the first meeting of the evaluation steering 
group was primarily aimed at producing a list of the main impacts expected from the 
said programme. The METAPLAN® tool was used for this purpose. 
Outside consultants, who had been trained to use the tool, were responsible for 
planning the session. The participants (13 persons) were mainly administrative officials 
from local authorities, including a representative of the commissioner of the evaluation. 
The session, which lasted about two hours, was intended to reveal the main 
impacts expected from the programme, bearing in mind that these impacts had to 
serve as a basis for the evaluation and, in particular, the formulation of specific 
evaluative questions. 

The three questions that served as a basis for discussions in the METAPLAN® 
session were formulated as follows: "What do you think of the usefulness of the 
programme?" "On which impact would you like information?" and, finally, "Which two 
impacts are to be evaluated?" The latter question was used to grade the priorities. 
Owing to the way the process was structured, the steering group soon reached 
consensus on a list of five impacts which appeared as particularly important to 
evaluate: "Reskilling", "Adaptation of training", "Attractiveness of the region", 
"Reconversion / Diversification" and "Local initiative jobs". 

Source : Rapport C3E (1998) L'évaluation intermédiaire des interventions 
structurelles dans la zone d'objectif 2 de Franche­Comté 

Principal evaluation  techniques and tools 

What  is the  purpose  of  the  tool? 

METAPLAN®  is used to  group together  and structure  information  based  on 
multiple  points  of  view,  and  to  produce  a synthesis  acceptable  to  most  of 
the  participants  in  the  meeting.  This  type  of  tool  meets  several  types  of 
need:  extracting  information  and  aggregating  it  by  means  of  questions 
asked  in  an  appropriate  way,  while  respecting  the  viewpoints  of  all  the 
members  of the  group; increasing the acceptance  of decisions taken  within 
the group  because all points  of view are taken  into account,  including  those 
which  are  not  dominant  within  the  group. 

In which  cases  can  it  be  used? 

This  is  a tool  used  for  discussion  rather  than  decision-making.  It  helps  to 
identify  the  obstacles  and  resistance  to  the  successful  accomplishment  of 
an action  or strategy,  and  highlights  the  likely  solutions  to  these  problems. 
It  is  used  to  structure  an  action  composed  of  multiple  priorities.  With  this 
tool,  it  is possible to  reduce tension within  a group,  particularly  through  the 
participation  of  all the  stakeholders  and  through  discussion  and  weighting 
techniques.  Finally,  this  type  of  tool  may  be  used  in the  construction  of  a 
common  representation  of  the  problems  and  solutions  to  be  implemented, 
which  can  help  stimulate  a common  will  to  act. 

How  should  it  be  implemented? 

The  sequence  of  steps  must  necessarily  be  entrusted  to  an  outside 
moderator  who  is  neutral  and  familiar  with  the  tool.  It  will  be  up  to  him  or 
her to  prepare  and  chair the  meeting, and to  present  the  results. 

Step  1.  Preparation  of  the  agenda:  The  agenda  consists  of  three 
questions  on  which  the  session  will  be  based.  The  formulation  and 
articulation  of  these  questions  is  important  in  so  far  as  they  must  be 
sufficiently  open  to  leave  room  for  a wide  range  of  answers,  and  must  be 
expressed  in such  a way that  the  participants  can and want  to answer. The 
order  in  which  the  three  questions  are  asked  is  also  important.  The  first 
question  or  "introductory  question"  is  used  to  illustrate  the  functioning  of 
the technique  and to  establish  a common  system  of  reference. The  second 
question  or  "card  question"  is  the  core  of  the  approach:  it  is  a  more  in-
depth  consideration  of  the  first  question.  Its  treatment  will  be  more 
elaborate  in so far as the answers will be discussed  in more detail. The third 

Tools for structuring  an  evaluation 

question, known  as a "weighting  question", will be used to choose  between 
the  answers  given,  in  order  to  establish  priorities  for  action.  These  three 
questions  must be prepared in advance by the moderator who ensures their 
appropriate  articulation  and the  progressive  nature  of the  procedure. 

Step  2. The  work  session: The  meeting  as such  consists  of  a single  work 
group.  Depending  on  the  size  of  the  public,  several  sub-groups  may  be 
created  during  work  on the  second  question  (card question).  It is,  however, 
important  that  the  work  organised  around  the  first  and  third  questions 
(introductory  question  and weighting  question)  involve  all the  participants. 

The  moderator  poses  the  "introductory  question"  and  asks  for  answers  in 
the  form  of  simple  assertions  that  he  or  she  writes  down  on  Post-its 
(maximum  30).  These  are  stuck  onto  a  paperboard,  forming  "blocks"  by 
grouping  together  answers  that  are  similar  or  underpinned  by  the  same 
logic.  This  initial  question  enables  the  participants  to  create  a  common 
representation  of  the  problems,  to  give  the  same  names  to  the  same 
concepts,  and  to  understand  the  method  of  discussion. 

The  "card  question"  is  the  most  important  because  it  deals  with  the  true 
object  of the  meeting  and  will serve as a basis for the group discussion. To 
answer  this  question,  the  participants  are  divided  into  groups  of  two  or 
three  persons  and  are  asked  to  provide  answers  in  the  form  of  short 
sentences  which  one  person  in the  group  writes  on the  Post-its  (3 to  5 per 
group). The  moderator  collects  these  answers  and  then,  reading  each  one 
out  aloud, displays  them  on the  board, trying  to  group  them  into  "blocks". 
The  participants  may  contest  the  answers  or  ask  for  further  details.  In this 
case the  moderator  puts  a  "flash"  sign  on the  Post-it  concerned  (the  flash 
rule). Further details on the question may be provided  and  may  be noted  on 
a  Post-it  of  a  different  colour  stuck  next  to  the  flash.  Gradually,  with  the 
participants'  help,  the  moderator  completes  the  "blocks"  of  answers  (6 
post-its  per  "block"  at the  most)  which  should  be  underpinned  by a similar 
logic. The participants  must then consult each other to find titles  describing 
the content  of the  different  "blocks". 

The  "  weighting  question"  will  be  used  to  show  the  importance  that  the 
different  participants  attribute  to  the  different  "blocks".  The  participants 
answer  the  question  by  pasting  stickers  onto  those  blocks  that,  in  their 
view,  provide  the  best  answer. 

Principal evaluation techniques and tools 

The following is an example for these three types of question: 

1. Introductory question 

2. Card question 

3. Weighting question 

"What do you expect 
from a training 
policy?" 

"On which specific 
points would you 
like the evaluation 
of a training 
policy to focus?" 

"Which elements should 
be treated as a priority 
by the evaluation?" 

The first question serves to launch the debate on the impacts of a training 

policy and to initiate reflection on possible typologies of effects. The 

answers given will also be used to guide the answers subsequently given 

to the framework q u e s t i o n. 

The s e c o nd question serves to consider in more d e p th the points 

addressed in the first question, but also to a dd information. 

The third question is used to select elements of answers that seem to be 

priorities. 

The following diagram summarises the last t wo phases of a METAPLAN®: 

Box 23 ­ Structuring an evaluation using the M E T A P L A N® technique 

Answers to the "card 
question" are noted on 
Post­it notes and placed 
on the paper board 

Answers are grouped 
together into "clusters" 
with a common logic. 
A title is given to each cluster 

The participants grade 
the clusters by weighting 
them with stickers 

Cluster 
Α 

Cluster 

Π 
π  π 
EP 
π u 
□ π 

Tools for structuring  an  evaluation 

•  Application  to the  Structural  Funds 

METAPLAN®  is a tool  with  several  possibilities  for  use in the framework  of 
a Structural  Funds evaluation: 

It can  serve to  clarify  the  logic  underlying  actions  and  to  identify  measures 
that  need  to  be  considered  in  more  depth.  It  is  a structuring  tool  that  may 
be  used  by  programme  managers  during  preparatory  meetings  before  the 
launching  of  evaluations. 

METAPLAN®  can also serve as a basis for the definition of evaluation  plans 
and  terms  of  reference.  It  facilitates  the  definition  of  evaluation  questions 
(e.g.  based  on  the  questions  proposed  above)  and, from  these  questions, 
the  selection  of  evaluation  criteria. 

The  advantage  of  this  tool  is  well  suited  to  a  pluralistic  approach  to 
evaluation  in  which  the  stakeholders  are  involved  in  the  selection  of 
evaluation criteria. This helps to  increase the  use of  evaluation  results  since 
the  criteria  are accepted  by a majority  of  stakeholders. 

Strengths  and  limits  in  practice 

•  METAPLAN®  is  a  structuring  and  formalising  tool  that  promotes  the 
appropriation  of  conclusions  by  those  who  have  participated  in  the 
meeting,  irrespective  of  the  number  of  people.  Thus,  it 
facilitates 
consensus  building. 

•  This  tool  allows  for  the  identification  of  points  of  conflict  and  makes  it 
possible  to  overcome  situations  of  deadlock  which  tend  to  appear  in 
meetings.  Since  each 
into  account  and 
discussed,  it  is  generally  possible  to  avoid  situations  in  which  a  few 
individuals  dominate  the  entire  group. 

idea  proposed 

is  taken 

•  Whereas  METAPLAN®  is a  tool  that  facilitates the  formalisation  of  ideas, 
it is not a tool for creating  new ideas. It expresses  nothing  more than what 
is  in  the  minds  of  the  participants  during  the  meeting.  If  implicit 
agreement  exists  on  solutions  to  implement  (e.g.  same  idea),  false 
solutions  on  which  there  is consensus  may  emerge.  In other  words,  this 
type  of tool  can  also  promote  deadlock. 

•  METAPLAN®  requires  the  presence  of  a  qualified  person  if  it  is  to 
produce  sound  results. The formulation  and  sequence  of  the three  types 
of  question  are  important  and  have  to  be  prepared  in  respect  of  both 

Principal  evaluation  techniques  and  tools 

content  and  form.  The  same  applies  to  the  group  leadership.  In  view  of 
its apparent  simplicity,  it  may  be tempting  to  organise work  sessions  with 
this  tool,  despite  a  lack  of  information.  In  such  cases  the  results  of  the 
work  are  likely  to  be  disappointing  and  to  have  repercussions  on  the 
entire  group. 

Further  reading 

Schnelle,  E.  (1979),  The  METAPLAN®-Method  :  Communication  Tools  for 

Planning  and  Learning  Groups.  Hamburg:  Quickborn 

Tools for observing changes in the field 

The  questionnaire  survey  is  the  simplest  observation  tool.  Administered 
through  the  post,  by  telephone  or,  more  recently,  by  e-mail,  it  is  used  to 
collect  numerous  homogeneous  data  that  can  serve  for  statistical 
purposes.  This  tool  is  not  suitable  for  ex  ante  evaluation.  Several  tools 
presented  above  are  also  suited  to  observation  in  the  field,  in the  case  of 
in-depth  evaluation:  individual  interviews,  focus  group  interviews  and 
case  studies. 

Finally, ethnographic  observation  is used to collect  complete  and  detailed 
information  in cases where the  populations  are difficult  to  reach or  reluctant 
to  participate  in a survey. 

Tools for observing changes 

Questionnaire  survey 

The  questionnaire  survey  consists 

of  putting  a  series  of  standard 

questions  in a structured  format  to 

a  sample  of  individuals  who  are 

usually 

selected 

as 

being 

representative  of  the  population 

under 

observation. 

It  may 

sometimes 

be 

exhaustive, 

covering 

the  whole 

of 

this 

population. 

A 

In-depth 
evaluation 

S 

Medium 
technical 
complexity 

O 

Φ} 

High 
cost 

Θ 

Retrospective 
use 

Use 
stabilised 

The  development  of  questionnaire  surveys  and  sampling  was  based  on  the 

first  precise  forecasts  of  election  results  obtained  during  the  inter-war 

period  by  opinion  poll  institutions.  Considered  as  one  of  the  basic  tools  of 

the  social  sciences,  questionnaire  surveys  are  today  used  by  most  public 

and  private  research, survey,  statistics  and  evaluation  institutions. 

Example: A survey  carried out among  employers  on the effects  of employment 

policies 

This  example  corresponds  to that  of Box  19. 

The  French  Ministry  of  Labour,  Employment  and Occupational  Training  frequently 
uses  questionnaire  surveys  (generally  telephone  surveys)  to  evaluate  interventions 
aimed  at  promoting  employment.  A  survey  was carried  out in  1990 to  evaluate  a 
measure  aimed  at promoting  the employment  of the long-term  unemployed,  in the 
form  of  an  exemption  of  social  charges  for  employers.  A  questionnaire  was 
submitted  telephonically to the employers  of a representative  sample of about 1,000 
jobless  persons  who had benefited  from  the  measure.  The sample  covered  only 
employers  and  unemployed  persons  affected  directly  by the measure. 
In each firm  the person  questioned  had enough  knowledge  about  the decision  that 
had been  taken to recruit the jobless  persons  and to give  information  about the role 
played  by the offer  of exemption  from  social  charges. To avoid  biased  answers the 
employers  were  advised  that  the questionnaire  would  be treated  anonymously. The 
questionnaire  included  several  cross-related  questions  designed  to  reveal  what 
would  have  happened  in  the  counterfactual  situation  i.e.  in  the  absence  of the 
exemption.  These  included  no additional  recruitment,  recruitment  of a  person  with 
different  characteristics,  recruitment  of the same  person  but in different  conditions, 
etc. The survey  results were  used to estimate, for each firm, the level of  deadweight 
(i.e. the number of unemployed  people who would have been employed even without 
the  exemption)  and  substitution  effects  (i.e. the  extent  to  which  the  exemption 

Principal evaluation  techniques and tools 

encouraged  the  employer  to  recruit  an  eligible  jobless  person  rather  than  another 
non-eligible  person). 

Observation  of  the  entire  sample  provided  an  estimation  of  the  average  coefficients 
of  deadweight«  and substitution  effects. Since the sample  was  representative,  these 
coefficients  were  generalised  to  all the  firms  benefiting  from  the  exemption,  in  order 
to  estimate  the jobs  created  by  the  measure. 

Source:  "Aides  publiques  à  l'embauche:  quels  effets  sur  l'emploi"  -  Premières 
synthèses,  N°6,  Mai  1990,  Ministère  du  Travail  de  l'Emploi  et  de  la  Formation 
Professionnelle 

What  is the  purpose  of  the  tool? 

In the  evaluation  context,  questionnaire  surveys  are used  mainly  to  collect 
addressees'  opinions.  The  questions  asked  may  be  descriptive  (e.g.  How 
did you  learn about the existence of the support?), normative (e.g. Does the 
service  provided  correspond  to  a  priority  need  in  your  organisation?)  or 
causal 
the  evaluated 
intervention?). 

(e.g.  What  would  you  have  done  without 

In which  cases  can  it  be  used? 

The  advantage  of  questionnaire  surveys  is  that  they  produce  results 
applicable  to  the  whole  of  the  observed  "population",  either  directly  if  the 
survey  is  exhaustive  or  by  extrapolation  from  a  sample.  However,  the 
minimum  size  of  a  sample  that  can  be  generalised  in  this  way  is  around 
1,000  units. A questionnaire  survey  must therefore  be  used  in cases  where 
the  population  to  be  observed  is  large  and  homogeneous  (e.g.  trainees, 
firms, travellers, etc.). 

Questionnaires  are  most  useful  when  the  investigator  has  a  precise  and 
clear  idea  of  what  he  or  she  wants  to  observe.  In  this  case  the  simplest 
survey  consists  of  closed  questions  to  which  a  series  of  replies  are  given 
from  a  number  of  predetermined  responses.  There  are also  cases  where  it 
is  useful  to  use  questionnaires  that  allow  responses  to  be  directed, 
particularly  when  the  investigator,  prior  to  the  survey,  does  not  have  clear 
ideas  about  the  opinions  of  respondents  or  the  way 
the 
intervention  works. 
In  this  case  a  combination  of  open  and  closed 
questions  can be useful. It must however be noted that the analysis of open 
questions  is more complex  and expensive since, once the primary data has 
been  collected, the  investigator  has to  determine, group  and codify  similar 

in  which 

Tools for observing changes 

is  particularly  well  suited  to  the  production  of 
responses.  This  tool 
descriptive  information  and  to  make  classifications  in  different  categories 
(e.g. a  breakdown  of  assisted  firms  in terms  of their  size, sector  of  activity, 
age,  nature  of  their  market,  etc.).  It  can  also  be  used  to  gather  both 
qualitative  and  quantitative  information.  However,  subjective  data  must  be 
carefully  interpreted  when  the  evaluation  tries  to  establish  causality  or 
provide  normative  information. 

A single sample (panel) may be questioned  in two or more successive  periods 
in  order  to  provide  longitudinal  data  allowing  the  monitoring  of  beneficiaries 
and the estimating  of the sustainability  of effects.  For example, panel surveys 
are  used  to  evaluate  the  sustainability  of  results  obtained  in  the  field  of 
employment  (e.g. trainees are questioned  before  and after their training. 

How  should  it  be  implemented? 

If a questionnaire  survey  is to  be effective, it should consist  of the  following 
four  steps: 

Step  1. Designing  the  questionnaire: The first  step  is to  rigorously  define 
the  objectives  of  the  survey  and  devise  appropriate  questions.  Questions 
need  to  be  stated  clearly  and  formulated  in  an  interesting  way.  It  is 
important  not  to  multiply  secondary  questions  which  could  confuse 
respondents  and/or  interviewers thus undermining the quality  of the survey. 
Where  comparisons  with  other  data  sets  are to  be  used, the terms  used  in 
a questionnaire  must  be carefully  chosen  so  as to  ensure their  coherence. 

Most  questionnaires  are designed to ask primarily closed  questions  (choice 
between  a limited  number  of  answers), though  many  combine  these  with  a 
small  number  of  open (spaces to  be filled  in by the  respondent  themselves) 
or  semi-open  questions  (a  list  of  options  with  the  answer  "other"  included 
to allow  respondents  to give other opinions/answers  to those envisaged  by 
the  investigator).  In  the  latter  two  cases  however  the  need  to  codify 
answers after the data  is collected  may pose problems of  interpretation  and 
comparability  if  respondents  raise  very  different  issues  from  each  other. 
However  it is also often a useful ways of ensuring that  issues overlooked  by 
the  investigator  at the  outset  are not  excluded  from the  eventual  analysis  if 
they  emerge  as  being  important  for the  addressees  of  a  programme. 

It  is  essential  to  ensure  that  the  formulation  of  the  questions  does  not 
influence  the  answers.  It  should  also  be  noted  that  questions  requiring 
respondents  to  recall  past  events,  situations  and  decisions  are  usually 

Principal  evaluation  techniques and tools 

'double  negative'  questions.  The 
considered  problematic.  As  are 
importance of  phrasing questions depends on the method that will be used. 
Thus  in  the  case  of  postal  surveys  or  other  self-completion  surveys  it  is 
absolutely  vital  to  ensure  that  questions  can  not  be  misunderstood  by 
respondents.  This  may  be  less  important  where  surveys  are  administered 
face  to  face  and  the  interviewer  has  an  opportunity  to  clarify  questions  or 
prompt  respondents  where  they  fail  to  understand  the  question.  The 
capacity  to  do  this  though  will  depend  on  the  level  of  experience  and 
understanding  of  the  interviewers  and,  unless  they  are  experienced  and 
senior  evaluators,  it  is  unwise  to  rely  on  them  to  resolve  ambiguities  in 
question  design. 

The  questions  may  be  nominal  (e.g.  Activity  of  your  organisation: 
agriculture,  industry,  trade,  other),  ordinal  (e.g. What  did  you  like  most  in 
this training:  its  quality,  price,  relevance?)  or  numeric  (e.g. How  many  days 
of training  did  you  receive?). 

If  the  questions  are  normative  or  causal,  it  will  be  necessary  to  use 
triangulation. Triangulation  consists  of  asking  several  similar  questions  that 
make  it  possible  to  define  the  limits  of  an  answer  that  presents  a  risk  of 
bias.  For  example,  the  following  three  questions  allow  one  to  estimate 
deadweight:  Without  the  support,  would  you  have  made  an  investment  of 
the  same  size,  a  little  smaller,  much  smaller?  ... At  the  same  date,  a  little 
later, much  later?  ... With the same technology,  with  a different  technology? 

Step 2. Sampling: The population  to  survey  must  be defined  precisely (e.g. 
all firms with  between  1 and 50 employees, which  received  assistance  over 
the  past  5  years)  and  available  information  on  the  structure  of  this 
population  must  be gathered. One must  also  check  whether the  population 
is known  (names, addresses  and telephone  numbers)  or  unknown. 

When  a  sample  is  used,  it  constitutes  a  model  of  the  entire  population.  In 
order  to  be  representative,  it  must  be  large  enough  and  must  comply  with 
precise  statistical  rules. The following  indications  are highly  simplified  rules 
that  must  be  verified  on  a  case  by  case  basis  by  a  professional.  The 
standard size of a representative sample is about  1,000  individuals. The size 
of the sample does not depend on that  of the  population; a sample of  1,000 
persons  may  represent  a  population  of  2,000,000  persons  or  one  of  6,000 
persons.  The  size  of  the  sample  does,  however,  depend  strongly  on  the 
required  precision.  One  has  to  multiply  the  number  of  questionnaires  by 
four  to  double  the  precision.  Finally,  the  size  of  the  sample  must  also  take 

Tools  for observing changes 

into  account  the  heterogeneity  of  the  answers.  For  the  most  important 
questions,  an  early  estimation  of  the  proportion  of  respondents  who  are 
expected  to  give  a  particular  answer  must  be  made.  If  20  to  80%  of  the 
respondents  are expected to choose one of the answers, a sample of  1,000 
individuals  will  be  enough  to  make  the  answer  valid.  If  5%  or  fewer 
respondents  choose  one of the  answers, the sample will  have to  be  bigger. 
The survey  provides  a valid figure for an answer  if at  least  100  respondents 
have  chosen  this  answer. 

Several  techniques  are  proposed  in  the  following  table  for  constructing  a 
representative  sample.  Only  the  last  two  are applicable  if the  population  is 
unknown  or  hardly  known. 

Box  24  -  Sampling  tecl 

iniques 

Technique 

Principle 

Simple  random 
selection 

Systematic  selection 

Stratified  random 
selection 

Selection  by  clusters 

By  quota 

"Snowball" 

Each  observation  unit  is  selected  randomly.  It  has 
an  equal  chance  of  being  part  of  the  sample  and 
can  be  selected  only  once 

1  individual  out  of  10  or  100  is  chosen;  the  first 
individual  is chosen  randomly 

the 

population 

When 
particularly 
homogeneous,  it  is  divided  into  homogeneous  sub­
groups  and  a  sample  is  then  taken  in  each  stratum 
thus  constituted 

not 

is 

The  population  is  divided  into  sub-categories  or 
clusters  (e.g.  Firms  assisted  by  the  municipality); 
some  clusters  are chosen  randomly  and  then  all the 
individuals  in a cluster  are  questioned 

A  reduced  model  is  constructed,  consisting  of  the 
known characteristics  of the  population  under  study; 
the  researchers  then  have  to  find  the  individuals 
corresponding  to  these  characteristics 

Based on a sample with a limited number of  persons, 
more  units, related  to the first  ones, are  added 

Step  3.  Pre-test:  A  pre-test  or  pilot  of  the  questionnaire,  on  a  limited 
number  of  respondents,  is  essential.  It  provides  a critical  view  of  the  form 
and content  of the questionnaire, thus enabling  one to assess  its  relevance. 
The  final  version  of  the  questionnaire  will  then  be  drawn  up.  This  is 

Principal  evaluation  techniques and tools 

particularly  important  in  the  case  of  self-completion  surveys  and  those 
which  involve  complex  answers  or  'filters'  (ways  of  steering  respondents 
through  a survey to those  questions  which  are relevant to their  experiences 
-  thus  for  example  ensuring  that  only  the  users  of  service  give  opinions 
about  it  and that  those who  have  no  basis for  making  judgements  do  not). 

Step  4. Administration  of  the  questionnaire:  The questionnaires  may  be 
sent  by  mail or  Internet  or distributed  by  hand - for  example  at  a  particular 
event  which  the  researcher  is  seeking  feedback  about.  In  these  cases 
surveys  are self-administration  or self-completion  questionnaires.  In  others 
they  may  be  completed  with  the  help  of  an  interviewer,  for  example  being 
carried  out  over  the  telephone  or  face-to-face. 
In  the  case  of  self-
administration  a low rate of response (sometimes  20% for answers to a first 
batch)  must  be  expected.  The  rate  of  response  may  be  improved  by 
reminders  and,  where  relevant,  by  information  in  the  specialised  press. 
Postal  sun/ey  responses  may  be  boosted  by  providing  respondents  with 
pre-addressed  and  postage  paid  envelopes.  Depending  on the  difficulty  of 
the  context,  a  high  rate  of  response  ranges  from  60  to  90%.  In  case  of 
repeated  surveys  on  the  same  panel,  a  loss  of  around  10%  per  year  is 
considered  inevitable. 

Potential  respondents  are  more  likely  to  participate  if  the  purpose  of  the 
survey  is  explained  to  them  in  a  way  that  demonstrates  the  value  of  their 
contribution.  Where  feasible  respondents  should  be  given  a  guarantee  of 
the  confidential  treatment  of  her/his  answers.  The  interviewers  must  be 
informed  of  the  "rules  of  conduct"  and  be  able  to  make  this  clear  to 
interviewees.  Respondents  may  also  be  offered  incentives  to  participate 
and  to  return  forms  promptly.  Many  survey  organisations  offer  the  chance 
to enter a prize draw  or a similar  incentive. This is less common  in the public 
sector, but  has been used in some contexts.  In some cases it may be useful 
to offer  respondents  the opportunity  to  receive a summary  of the  evaluation 
report.  It  is  good  practice  to  follow  up  respondents  with  a  letter  thanking 
them for their  assistance. A random  check  is generally  carried  out  after  the 
survey  to  check  that  the  interviews  actually  took  place. 

Step  5.  Codifying  the  data:  The  next  stage  is  to  codify  and  record  the 
answers  - almost  always  in computerised  form  for  subsequent  analysis. At 
this  stage  adjustments  are  generally  made 
into  account 
imperfections  in the  sample,  and  especially  to  correct  the  bias  introduced 
by  non-answers. 

take 

to 

Tools  for observing  changes 

Step  6.  Interpreting  and  disseminating  the  results: The  final  stage  is  to 
conduct  an  analysis  of  the  meaning  of  the  results  of  the  survey.  This 
involves  understanding  the  purpose  of  the  survey  and  it  limitations  -  for 
example  constraints  related  to  sample  size, the  reliability  of  the  responses 
and  so  forth.  It  is therefore  important  that  the  analysis  phase  is  led  by  an 
experienced  researcher who  understands these problems and was  involved 
in the survey  design at the outset.  For the survey to be of  any  practical use, 
its  results  have to  be  presented  in a way  that  is accessible  and  meaningful 
to  policy  makers.  Numerous specialised computer  programmes can  help to 
undertake this  step.  It  is usually  important  therefore  to  ensure that  the final 
report  provides  sufficient  details  of  the  way  in  which  the  survey  was 
performed  (provided  in  a technical  annex  for  those  who  wish  to  check  its 
validity)  but restricts  reporting  in the main text to a straightforward  and clear 
explanation  of  the  findings. 

Application  to  Structural  Funds 

The  questionnaire  survey  is  suited  to  the  observation  of  the  results  and 
impacts  of  an  intervention  affecting  a  large  and  sufficiently  homogeneous 
population.  It  is therefore  likely  to  be  a tool  reserved  for  the  retrospective 
evaluation  of  a  simple  intervention,  for  example,  the  evaluation  of  training 
for  the  unemployed.  However,  it  may  also  play  an  important  role  in  'before 
after'  analyses  where  it  is  used  to  establish  a  baseline  at  the  start  of  a 
programme  against  which  progress  is  measured  during  or  after  the 
implementation  of  an  initiative. 

Strengths  and  limits  in  practice 

•  The  questionnaire  survey  has  the  advantage  of  producing  structured, 
quantified  and  generalisable  information,  providing  the  sample  has  been 
properly  selected.  It  is  therefore  particularly  useful  for  quantifying 
indicators,  including  monitoring  indicators. 

•  This  technique  is  costly  since  a  sample  has  to  contain  more  than  1000 
units to ensure that the  information  provided  is statistically  significant.  On 
the  other  hand,  it  allows  a wide  population  to  be  reached  at  a  relatively 
low  cost. 

•  It  is  sometimes  difficult  to  know  the  population  to  be  questioned, 
especially  if the  population  is large  and  not  particularly  well  identified  by 
the  monitoring  system.  In  this  case  the  tool  loses  one  of  its  main 
advantages: the  generalisability  of the  observations. 

Principal  evaluation  techniques and tools 

•  The need to standardise  responses through closed  questions  may lead to 
simplification.  For  this  reason  the  questionnaire  is  usually  reserved  for 
situations  where  the  effects  of  the  intervention  have  been  identified 
beforehand  and  are  clearly  understood.  By  contrast  the  impacts  of  a 
highly  innovative action  would  be best evaluated through  open  questions 
and a variety  of  other  non-survey  based  techniques. 

•  The  quality  of  a  questionnaire  survey  may  be  undermined  if  some 
answers are wittingly  biased  by  respondents  who  are  usually  in favour  of 
the  intervention  from  which they  benefit. 

•  Research  has  shown  also  that  many  people  answer  questions  in  an 
attempt  to  be  polite or avoid  appearing  ignorant when they  in fact  do  not 
know  the  answer  or  feel  unable  to  give  a  judgement.  To  be  reliable  a 
questionnaire  should  therefore  consist  of  questions  that  all  respondents 
interpret  in the same  way  and  can  answer  accurately. 

Further  reading 

Dussaix  A.-M.,  Grosbras  J.-M.  (1996),  'Les  sondages:  principes  et 
méthodes.'  collection  Que  sais-je?  Paris:  Presses  Universitaires  de 
France 
Practical  and  accessible,  this  book  contains  the  most  essential 
information  on  polling techniques  and  practice. 

Fink A.  (1995),  'The Survey  Handbook'  Sage 
A complete  and  practical  guide. 

Grange  D.,  Lebart  L.,  ed.  (1994),  'Traitements  statistiques  des  enquêtes' 

Dunod 
A  large  body  of  technical  and statistical  information,  provided  in an 
accessible form. 

Henerson  M.E.,  Morris  L.L.,  Fitz-Gibbon  C.T.  (1987)  'How  to  measure 

Attitudes?'  Sage 
Covers  the  construction  and  use of  scales  of  measurement. 

Javeau  C.  (1992)  'L'enquête  par  questionnaire,  manuel  à  l'usage  du 

praticien',  Edition  de  l'Université  de  Bruxelles 
A  practical  guide  in fifteen  steps. 

Tools  for observing  changes 

Ethnographic  observation 

The 

e t h n o g r a p h ic 

o b s e r v a t i on 

technique  consists  of  observing 

the  daily  activity  of  the  actors  w ho 

implement  the  programme,  or  the 

beneficiaries  w ho 

enter 

into 

contact  with  the  programme.  Thus, 

the  observer  is  present  during  the 

selection  of  projects  or  participates 

in  a  training  course,  for  example. 

He  or  she  thus  tries  to  observe  and 

A 

In-depth 
evaluation 

0 

Medium 
technical 
complexity 

O 

Φ 

Medium 
cost 

© 

Retrospective 
use 

Innovative 
use 

understand  the  situation  "from  the  inside". 

This  technique  is  particularly  useful  when  one  has  to  observe  a  situation 

about  which  there  is  little  knowledge,  or  when  it  is  suspected  that  the  same 

situation  is  understood  very  differently  depending  on  whether  the  point  of 

view 

is  "external"  or  "internal".  Ethnographic  observation 

is  the  only 

technique 

that  allows  observations 

to  be  made 

in  fields  where  overt 

research  activity  is  a  serious  problem.  The  technique  was  born  over  a 

century  ago,  with  the  science  of  ethnography. 

Example:  Evaluation  of  a  public  housing  service 

Ethnographic  observation  was  applied  to  the  evaluation  of  social  housing  policy  in 
France. The aim  was to  analyse the  discrepancy  between  the  national  objectives  of 
this  policy  and  its  implementation  in  the  field  by  civil  servants  in  contact  with  the 
public. 

For several  months  the evaluator  accompanied  teams  of  civil  servants  employed  by 
the  agencies  managing  housing  for 
these 
authorities,  he  visited  these  homes  when  the  tenants  arrived  and  left.  These  visits 
were  intended  to  judge  the  state  of  the  housing  and  to  determine  the  amount  of 
reparation  work  to  be  paid  either  by  the tenant  or the  managing  agency. 

low-income  groups.  Together  with 

The  position  as  an  observer  enabled  him  to  "melt  into  the  background"  and  to 
observe  about  fifty  situations  in  which  the  national  policy  was  applied  in  concrete 
situations  by  the  civil  servants.  The  situations  observed  were  recorded  and  about 
twenty  typical sequences  were  selected. These sequences were written  up, with  due 
respect  to confidentiality,  and then  presented  to  policy  actors  in meetings  organised 
separately  at different  levels (civil servants in the field, heads of the agencies,  national 
authorities). 

The evaluation  helped to show  how  local authorities  apply the rules, how the  tenants 
interpret  them,  and  how  the  latter  influence  their  application. 

Principal evaluation techniques and tools 

The  evaluation  highlighted  the  fact  that  tenants  had  difficulty  understanding  the 
reasons for the rules. In view of these results, several amendments  were made: the 
heads of the agencies  revised some rules and initiated a review  of the  recruitment 
and training of their agents. 
Source:  Philippe  Warin,  "  L'évaluation  des  usagers  au  cours  des  relations  de 
service ", Les annales de la recherche urbaine n°47 (1990). 

What  is the  purpose  of the  tool? 

Ethnographic  observation  serves  to  collect  in-depth  information  on  a  few 
typical  situations  in  the  implementation  of  an  intervention.  It  provides 
detailed  insights  into  the effects  of  the  intervention  and the influence  of  the 
context,  and  is  sensitive  to  the  viewpoints  of  the  key  actors  and  the 
beneficiaries. 

In which  cases  can  it be  used? 

Ethnographic  observation  has  been  used  to  understand  the  functioning  of 
policies  in education,  health  (e.g. diagnosis,  care), justice,  police,  scientific 
research,  urban  transport,  etc.  In the  evaluation  framework,  this  technique 
is recommended  particularly  for the observation  of  processes  of  interaction 
between the  administrations  and  their  public. 

In  the  example  used  above  to  present  ethnographic  observation  (the 
evaluation  of  a  public  housing  service)  the  researcher  also  carried  out  an 
ethnomethodological  analysis.  Ethnomethodology  appeared  in  the  1960s 
and  developed  in  the  early  1980s  in  Europe.  It  studies  the  practical  daily 
reasoning  used  by  individuals  (civil  servants,  beneficiaries,  users,  etc.)  to 
judge the situations they encounter  and the collective  actions  in which  they 
are  involved. 

If  carried  out  properly  this  technique  does  not  disturb  anyone  or  influence 
his  or  her  behaviour.  It  can  therefore  be  used  to  observe  the  spontaneous 
behaviour  of  populations  who  are  reluctant  to  accept  the  formalism  of 
questionnaire  surveys.  It  is  the  only  technique  applicable  when  there  are 
serious  difficulties  of  gaining  access  to the  field, for example  in the case  of 
conflict within the organisations  responsible for implementation or when the 
beneficiaries'  behaviour  is  partly  illegal  or  irregular.  The  technique  is  then 
particularly  interesting  if the evaluation  is launched with a view to  amending 
the  regulations  suspected  of  being  ineffective. 

Tools for observing changes 

Due to  its  inductive  character,  ethnographic  observation  serves  to  observe 
the  results  of  an  intervention  of  which  the functioning  is  not  well  known.  It 
is particularly  useful when  it is suspected  that the  reality  is not  perceived  in 
the  same  way  by  those  who  implement  the  intervention  and  by  the  public 
who  receives  it or  benefits from  it. 

On  the  other  hand,  the  technique 
is  time-consuming  and  since  the 
observations  focus  on  a  few  restricted  situations  they  cannot  easily  be 
generalised.  The  tool  is  therefore  best  suited  to  the  evaluation  of  an 
requires 
extensive  and  relatively  homogeneous 
considerable  skill  on  the  part  of  the  researcher  to  absorb  and  reflect 
accurately  the  behaviour  of  the  key  actors,  and  it  may  take  time  for  the 
researcher  to  'melt  into  the  background'  and  therefore  for  participants  to 
behave  in a normal  way. 

intervention. 

It  also 

How  should  it be  implemented? 

Observation  has generally  been  carried  out  in the following  steps: 

for 
Step  1.  Choice  of  situations  for  observation:  The  situations 
observation  are defined  beforehand  in relation to the  preoccupations  of  the 
evaluation  commissioners.  They  consist  of  situations  of  interaction  or  of 
negotiation  between the public actors and the beneficiaries  of the  evaluated 
policy. 

Step  2. Observation: The observer  follows  and  registers  the course  of  the 
interaction,  taking  care  to  disturb  the  behaviour  of  the  actors  as  little  as 
possible. This work  consists  of  note-taking  and audio-visual  recordings  (as 
discretely  as  possible). 

This  step  cannot  be  limited  merely  to  a  simple  observation;  it  must  be 
completed  by  an  institutional  analysis  so  as  to  identify  the  areas  of 
constraints  bearing  on the relations  between the actors. The observer  must 
collect  a  maximum  of  information  and  capture  a  particular,  inside  view  of 
the  situation. 

Stem  3.  Identifying  and  cutting  up  typical  sequences:  The  events 
observed  during  the  second  step  are  analysed  in  terms  of  characteristic 
sequences.  Each  recording  is  'cut  up'  just  as  one  would  edit  a  film  into 
sequences. 

The  observer  identifies  the  "evaluative  assertions",  that  is  to  say,  the 
sentences  which  convey  an  explicit  or  implicit  value  judgement.  Typical 

Principal evaluation  techniques and tools 

sequences  and  their  analysis  are  concentrated  on  these  assertions,  and 
reveal  the  way  in  which  the  policy  is judged  in the  field.  Used  in  this  way, 
the  tool  can  shed  important  new  light  on  the  validity  and  effectiveness  of 
the  policy. 

Step  4.  Analysis  of  typical  sequences  with  the  actors:  The  typical 
sequences  and  assertions  are  rewritten  or  modified 
to  make  them 
anonymous. They  are then  given  to  the  people  representative  of  those  that 
have  been  observed,  for  the  purpose  of  collecting  their  comments  and 
reactions. This step serves to verify that  no bias has been created  by taking 
the  sequences  out  of  their  context.  It  gives,  for  each  sequence,  keys  for 
interpretation  which  are  recognised  and  validated  by  the  "community" 
under  study. 

Application  to  the  Structural  Funds 

To our knowledge, ethnographic  observation  has been used very  little in the 
context  of  the  evaluation  of  socio-economic  programmes.  Yet  this 
technique  is  well  suited  to  certain  Structural  Funds  interventions,  for 
example those intended for publics that are difficult  to observe  by means of 
more traditional  inquiry  techniques  (e.g. the  long-term  unemployed). 

Ethnographic  observation  is  a  generic  tool  that  involves  the  collection, 
interpretation  and  comparison  of  data.  It  shares  these  characteristics  with 
the  case  study  technique.  It  is  therefore  particularly  well  suited  to  the 
analysis  of the effects  of  an intervention  that  is innovative or unfamiliar,  and 
especially 
the 
influence 
apparent  success  or failure  of  the  community  interventions  evaluated. 

the  clarification  of  confounding 

factors  that 

Strengths  and  limits  in  practice 

•  Ethnographic  observation  serves  to  reveal  the  discrepancy  between  the 
way  in  which  public  intervention  is  understood  high  up  at  the  decision­
making level, and the way  in which it is understood  in the field; it  highlights 
the  interpretation  made  of  it  by  individuals  in an operational  situation. 

•  The observation  is limited  to  a small  number  of situations.  Generalisation 
is therefore  possible  only  if the  intervention  is sufficiently  homogeneous. 

•  It  is based on spontaneous  information, gathered  by an  independent  and 
experienced  observer.  The  reliability  of  the  observation  depends  to  a 
large  extent  on the  professional  know-how  of  the  observer-analyst. 

Tools for observing changes 

•  Despite  its  advantages,  ethnographic  observation  requires  meticulous 
preparation  to  enable  the  observer  to  fit  into  the  observed  context 
without  disturbing  anyone,  as  well  as  considerable  time  for  data 
collection  because  the  observer  has  no  control  over  the  schedule  of 
her/his  observations. 

•  This  technique  allows  data  to  be  collected  in  difficult  situations  where 

other  survey  techniques  cannot  be used. 

Further  reading 

Coulon  A. (1987),  L'ethnométhodologie,  Paris:  PUF, Que  sais-je? 

An  approach  which  is  both  theoretical  and  practical,  illustrated  by 
case  studies. 

Garfinkel  H.  (1967),  Studies  in  ethnomethodology,  Englewood  Cliffs,  NJ: 

Prentice  Hall 
A  return to the  origins  of  ethnomethodology. 

Jorgensen  D.L.  (1989),  Participant  Observation,  a Methodology  for  Human 
Studies,  London:  Sage  Publications,  Applied  Social  research 
methods  Series,  No  15, 
Complete  manual  on  a  variation  of  ethnographic  observation  in 
which  the observer  puts  her/himself  personally  in the  situation of  an 
actor  or  addressee  of  the  programme. 

Tools for analysing data 

When an evaluation  concerns  a limited domain, it is possible to carry out an 
in-depth  analysis  of  the  causal  links  between  the  action  and  its  effects. 
Several tools  may  be  used  in this  context: 

•  Variance  analysis (not  presented  in this Volume) and factor  analysis are 
used  to  reveal  similarities  and  dissimilarities  within  a  sample  of 
observations,  to  create  typologies,  and  to  identify  exogenous  factors 
associated  with  the  production  of the  impacts. 

•  The Delphi survey is included  in this section  because  it was designed  for 
estimating  impacts.  It  is  well  suited  to  ex  ante  evaluations  that  rely  on 
secondary  data.  The  tool  mobilises  and  analyses  data  through  the 
intervention  of  experts.  It  is  therefore  similar  to  the  expert  panel  that  is 
also  suitable  for  data  analysis. 

•  Comparison  groups  are  used  to  estimate  net  effects  by  noting  the 
difference  between  a  group  of  addressees  and  a  group  of  non-
addressees. 

•  Regression  analysis  is  used  to  estimate  net  effects  and  to  determine 
whether  the  causal  links  between  the  intervention  and  its  effects  are 
statistically  significant. 

•  Case  studies,  group 

interviews  and  ethnographic  observation 
mentioned  above  as  tools  for  observation,  are  also  tools  which  can  be 
used  flexibly  for  comparing  and  analysing  data.  It  is  even  possible  to 
estimate  effects  (in  the  form  of  a  range,  from  minimum  to  maximum 
effects)  by  means  of  case studies  carried  out  on  a selection  of  projects. 

Tools for analysing data 

Factor analysis 

Factor analysis is a multidimensional 

statistical analysis technique used to 

sort large volumes of quantitative 

data concerning groups of 

individuals about whom there is 

information about numerous 

characteristics (variables). The tool 

identifies similarities or links (factors) 

between individuals and between 

variables. It enables one to "trade off 

Λ 

In­depth 
evaluation 

S 

Medium 
technical 
complexity 

Φ 

Medium 
cost 

Θ 

O 

O 

Use 
stabilised 

Retrospective 
use 

Prospective 
use 

information for legibility", by producing a simplified graphic representation of 

available data, showing the phenomena to be analysed. 

Although its origins can be traced back to the beginning of the century, its 

recent development can be attributed to significant advances in data 

processing, owing to computer technology. 

An example: Employment impacts in the Catalan co­operative sector 

This example corresponds with Box 20. 

The use of factor analysis can be illustrated with respect to an ex post evaluation 
project, oriented towards the employment effects (creation and maintenance of jobs) 
of subsidies in the sector of Catalan co­operatives. Evaluation in this case was 
complicated by the diversity of co­operatives and the consequent heterogeneity of 
the population. The problem was twofold: first, the relevance of measures taken by 
the region had to be determined in view of this situation; secondly, the importance of 
impacts on those co­operatives which received assistance had to be assessed. 
Given the variety, as well as the lack of knowledge of the population and of impacts, 
factor analysis was chosen for its ability to reveal the main characteristics of the 
population under consideration (about 3,000 co­operatives). 

The first step in the evaluation consisted of sending a questionnaire to all the co­
operatives. This was completed through interviews. The data thus gathered were first 
analysed using traditional statistical methods (search for the main characteristics of 
the population, drawing up of cross­referenced tables). The second step in the data 
analysis consisted of a factor analysis with a view to isolating variables likely to 
explain variations in employment in the co­operatives: including the size of the entity, 
age, sector of activity, technological level, etc. 

Source: Comité Assessor para l'Estudi de l'Organització de l'Administració, 
Generalität de Catalunya, 1995 

Principal evaluation  techniques and tools 

What  is the  purpose  of  the  tool? 

Factor  analysis  can  serve  various  purposes.  In the  evaluation  context  it  is 
used to: 
•  analyse  in  detail  the  reactions  and  characteristics  of  the  beneficiaries, 
and  thus  gain  a  better  understanding  of  the  relative  efficiency  of  the 
interventions; 

•  explore the correlation  between phenomena and thereby  identify  possible 
confounding  factors  that  have  amplified  or  reduced  the  impact  of  the 
programme. 

In  which  cases  can  it  be  used? 

Factor  analysis  is a data  processing  tool  used  by  national  statistics  offices 
and  survey  institutes.  It  has  the  advantage  of  processing  quantitative  data 
in such a way that typologies  can  be constructed. The tool  is therefore very 
useful  when  the  evaluation  has  to  construct  classes  within  the  population 
concerned  (e.g.:  qualification,  socio-professional  category,  criminality, 
urban  concentration, etc.). 

It  is important  to  note that the  classifications  established  by factor  analysis 
are  not  chosen  randomly  by the  evaluation team. On the  contrary,  they  are 
drawn  from  a  vast  corpus  of  data,  and  constructed  in  such  a  way  as  to 
allow  the  best  discrimination  possible  within  these  data. 

How  should  it  be  implemented? 

Step  1. Constitution  of  a  table  of  data: the  data  processed  are  provided 
by  monitoring  information  or, more commonly, from  a questionnaire  survey. 
All  the  data  collected  are  grouped  together  in  a  data  matrix,  like  that 
resulting  from  each  survey  or  census.  A  row  corresponds  to  the  data 
relative  to  a  person  (firm,  etc.)  and  a  column  groups  together  values  of  a 
single  variable  for different  individuals. 

Steps  2  and  3.  Choice  of  a  space  of  individuals  and  a  space  of 
variables:  These  two  steps  are  fairly  similar.  In  both  cases  the  sample  is 
examined,  extreme  or  incorrect  values  sought,  and  the  variables  and 
groups  of  persons  relevant  to  the  analysis  extracted.  It  is  here  that  the 
analysis  is oriented  towards  those variables  on  which  the study  is to  focus. 

Step  4.  Multi-dimensional  analysis:  This  analysis  is  a  step  that  is 
facilitated  today  by  the  existence  of  specialised  statistical  software.  It  is, 

Tools for analysing  data 

however,  a crucial  one, particularly  because several technical choices  must 
be made in relation to the available data and desired results. In this step the 
number  of  observations  will  be  reduced  and  relevant  factors  identified. 
These will then  be  used for  differentiating  the  population. 

Step  5.  Projection  into  two  (sometimes  three)  dimensional  space: 
Statistical  software  produces  part  of  its  results  in the form  of  a card  which 
may feature  individuals  and/or  variables. 

Box 25 -  Example  of  result of factor  analysis 

Example of the  evaluation of aid to Catalonian cooperatives 

u 
ω  xxxx 

xxxx 

xxxx 

Cooperatives which ... 

invest 

., 

receive aid 

y''' 
are medium-
sized 

ι 

XXXX 

X3CX 

< 

xxxx 

are far from the coast 

have been In operation for years 

factor 2 

,'xxxx 
, 

xxxx 

*»„ 

', 

'  xxxx 

xxxx 

Step  6.  Interpretation  of  results:  factor  analysis  does  not  reveal  the 
significance  of  the  factors  nor  that  of  the  groups  formed  on  the  map.  The 
analyst  has  to 
interpret  them  and  to  name  them.  Ideally,  the  key 
stakeholders  in the  evaluation  should  be associated  with  this  step, since  it 
provides  an  opportunity  for  them  to  enhance  their  understanding  of  the 
functioning  of  the  intervention. 

Application  to the  Structural  Funds 

Factor  analysis  may  be  used  as  an  ex  ante  evaluation  tool,  for  example  for 
identifying  typologies  of  the  target  population,  based  on  the  population  as  a 
whole. It can thus be used to set up procedures for selecting beneficiaries or for 
identifying the nature of the socio-economic  problems facing this population. 

Factor analysis may also be used to highlight the similarities and  differences 
between  groups  of  individuals  or  geographical  areas.  This  is  particularly 

Principal  evaluation  techniques and tools 

useful  for  making  comparative  analyses  of  regional  development  or  for 
evaluating the  reactions  of  the  beneficiaries  of  specific  actions. 

On  the  other  hand,  factor  analysis  can  not  be  used  to  estimate  socio­
economic  effects.  It  is  above  all  a tool  for  data  comparison  and  inductive 
analysis. 

Strengths  and  limits  in  practice 

•  Factor  analyses  are  a tool  for  clarifying  raw  material  resulting  from  large-
scale  surveys  because  they  help  to  reveal  relevant  dimensions  (factors 
explaining  the  differentiation  or  resemblance  of  a group  of  data). 

•  The results of factor analysis are easily accessible and are presented very 

clearly  (diagrams),  aggregating  a large  number  of  data. 

•  Although  factor  analysis  is  used  to  illustrate  the  existence  of  groups  of 
variables  around  certain  categories  of the  population, it  is  not  a tool  that 
can  explain  the  existence  of  cause  and  effect. 

•  Factor  analysis  requires  access  to  large  amounts  of  data,  which  may 

prove time  consuming  and costly  to  collect. 

•  Factor  techniques  require  specific  qualifications  and  methodological 

precautions.  Used  wrongly, they  could  provide  tautological  results. 

•  Factor analysis cannot  be conceived  without the  use of a large number  of 

data, which  may  prove  long  and  costly  to  collect. 

Further  reading 

Bry  X. (1995),  'Analyses  factorielles  simples',  Paris:  Economica,  11 l p. 

Although  this  book  is  of  general  interest,  it  is  preferable  to  have 
some  knowledge  of  statistics. 

Dazy  F  and  Le  Barzic  J.  F.  (1996),  'L'analyse  des  données  évolutives, 

méthodes  et applications',  Paris: Technip 
Written  form of  a training seminar,  requiring  sound  basic  knowledge 
in statistics. 

Grange  D.  and  Lebart  L.  (1994),  'Traitements  statistiques  des  enquêtes', 

Paris:  Dunod 
A  book  which  focuses  on the  essential  points. 

Tools for analysing data 

Jaccard,  J.  (1998),  Interaction  Effects  in  Factorial  Analysis  of  Variance, 

London:  Sage  publications. 

Kim,  J-O.,  Mueller,  C.W.  (1979),  Introduction 

to  Factor  Analysis,  London: 

Sage  publications. 

Rossi, P.H. (1997),  'Advances  in quantitative  evaluation,  1987-1996',  in New 
directions  for  evaluation, Winter  1997  (76). San  Francisco  : Jossey-
Bass  Publishers. 
Advances  in  data  collection,  measurement  and  statistical  analysis 
which  are appropriate  to the  evaluators'  needs. 

Tools for analysing data 

Delphi Survey 

Evaluations that rely on the views 

of experts may be carried out in 

the framework of ex ante 

evaluation. The Delphi method is a 

case in point. It is presented in the 

form of an anonymous and 

iterative consultation procedure by 

means of postal surveys. The 

absence of prior consultation 

within the group allows for the 

A 

In­depth 
evaluation 

Β 

High 
technical 
complexity 

G 

φ) 

High 
cost 

© 

Prospective 
use 

Innovative 
use 

clarification of differences of opinion between experts, and thus for a better 

understanding of the nature of each person's expertise. 

The method is recommended when the questions posed are simple (a 

programme with few objectives, of a technical nature) and for the purpose 

of establishing a quantitative estimation of the potential impacts of an 

isolated intervention (e.g. increase in taxes or in the price of energy). 

Example: AIDS vaccination policy: a scenario analysis using the Delphi method 

The Delphi method was used to examine likely answers to the introduction of a first 
AIDS vaccination in Switzerland. Thirty persons concerned by this vaccination 
strategy contributed anonymously to the study. The study was based on a scenario 
developed beforehand, describing the characteristics of a first preventive, partially 
effective, vaccination against AIDS. 

The Delphi consultation was carried out in three stages: 

• list the objectives to be achieved in the first five years, 

• evaluate the acceptability and feasibility of proposals concerning the development 

of a public health strategy and the AIDS vaccination, 

• estimate the potential use of the vaccination by different groups of users. 

These results allowed for a) significant conclusions to be drawn on the strategies and 
recommendations to be developed in the framework of AIDS prevention campaigns 
and b) the provision of a frame of reference for the setting up of a future AIDS 
vaccination strategy. 

Source: Zuber, P. (1994) Introducing a first AIDS vaccination in Switzerland: A Delphi 
Policy Analysis. Lausanne: Institut de médecine sociale et préventive. 

Principal  evaluation  techniques and tools 

What  is the  purpose  of  the  tool? 

Like  the  other  expert-based  approaches,  the  Delphi  survey  is  useful  for 
producing,  in  the  evaluation  context,  recommendations  of  which  the 
credibility  depends  on the  legitimacy  of the  experts  themselves. 

It  may  be  used  in  forward  planning  to  establish  hypotheses  about  the 
realisation  of  events  and  on their  socio-economic  implications. 

Above all, the  method serves to shed  light on the evolution of a situation, to 
identify  priorities  or to  draw  up  prospective  scenarios. 

In which  cases  can  it  be  used? 

This  technique  may  be  used  when  significant  expertise  exists  on  the 
subject,  i.e.  in  the  case  of  programmes  that  are  not  innovative.  It  is  also 
recommended  in  an  ex  ante  evaluation  context  if  the  evaluation  concerns 
public  intervention  of a technical  nature. Thus, it was very  often  used  in the 
framework  of  energy  policies,  for  example,  for  prospective  studies  on  the 
impact  of  changes  in taxation. 

It  may  be  used  to  ensure  that  experts  do  not  influence  one  another. 
Anonymity  and the absence of interaction favour  an in-depth analysis of the 
reasoning  behind  the opinions  expressed. 

How  should  it  be  implemented? 

The approach  consists  of  questioning  the  experts  by  means  of  successive 
questionnaires,  in  order  to  reveal  convergence  and  any  consensus  there 
may  be. The  main stages  of  this  process  are: 

Step 1. Determination and formulation of questions: Particular care must 
be given to the  choice  and formulation  of  questions,  so  as to  obtain  useful 
information. 

Step  2.  Selection  of  experts: They  must  have  specific  knowledge  on  the 
subject  and  be prepared  to  become  involved  in this type  of  procedure. The 
panel  is generally  composed  of  about  fifty  persons. 

Step  3.  Formulation  of  a first  questionnaire  that  is sent to the  experts: 
The first  questionnaire  must  contain  a  reminder  of  the  nature  of  the  study 
and  include two  or three  semi-open  and  open  questions. 

Tools  for analysing data 

Step 4. Analysis  of the  answers  to the first questionnaire: The  answers 
are  analysed  in  order  to  determine  the  general  tendency  and  the  most 
extreme  answers. 

Step  5.  Formulation  of  a  second  questionnaire  that  is sent  to  experts: 
Each expert  informed  of  the  results  of  the  first  round  is asked  to  provide  a 
new  answer  and  to justify  it  if  it differs from  the  general  tendency. 

Step  6.  Sending  of  a  third  questionnaire:  This  questionnaire  is  intended 
only  for  those  experts  whose  answers  were  "extreme".  They  are  asked  to 
criticise the  arguments  of those  who  supported  the  opposite  point  of  view. 
The  comparison  of  opinions  has  a moderating  influence  and  facilitates  the 
appearance  of convergence  between  the  points  of  view. 

Sufficient  convergence  of  opinions  generally  appears  with  the  fourth 
questionnaire.  If that  is  not the  case, the cycle  continues. 

Step 7: Summary  of the  process  and drawing  up of the final  report. 

Application  to  the  Structural  Funds 

In the case of  the  Structural  Funds, the  Delphi  inquiry  is recommended  for 
obtaining  macro-economic  estimations  when  the  phenomena  involved  are 
complex;  for  example,  to  quantify  the  impact  of  a  major  infrastructure 
project.  It  may  also  be  used  to  specify  relations  of  causes  and  potential 
effects  in the  case of  innovative  interventions. 

It  is  particularly  useful  when  a very  large  territory  is  being  dealt  with  since 
there  are no experts'  travel  expenses,  only  postage  costs. 

Strengths  and  limits  in  practice 

•  Unlike  the  panel  approach,  the  conclusions  drawn  from  this  type  of 
approach facilitate the taking  into account  of  minority  points  of  view. The 
method  furthermore  allows  for a better  understanding  of the elements  on 
which  the experts'  opinions  are based. 

•  The  use  of  this  technique  has  limited  flexibility  because  the  various 

iterations  are time-consuming  (over three  months  in all). 

•  As  in  the  case  of  expert  panels,  the  reliability  of  the  Delphi  survey 
depends  on  the  subjectivity  of  the  experts'  opinions,  itself  a function  of 
their  knowledge. 

Principal  evaluation  techniques and tools 

Further  reading 

Godet  M.  (1985),  Prospective  et  planification.  Paris:  Economica,  pp.  119-

125. 
The Delphi method  is presented and illustrated  in the first  part of this 
book  devoted  to  forecasting. 

Nadeau  M-A.  (1988).  L'évaluation  de  programme.  Laval, Québec:  Presses 

de  l'université  de  Laval, pp  349-352. 
A  manual  presenting  the  methods  and 
instruments  used 
programme  evaluation. The  Delphi  method  is described  briefly. 

in 

Witkin  B.R  and  J.W  Altschuld  (1995),  Planning  Conducting  Needs 

Assessments,  Thousand  Oaks: Sage,  pp  193-203. 
Collection  of  methods  to  be  used  in the  evaluation  of  needs. A  part 
of the chapter on group techniques  is devoted to the Delphi method. 

Tools for analysing  data 

Comparison  groups 

The  comparison  group  technique 

involves  comparing  a  "participant 

group"  (e.g.  SMEs  benefiting  from 

services)  with  a similar group  which 

has  been  unaffected  by 

the 

intervention. Ideally, the two  groups 

should  be  identical  in  all  respects 

except  for  their  participation/non-

participation 

in  the  programme. 

Where  this  applies,  the  group  that 

A 

In-depth 
evaluation 

0 

Medium 
technical 
complexity 

Θ 

© High 

cost 

O 

Use 
stabilised 

Retrospective 
use 

has not  been affected  by the intervention  is referred to as the  'control  group' 

and fulfils the same function  as a 'control' in the natural sciences.  In practice 

however  it  is  rare to  find  groups  which  are  identical  in  every  respect  and  in 

the  social  sciences  and  impossible  in  the  context  of  European  Structural 

Funds,  which  is  why  it  is  not  considered  in  this  document.  A  modified 

version  of  the  approach  is to  use the  "matching  pairs"  technique. 

Example: Estimation  of the  impact of a training  and consultancy  programme in 

firms in the  UK 

This  example  corresponds  to that  of Box  21. 

The TEC  (Training  and Enterprise  Council)  programme  in England was evaluated by 
means  of  the  comparison  group  technique.  The  effect  of  the  programme  was 
estimated  through  a  "before  and  after"  comparison  between  firms  using  the 
programme  services  and  a  set  of  matched  pairs  of  companies  which  had not 
participated  in TEC programmes.  Matches  were  made  according  to  the  following 
criteria: type of activity of the firm, size of the firm in terms of number  of employees, 
age and geographical  location. A total of 180 pairs of beneficiary  and  non-beneficiary 
firms were  questioned. 

The report  examines  both the practical  and  theoretical  difficulties  of the comparison 
group  techniques.  The main  practical  difficulty  identified  was that  of  matching the 
pairs. Theoretical  problems  related to the fact  that,  even  if the pairs  linked  together 
were  apparently  exact  matches,  there  remained  imperceptible  differences  between 
them  which  might  account  for differences  in performance  which  were  unrelated to 
the programme being evaluated differently. Despite these difficulties, the comparison 
group  technique  made  it  possible  to  define  norms  for  other  regional  or  national 
factors  that  had influenced  the results  of the firms. 

Two  data  collection  techniques  were  applied.  One consisted  of a questionnaire on 
the  employment  impact  of the TEC. To avoid  results  being  biased  by  'instrumental, 

Principal evaluation techniques and tools 

self  interested  "strategic"  responses'  (e.g.  a  positive  reply  given  in  the  hope  of 
obtaining additional support), firms were asked not only to estimate the employment 
effects  of the  TEC services,  but  also to  give details  of  their  turnover  and changes 
relating to employment. This was designed to enable objective comparisons between 
the matching samples. 
Source: MEANS Handbook N°3, pp. 41-42 

What  is the  purpose  of  the  tool? 

The aim of the comparison  group technique  is to  estimate the net effects  of 
a  public  intervention,  that  is to  say,  the  effects  directly  attributable  to  this 
intervention,  and  not  those  which  are  the  result  of  exogenous  or 
confounding  factors. 

In which  cases  can  it  be  used? 

The  comparison  group  technique  is  usually  used  when  the  programme 
offers  subsidies  or  services  to  a  relatively  homogeneous  group  of 
beneficiaries  and  when  these  beneficiaries  have  already  been  reached 
(retrospective  situation)  and there exists a similar group which has not  been 
offered  or  has  not taken  up  assistance from  the  programme  which  is  being 
evaluated. 

To  produce  credible  results,  the  two  groups  being  compared  must  be 
sufficiently  similar  in respect of  likely  'confounding  factors'. This means that 
one  can  identify  and  grade  these  factors  before  constituting  the  groups  to 
the 
compare.  The  tool 
mechanisms  producing  effects  are already  fairly  well  known. 

is  therefore  reserved  for  situations 

in  which 

The comparison group technique does  not require the use any specific  data 
collection  technique.  In fact  several  approaches  to  primary  data  collection 
(questionnaire  surveys,  interviews,  etc.)  and/or  secondary  analyses  are 
equally  applicable. 

How  should  it  be  implemented? 

Step  1.  The  construction  of  comparison  groups:  Three  ways  of 
constructing  comparison  groups  exist. 

•  Creation  of  matching  pairs  by  selecting,  for  each  individual  in  the 
beneficiary  group,  an  individual  not  affected  by  the  programme  but  as 

Tools for analysing data 

similar as possible in all other respects. The difficulty  is finding  individuals 
not  affected  by  the  programme.  It  is  for  example  likely  that  a  firm 
operating  in a similar  market  to a participant  of the  programme  may  have 
been  affected  adversely  by  assistance  given to  what  is,  in effect,  one  of 
its  competitors.  Equally,  \jery  little  is  likely  to  be  known  about  non-
addressees,  and  since  checking  out  their  characteristics  and  finding 
individuals  to  fit  a  specific  profile  is  likely  to  be  difficult  and  long,  the 
creation  of  matching  pairs  may  be  expensive. 

•  Comparison  with  individuals  known to the  programme  but  not  benefiting 
from  it  (candidates  who  withdrew  their  application,  candidates  not 
selected,  selected  candidates  who  dropped  out  of  the  programme 
midway).  In this case, the group of  non-beneficiaries  must  be adjusted  to 
resemble  the  group  of  beneficiaries  as  closely  as  possible.  However,  a 
bias  remains, because there are likely to  be reasons  that  caused  them  to 
withdraw  or to  be  rejected  which  may  also  affect  their  performance  as  a 
whole. 

•  A  generic  comparison  may  be  made  with  a  hypothetical  'comparison' 
group  based on national statistics. This is done when the  programme  has 
affected  only  a small  group  of  beneficiaries  very  specifically,  and  the  rest 
of the  country  has  not  benefited from  similar  programmes. 

Step  2.  The  actual  comparison:  The  principle  consist  of  comparing  the 
changes  observed 
in  the  two  groups  and  then  deducing,  from  the 
difference, the  net  effect  of  the  programme. 

Different 
disadvantages,  and  varying  degrees  of  validity. 

forms  of  comparison  exist,  each  with  its  advantages  and 

•  An  "afterwards  only"  comparison  involves  analysis  of  the  differences 
between  the  two  groups  after  the  subsidy  or  service  has  been  received. 
Its advantage  is its simplicity. The disadvantage  of this approach  is that  it 
disregards  differences  that  existed  between the two  groups at the  outset. 

Participation  in 
the  programme 

Situation  after 
the  programme 

Participant  group 

Comparison  group 

yes 

no 

X2 

X'2 

Estimation  of the  effect 

X 2 - X '2 

Principal  evaluation  techniques and tools 

A "before-after"  comparison avoids this problem  by taking into  account 
possible  differences  between  the  two  groups  before  as  well  as  after  the 
intervention.  It  therefore  focuses  on  the  relative  improvement  of  the 
beneficiaries  compared  to  non-beneficiaries.  Its  drawbacks  are  that  it 
requires  the  availability  of  data  on  the  situation  before  the  application  of 
the  treatment  (e.g. through  monitoring  or  in  the  form  of  statistics),  and 
such  data  does  not  always  exist.  Moreover,  there  are situations  in  which 
the evaluation  does  not start  until the treatment  is underway  and in these 
cases  the  'before'  state  may  not  be  known.  There  are  also  many 
programmes  in  which  the  beneficiaries  and  non-beneficiaries  are  not 
known  at  the  outset  and  information  about  their  differences  prior  to  the 
programme  are  not  available.  In  all  these  cases  data  about  the  'before' 
situation  have to  be gathered  retrospectively  and are therefore  unlikely  to 
be entirely  reliable. 

Participation  in 
the  programme 

Situation  before 
the  programme 

Situation  after 
the  programme 

Participant  group 

Comparison  group 

Estimation  of 
effects 

yes 

no 

(X2-X1) 
( Χ ' 2 - Χ Ί) 

X1 

ΧΊ 

X2 

X'2 

•  The  "before-after"  trend  comparison  is  considered  more  reliable.  The 
performance  over  time  of  both  groups  is  compared  in  order  to  test  the 
extent  to  which  beneficiaries  display  greater  improved  performance 
relative to  their  own  previous  situation  than  do  non-beneficiaries. 

Application  to  the  Structural  Funds 

The  comparison  groups  technique  is  suitable  for  ex  post  evaluations  for 
which  the  identification  of  impacts  is  essential.  In  particular,  it  enables  the 
estimation  of  deadweight,  that  is  to  say,  changes  which  would  have 
occurred  in the target  group  in any case, even without  the  programme,  and 
which  therefore  can  not  be considered  as  an  impact  of the  programme. 

In  the  context  of  the  Structural  Funds,  it  is  however  often  difficult  to 
determine  comparison  groups.  The  public  concerned  must  be  sufficiently 
homogeneous  and  a  significant  proportion  of  the  potential  beneficiaries 
must  not  have  participated  in  the  programme  (if  this  is  not  the  case,  it  is 
said that there is  "saturation " of the population exposed to the programme). 

Tools for analysing data 

One approach is to compare the performance of beneficiaries with those of 
select comparison groups from outside eligible areas. However, few studies 
have so far employed this technique. 

Strengths and limits in practice 

• A well­designed comparison group analysis provides a convincing 

estimate of a range of programme effects. 

• One of the problems posed by this tool stems from the need to take into 

account, from the outset, factors that are likely to have a decisive influence 
on the production of effects. If important variables are overlooked (often 
factors outside the influence of the programme) it becomes difficult to 
establish strong causal inferences and the analysis loses some of its validity. 

• Another shortcoming stems from its inability to explain the observed 
effects. It focuses on the measurement of effects rather than on 
mechanisms that produced them (implementation, quality of services 
provided, reactions of beneficiaries). The programme and its effects are 
treated as a "black box". This may be partially compensated by using other 
tools (e.g. case study), giving the comparison a qualitative dimension. 

Further reading 

Boruch, R.F. (1997), 'Randomized Experiments for Planning and Evaluation. 

A practical guide'. In Applied Social Research Methods Series, 44. 
London : Sage Publications. 
The book deserves to be called what is written as its second title "a 
practical guide". 

Matalon Β. (1988), Décrire, Expliquer, Prévoir­ démarches expérimentales 

et terrain. Armand Colin ­ collection U. pp. 219­252 
More detailed and technical volume ­ see the third part. Annex 1 
considers the particular case of the construction of comparison 
groups by drawing lots. 

Mohr, L.B. (1995) 'Impact Analysis for Program Evaluation' (2nd edition), 

London: Sage Publication. 

Monnier E. (1992), Evaluation de l'action des pouvoirs publics. Paris: 

Economica, pp. 9­51 
Good illustration of comparative techniques. 

Treasury Board of Canada, Office of the Contrôler General (1991), 'Program 

Evaluation Methods'. Treauusury Board of Canada. 
General approach ­ see chapter 3.1 ­ 3.35 

Principal  evaluation  techniques and tools 

More  in-depth: control  groups  and  the  experimental  approach 

In order to eliminate the  risk  of  bias, individuals, families or  enterprises  can 
be randomly  assigned  to  one or the other  of the two  groups. This  practice, 
known  as  the  experimental  approach,  is  the  most  rigorous  way  of 
establishing  supposed  causal  links  between  public  interventions  and  their 
expected  impacts.  This  approach  assumes  the  comparison  of  a  group 
exposed  to  the  programme  (experimental  group)  and  a  strictly  identical 
group (control group) that allows what would  have happened in the absence 
of  the  programme  to  be  observed. 

The  experimental  approach  is  used  to  verify  the  relations  of  cause  and 
effect  between  observed  socio-economic  changes  and  the  evaluated 
programme.  It  allows  the  state  of  a  given  social  situation  following  the 
implementation  of  a programme to  be empirically  compared  to  what  would 
have  been  the  situation  without  the  benefit  of  the  programme.  This 
approach  thus  allows  the  evaluation  of  the  net  impact  of  a  programme  by 
controlling 
factors.  Since  the  use  of 
comparison  groups that  are statistically  equivalent  strengthens  the  internal 
validity  of .the  technique,  it  can  be  considered  the  most  rigorous  way  of 
establishing  causal  relations  in relation to  the  results  of  a  programme. 

influence  of  confounding 

the 

to 

combat  alcohol 

Experimental  approaches  can  be  used  to  evaluate  a  wide  range  of  public 
interventions,  but  have  been  mainly  used  in  educational  programmes 
(programmes 
consumption,  smoking,  etc.). 
Experimentation  can  be  appropriate  when  it  is  possible  to  perform  a 
random  assignment  without  fear  of  social  injustice  occurring  from  the 
exclusion  of  the  control  group  from  the  benefits  of  public  intervention.  It is 
equally justified  in the  case  of  costly,  innovative  programmes  for  which  the 
most  reliable data  possible  are required. 

Implementing  an  experiment 

Experimentation  is based, first  and foremost,  on the  random assignment  of 
the  units  of  observation,  divided  between  the  experimental  group  and  the 
control  group  so  as  to  constitute  groups  that  are  statistically  equivalent. 
Thus,  the  probability  will  be  very  high  that  the  descriptive  statistical 
parameters  of  any  variable  describing  these two  groups  will  be similar.  The 
second  step  consists  of  applying  a  given  treatment  uniquely  to  the 
experimental  group (the only difference  between  the two  groups). This  step 
is followed  by  making  measured  observations  of the two  groups  before the 

Tools  for analysing data 

application  of the treatment  (E (1) and E' (1)), and then after  (E (2) and  E' (2)). 
Finally, the gap  between the two  groups'  initial  and final states  is compared 
to  provide  a measure of  the impact  of the treatment.  Whether  the  impact  is 
significant  or  not  is judged  on the  basis  of  the  extent  of the  difference. 

The table  below  summarises  the  implementation  of this  approach: 

Initial 
measurements 

Application 
of  treatment 

Measurement 
in  fine 

Variation  in 
measurements 

E(1) 

X 

E (2) 

E(2)-E(1) 

Experimental 
group 

Control  group 

E'(1) 

E'(2) 

E' (2) -  E' (1) 

The  limits  of  experimentation 

to 

is  difficult 

The  experimental  approach 
implement.  Often,  random 
assignment  cannot  be  achieved  since  it  is  sometimes  impossible  to  define 
the  variable  that  would  allow  the  constitution  of  a  comparison  group  (e.g. 
when the target population benefits  in totality from the programme).  Random 
selection  of  the  members  of  the  groups  studied  is  rarely  possible  for  the 
totality  of  the  target  population,  while  the  use  of  volunteers  is  a  source  of 
bias,  known  as  "autoselection".  Furthermore,  the  artificial  creation  of  a 
control  group  harms  the  external  validity  of  the  experimental  approach. 
Whatever  method  is used to select the participants, external validity  may  be 
compromised.  Contextual  factors  linked  to  the  choice  of  sites  also  limit  the 
representativeness  of the experiment.  Many elements therefore confirm  that 
the  experimental  approach  reduces  the  extent  to  which  findings  can  be 
generalised  to  an  entire  programme  or to  other  contexts. 

Ethical  and  political  problems  can furthermore  arise, when  it  is  considered 
the 
that  only  the  members  of  the  experimental  group  benefit  from 
programme  to  the  detriment  of  individuals  in the  control  group.  Moreover, 
the  experimental  approach  requires  significant  financial,  intellectual  and 
institutional  resources,  and  the  constitution  of  statistically  equivalent 
groups is in itself  a demanding  task.  Finally, this approach does  not  provide 
information  about  the effects  of  sudden  events. 

How  to  use  the  tool  for  evaluation 

The  experimental  approach  is justified  in  a certain  number  of  cases  where 
causal  relations  are  tew,  simple  and  known,  when  the  objective  is  to 

^ ^^ 

Principal  evaluation  techniques and tools 

measure  impacts  that  are  stable,  and  not  to  explain  complex  phenomena 
that  are in a state  of  flux. 

This  tool  is  therefore  not  particularly  well  suited  to  evaluating  Structural 
Funds  and  an  example  of  its  application  within  this  framework  cannot  be 
the 
found.  Practical  difficulties 
uncertainty  of  its  usefulness  can  explain  this  absence.  Structural  Fund 
programmes  are applied over a relatively long period. Thus, the two  groups, 
experimental  and  control,  could  be  exposed  to  certain  variations  and 
obvious  differences  would therefore  occur  between  the  groups. 

implementation  and 

relating 

its 

to 

Further  reading 

Boruch  R.  F. (1997), Randomized  Experiments  for  Planning  and  Evaluation, 

Thousand  Oaks, CA:  Sage, 265  p. 
A book  which  presents  a measured  argument  in favour of the  use of 
the  experimentation  technique. 

Tools  for analysing  data 

R e g r e s s i on  a n a l y s is 

Regression  analysis 

is  used 

to 

obtain  a  quantitative  estimation  of 

the  influence  of  several  explanatory 

factors 

(e.g. 

intervention 

to  be 

evaluated,  confounding  factors)  on 

a  variable  that  has  to  be  explained 

(e.g. the  result  of  an  intervention).  A 

logical 

explanatory  m o d el 

is 

required  if  regression  analysis  is  to 

be  used.  The  tool  calculates 

the 

O 

Overall 
evaluation 

0 

Medium 
technical 
complexity 

© 

φ 

Medium 
cost 

O 

Innovative 
use 

Retrospective 
use 

parameters  of  this  model,  adjusting  them  as  accurately  as  possible 

in 

relation  to  developments  observed  in  the  past.  Regression  analysis  can  be 

applied  in  various  forms,  depending  on  whether  the  variables  of  the  model 

are  continuous  or  not,  and  on  whether  their  relations  are  linear  or  not. 

The  tool  has  been  used  for  about  a  century  by  statisticians,  particularly  in 

biology.  It  was  generalised  with  the  development  of  microcomputers  and 

statistical  software  enabling  users  to  easily  adjust  c o m p l ex  models  to  fit 

numerous  data. 

Example: Evaluation  of  ESF training  courses  in  Spain 

In  1995  the  Departments  of  Economic  Analysis  and  Applied  Economics,  of  the 
Faculty  of  Economic  Science  at the  Independent  University  in Madrid, evaluated  the 
effects of certain training  measures set up with the assistance of the European  Social 
Fund. This evaluation  was to  provide answers  to  the following  questions:  How  many 
participants  found  a  job  after  attending  the  course?  What  is  the  profile  of  the 
participants  who  found  a job?  What  is the  influence  of the  regional  context? 

A survey  was carried out  on a sample  of  13,127  real or  potential  trainees  throughout 
Spain  (with  the  exception  of three  regions). 

An explanatory  model  was  constructed  with the following  variables: 

Variables to  be explained  (dependent  variables): the employment  rate and the  length 
of  time  looking  for  a job; 

Explanatory  variables:  gender,  age,  qualifications,  place  of  residence,  type  of 
unemployment  (long-term, search  for  a first job), and  professional  experience; 

A special  explanatory  variable  was  added:  participation  in the  programme  (duration 
of  ESF training). 

This  model  comprises  continuous  variables  such  as  age  or  length  of  time  seeking 
employment,  as well as discontinuous  variables such as gender or the fact of  finding 

Principal  evaluation  techniques  and tools 

a  job.  Regression  analysis  was  used  to  adjust  the  model  to  the  results  of  a 
questionnais  survey  carried  out  on  a  sample  of  13,127  real  or  potential  trainees 
throughout  Spain  (with the exception  of  three  regions). 

The  regression  analysis  was  used  to  estimate  the  impact  of  the  programme  on  the 
employment  of the trainees. Moreover,  it was  used to produce additional  estimations 
such  as: 

•  Men  improve  their  chances  of  finding  a job  by  5%; 

•  Regional  locality  is  important:  trainees  living  in  the  Balearic  Islands  have  about 

20%  more  chance  of  finding  a job  than those  in Galicia; 

•  People with  secondary  school  education  improve  their  chances  of finding a job  by 

over  12% compared  to those  who  only  have  primary  school  education. 

Source: F. Saez-Fernandez,  M. Toledo, "Education, Labour Market  and  Employement 
Policy  -  Empirical  Evidence  and  Implications',  7th  EALE Conference,  Madrid  1995. 

What  is the  purpose  of the  tool? 

When  applicable,  regression  analysis 
for 
estimating effects.  It serves to  attribute  part of the changes observed to  the 
evaluated  programme 
to  explanatory  or 
confounding  factors. 

(net  effect)  and 

is  an  excellent 

technique 

the  rest 

In which  cases  can  it  be  used? 

The  tool  is  applicable  only  if  one  already  has  a  credible  and  sound 
explanatory  model.  It  must  then  be  used  in  cases  where  the  effects  of  the 
intervention  are  well  identified  and  where  the  production  process  of  the 
effects  is  known. 

Regression  analysis  may  typically  be  used  when  one  of  the  effects  of  the 
evaluated  programme  is given  precedence  (e.g. placement  rate). 

In  order 
to  apply  the  tool,  a  considerable  volume  of  quantitative 
observations  is  required:  between  one  thousand  and  ten  thousand  units 
(trainees,  enterprises,  farms,  etc.).  The  observations  must  be  detailed 
according  to  all the  variables  of  the  explanatory  model. 

How  should  it  be  implemented? 

Step  1.  Construction  of  the  causal  model:  Construction  of 
the 
explanatory  model  is  a  difficult  step  that  must  be  carried  out  in  line  with 
reflection  on  the  action  theory  of  the  intervention.  Several  kinds  of  variable 

Tools  for analysing data 

may  exist. They  may  be specially  created, for example to take  into  account 
the  fact  that  an  individual  has  benefited  from  support  or  not  (a  dummy 
variable, taking values  0 or  1). A variable  may  also  represent  an  observable 
characteristic  (having  a job  or  not)  or  an  unobservable  one  (probability  of 
having  a job). The  model  may  presume  that  a particular  variable  evolves  in 
a  linear,  logarithmic  or  probabilistic  way.  All  the  explanatory  models  are 
constructed  on the  basis  of  a  model  such as: 

Y:f(X0,  X1, X2...,Xn)  in which: 

Y  is the  change  that  the  programme  is  mainly  supposed  to  produce  (e.g. 
employment  of  trainees) 

f  is a function  of  several  explanatory  variables 

XO is a variable  representing  the fact  that  an individual  benefits  or  not  from 
the  intervention. 

X1,...,  Xn are the other  variables  likely  to  explain the  change. 

Box 26-  Performing  a regression  analysis 

S t e p l.  Construction  of 
the causal  model 

«-  ►! 

Y  =f  (Χ1.Χ2 
This relation  is not  given 
by the method 

Xn) 

; 
; 
; 

I 

Step 2.  Construction  of 
a sample 

I 

Step 3.  Collection of 
data 

I 

Step 4.  Calculation  of 
the coefficients of  the 
relationship  model 

i 

Step 5.  Test of  the 
coefficients  obtained 
(adjustments  quality) 

I 

See questionnaire 
survey 

» 

J  Significance  of  parameters 

Analysis  of  residue 

' 

Step 6.  Generalisation to 
the total  population 
(inference) 

Principal  evaluation  techniques  and tools 

A  good  model  has  to  select  few  variables  with  substantial  explanatory 
power.  Important  factors  must  not  be  forgotten.  Causal  relations  that 
function  both  ways  must  be  avoided, e.g. X1 explains  Y  but  Y explains  X1 
(circular  causality). 

If  two  explanatory  variables  are  linearly  dependent  and  provide  the  same 
information,  it  is futile  to  consider  both  of  them.  In fact  phenomena  of  co-
linearity  weaken  the  adjustment.  For  example,  if  the  women  questioned 
have  experienced  periods  of  previous  unemployment  which  are 
systematically  longer  than those  of  men, it  will  not  be  possible  to  separate 
the  influence  of  the  two  explanatory  factors:  gender  and  duration  of 
previous  unemployment. 

Some  forms  of  regression  analysis  (path  analysis)  allow  for  the  separation 
of  direct  and  indirect  effects.  Indirect  effects  occur  through  one  or  more 
intermediary  variables.  Given  the  complexity  characterising  most  socio­
economic  phenomena, this  priority  is  important. 

Step  2.  Construction  of  a  sample:  To  apply  multiple  regression,  a  large 
the 
sample 
questionnaire  survey  for the  construction  of  samples. 

is  usually  required  (2,000  to  15,000 

individuals).  See 

Step  3.  Data  collection:  Data  are  generally  taken  from  the  monitoring 
system,  from  a questionnaire  survey  or from  a combination  of  both. 

Step 4. Calculation of coefficients: This  phase in the analysis  is facilitated 
to  a  large  extent  by  statistical  software  that  is  both  affordable  and  can  be 
used  on  a microcomputer  (e.g.:  SAS, SPSS, SPAD, etc.). 

Step  5. Test  of the  adjustment  made: The  preparation  and  adjustment  of 
the  model  are  aimed  at  trying  to  explain  as  much  of  the  variability  of  the 
observed  changes  as  possible.  The  part  that  remains  unexplained 
represents  the  residue. Thus, the  smaller  the  residue, the  better  the  quality 
of the model and its adjustment. The analysis of residues  is a very  important 
step:  it  is  here  that  one  sees  the  degree  to  which  the  model  has  been 
adapted  to  the  phenomena  one wants  to  explain.  It  is the  residue  analysis 
that  enables  one  to  tell  whether  the  tool  has  made  it  possible  to  estimate 
the  effects  in a  valid  way  or  not.  If  significant  anomalies  are detected,  use 
of  the  tool  must  be  considered  as  a  failure  and  the  evaluation  must  be 
stopped  before  Step  6. 

Step  6.  The  purpose  of  this  last  phase  is  to  generalise  the  coefficients  of 
the  model  to  the  population  as  a whole.  It  is therefore  used  to  produce  an 
estimation  of the  effects. 

Tools  for analysing data 

Application  to the  Structural  Funds 

limits.  The 

In  the  context  of  the  Structural  Funds,  use  of  this  technique  soon 
encounters 
requires 
quantitative  knowledge  of  several  thousand  individuals.  It  is therefore  also 
interventions 
time-consuming  and  expensive  and  is  applied  only  to 
affecting  a  large  number  of  people. 

is  unwieldy  because 

technique 

it 

Strengths  and  limits  in  practice 

•  Use  of  regression  analysis  affords  an  opportunity  to  specify  hypotheses 
concerning  the  nature  of  effects  (action  theory),  as  well  as  confounding 
factors. 

•  When 

it  is  successfully  concluded  (statistically  valid  adjustment), 
regression  analysis  produces  a  quantitative  estimate  of  net  effects  and 
solves  one of  the  most  difficult  problems  in evaluation. 

•  The observations  must  present  sufficiently  contrasted  evolutions to  allow 
for  adjustment.  For  example,  if  all  the  observations  concern  the  30-40 
age  group,  it  will  not  be  possible  to  estimate  the  influence  of  age  on 
employment. 

•  Regression analysis  is likely to reach the conclusion  that there is a strong 
link  between  two  variables,  whereas  the  influence  of  other,  more 
important,  variables  has  not  been  estimated  (this  error  is  called  "data 
snooping"). 

•  Relations  between  the  different  explained  and  explanatory  variables  are 
often  circular  (X explains  Y and Y explains  X). In this case, the  method  is 
inapplicable. 

•  The  implementation  of the  tool  is both time-consuming  and  expensive. 

•  The  tool  has to  be  used  with  great  care: errors  in  data  collection  and  in 
the  explanatory  model  affect  conclusions  and  this  can  easily  be 
overlooked. 

Further  reading 

Wonnacot  T.H.,  Wonnacot  R.J.  (1984),  Statistique,  Economie,  Gestion, 

Sciences,  Médecine,  Paris:  Economica,  790  pp. 
Manual  explaining  the  basic  principles  of  statistics  in  the  social 
sciences. Clear  and  illustrated  with  numerous  examples. 

Principal evaluation techniques and tools 

Mohr  L.B.  (1995),  Impact  analysis  for  program  evaluation,  London:  Sage 

Publications,  310  pp. 
This  book  addresses 
the  main 
evaluation.  Clear,  avoids  a mathematical  mode  of  expression. 

techniques 

for  programme 

Tools for making a judgement 

In  the  framework  of  in-depth  evaluation,  evaluators  can  use  any  available 
tool  for  making  a judgement,  particularly  those  that  have  been  developed 
for  project  evaluation. 

•  Cost-effectiveness  analysis judges  the  efficiency  of  the  intervention  by 
quantitative  comparison  with  one  or  more  other  interventions.  It  favours 
an  objective  considered  to  be a priority.  It  is particularly  well suited  to  ex 
post  evaluation. 

•  Benchmarking  judges  the  efficiency  of  the  intervention  by  qualitative 
comparison  with  another  intervention  recognised  as  being  excellent  (i.e. 
a  benchmark).  With  benchmarking,  several  effects  can  be  taken  into 
account  and  synthesised. 

•  Cost-benefit  analysis judges the  efficiency  of  the  intervention  by  taking 
into  account  all  its  effects  and  by  making  a quantitative  synthesis.  Cost-
benefit  analysis  can  judge  an  intervention  in  the  absolute,  without  any 
comparative  reference to  other  interventions.  It is generally  used ex  ante, 
but  all the conditions  required for  its efficient  application  are rarely  met. 

•  Multicriteria  analysis (presented above) compares the effectiveness  and 
efficiency  of  several  projects or interventions. The tool takes  into  account 
and  synthesises  several  effects,  quantitatively  and/or  qualitatively.  This 
tool can be used to take into account any differences  in the points of view 
of  the  partners. The  tool  is generally  used  ex  ante.  Its flexibility  makes  it 
suitable  for  use  in many  situations. 

•  The  expert  panel  (presented  above)  is  a very  flexible  tool  which  can  be 
used  to  produce  a  synthetic 
judgement  based  on  qualitative  and 
quantitative  data,  even  if  these  are  incomplete.  With  this  tool  any 
differences  between  the  points  of  view  of the  partners  can  be taken  into 
account.  To  a  lesser  extent,  the  Delphi  survey  (presented  above)  can 
provide  similar  assistance  in the formulation  of  a judgement. 

•  Finally,  the  colour  vote  (presented  above)  is  a tool  which  may  be  used 
during the final synthesis  of  an evaluation, for validating conclusions  and 
recommendations. 

Tools for making  a  judgement 

Cost-Effectiveness 
Analysis 

Cost-effectiveness 

analysis 

originates  from  certain  types  of 

evaluations, 

notably 

in 

the 

domains  of  health  and  road  safety. 

Characterised  by its pragmatism,  it 

focuses  directly  on  the  main  effect 

of  a  programme  in  relation  to  its 

cost,  purposely  excluding  other 

potential  effects.  It  is  a  simplified 

A 

In-depth 
evaluation 

Θ 

Q 

Medium 
technical 
complexity 

O 

Φ 

Medium 
cost 

O 

Use 
stabilised 

Retrospective 
use 

Prospective 
use 

form  of  "value-for-money"  evaluation  (VFM)  in  which  the  "value"  of  the 

programme  is expressed as a single dimension. Cost-effectiveness  analysis 

differs  from  cost-benefit  analysis,  particularly  in  so  far  as the  effect  of  the 

programme  is  expressed  in  physical  quantities  (e.g.  number  of  deaths 

avoided)  and  not  as  a monetary  value. 

Example: Creating  and safeguarding jobs  in Wales 

This example  corresponds  to that  of Box 3. 

The  British  National  Audit  Office  (NAO) carried  out an evaluation,  based  on  cost-
effectiveness  analysis, of policies for the creation  and  safeguarding  of jobs  in Wales. 
These policies  had been implemented  over the preceding twenty  year period by three 
key  agencies: the Welsh  Office  (WO)  responsible  for urban  development,  the Welsh 
Development  Agency  (WDA)  responsible  for  industrial  development  and  the 
Development  Board  for  Rural  Wales  (DBRW)  charged  with  social  and  economic 
development  in the Northern  and Central  regions  of Wales. The evaluation  was  set 
up jointly  by the NAO and the WO and  lasted for 5 months. It focused on the effects 
of these  policies on a sample of 303  beneficiary  firms, which  were questioned  about 
these  effects. 

The results of the interviews were used to calculate the cost  per job of the  measures, 
on  the  basis  of  the  foreca .t  number  of  jobs  created,  the  deadweight  and 
displacement  effects  and the beneficial  effects on suppliers of these enterprises. The 
overall  cost  of  the  programme  was then  divided  by the  net  benefits  to  obtain  a 
measure  of  cost-effectiveness;  i.e.  cost-effectiveness  =  cost  of  the  programme  / 
(number  of gross jobs  created -  deadweight  -  displacement  + supplier  effects). 

The absence  of uniformity  in the implementation  of regional  development  actions in 
Wales  made  comparisons  of costs  per job created  difficult.  Nevertheless,  the  NAO 
concluded  that  the  employment  effects  of  the  programmes  were  often  over-
evaluated,  particularly  since  deadweight  and displacement  effects  were  not  taken 

Principal evaluation techniques and tools 

into account, but that the programmes had nevertheless created a significant number 
of new jobs. 
Source: S. Roberts and Ch. Pollitt, (1994) Audit or Evaluation? a National Audit Office 
VFM Study. Public Administration 

What  is the  purpose  of the  tool? 

Cost-effectiveness  analysis  is  used  to  evaluate  the  effectiveness  of  a 
programme  in terms of a single criterion that  is considered  preponderant  or 
the  key  objective  of  the  programme.  On  the  basis  of  this  criterion,  it  is 
possible to compare the effectiveness  of the  programme with that  of  similar 
programmes  in  other  regions  or  alternative  approaches  focusing  on  the 
creation  of  similar  results  in the  same region. 

the 
Cost-effectiveness  analysis  may  constitute 
benchmarking  of  programmes  or  actions  implemented  in similar  contexts, 
but  which  apparently  have  a  better  cost-effectiveness  ratio  than  the 
programme  under  analysis. 

first  step 

the 

in 

In which  cases  can  it be  used? 

Cost-effectiveness  analysis  is used  in both  ex  ante and  ex  post  evaluation. 
It  is  used  when  all  the  expected  effects  have  been  defined  and  are 
homogeneous  and/or  can  be  measured  in  terms  of  a  key  result  (e.g.  the 
number  of jobs  created). 

How  should  it be  implemented? 

Step  1. Determination  of the  programme  objective: This first  step  is one 
of  explanation  of  the  programme 
logic.  The  following  questions  are 
answered:  What  are  the  goals  to  be  achieved?  What  are  the  programme 
outputs?  What  are the expected  impacts?  Among them, which one may  be 
considered  predominant? 

Step  2.  Definition  of  the  total  cost  of  the  programme:  At  this  stage  the 
cost  of  all the  public  resources  of  the  programme  are added  up to  obtain  a 
total  cost.  Generally  speaking,  only  direct  resources  that  have  a  well-
defined  monetary  value are  included.  Less tangible  costs, such  as the  time 
passed  by  staff,  and  in  particular,  supervision  costs,  are  not  usually  taken 
into  account.  The  cost  of  a programme  is sometimes  measured  in  relation 
to  the  benefits  that  could  have  been  obtained  by  allocating  public  monies 
to  other  ends  (opportunity  costs). 

Tools for making  a  judgement 

Step  3.  Measuring  the  impact:  This  step  is  the  trickiest  one.  Numerous 
studies,  including  the  evaluation  of  the  NAO,  presented  above;  use 
empirical  approaches  based  on  the  collection  of  primary  data,  in  order  to 
gather  information on the positive effects  of a programme  under  evaluation. 
It is also  possible,  however, to  estimate  impacts  on the  basis of  secondary 
data  and/or  the  modelling  of  the  implementation  of  the  programme. 
Whatever the  method  used, it is important to  have an exact  picture  not only 
of  positive  effects,  but  also  of  deadweight,  displacement  effects,  crowding 
out  effects,  etc.  Certain  evaluators  have  estimated  that  indications  should 
be given  about  multiplier  and  unexpected  secondary  effects.  In  reality,  it  is 
extremely  difficult 
in  practice  to  accurately  evaluate  these  secondary 
effects.  Consequently,  the  majority  of  evaluations  does  not  take  them  into 
account  or  gives estimations  of  their  level. This  solution  is  preferable  given 
the importance of secondary  effects. The lack of accuracy  of the  estimation 
of  these  impacts  can  be  attenuated  by  the  use  of  sensitivity  analysis  to 
indicate  the  probable  level of  benefits. 

Step 4. Calculation  of the  cost-effectiveness  ratio: The last  phase is that 
of  the  calculation  which  will  give  the  final  result.  It  consists  of  a  simple 
division, for  which  the formula  will  be of  the following  type: 

Programme  cost  /  (gross  effects  -  deadweight  and  displacement  effects, 
and should the occasion arises + multiplier  effects  on suppliers and  indirect 
addressees). 

Application  to  the  Structural  Funds 

The  cost-effectiveness  technique  is  difficult  to  apply  to  Structural  Fund 
programmes  that  often  have  multiple  objectives.  From  this  point  of  view, 
multicriteria  analysis  is  often  more  appropriate.  It  may,  however,  be  a 
focused  on 
suitable 
employment  effects,  effects  on  the  creation  of  SMEs,  etc.,  or  to  compare 
approaches  that  are  aimed  at  producing  similar  impacts.  It  is  nevertheless 
important  to ensure that the basis of the calculation  and the reliability of  the 
data  relating to the  costs  and  net  benefits  are  consistent. 

thematic  evaluations  such  as  those 

tool 

for 

Strengths  and  limits  in  practice 

•  The  analysis  must  be  preceded  by  reflection  on  the  programme 
objectives,  on  its  main  objective  and  on  the  indicator  applied  to  that 
objective. 

Principal  evaluation  techniques and tools 

•  Cost-effectiveness  analysis tends, by its nature, to  focus  on direct  results 
that  occur  over the short  to  medium term, but does  not  look  in general at 
more  long-term  impacts.  Yet  it  on  these  that  the  overall  effectiveness  or 
the  lack  of  effectiveness  of  programmes  and  policies  depend.  For 
example,  the  effectiveness  of  Structural  Fund  programmes  depends  on 
achieving  a  combination  of  multiple  objectives  aimed  at  promoting 
growth  and  autonomous  development  within  a region. 

•  Cost-effectiveness  analysis  can  serve  to  compare  programmes  only 
when  their  implementation  is  straightforward  and  their  impacts  of  a 
similar  nature. This type  of  case  is far  rarer  than  one  might  imagine.  Is it 
possible  to  compare  jobs  created  without  introducing  differentiation 
concerning  their  stability,  quality,  required  qualifications,  etc.? 

•  Since  cost-effectiveness  analysis  is  based  on  an  estimation  of  the 
programme  impact  in  relation  to  its  main  objective,  it  presents  the 
advantage  of  producing  easily  understood  findings,  which  concentrate 
on  the  main  preoccupations  of  key  groups  (including  politicians  and 
decision-makers). 

•  If the analysis  is based  on  specific  links  between  inputs  and outputs  that 
are relatively well established, the tool can facilitate the description  of  the 
actual  functioning  of  programmes.  This  can  be  useful  for  the  refining  of 
existing  policies  or  improving  the  effectiveness  of  future  interventions. 

•  Cost-effectiveness  analysis  can  only  be  used  to  compare  programmes 
that are simple to  implement  and of which the impacts are the same type. 
Yet, this situation  is much  rarer than  is imagined. Thus, for example, even 
programmes  targeted  on  an  identical  major  objective,  such  as  the 
creation  of  jobs,  can  create  employment  opportunities 
that  are 
qualitatively  different  (e.g.  in  terms  of  longevity,  security,  remuneration, 
probability  that  they  will  be  accessible  for  the  inhabitants  of  targeted 
regions, etc.). 

•  Establishing  causal links of this type  requires the ability  to obtain  detailed 
data  from  programme  managers  and  addressees.  When  these  are  not 
collected  through  the  monitoring  of  programmes  and  projects,  the 
evaluation  will  involve the  painstaking  collection  of  primary  data. 

•  Ideally,  cost-effectiveness  analysis  should  be  used  in  combination  with 
other  techniques  of  economic  analysis  in  order  to  analyse  longer-term 
impacts  on the  regional  GDP  and  competitiveness. 

Tools for making a judgement 

Further reading 

Creating and Safeguarding Jobs in Wales, A Value for Money Audit (1993), 

Brunei University, MEANS 
A pilot evaluation of the MEANS programme, 20 ρ 

Garrabé M., Ingénierie de l'évaluation économique, Paris: Ellipses, 255p. 

A book on economic evaluation, describing the main economic 
evaluation tools. Among these, features cost-benefit analysis, which 
includes cost-effectiveness analysis. Accessible. 

Gray A.W. (1995), EU Structural Funds and Other Public Sector 

Investments. A Guide to Evaluation Methods, Dublin: Gill & 
Macmillan,106 p. 
A book on economic evaluation techniques, specifically oriented to 
cost-benefit analysis. Contains a complete description of cost-
effectiveness analysis, its logical articulation and the conditions of 
its implementation. Highly accessible. 

Sanders J.R (1994), The Program Evaluation Standards, How to Assess 
Evaluations of Educational Programs, Second Edition, London: 
Sage Publications, 222 p. 
A manual devoted to the main steps in the evaluation of 
programmes. Contains a brief description of cost-effectiveness 
analysis. Highly accessible. 

Tools for making  a  judgement 

Benchmarking 

Benchmarking  w as  developed 

in 

the  1980s  by  the  major  American 

electronics 

firms 

that  wanted 

to 

improve  their  competitiveness  and 

quality 

in  the  face  of  Japanese 

c o m p e t i t i o n.  T he 

t e c h n i q ue 

is 

b a s ed 

on 

t he  e x c h a n ge 

of 

information 

relating 

to 

t he 

o u t c o m es  a nd  organisation  of 

certain  segments  of  production, 

O 

Overall 
evaluation 

A 

In-depth 
evaluation 

0 

Low 
technical 
complexity 

O 

@> 

High 
cost 

© 

Prospective 
use 

Innovative 
use 

between  firms  with  a  reputation  for  excellence  in a  given  field. 

In  the field  of  "New Public  Management"  this  practice  has been  translated 

into  the  management  of  public  services  a nd  municipal  administrations.  It 

may  also  be  relevant 

t h o u gh 

in  t he  o n g o i ng  evaluation  of  public 

interventions. 

Example:  evaluation  of urban  management 

The  Bertelsmann  foundation  in Germany  organised  a competition,  open  to  German 
towns. The  programme  judged  the quality of administrative  reforms  implemented by 
the candidate  towns, by comparing  them to three  benchmarks. 
In  order  to  have  a  standard  for  comparison,  the  foundation  called  on  a  panel  of 
experts  which  defined  the criteria  to be used  to judge  the quality  of  administrative 
reforms.  This  jury  then  identified  cases  of  "best  practices"  in terms  of the  chosen 
criteria,  i.e.  three  towns,  situated  in  New Zealand,  the  UK  and  Switzerland,  well 
known for the originality  and effectiveness of their administrative reforms. Each of the 
towns  identified was considered to be "exemplary"  and  served as a "benchmark" for 
at  least one of the criteria  defined  by the jury. 

The  criteria  were  as  follows:  modalities  of  commercialisation  of  public  services; 
quality strategy  applied to the public services; strategic  management  concerning the 
organisation  of services; and financial  management. 

Apart  from the fact  that  it provided  a clear set of criteria for judging  the competition, 
the  benchmarking  technique  made  it  possible  to  evaluate,  on a  broader  scale, the 
progress  of administrative  reform  in many  German  towns. The process  was seen as 
highlighting the backwardness of German towns  in implementing  reforms of this  kind. 
Source:  Frieder  Naschold, 
"Internal  Modernisation,  Competition,  Fiscal 
Consolidation.  International  Experiences  with  Local  Government  Administrative 
Reform", WZB,  1996. 

Principal evaluation  techniques and tools 

What  is the  purpose  of  the  tool? 

The  aim  of  benchmarking 
is  the  identification  of  the  strengths  and 
weaknesses  of  an  organisation  by  comparing  it  to  other  organisations 
renowned  for  being  the  best  in the  field  in  which  progress  must  be  made. 
The  approach  consists  of  the  comparison  of  performances  situated  at 
different  levels, depending  on  the  benchmark  chosen: 
•  External  benchmarking  is  used  to  compare  the  different  functions  of  the 
production  process  of  an  organisation  to  similar  functions  in  other 
organisations. 

the  methods  applied  within 

•  Internal  benchmarking  is  used  to  study  best  practice  and  to  use  it  to 
improve 
Internal 
benchmarking  is  applied  to  the  divisions  of  a  parent  organisation.  It  is 
sometimes  considered  to  be  a  less  demanding  approach  because  it 
involves  comparisons  with  a  relatively  small  group  as  opposed  to  the 
'best  of  the  rest'. 

the  organisation. 

The advantage of  benchmarking  lies in the decomposition  of the  productive 
logic  into  elementary  functions  the  performance  of  each  of  which  is  then 
compared. The exchange of information  on these elementary functions  thus 
becomes  possible  between  and within  very  different  organisations. 

This type of  approach  provides a more objective  view  of current  practice  in 
an  organisation,  by  comparing  it  to  practices  elsewhere.  It  is  a  way  of 
learning from others' experience. Practised on a larger scale,  benchmarking 
may  promote  the  spread  of  good  practice. 

Although 
benchmarking  may  be  used for  the evaluation  of  measures  and  projects. 

it  was  originally  designed  to  be  applied  to  organisations, 

• 

In which  cases  can  it be  used? 

As  an  evaluation  tool,  benchmarking 
is  applied  to  the  evaluation  of 
measures.  It  consists  of  identifying  measures  (or  projects)  recognised  as 
best  practice  and  using  them  as  a  standard  of  comparison  for  evaluating 
another  measure  (or  project). 

This  enables  one  to  clarify  the  functioning  of  the  programme,  by  detailing 
the  different  measures  characterising  it,  and  to  identify  the  strengths  and 
weaknesses  of  particular  approaches. 

The  technique  may  be  used  to  determine  benchmarks 
in  terms  of 
effectiveness  and  efficiency  -  that  is  to  say,  programmes  recognised  for 
their  quality  -  and  thus  to  implement  the  changes  to  be  made  in  the 

Tools for making  a  judgement 

framework  of  the  evaluated  programme  to  enable 
performance  of the  best  performers. 

it  to  match 

the 

How  should  it  be  implemented? 

Box 27  -  Benchmarking  in four  steps 

1. 
Select  those  parts 
of  the process 
which  will be 
subject  to 
benchmarking 

-

2. 
Identify 
a  partner 
for  the 
benchmarking 

-

3. 
Observe 
the  process 
in the  partner's 
situation 

* 

4. 
Analyse 
the  discrepancies 
in  performance 

Step  1.  In  the  preliminary  phase  the  nature  of  the  problem  must  be 
identified  and  the  problem  presented  clearly.  A  first  step  consists  of  the 
conceptualisation  of the  process to  be analysed. The  benchmarking  will  be 
able  to  focus  only  on  some  of  the  steps  in  the  process.  These  will  be 
chosen  in  relation  to  the  nature  of  the  problem:  they  are  the  benchmark 
criteria.  A  second  step  consists  of  constructing  indicators  which  can  be 
used to  make the  evaluation  criteria  measurable. 

Step 2. The choice of the organisation(s)  with which the comparison  will  be 
made  depends  on  the  criteria  chosen  for  the  benchmarking.  The 
comparators  need  not  necessarily  have  similar  structures,  but  will  be 
chosen  because they are seen as relatively good  performers  in terms  of the 
benchmarking  criteria.  Benchmarking  is,  however,  based  on  a  form  of 
reciprocity.  Both  partners  must  get  something  out  of  it.  If  this  is  not  the 
case,  the  benchmarking  will  be of  limited  interest. 

Step  3.  It  is  at  this  point  that  the  benchmarking  takes  place.  It  consists  of 
an in-depth analysis of the criteria identified  in the partner organisation, with 
a  view 
the  solutions 
implemented  to  meet  these  criteria. 

to  establishing  a  systematic  comparison  of 

Step 4. This analysis may have different  forms: face-to-face contact  through 
visits  in the  organisation,  the  sending  of  a  questionnaire,  use  of  specialist 
consultants  or  panels  composed  of  members  of  the organisations  involved. 

Whilst  all benchmarking  exercises  include  these  four  stages, they  can  take 
a variety  of  different  forms.  In some  cases  benchmarking  is used  simply  to 
compare  the  outputs  or  outcomes  achieved  by  different  organisations  or 

Principal evaluation  techniques and tools 

policies.  Here  the  focus  is  upon  assessing  relative  effectiveness  and  the 
result is often some form of ranking of  performance through for example the 
publication  of  'league  tables'.  More  and  more agencies  are though  going  a 
step further,  using benchmarking  as a diagnostic  tool to enable  programme 
managers  not  just  to  observe  differences  in  performance  but  also  to 
analyse the reasons for these and to  learn from  each other  about  strategies 
for  enhancing  effectiveness.  This  latter  approach  is  known  as  'process 
benchmarking'.  It  requires  analyses  of  the  different  management  styles, 
organisational  cultures  and  environmental  contexts  within  which 
programmes  operate  as  well  the  means  by  which  policies  are  being 
implemented.  Its  value  lies  in  the  ability  to  identify  good  practice  and  to 
transfer  this  to  or  modify  it for  implementation  in other  settings. 

It  is often  argued  that  comparisons  between  similar  programmes  operating 
in  comparable  environments  are  the  most  valuable  because 
their 
performance  might  be expected to  be broadly comparable  and they  benefit 
from  adopting  similar  practices.  Unlike  output  based  benchmarking 
however  this  is  not  always  the  case  and  some  programme  managers  have 
found  that  they  are able to  gain  important  new  insights  from  benchmarking 
against  organisations  which  are  quite  different  to  their  own.  Exchanges  of 
good  practice  between  public  and  private  sector  agencies  represent  an 
increasingly  common  example  of the  latter.  In the  context  of  the  structural 
funds  examples  might  include  process  benchmarking  to  compare  the 
operation of similar  programmes  in different  member states or  comparisons 
of EU funded programmes  and domestic  initiatives within the same country. 

Application  to  the  Structural  Funds 

The  application  of  benchmarking  to  the  Structural  funds  consists  of  the 
identification  of  projects  recognised  for  their  good  practice,  and  of  their 
comparison  with  projects  implemented  in  the  framework  of  the  Structural 
Funds. 
International 
Organisations,  Universities  or  other  public  or  private  organisations. 

It  may  concern  measures 

implemented 

in 

It  is  also  possible  to  use  benchmarking  during  the  preparation  of  a  new 
cycle  of  Structural  funds,  which  may  result  in  changes  of  a  regulatory, 
strategic  and  organisational  nature.  The  tool  may  also  be  applied  for  the 
purpose  of  improving  the evaluation  systems  themselves. 

Tools  for making  a  judgement 

Strengths  and  limits  in  practice 

•  Benchmarking is a flexible method which can be used to formulate solutions 

to  organisational  problems  and to  highlight  strengths  and weaknesses. 

•  The  implementation  of  benchmarking  must  involve  a  team  that  is  big 
enough  for  the  results  to  be  accepted  within  the  organisation,  but  small 
enough to  maintain  its flexibility  and adaptability  vis-à-vis the changes  to 
be  made.  In large  institutions  this  poses  problems. 

•  The  procedure  must  be  initiated  within  an  organisation  and  be 
constructed  on  a  participative  basis  if  it  is  to  be  effective.  If the  work  is 
formal  and  is  carried  out  by  outside  consultants  without  any  real 
participation  from  within,  the  benchmarking  will  inevitably  be  a  limited 
exercise  with  a very  limited  impact  in the  organisation. 

•  The identification of a partner for the benchmarking  may be  problematical 
in the framework  of  public  programmes,  and  it  may  be  necessary  to  use 
panels  of  experts  to  identify  the  best  practices. 

•  Benchmarking  is  often  based  on  the  sharing  of  strategic  information.  It 
may  therefore  constitute  a  risk  for  the  organisation  that  agrees  to 
participate  if it is a public enterprise or a research  laboratory  in a strategic 
political  or  economic  sector. 

Further  reading 

Balm,  G.  J.  (1994)"Evaluer  et  améliorer  ses  performances.  L'étalonnage", 

AFNOR,  Paris,  159  p. 
Sets  out  the  steps  of  benchmarking  as  undertaken  by  IBM  in  the 
United  States.  Sometimes  a little  dense. 

Camp,  R. (1990)  'Benchmarking:  the search  for  industry  best  practices  that 

lead  to superior  performance',  ASQC  Quality  Press. 

Chang,  R.  Y,  Kelly,  P.  K.  (1995)  "Progresser  avec  le  Benchmarking",  Les 

presses  du  management,  Paris,  122p. 
A  narrative  description  of  the  setting  up  of  benchmarking  in a  firm. 
Very  simple  and  lively. 

Karlöf  &  Partners  (1995).  "Pratiquer 

le  Benchmarking",  Les  éditions 

d'organisation,  Paris, 144p. 
Practical  guide  which  provides  a  lively  illustration  of  the  different 
steps  in benchmarking, from the  viewpoint  of  firms. 

Principal evaluation techniques and tools 

Naschold,  F. 

(1996), 

'Internal  Modernisation, 

Fiscal 
Internal  Experiences  with  Local  Government 

Consolidation: 
Administrative  Reform'.  Berlin: Science  Center. 
An  article  which  shows  how  to  use  benchmarking  as  an  evaluation 
method. 

Competition, 

Tools for making  a  judgement 

C o s t - b e n e f it  analysis 

A 

Θ 

© 

Medium 
cost 

In-depth 
evaluation 

Medium 
technical 
complexity 

Cost-benefit  analysis  (CBA)  is  a 
tool  for  analysing  public  projects. 
Its purpose is to determine  whether 
is 
realisation  of  a  project 
the 
desirable  from  the  point  of  view  of 
society  as  a  whole.  The  technique 
takes  into  account  and  assigns  a 
monetary  value  to  all  the  positive 
that 
and  negative  effects 
the 
project  has  on 
the  different 
components  of  society,  throughout 
the life of the project.  In order to validate the appropriateness  of the  project, 
it is important  not only to assess its feasibility  but also to evaluate  alternative 
options  in order to  select  the  most  appropriate  solution  for the  community. 

Θ 

G 

Prospective 
use 

Use 
stabilised 

Use  of  cost-benefit  analysis  was  generalised  during  the  1960s  in  the 
framework  of  the  PPBS  (Planning  and  Programming  Budgeting  System)  in 
the  United  States.  As  a decision-making  tool,  it  can  be  used  to  justify  the 
choice  of  a  project  among  a  set  of  possibilities.  Some  experiences  in  the 
systematic  use  of  CBA  have  been  carried  out,  for  example  by  the  World 
Bank  (in  the  1970/80s)  and  the  British  government  (in  the  1980s).  Today 
cost-benefit  analysis  plays a significant  role only  in the  evaluation  of  large-
scale  projects  such  as  major  infrastructure  projects. 

Example:  The  bullet  train  in  the  Madrid-Seville  corridor  -  a  cost-benefit 

evaluation 

The  Spanish  bullet  train  (AVE)  connected  Madrid  to  Seville  in  1992.  During  its 
construction,  several  technological  options  existed.  There  was  a  choice  between  a 
complete  high-speed  train  requiring  a  specific  infrastructure,  and  using  pendular 
technology  necessitating  less  investment  in  infrastructure  but  resulting  in  lower 
travelling  speeds  than  those  of  the AVE. 

Cost-benefit  analysis  was  applied  retrospectively  to this  project, taking  into  account 
the  structure  of  the  costs  and  rates  observed  since  1987. The  following  costs  and 
benefits  were  taken  into  account: 

•  cost  for the  Spanish  and  European tax  payer 

•  cost  for  the  passengers  of the  train 

•  cost  in terms  of  pollution  for  people  living  near  the  railway  line 

•  advantages  in  terms  of  time  saved  by  passengers  (calculated  as  a  monetary 

equivalent). 

Principal evaluation  techniques and tools 

Given  that  costs  are  concentrated  mainly  during  the  construction  and  that  benefits 
are  spread  out  over  a  long  period,  all  the  costs  and  benefits  were  calculated  by 
measuring  their  net  present  value. 

The  study  resulted  in  a  retrospective  validation  of  the  project  by  showing  that  its 
benefits  were, on  the  whole, greater  than  its  costs. 

Source: G.  De  RUS, Facultad  de Ciencas  Económicas  de Tarifa,  1996 

What  is the  purpose  of  the  tool? 

Cost  benefit  analysis  is  used  essentially  to  compare  all  the  impacts  of  a 
public  project  during  its  lifetime,  when  these  impacts,  even  if  they  are 
widely  diverse, can  be  measured  in  monetary  terms. 

In which  cases  can  it  be  used? 

Cost-benefit  analysis  is widely  used  in the framework  of  procedures  prior  to 
the  construction  of  transport  infrastructure.  It  is  used  when  the  expected 
impacts of a project extend  beyond the private sphere, are differentiated, and 
concern a heterogeneous  population. It is carried out within the framework of 
projects where the  addressees and those who  stand to  lose from the  project 
can be identified, and projects for which all the effects, even those not related 
to the  market,  can  be identified  and attributed  monetary  values. 

How  should  it  be  implemented? 

Step  1. Definition  of  the  project  and  its  cost:  This  first  phase  serves  to 
situate  the  project  in  its  context.  It  is  during  this  step  that  the  inherent 
constraints  in  the  realisation  of  the  project  are  determined,  and  that  the 
different  possible  variants  and  the  duration  of  the  project  are  examined. 
This  is the  phase  of  the determination  of  costs, with the  application,  where 
relevant,  of  fictive  prices  (e.g.  the  fictive  price  of  unskilled  labour  is  nil  if 
there  is a high  level  of  unemployment). 

Four  categories  of  variables  define  the  CBA: 

•  costs  (C):  not  only  the  monetary  but  the  social  costs  generated  by  the 
project  must  be  taken  into  account  (e.g.  the  real  cost  of  renting  land 
made  available free  of  charge  by a  public  institution  to  a user); 

•  benefits  (B): both  the  strictly  economic  spin-off  and  the  social  impact  of 
into  account  (e.g.  reduced  congestion 

the  project  must  be  taken 
following  the  construction  of  a highway); 

Tools for making  a  judgement 

•  time (t): since any investment  is characterised  by a lifespan, it is important 
to  extend  the analysis  to the  entire  period concerned  by the future  of  the 
project  (e.g. each  of  the  years  during  which  a  hospital  will  function);  the 
time  itself  is limited  by  the  lifespan  of  the  investment  which  is equal  to  n 
(e.g. 20 years for  infrastructure  and  10 years for  productive  investments); 

•  discount  rate (r): since the costs  and  benefits are spread  out  over time, it 
is  essential  to  discount  values  so  as  to  be  able  to  total  the  values 
calculated  (e.g. the  assessment  of an advantage at  its current  price  does 
not  allow  for  the  correct  comparison  of  two  distinct  periods,  given  the 
phenomenon  of  inflation). 

Box 28  -  Steps in a  cost-benefit  analysis 

Step 1. Identifying the project, 
checking  for  its  cost 

., 

' ' 
Step 2. Identifying  benefits and 
losses for social  groups 

' 
r 
Step 3.  Valuing benefits  and 
drawbacks  in euros 

1 
r 

Step 4.  Evaluation of  the 
project 

I  What are the  possible costs  of  running the 

^ 

are the  limits of  the project? 

Ί  How  long Is the  envisaged  duration  of  the 
I  project? 

i 

¡ 

J 

I 

Identification of  impacts for winners  and 
losers. Quantitative  estimates  of  impacts. 

! 

I  Use of social values, willingness to  pay and 
,  discount  rate 

; 

I  Add  present value  of all  gains and  losses In 
accordance  to the  compensation  principle. 

J 

;  Net present  value of the  project. 
;  Social  rate of  return. 

' 

! 
; 

Step 2. Identification and physical quantification of the advantages  and 
disadvantages  of  the  project. This  phase  consists  of  the  identification  of 
the  advantages  and  disadvantages  of  the  project,  which  may  affect 
different  groups  of  persons  (e.g. the  time  saved  by  people  using  the  new 
railway  line  or  the  amount  of  disturbance  suffered  by  the  people  living 
nearby).  CBA  differs  from  the  evaluation  of  private  projects  on  two 
fundamental  points: 

Principal evaluation  techniques and tools 

•  the  costs  and  benefits  to  all  members  of  society  are taken  into  account, 
income  of  the  agency 

and  not  only  the  monetary  expenses  and 
responsible  for  it; 

•  the  social  rate  of  return  may  differ  substantially  from  the  private  rate  of 
return  after  phenomena  of  externalities  have  been  taken  into  account, 
because  the  latter  concern  the  positive  or  negative  effects  that  society 
has to  accept  as a consequence  of  the  realisation  of  the  project. 

Externalities  concern  the  positive  or  negative  effects  that  society  has  to 
accept  as  a  consequence  of  the  realisation  of  a  given  economic  activity. 
They  refer  to  the  positive  or  negative  impacts  on  society,  without  these 
impacts  being  taken  into  consideration  uniquely  within  the  functioning  of 
the  market.  For  example,  the  improvement  of  transport  infrastructure  is 
reflected  in a reduced  accident  rate, time  savings,  reduced  transport  costs 
and  less  pollution.  These  effects  benefit  everyone,  without  everyone 
necessarily  having  contributed  to the  realisation  of the  project. 

Step  3.  Monetarisation  of  advantages  and  disadvantages:  This  phase 
must  not  be confused with that  of the definition  of costs.  Here the  monetary 
value  must  be  given  to  the  different  impacts  (positive  or  negative)  of  the 
project,  the  main  ones  being  outside  the  sphere  of  market  relations.  The 
principle  of  cost-benefit  analysis  is  that  each  impact  is  given  a  value  in 
proportion  to  what  those  who  benefit  from  it  are  prepared  to  pay  to  obtain 
it  (presuming  they  are  in  favour  of  the  impact)  or  to  avoid  it  (if  they  are 
against  the  impact).  For example, a passenger  is prepared  to  pay  100 Euro 
to  save  an  hour, a  resident  is  prepared  to  pay  10,000  Euro less for  a  house 
exposed  to  pollution,  etc.  If  an  advantage  or  disadvantage  cannot  be 
evaluated  in the form  of willingness  to  pay  (What  is the value  of  a  biotope? 
What  is the value of  a human  life?  What  is the  value of  knowledge  gained? 
etc.),  social  values  fixed  by  discretion  are  used.  Social  value  should  be 
taken  here  to  mean  the  value  fixed  through  an  explicit  political  process 
(though  more often  implicit). The advantages  and disadvantages  are spread 
out  over  time  and  have to  be  related  to  the  different  stages  of  the  project. 
For  this  purpose,  time  itself  is  considered  to  have  a  social  value,  i.e.  the 
discount  rate. 

Step 4.  Evaluation of the  project: Once all the preceding  steps  have  been 
completed, the evaluation as such can be started. This is when a judgement 
will  be  made  on  the  project.  It  is  at  this  stage  that  the  principle  of 
compensation  applies.  If the  project  is soundly  based, the  winners  should 

Tools for making a judgement 

be  able  to  compensate  the  losers,  which  translates  into  the  following 
principle:  the  monetary  value  of  the  disadvantages  and  costs  must  not 
exceed the monetary  value of the advantages. This  principle is obviously  of 
an  abstract  nature, for  compensation  is  never  really  paid.  Since the  results 
of  CBA  may  vary  considerably,  depending  on the  working  hypotheses,  it is 
necessary  to  complete  the  evaluation  of  the  project  with  a  sensitivity 
analysis.  This  will  be  used  to  check  the  extent  to  which  the  profitability  of 
the  project  is affected  by  variations  in essential  variables. 

Two  indicators  summarise  the final evaluation  of  the  project 

•  the  net  present  value  of  the  realisation  of  the  project,  which  is  no  more 
than  the  difference  between  the  benefits  and  the  socio-economic  costs 
of  the  realisation  of  the  project,  measured  by  taking  account  of  the 
duration of the project, and calculated as comparable values on the basis 
of  the  discount  rate; 

•  The 

the 
internal  rate  of  return  of  the  project,  which  relates  to 
determination  of  the  discount  rate for  which  the  present  net  value  of  the 
project  is  nil, and  which  makes  it  possible  to  assess  the  social  utility  of 
the  project. 

Application  to  the  Structural  Funds 

Cost-benefit  analysis  is  applied  to  the  evaluation  of  large  projects.  It  is  a 
regulatory  obligation for the  prior appraisal  of  major  European  projects. It is 
carried  out  by  the  Member  States  that  have  to  submit  the  analysis  to  the 
Commission  before  any decision  can  be taken to finance the  project.  Cost-
benefit  analysis  is  rarely  used  in other  situations. 

For  the  period  1989-1993,  the  internal  economic  rate  of  return  of  major 
projects  financed  by the  Structural  Funds  are as follows: 

Sectors 

Average  rates 

Number  of  projects 

Energy 
Water  and  environment 
Transport 
Industry 
Other  services 

Total 

13.7 
21.7 
19.2 
18.4 
18.7 

19.3 

5 
21 
63 
14 
5 

108 

Source: Guide de l'analyse des coûts et avantages des grands projets, Commission 
européenne 

Principal evaluation techniques and tools 

Strengths and limits in practice 

• The setting up of a cost­benefit analysis forces designers to ask essential 

questions such as: Who are the persons concerned? What are the 
advantages and disadvantages of the project? What part of the population 
wins and loses and how? In this sense, it helps to clarify the project. 

• Cost­benefit analysis makes it necessary to ask about the impacts of a 

project in clear terms. However, it cannot serve to analyse impacts, which 
have to be examined using other evaluation techniques. 

• It is not always possible to give a credible and legitimate value to certain 
advantages and disadvantages (e.g. the destruction of a rare biotope). 

• Cost­benefit analysis does not directly take into account the objectives of 
the policy being carried out. All the advantages and disadvantages are 
included, whether they are important or not in relation to the political 
objectives. 

• The full complexity of a large project, the number of persons concerned, 
the definition of impacts, the indirect effect, etc. make the quantification 
of advantages and costs very difficult. 

• The principle of compensation is never applied in reality, the use of this 

technique is therefore far removed from the pure theoretical constructions 
(public economy and welfare economy) which serve as its basic justification. 

Further reading... 

Chervel M. (1987), Calculs économiques publics et planification. Les 

méthodes d'évaluation de projets, Paris: Publisud, 268 p. 
A detailed presentation of all techniques for public economic 
calculation. The main aspects are addressed and a classification 
proposed. 

Commission Européenne (1997), Guide de l'analyse Coût­Avantages des 

grands projets dans le cadre de la politique régionale 
communautaire, DG XVI, Bruxelles, 90 ρ 
A guide intended for cost­benefit analysis prior to the 
implementation of major projects. Describes the main steps of the 
method. 

Diamond D; and Spence N. (1983), Regional Policy Evaluation. A 

Methodological Review and the Scottish Example, Gower 
International Library of Research and Practice, Aldershot, 170 p. 

Tools  for making  a  judgement 

A  presentation  of  the  main  evaluation  tools  taken  from  economic 
theory applied to  regional development  policy. A critical approach  to 
cost-benefit  analysis.  Highly  accessible. 

Gauthier  G  and  Thibaut  M  (eds.)  (1993), L'analyse coût-avantages,  défis  et 

controverses,  Paris: Cetai,  Economica,  526  p. 
Accessible  only  to  the  initiated.  A  compilation  of  texts  by  various 
specialists  on  the  advantages  and  limits  of  cost-benefit  analysis, 
particularly  the  epistemologica!  aspects. 

Nas  T.F.  (1996),  Cost  benefit  Analysis.  Theory  and  application,  London: 

Sage  Publications 
A  manual  that  considers  all  the  micro-economic  foundations  of 
cost-benefit  analysis  and  includes  case  studies.  Accessible  only  to 
the  initiated. 

More  in-depth: contribution  of  the  technique 
to  the  evaluation  of  projects 

Once  the  project  has  been  clearly  identified,  its  objectives  defined,  its 
feasibility  established  and  alternative  options  considered,  the  evaluation 
team  is in  a  position  to  launch the  different  phases  of  quantification  of  the 
project:  financial  analysis,  cost-benefit  analysis  and  sensitivity  analysis. 
This description of concepts is completed  by a synthetic  presentation of the 
observations  made  on  the  basis  of  an  analysis  of  the  major  projects  co-
financed  by the  Structural  Funds. 

Financial  analysis 

the 

that 

investment  project  requires 

The  financial  analysis  of 
the 
programming of this investment  and the expected  income and expenses  be 
taken  into  account  in such  a way  as to  assess  as satisfactorily  as  possible 
the annual inputs and outputs  related to the realisation and  management  of 
the  project.  This  approach  is  used  to  evaluate  the  financial  weight  of  the 
project  and  to  ensure that  there  is  no  risk  of  it  being  stopped  at  any  stage 
due to  a lack  of  liquidity.  It is on the  basis of the  results of this  analysis  that 
into  account  numerous 
the  CBA  will  be  carried  out,  by 
complementary  components  of  a  socio-economic  nature,  through  which 
the  social  impact  of the  project  can  be measured. 

taking 

To assess the net  present  financial value of  a project, the values  updated  to 
a discount  rate "r"  of the future financial  inputs (I) and outputs (O), (including 
the  residual  value  of the  investment  at the end of the  period)  are taken  into 
account: 

Principal evaluation techniques and tools 

VFAN  = Øo-Cg+^-Cg  /(1+r)+(l2-02)  /(1+r)2  + 

(On-On) /(1+r)n 

The  discount  rate  represents  the  opportunity  cost  of  capital,  i.e.  the 
economic  value  of  the  capital  invested  in  its  optimum  alternative  use. The 
rate  used  in  practice  is  often  equivalent  to  the  long-term  rate  of  return  on 
the  capital  invested  in  financial  markets.  Sometimes  the  discount  rate  is 
chosen  in  a  rather  arbitrary  way.  When  comparing  several  projects,  it  is 
worth  using the  internal financial  rate of  return  "rr",  which  is the  value of  "r" 
which  satisfies  the following  equation: 

(lo-Cg+fjVCg  /(1+r)+(l2-02)  /(1+r)2  + 

(ln-On)  /(1+r)"=0 

factors, 

In  the  case  of  public  investments,  it  often  occurs  that  the  rates  of  return 
observed  are weak  or even  negative. This  can  be explained  by  a  multitude 
including  price  distortions,  problems  with  pricing,  and 
of 
externalities.  Hence,  the  need  to  extend  the  analysis  to  criteria  other  than 
financial  variables,  in  order  to  judge  if  the  public  investment  is  opportune. 
This involves  the identification  and  monetary  evaluation  of all the  costs  and 
benefits  of the  project. 

Cost-Benefit  analysis 

In  a  similar  way  to  financial  analysis,  the  net  present  economic  value  is 
calculated  by  means  of  the following  formula: 

VEAN  =  ( B Q - C O M B ^ C ,)  /(1+r)+(B2-C2)  /(1+r)? + 

(Bn-Cn)  /(1+r)" 

in  which  B¡  represents  all  the  benefits  expected  in  period  i  and  C¡ all  the 
costs to  cover  during the  same  period. If the  net present  economic  value  is 
more than zero, then the evaluation  concludes  positively  about  the  project. 
The coefficient  "r"  represents the social discount  rate used to update all the 
costs  and  benefits  as  a function  of  the  reference  year, defined  in this  case 
as  year  0. The  analysis  of  major  European  projects  of  the  first  and  second 
generations  has  shown  discount  rates varying  from  3%  to  10%.  Generally, 
the  advised  rate  is  5%. 

Still  by  analogy  with  financial  analysis, the  rate  of  internal  economic  return 
on the  project  "rr"  is the value  of  "r"  which  satisfies  the following  equation: 

( B o - C J + ^ - C ,)  /(1+r)+(B2-C2)  /(1+r)2  + 

(Bn-Cn)  /(1+r)"=0 

Since all the costs and benefits of a project cannot systematically be translated 
into  monetary  terms, the analysis will  be complemented  by an in-depth  study 
on the  non-monetary  aspects  related to the  realisation of the  project. 

Tools  for making  a  judgement 

Analysis  of  sensitivity  and  risk 

Depending  on  the  case,  the  results  of  the  cost-benefit  analysis  can  be 
extremely  sensitive to  a change  in the working  hypotheses. The analysis  of 
sensitivity  consists  of  varying  the  value  of  several  key  variables,  so  as  to 
assess the  impact  of  changes  in hypotheses  on the  project's  rate of  return. 
If the modification  of a key variable results in a significant  change  in the  rate 
of  return, it is essential to  improve the  reliability  of the  measure obtained  for 
this variable. 

Major  public  projects  are no more exempt from  uncertainty  than are  private 
investments.  Since  cost-benefit  analysis  is  based  on  assumptions  about 
the  expected  impacts  of  the  project,  it  is essential  to  assess the  degree  of 
risk  involved  in  the  project.  Risk  analysis  will  be  carried  out  by  linking 
probabilities  to  alternative  values  of  crucial  variables,  by  determining  the 
economic  rates  of  return associated  with them, and then  by calculating  the 
distribution  of  probability  of these rates of return. This permits an evaluation 
of  the  probability  of  the  project  having  an  economic  rate  of  return  that  is 
smaller  than  the  minimum  rate  of  return  required.  Risk  analysis  enables 
decision-makers to make their final decision with full knowledge of the risks 
facing  the  project. 

Rafes  of  return  of  European  projects 

Sectors 

Mid­
term 

Number 
of 
projects 
(*) 

Average 
internal 
financial 
rate 
of  return 

Number 
of 
projects 
(*) 

Average 
internal 
economic 
rate 
of  return 

Number 
of 
projects 
(*) 

Energy 

Water  and 
environment 

Transport 

Industry 

Other 
services 

Total 

24.7 

29.1 

26.6 

8.8 

14.2 

20.1 

9 

47 

127 

96 

10 

289 

7.0 

-0.1 

6.5 

19.0 

4.2 

11.5 

6 

15 

55 

68 

5 

149 

12.9 

15.8 

17.1 

18.4 

16.3 

16.8 

6 

51 

152 

14 

10 

233 

(')  Projects  for  which  data  is  available 

Source:  Guide  de  l'analyse  des  coûts  et avantages  des  grands  projets,  Commission 
des  communautés  européennes 

Principal  evaluation  techniques and tools 

For the  programming  periods  1989-1993  and  1994-1999  corresponding  to 
the  first  and  second  generations  of  programmes,  resulting  from  the  reform 
of  the  Structural  Funds,  the  mid-term  horizon  (economic  lifespan  of  the 
project), the average internal financial  rate of return and the average  internal 
economic  rate  of  return  of  a  set  of  400  major  projects  co-financed  by  the 
Structural  Funds  were analysed. 

This  highly  instructive  synthesis  enables the  most  important  aspects  of  the 
contribution  of  CBA  to the  analysis  of  large  projects  to  be  pinpointed. 

Firstly, the average  lifespan of  projects varies considerably from one field to 
the  next.  With  an  average  of  20  years,  the  lifespan  varies  from  9  years  for 
industrial  infrastructure  to  30  years  for  environmental  infrastructure.  An 
assessment  of the  lifespan of a project  is important  if the  potential  mid- and 
investments  are  to  be  correctly 
long-term 
estimated. 

impacts  of  the  planned 

Secondly,  the  estimation  of  the  internal  financial  rate  of  return  is  used  to 
define  the  financial  needs  of  the  project  throughout  its  life.  This  financial 
analysis  is also vital for the  programming  of  investments. Whereas  financial 
returns  are  high  for  productive  infrastructures,  which  usually  generate 
substantial  income  (rents  or  selling  prices  of  property  and  facilities,  selling 
price  of  services,  etc.),  and  are generally  around  19%, the  financial  rate  of 
return  of  environmental  facilities,  which  are  very  rarely  priced  at  market 
prices,  is very  low  (-0.1 %  on  average).  For  projects  characterised  by  a  low 
financial  rate  of  return,  financial  programming  is  all  the  more  important 
when  the  question  of  the  viability  of  the  project  and  of  its  real  impact  on 
public  finances  is asked. 

Thirdly, the assessment  of costs and socio-economic  benefits of the  project 
is used to evaluate  its real economic  value and its contribution  to  economic 
and social  welfare  - the  ultimate  aim of  a CBA. An  initial  observation  is that 
the  internal  economic  rate  of  return  differs  significantly  from  the  internal 
financial  rate  of  return.  For  projects  co-financed  by  the  Structural  Funds, 
the  average  internal  economic  rate  of  return  varies  between  12.9%  in  the 
energy  sector to  18.4% for the  industrial  sector.  These figures  highlight  the 
advantages  of  the  cost-benefit  approach  which  enables  the  analysis  to  be 
extended  beyond  the  strict  framework  of  financial  analysis  by  taking 
account,  in  particular,  of  the  externalities  that  the  project  will  generate. 

IV 
Recommendations for selecting 
and assembling tools 

(1)  prospective 

The most  suitable  tools  must  be identified  for  each specific  evaluation.  Part 
four 
IV  provides  assistance  for  designers  of  evaluations,  by  describing 
typical  situations: 
examination  of  a  multi-sectoral 
programme:  (2) retrospective  examination  of  a  multi-sectoral  programme; 
(3) more  in-depth 
in-depth 
treatment  of  a  retrospective  analysis.  In  each  of  these  four  situations,  the 
appropriateness  of  tools presented  in  this  Volume is reviewed.  This Volume 
concludes  with  recommendations 
for  the  assembly  of  several  tools  within 
the same  evaluation. 

treatment  of  a  prospective  analysis;  (4) more 

Guide for identifying the most appropriate tools 

The identification  of  the most  appropriate  tools  must  be done on a case  by 
case basis, in relation to the nature of the object to  be evaluated and, above 
all, in relation to the questions asked by the commissioner.  It is therefore out 
of  the question  to  provide  a general  "blueprint".  However, the  designers  of 
evaluations  can  be assisted  by the description  of  several typical  situations. 
First,  the  object 
to  be  evaluated  may  be  an  entire  multi-sectoral 
programme  or, by contrast,  a particular  evaluative  question that  needs to 
be considered  in greater  depth. 

From the point  of view of the choice of tools, it is also  important to take  into 
account  the  time  dimension  of  the  question  posed.  Is  the  analysis  to  be 
prospective  or  retrospective?  An  analysis  is  retrospective  if  it  can  be 
answered on the basis of observations of effects  in the field. This is the case 
of ex post  evaluation. It is prospective  in the opposite  case, particularly  for 
ex  ante  evaluation.  Intermediate  evaluation  may  comprise  a  mixture  of 
prospective  analysis (e.g. what  is the expected  impact  of projects that  have 
already  been  selected?)  and  retrospective  analysis  (e.g. what  are  the  first 
results  obtained  in the field?). The combination  of  these  distinctions  allows 
the definition  of the  evaluation  situations  presented  in  Box  29. 

Box 29  - Typical  situations 

Overall  evaluation  of 
a  multi-sectoral 
programme 

More  in-depth  study 
of an  evaluative 
question 

prospective  analysis 

retrospective  analysis 

Situation 1 

Situation  2 

for  example: 
ex  ante  or  intermediate 
evaluation  of  a  regional 
programme  or a CSF 

for  example: 
intermediate  or ex  post 
evaluation  of  a  regional 
programme  or a  CSF 

Situation  3 

Situation  4 

for  example: 
ex  ante  or  intermediate 
evaluation  of  a  measure 
or a sectoral  programme 

for  example: 
intermediate  or  ex  post 
evaluation  of a  measure 
or sectoral  programme 

The  guide  presented  below  proposes  an  aid  for  the  selection  of  tools  in 
these  four  situations.  It  is  limited  to  those  tools  figuring  in  an  information 

Principal  evaluation  techniques  and tools 

sheet  in  this  Volume.  The  guide's  approach  could  easily  be  extended  to 
other  tools.  It  does  not  recommend  the  systematic  application  of  any 
specific  tool  to a particular situation, but  proposes several tools  from  which 
only  those  (or  the  one)  best  suited  to  the  situation  in  question  should  be 
chosen. 

Tools for  the  overall  evaluation  of  a  programme,  prospective  analy­
sis 

Box 30 -  Overall  evaluation  of  a programme,  prospective  analysis 

là 

SWOT  Analysis 

Impact  mapping 

Colour  vote 

O 
<> 

-Ó-

Individual  interview 

<> 

-o-

-Ó-

Focus  group 

Case  study 

Geographical  information  system 

Input-output  model 

Macro-economic  model 

Expert  panel 

Multicriteria  analysis 

-o  o o 

-ó-
O 

Tools 

(  J  Appropriate  O  Application subject  to certain  conditions 

In  this  situation,  the  main  lines  of  the  programme  have  already  been 
proposed. The evaluation  must  help to  ensure that the  planned  programme 
has  objectives  that  are  soundly  justified,  clear,  graded,  and  logically 
connected  to  the  planned  actions.  If necessary, the  evaluation  may  include 
field  observation  to  check  the  nature  and  importance  of  needs.  It  may  also 

Selecting and assembling tools 

include a forecast  of  effects,  particularly  for  helping  to  quantify  objectives. 
An evaluation  of this  type  warrants  the  use of  fools,  if adequate funds  and 
time are available. 

Tools for the  overall  evaluation  of  a  programme, 
retrospective  analysis 

Box 31 - Overall evaluation  of a programme,  retrospective  analysis 

li 

SWOT  Analysis 

<y-

Impact  mapping 

Colour  vote 

Individual  interview 

Focus  group 

Case  study 

-<>-

<^ 

Geographical  information  system 

—Ò-

Shift-share  analysis 

Input-output  model 

Macro-economic  model 

Expert  panel 

Multicriteria  analysis 

Benchmarking 

-o-
<^-o-
o-
-Ò-

Tools 

( 

)  Appropriate 

O  Application subject to certain conditions 

In this situation, the evaluation  must observe, analyse and judge the  effects 
produced  by  the  programme  In the  field.  These  effects  are  numerous  and 
are  the  result  of  multiple  processes  that  differ,  depending  on  the 
components  of the  programme.  Moreover, the evaluation  may  have to  take 
into  account  the  differences  in the  partners'  points  of  view.  This  situation 
presents constraints, which limit, to an important extent, the choice of tools. 

Principal evaluation  techniques and tools 

Tools  for  in-depth  evaluation,  prospective  analysis 

intended 

In this  situation, an aspect  or  part  of  a complex  programme  is evaluated  in 
depth. This  corresponds  to  the  case  of  a simple  intervention, consisting  of 
similar  actions, 
for  a  relatively  homogeneous  public  and 
possessing  a primary objective. The prospective  analyses that are generally 
performed  in this  situation  examine  the  feasibility  and  the  relevance  of  the 
planned  intervention.  Consequently,  the  evaluation  must  help  to  explain 
how  this  intervention  will  produce  these  effects,  what  the  estimated  extent 
of  the  effects  will  be and to  what  extent  the  planned  intervention  responds 
to  a real need. 

Box 32  -  Tools for  in-depth  evaluation, prospective  analysis 

J 
55 

O 
<> 

<> 

Logical  framework 

Metaplan® 

Colour  vote 

Delphi  survey 

Individual  interview 

Focus  group 

o-
-o—o-

Cost-effectiveness  analysis 

-Ó-

<y-

Benchmarking 

Cost-benefit  analysis 

Multicriteria  analysis 

Expert  panel 

-Cr-

-o— 

Tools 

Cj  Appropriate 

O  Application subject  to certain conditions 

Tools for  in-depth  evaluation,  retrospective  analysis 

In  this  situation,  the  evaluation  observes,  analyses  and  judges  the  effects 
obtained  in the field.  There  are few  of  these  effects  and they  are  produced 

Selecting and assembling tools 

by  processes  that  can  be  schematised.  It  is  therefore  possible  to  study 
them 
in-depth.  Time  may  also  be  devoted  to  looking  for  suitable 
benchmarks  for  judging  whether  the  effects  obtained  are sufficient  or  not. 
Most  classical  evaluation  tools are applicable  to this situation. 

Box 33 -  In-depth evaluation, retrospective  analysis 

fiS 

Logical  framework 

Metaplan® 

Colour  vote 

Questionnaire  survey 

Focus  group 

Case  Study 

Ethnographic  observation 

Factor  analysis 

Comparison  group 

Regression  analysis 

Cost-effectiveness  analysis 

Benchmarking 

Expert  panel 

-o-
<> 
<y-

-o-

-o—o 
-o-

o—o 

Tools 

\^)  Appropriate 

O  Application subject to certain conditions 

Recommendations  for assembling  several  tools 

Procedure  for  assembling  tools 

The  choice  of  tools  and  their  assembly  is  generally  a  process  of  trial  and 
error. At first, the  nature  of the evaluation  (a multi-sectoral  programme  or a 
less  complicated  intervention)  and  that  of  the  analysis  to  be  performed 
(prospective  or  retrospective)  suggest 
that  certain  tools  might  be 
appropriate. The preceding  guide can  help to  identify  these tools. 

At this stage, a temporary method is constructed  and its quality checked by 
means of the following  questions: 

•  Are the  eight  standard  functions  of  an  evaluation  dealt  with?  Are  some 

functions  irrelevant?  Have some functions  been  forgotten? 

•  Which  tools  have  been  chosen?  What  functions  do  they  fulfil?  Are  they 
well suited to the functions to  be dealt  with?  Does their  use comply  with 
the established  procedures? 

•  Is the  method  as a whole sound  and  homogeneous? 

The latter question  is perhaps the most important, for an evaluation  method 
can  be compared  to  a chain. Each function  is a link  in the  chain  and  each 
link  is  stronger  when  treated  with  an  appropriate  tool.  The  metaphor  of  a 
"technical  chain"  highlights  the  interdependence  of  the  different  tools 
assembled  within  an evaluation  method. 

A  link  is strong  if the technical  underpinning  of the corresponding  function 
is sufficient  in relation to the  risk of the evaluation  results  being challenged. 
A  link  does  not  necessarily  have  to  be  "tool(s)-based"  to  be  strong. 
However,  if  it  is, its strength  depends  on the appropriateness  of the tool  in 
relation to the question  posed  and the situation. 

The entire technical  chain  is as strong  as  its  weakest  link.  It  is therefore  of 
little  use to  employ  a sophisticated  and expensive tool to estimate  effects, 
if the  data  to  which  the tool  is applied  has  been collected  in a  contestable 
manner. The soundness  of  an evaluation  method  is only  partially  related  to 
the  quality  of  the  tools  used.  It  is  less  the  highly  technical  way  in  which 
certain  functions  are  dealt  with,  than  the  fact  that  all  the  functions  are 
treated  with  the  same  level  of  technical  underpinning,  which  makes  an 
evaluation  good.  Thus,  when  a  temporary  evaluation  method  has  been 
constructed,  it  is essential first to  check  whether  it  has any weak  links. 

Principal evaluation techniques and tools 

Box 34  - The technical  chain  is as strong  as its weakest  link 

Clarifying/grading  effects  and  criteria 

Choosing  observation  Instruments 

Defining the observation  field 

Collecting  data 

Comparing  data 

Estimating  effects 

Judging  in terms  of the  different  criteria 

Formulating  a synthetic  judgement 

[  ) -— 

-  Weak link 

Tools 

fj  Strong link 

The  evaluation  method  is  then  constructed  by  successive  improvements, 
until the  following  requirements  are  met: 

•  all the  relevant functions  have  been  dealt  with 

•  the  selected  tool(s)  are suited to the situation  and the function  they  fulfil 

•  the  method  constructed  is homogeneous  and  has  no weak  points 

•  the method  chosen  is compatible with the cost  and schedule  constraints 

of the  evaluation. 

It  is  not  possible  to  seek  maximum  robustness,  nor  to  choose  numerous 
and  sophisticated  tools,  If  the  time  and  budgetary  constraints  are  too 
severe. The realism of a method demands that  its strength, but also its cost, 
are consistent with the importance of the evaluation for its clients. When the 
method  proposed  does  not  meet  this  demand  for  realism,  it  must  be 
adapted  by  eliminating  certain  tools  or  replacing  them  by  others  that  are 
easier to  implement. 

Selecting and assembling tools 

A  problem  that  is  often  forgotten  is that  a  method  is  not  constructed  for 
each  evaluation  but  for  each  question  within  an  evaluation.  Assuming  a 
single  evaluation contains the following two  questions: 

A.  Was  the  impact  of  the  programme  sufficient  in  terms  of  reducing 

unemployment  in the  region? 

B.  Which  measures  of  the  programme  are  the  most  efficient  for  creating 

jobs, and  why? 

Question A must  be treated by estimating the impact  of all the measures, if 
necessary  by  means  of  an  econometric  model.  On  the  other  hand,  for 
question  Β  it  would  be  better  to  use a  method  based  on  observation  and 
analysis  of  particularly  successful  projects  in  terms  of  job  creation  (case 
study). In some cases a single method  may answer two or more questions, 
but  in general  the  answers  will  not  all  be  equally  robust.  It could  therefore 
be said that the cost of an evaluation increases  in proportion to the  number 
of questions  in the  terms  of  reference  (and to which  a real answer is given). 

Three  examples  of  assembling  tools 

The following  three  propositions  are  based  on  real examples  presented  in 
the  preceding  chapters. 

The  first  example  is  the  overall  evaluation  of  the  Hainaut  regional 
development  programme,  presented  in  the  information  sheet  relating  to 
multicriteria analysis (page 113) and in Volume 4 of this Collection,  devoted 
to  innovative  techniques  developed  in  the  framework  of  the  MEANS 
programme.  This  intermediate  evaluation  was  intended  to  compare  the 
effectiveness  of  the  main  measures  of  the  programme.  It  ended  with  a 
multi-criteria  analysis  that  was  used  to  make  a  synthetic  judgement.  This 
judgement  was  based  on  a  qualitative  estimation  of  the  effects  of  each 
measure. 

In  order  to  strengthen  the  estimation  of  impacts,  the  method  could  be 
developed  in the following  way: 

•  clarification  of the  effects to  be evaluated  by  means of  an impact  map, 

•  collection  of  indicators  and  available  secondary  data  on  each  measure 

evaluated, 

•  organisation  of  one  or  more  focus  groups  for  each  of  the  evaluated 

measures; qualitative  estimation  of  impacts, 

Principal evaluation techniques and tools 

•  comparison  between  measures  and  synthesised  judgements,  by  means 

of  a multicriteria  analysis. 

Box 35  -  Example  of the combination  of tools 

for the  intermediate  evaluation  of an entire  programme 

Clarifying and grading the 
effects to be evaluated, defining 
criteria 

Choosing  observation 
instruments 
Defining the observation field 

METHOD 

Impact  mapping 

\ 

Monitoring indicators 

Collecting data 
Comparing data 

Secondary  data 

Tools 

Estimating  effects 

Focus groups 

Judging in terms of the different 
criteria 
Formulating a synthetic 
judgements 

Multicriteria analysis 

The  second  example  is  based  on  the  evaluation  of  job  creation  in Wales, 
described  on  page  178  (see  the  information  sheet  on  Cost-effectiveness 
analysis) and in Box 3. This ex post evaluation aimed at making an  in-depth 
analysis  of  an  impact.  It  included  a  questionnaire  survey  on  assisted 
businesses,  used to  estimate  deadweight  and  displacement  effects,  and a 
cost-effectiveness  analysis, used to judge the efficiency  of the  programme, 
compared  to a Scottish  programme. The weak  links  in this  evaluation  were 
probably  the  estimation  of  deadweight  effects  and  the  comparison  of 
efficiency. 

To  enhance  the  evaluation  on  this  point,  the  method  could  have  been 
improved  in the following  way: 

•  definition  of  the  field  of  observation  in  three  groups:  Welsh  businesses 
assisted  by  the  programme,  non-Welsh  businesses  assisted  by  other 
programmes  considered  to  be  efficient,  and  Welsh  businesses  not 
affected  by the  programme. 

Selecting and assembling tools 

•  collection  of data  by means  of  a questionnaire  survey 

•  estimation  of  effects  by  means  of  comparison  groups 

•  cost-effectiveness  analysis  and  judgement 

in  relation  to  other 

successful  programmes. 

This  method  would  of  course  have  been far  more expensive. 

Box  36  -  Example  of  the  assemblage  of  tools 

for  an  in-depth  ex  post  evaluation 

Clarifying  and  grading  the 
effects to be evaluated, defining 
criteria 

Choosing  observation 
instruments 

Defining the  observation  field 

M E T H OD 

Indicator  of 
job  creation 

Recipient  and 
non-recipient  firms 
inside  and  outside 
the  eligible  area 

Collecting  data 
Comparing  data 

Questionnaire  M 
survey 

ΤηηΙς 

Estimating  effects 

Comparative  * 

analysis 

/ 
. 
/ 

Formulating  a  synthetic 
judgements 

Cost-effectiveness 
analysis 

v 

> 

impact  of 

the  probable 

The third  example  is based on the  ex ante evaluation  of  European  support 
to  the  Republic  of  Ireland,  presented  in  Box  13. This  evaluation  aimed  at 
estimating 
the  assistance  on  economic 
development,  particularly  in  terms  of  the  per  capita  GDP. The  evaluation 
comprised  a first  series  of  micro-economic  estimations  (impact  on  public 
spending,  on  productivity  and  on  production  costs).  These  estimations 
were then introduced  into a macro-economic  model. Estimations  of  micro-
economic  impacts  on  productivity  and production  costs  were probably  the 
weak  links  of  this  evaluation.  Moreover,  the  evaluation  stopped  at  the 
estimation  of  macro-economic  effects  and  did  not formally  go  so far  as  to 
judge  whether  the  effects  were sufficient  or  insufficient. 

Principal evaluation techniques and tools 

To  improve  the  homogeneity  of  the  evaluation  method,  a  little  less  money 
could have been spent on the econometric  model, and the weak links could 
have been  strengthened  in the following  way: 

•  clarifying  and  grading  the  probable  micro-economic  impacts  of  the 

different  measures, if  necessary  by  means  of an impact  matrix; 

•  requesting  an  expert  panel  to  estimate  the  main  micro-economic 

impacts; 

•  estimating the economic consequences  at the national level, by means of 

a macro-economic  model as was done  in practice; 

•  forming  an  expert  panel  and  asking  it  to  decide  whether  the  probable 

impacts  were sufficient  or  insufficient. 

At  little  extra  cost,  this  method  would  have  been  more  credible  and 
conclusive. 

Box 37  -  Example  of the assemblage  of tools 

for the  ex ante evaluation  of an entire  programme 

METHOD 

Clarifying  and grading the 
effects to be evaluated, defining 
criteria 

Identification of 
major  macro-economic 
impacts 

Estimating  effects 

Expert panel + 
Econometric  model 

Tools 

Formulating a synthetic 
judgement 

Expert panel 

Conclusion 

The use of tools has numerous  advantages in terms of providing  assurances 
about  the  results  to  be  obtained,  of  certainty  about  the  contents  of  the 
evaluation,  of  controlling  costs,  of  comparability  of  results  and  of  quality 
control.  A  more  technical  evaluation  is  often  more  straightforward 
to 
commission,  easier  to  control,  more  credible  in  the  eyes of  the readers  of 
the report  and more predictable  for the commissioner.  It can be  understood 
that  all these advantages make evaluation  tools  very  attractive. 

technical  complexity 

Since 
the  value  of  tools  used  in 
is  attractive, 
evaluations is often  exaggerated. An  entire  work of evaluation is  sometimes 
summarised  by the name of the tool employed  (e.g. cost-benefit  evaluation, 
macro-economic 
takes  on  an  emblematic 
dimension. 

evaluation),  which 

thus 

It is however  dangerous  to overstate  the value of a tool  while forgetting  that 
it  most  often  fulfils  a  single  evaluation  function,  is  adapted  to  a  certain 
context,  presents  constraints  specific  to  the  availability  of  data,  and  has a 
certain  cost  and  duration  of  use that  cannot  be  compressed.  To choose  a 
tool  without  taking  all  these  dimensions  into  account  can  have  serious 
consequences:  time overruns, absence  of conclusions,  conclusions  lacking 
in credibility,  irrelevant  conclusions,  etc. 

This  Volume  should  therefore  not  be  considered  as  an  advertisement  for 
highly  technical  evaluations.  Its  ambition  is  rather  to  adapt  the  level  of 
technical  complexity  to the strategic  importance  of the questions posed  and 
the  budget  and  time  scale  allocated  for  the  evaluation.  It  should  thus  be 
considered  as assisting  the mastery of the technical  aspects  of evaluations. 

Index 

Index 

analysis  of  causality  27 
approach  18 

Β 

benchmarking  115,  193,  201 
Beutel  model  98 

casestudy  59,  73, 8 1, 132,  151, 167 
colour  vote  39,  55,  115,  135,  193 
comparison  group  8 1, 132, 167, 179, 

229 

concept  mapping  of  impacts  39,  47, 

135,  227 

cost-benefit  analysis  193,  207 
cost-effectiveness  analysis 

19,  23, 

193,  195,229 

criterion  25,  28 

deductive  approach  27 
Delphi  survey  167,  175,  193 
descriptor  26 
design  18 

factor  analysis  8 1,  131,  167,  169 
field  26 
focus  group  59,  67,  227 
formative  evaluation  29 
function  18,  24,  225 

generic  tool  24 
geographical  Information  system  36, 

81,83 

group  interview  151,  167 

Η 

HERMIN  model  29,  112 

I 

impact  matrix  39,  135,  230 
in-depth  evaluation  30, 219, 222,  228 
indicator  26 
individual  interview  59,  6 1,  132,  151 
inductive  approach  27 
input-output  model  8 1,  95 
intermediate  evaluation  30,  219,  227 

logical  framework  40,  135,  137 

economic  technique  12 
ethnographic  observation  151, 161, 

M 

167 

ethnomethodology  162 
evaluative  question  25 
ex  ante  evaluation  30,  219,229 
ex  post  evaluation  30,  219,  228 
experimental  technique  12 
expert  panel  115,  117,  131, 193,  230 

macro-economic  model  29,  37,  8 1, 

103,  230 

METAPLAN®  20,  135,  145 
method  18,  225 
methodology  19 
multicriteria  analysis  115,  123,  193, 

228 

Principal evaluation techniques and tools 

N 

naturalistic  approach  12 
net  effect  27 

recapitulative  evaluation  29 
regression analysis  20,  8 1,  167,  187 

overall  evaluation  30,  219,  220, 221, 

227,  229 

secondary  data  27 
shift-share  analysis  8 1,  89 
specialised  tool  24 
structuring  25 
SWOT  analysis  2 1,  39,  4 1,  115 

pragmatic  approach  12 
primary  data  26 
procedure  18 

qualitative  analysis  27 
quantitative  analysis  27 
QUEST  model  111 
questionnaire  survey  59,  131,  151, 

target  public  31 
technical  chain  225 
technique  18 
thematic  evaluation  31 
tool  18 

V 

153,229 

variance  analysis  167 

Acknowledgements 

The  preparation  of  this  Volume  took  place  over  several  years  from  1995  to  1998. 
Preparatory  works  included the following  steps  : 

•  Five case studies were carried out on the basis of evaluations selected for their use 
of  well-known  evaluation  techniques.  Five  European  scholars  took  part  in  these 
studies  which  allowed  procedures  for the use of the evaluation tools  presented  in 
this  Volume to  be determined. 

•  Nearly  300  desks  officers  from  the  European  Commission  attended  training 
seminars  which  included  a  session  specially  devoted  to  evaluation  tools.  The 
contents  of these training seminars  evolved  over time to  meet  participants'needs. 
This Volume  includes  the results  of these  interactions. 

•  Several  experts  participated  in  the  writing  and  verification  of  the  information 

sheets. 

The  production  of  this  Volume  was  coordinated  by  Miguel  Benito  Alonso  (D.G. 
XVI.G.2  :  Coordination  of  Evaluation)  and  involved  several  Directorate-Generals, 
notably  D.G.  II, V,  VI,  XIV  and  XVI  and  in  particular  Philippe  Goybet,  Anastassios 
Bougas,  Martin  Hallet, Andrea  Mairate and Julius  Op de Beke. 

This  Volume  has  been  written  by  the  Centre  for  European  Evaluation  Expertise, 
notably  with  the  participation  of  Kristian  Colletis-Wahl,  Olaf  Rieper  and  Kevin 
Williams.  It was translated  into  English by  Liz  Libbrecht. 

The  authors  wish  to  thank  all those  who  contributed,  either  directly  or  indirectly,  to 
the  production  of  this  Volume,  and  in  particular  the following  experts:  Henri  Capron 
(Université  Libre  de  Bruxelles,  Belgium),  Dr.  Basil  Cracknell  (consultant,  UK),  Prof. 
Massimo  Florio  (CSIL,  Milan),  Christophe  Kellerhals  (Evaluanda,  Genève,),  Steve 
Martin  (Warwick  Business  School,  UK),  Thanos  Skouras  (Athens  University  of 
Economics  and  Business),  Evert Vedung (Uppsala  Universitet). 

European  Commission 

Evaluating  socio-economic  programmes 
Principal  evaluation  techniques  and  tools 

Luxembourg: Office for  Official  Publications  of the  European Communities,  1999 

1999 —  237  pp. —  14.8 χ 21  cm 

ISBN  92-828-6626-2 

Venta  ·  Salg  ·  Verkauf  ·  Πωλήσεις  ·  Sales ■ Vente · Vendita · Verkoop * Venda · Myynti · Försäljning 

BELGIQUE­BELGlE 

ÖSTERREICH 

Jean Do Lan noy 
Avenue du Roi 202/Koningslaan 202 
B­1190 Bru ielle s. Brussel 
Tel (32­2) 538 43 08 
Fax (32­2) 538 08 41 
Email: iean.de lanrtoyOinloboard.be 
URL. hrtpJ/wwwjean­de­iannoy be 

La librairie européenne/ 
De Europese Boekhandel 
Rue de la La 244/Welstraai 244 
B­1040 Bruì eil es/Brussel 
Tel (32­2) 295 26 39 
Fax (32­2) 735 08 60 
E­mail mailOlibeurop be 
URL http ' www hbeurop be 

Moniteur belge/Bel g lach Staatsblad 
Rue de Louvam 40­42 Le uve π so wog 40­42 
B­1000 Bruiellos,Brussel 
Tel (32­2)552 22 ti 
Fax (32­2) 511 01 B4 

DANMARK 

J. H. Schultz Information A/S 
Herstedvang 10­12 
DK­2620 Albertslund 
Tlf (45)43 63 23 00 
Fai (45) 43 63 19 69 
E­mail schultzOschuttz dk 
URL http //www.schuitz.dk 

CEU TSCHLArJD ^ _ ^_ 

Bundeaanzelger Vertag GmbH 
V e ri no b saoi e> lung 
Amsterdamer Straße 192 
D­50735 Köln 
Tel (49-221)97 66 80 
Fax (49­221)97 66 82 78 
E­Mail vertrieb θ bunde sa nzei ger de 
URL lmp_//www bundesanzeiger de 

ΕΛΛΑΔΑ/GREECE 

G. C. Eleftheroudakis SA 
I n le mal lonal Book store 
PanepistimKXj 17 
GR­10564 Alhina 
Tel (30­1)331 41  ΒΟ.Ί/2/3/4'5 
Fax  (30-1)  323 98  21 
E-mail  elebooksOnelor  gr 

ESPANA 

Boletín Oficial del Estado 
Trafalgar. 27 
E 28071 Maand 
Tel ( 3 4 ) 9 1 5 1 8 21 11 (Libros). 
913 B4 t7 15(Suscnp) 
Fax (34) 915 38 21 21 (Libros), 
913 84 17 14(SuSCnp) 
E­mail dientes β com boe es 
URL: http//www boe es 
Mundi Prensa Libros. SA 
Castellò. 37 
E ­28001 Madrid 
Tel (34)914 36 37 00 
Fax (34) 915 75 39 98 
E­mail: libreria βmuridrpronsa es 
URL http//www mu ndipr en sa com 

FRANCE 

Journal officiel 
Service des publications des CE 
26, rue Desaix 
F­75727 Paris Cedex 15 
Tel (33) 140 58 77 31 
Fax (33) 140 58 77 00 
URL http /fwww |oumal­orficiel gouv fr 

IRELAND 

Government Supplies Agency 
Pubi ca lion s Section 
4­5 Harcourt Road 
Dublin 2 
Tel (353­1)661 31 11 
Fax (353­1) 475 27 60 

ITALIA 

Licosa SpA 
Via Duca di Calabria. 1/1 
Casella postale 552 
1­50125 Firenze 
Tel (39) 055 64 83 1 
Fax (39) 055 64 12 57 
E­mail lieosaOtlbcc il 
URL hrtp /fwww ftbcc lUicosa 

LUXEMBOURG 

Messageries du livre SARL 

5. rue Raiffeisen 
L­2411 Luxembourg 
Tfri (352)40 10 20 
Fax(352)49 06 61 
E­mail ma.lOmdl lu 
URL http //www mdl lu 

NEPER LAN0 

SDU Servicecentrum Uitgevers 
Chnstorfel Plantons!raat 2 
Postbus 20014 
2500 EA Den Haag 
Tel (31­70) 378 98 B0 
Fax (31­70) 378 97 83 
E­mail sduOsdunl 
URL http /,­www sdu nl 

M a n i s c he Verlage­ und 
Universitatsbuchhandlung GmbH 
Kohl markt 16 
A­1014Wien 
Tel (43­1)53 16 11 00 
Fax (43­1) 53 16 11 67 
E­Mail bestellen O manz coat 
URL http//www manz at/mdex htm 

PORTUGAL 

Distribuidora de Livros Bertrand L d .' 
Grupo Bertrand. SA 
Rua das Torras dos Vales. 4­A 
Apartado 60037 
P­2700 Amadora 
Tel. [351­1)495 90 50 
Fax (351­1)496 02 55 

Imprensa Nacional­Casa da Moeda, EP 
Rua Marqués Sá da Bandella. 16­A 
Ρ­1050 Lisboa Codex 
Tel (351­1)353 03 99 
Fax (351­1) 353 02 94 
E­maJ defíneme mail.teiepac.pt 
URL: httpi/www incm.pl 

SUOMI FIMLAND 

Akateeminen Kirjakauppa/ 
Akademiska Bokhandeln 
Keskuskatu 1 /Centralgatan 1 
PL/PB 128 
FIN­OOlOt HelsinkiHelsingtors 
P/tfn (358­9) 121 44 18 
F/lax (358­9) 121 44 35 
Sähköposti akatilaus β akateeminen com 
URL: http^/www akateeminen com 

SVER ι .1 __ 
BTJ AB 
Traktorvägen 11 
S­221 82 Lund 
TIn (46­46) 18 00 00 
Fax (46­46) 30 79 47 
E­post btteu­pubObt) se 
URL htlp.//www bl| so 

UNITED KINGDOM 

The Stationery Office Ltd 
International Sales Agency 
51 Nine Elms Lane 
London SW8 5DR 
Tel. (44­171)873 90 90 
Fax (44­171) 873 84 63 
E­mail, ipa enquiñes θ tneso.co.uk 
URL http.//www lhe­stallone ry­o ff ice co uk 

ISLAND 

Bokabud Larusar Blondel 
SkóiavôrduslKj 2 
IS­IOt Reykiavik 
Tel (354)551 56 50 
Fax (354) 552 55 60 

NORGE 

S weis Norge AS 
Ostenjoveien 18 
Boks 6512 Etterstad 
N<06O6Oslo 
Tel (47­22)97 45 00 
Fax (47­22)97 45 45 

SCHWEIZ/SU ISSE.'SVIZZERA 

Euro Info Center Schweiz 
c/o OSEC 
Stampi e nbachsttaße 85 
PF 492 
CH­B03S Zurich 
Tel (41­1)365 53 15 
Fax (41­1)365 54 1t 
E­mail eicsOosoc eh 
URL http //www osee ch/eics 

BÀLGARUA 

Europres a Euromedia Lid 
59. Orvd Vitosha 
BG­1000 Sofia 
Tel (359 2)960 37 66 
Fax (359­2) 980 42 30 
E­mail Milena O mboxcit bg 

CESKA HEPUBLIKA 

ÜSIS 
NIS­prodejna 
H a velkova 22 
CZ­130 0 0 P r a h a3 
Tel (420­2)24 23 14 86 
Fd. (420­2)2­1 23 II 14 
E­mail nkpospOdecms cz 
URL htlpy/usiscrcz 

CYPRUS 

Cyprus Chamber of Commerce and Industry 
PO Box 1455 
CY­1509 Nicosia 
Tel (357-2)66 95 00 
Fax (357­2)66 10 44 
Email: demetrapOcco org cy 

EESTI 

Eesti Kau band u s ­ TM st us k oda (Estonian 
Chamber of Commerce and Industry) 
Toom­Kooli 17 
EE­0001 Tallinn 
Tel (372)646 02 44 
Fax (372) B46 02 45 
E­mail einfoOkoda ee 
URL http//www koda ee 

HRVATSKA 

Mediatrade Ltd 
Pavia H alza 1 
HR­10000 Zagreb 
Tet (385­1)481 94 1 
Fax (385­1) 481 94 1 

MAGYARORSZAG 

Euro Into Service 
Europa Haz 
Marnitszirjöt 
PO Box 475 
H­1396 Budapest 62 
Tel (36­1)350 80 25 
Fax (36­11350 90 32 
E­mail ouromfoOmail maiav.hu 
URL. http//www euromlo.hu/index.htm 

MALTA 

Miller Distributors Ltd 
Malta International Airport 
PO Box 25 
Luqa LOA 05 
Tel ( 3 5 6 ( 66 44 88 
Fax (356) 67 67 99 
Email: gwirthöusa net 

POLSKA 

A rt Polona 
Krakowskie Przedmiescie 7 
Skr pocztowa i 0 0i 
PL ­00­950 Warszawa 
Tel (48­22)826 12 01 
Fax (48-22)826 62 40 
E­mail ars_poiebovy hsn.com pi 

Euromedia 
Str. G r al Berihelot Nr 4i 
RO­70749 Bucuresti 
Tel (40­1)315 44 03 
Fax (40­1) 314 22 86 

ROSSIYA 

CCEC 
60­leliya Oktyabrya Av 9 
1 17312 Moscow 
Tel (7­095) 135 52 27 
Fax(7­Q95) 135 52 27 

Centrum VT1 SR 
Nam Slobody, 19 
SK­8l223Braiisiava 
Tel (421­7)54 41 Θ3 64 
Fax (421­7) 54 41 83 64 
E­mail, eutopOtbbi sltk.iluba sk 
URL htlp /'www sltk stuba sk 

SLOVENIJA 

Gos podara k i Vestn I k 
Dunajska cesta 5 
SLO­1000 Ljubljana 
Tel (386)613 09 16 40 
Fax (386)613 09 16 45 
E­mail europOgvestnik.si 
URL httpy/www.gvestnik.si 

TURKIYE 

Dunya Infoiai AS 
100. Yil Mahallossi 34440 
TR 80050 Bagcilar­Istanbul 
Tel (90­212)629 46 89 
Fax (90­212) 629 46 27 
E­mail. inlotelOdunya­gazelecom tr 

AUSTRALIA 

Hunter Publication! 
PO Box 404 
3067 Abbotsford. Victona 
Tel (61­3)94 17 53 61 
Fax (6t­3)94 19 71 54 
E­mail ipdaviesOozemailcomau 

CANADA 

Les éditions La Libert« Inc. 
3020, chemin Sainte­Foy 
G1X 3V Samte­Foy. Québec 
Tel (1­418)658 37 63 
F a i ( t ­ B 0 0 ) 5 67 54 49 
E­mail liberteOmediom qc ca 

Renouf Publishing Co. Ltd 
5369 Chemin Canolok Road Unit 1 
KU 9J3 Ottawa, Ontano 
Tel ( 1 - 6 1 3 ) 7 4 5 26 65 
Fax (1-613)745 76 60 
E­mail order dept β re nou (books com 
URL http //www renoutbooks com 

EGYPT 

The Middle East Observer 
41 Sherif Slreet 
Cairo 
Tel (20­2)392 69 19 
Fax (20­2) 393 97 32 
E­mail mafoudaOmeobserver com eg 
URL http//www meobserver com.eg 

INDIA 

EBIC India 
3rd Floor. Υ Β Chavan Contre 
Gen J BhosaloMaig 
400 021 Mumbai 
Tel (91­22)282 6064 
Fax (91-22)285 45 64 
E­mail ebicttgiasbmOl vsnl net in 
URL' httpy/www ebiandia com 

ROY international 

41. MishmarHayarden Street 
PO Box 13056 
61130 Tel Aviv 
Tol (972-3)649 94 69 
Fax (972­3) 648 60 39 
E ­ mail. royl β netvision. nel. il 
URL: httpy/www royint.co.il 

Sub­agent for the Palestinian Aulhonty: 

Index Information Services 

PO Box 19502 
Jerusalem 
Tel (972­2)627 16 34 
Fax (972­2) 627 12 19 

PSklapan 

Asahi Sanbancho Plaia «206 
7­1 Sanbancho, Chiyoda­ku 
Tokyo 102 
Tel (81­3)32 34 6 9 21 
Fax (81­3) 32 34 69 15 
E­mail booksOpsi­|apan co ¡ρ 
URL: httpjfvtww psi­japan com 

EBIC Malaysia 

Level 7, Wisma Hong Leong 
18 Jalan Perak 
50450 Kuala Lumpur 
Tel. (60-3)262 62 98 
Fax (60­3) 262 61 98 
E­mail ebic­kiemot.net my 

Mundi Prensa Mexico. SA de CV 

Rio Panuco No 141 
Colona Cuauhtemoc 
MX­O65O0 Mexico. DF 
Tol (52­5)533 56 58 
Fax (52­5) 514 67 99 
E­mail. 101545 23610compuseive.com 

EBIC Philippines 

19th Floor. PS Bank Tower 
Sen Gil J Puyat Ave. cor Ttndalo SI 
Makati City 
Metro Manilla 
Tel (63-2) 759 66 BO 
Fax (63-2) 759 66 90 
E­mail ocepcom© globo com.ph 
URL: htlp //www.eccp.com 

EBIC Sri Lanka 
Trans Asia Hoi el 
115 Sir chittam palam 
A Gardiner Mawalha 
Colombo 2 
Tel (94­1)074 71 50 78 
Fax (94­1)44 87 79 
E­mail ebicslOitmin.com 

EBIC Thailand 

29 Vamssa Building, eth Floor 
Soi Chidlom 
Ploenchit 
10330 Bangkok 
Tol (66-2)655 06 27 
Fax (66­2) 655 06 28 
E­mail: ebicbkkOksclS.thcom 
URL: htTpïwww ebicbkk org 

UNITED STATES OF AMERICA 

B erna n Associates 

4611­F Assembly Dnve 
Lanham MD20706 
Tel (1­800) 274 44 47 (loll (roe lelephono) 
Fax (1 ­800) 865 34 50 (Ion (roe lax) 
Email queryObeman.com 
URL: http://www bernan com 

Bitte wenden Sie alch an ein Büro Ihrer 
Wahl/ Please contact lhe s a l si o t i l e· 
ol your choleo/ Veuillez vous adresser 
au bureau de vente de votre choix 

Office for Officiât Publications 
ol the European Communities 
2, rue Mercier 
L­2985 Luxembourg 
Tel (352) 29 29­42455 
Fax (352) 29 29­42758 
E­mail: info mfoOopoce cec be 
URL' http://eur­op ou mt 

The  MEANS  collection: 
"Evaluating  socio-economic  programmes" 

This  six-volume  set  gives  wider  public  access  to  the  lessons  of  the  MEANS 
programme  which the  European Commission  introduced  with a view to  impro­
ving  and  promoting  evaluation  methods.  Its  publication  meets  a  growing 
need for  reliable  benchmarks  to deal  with  the  problems  of  evaluation  encoun­
tered when implementing  public  measures in general and Structural Fund ope­
rations  in  particular. 

In  this  context,  the  set  constitutes  a  unique  methodological  guide  that  can 
solve  the  technical  and  organisational  problems  arising  in  the  course  of 
evaluation  work  and  for  which  the  existing  literature  provides  little  help. 

The  collection  is  aimed  not  only  at  professional  evaluators,  for  whom  it  will 
provide a valuable handbook,  but also  at non-specialists,  in particular  manag­
ing  authorities  and  structural  programme  Monitoring  Committees. 

Evaluation  design  and  management 

Each  volume  is  designed  to  be  independent  of  the  others,  yet  they  are  also 
complementary. Together, they deal with all aspects of the evaluation of socio­
economic  programmes: 
I. 
II.  Selection  and  use of  indicators  for  monitoring  and  evaluation 
III.  Principal  evaluation  techniques  and  tools 
IV.  Technical  solutions  for  evaluation  within  a  partnership  framework 
V.  Transversal  evaluation  of  impacts  on  the  environment,  employment  and 

other  intervention  priorities 

VI.  Glossary  of  300  concepts  and  technical  terms 

OFFICE FOR OFFICIAL  PUBLICATIONS 

OF THE EUROPEAN  COMMUNITIES 

L-2985  Luxembourg 

uctural  Funds 

Evaluating socio-economic  programmes 

Technical  solutions for  evaluation 
3worl? 

MEANS 
Collection 

Volume 4 

EC Structural Funds 

Evaluating socio-economic programmes 

Technical  solutions  for  evaluation 
within  a partnership  framework 

Volume  4 

European Commission 

MEANS is a programme  of  the European Commission. 
It is aimed  at improving  Methods  of  Evaluating Structural  Policies. 
The MEANS Programme  and  Collection  was supervised by 
Phillipe  Goybet and Miguel Angel  Benito  Alonso, 
Unit for  the Coordination  of Evaluation in the Directorate  for  Coordination 
and  Evaluation of  Operations of  DG  XVI 
Coordination-evaluation@fmb. dg 16.cec.be 

The realisation  of  the Programme  was entrusted  to the 
Centre for European Evaluation Expertise (C3E) 
under  the direction  of  Eric Monnier  and Jacques  Toulemonde. 
Responsibility  for  the opinions  expressed in this 
book lies solely with  the authors  of  the text: 
C3E  -  13 bis  Place Jules Ferry  -  F  -  69006 Lyon  -  France  -  c3e@c3e.fr 

A great deal of additional information on the European Union is  available 
on the Internet. 
It can be accessed through the Europa server (http://europa.eu.int) and 
on the Inforegio Website (http://inforegio.cec.eu.int). 

Cataloguing data can be found at the end of this  publication. 

Luxembourg: Office for Official Publications of the European 
Communities, 1999 

ISBN  92-828-6626-2 

©  European Communities, 1999 
Reproduction is authorized, except for commercial purposes,  provided 
the source is acknowledged. 

PRINTED  ON  WHITE CHLORINE-FREE  PAPER 

Printed in Italy 

^r 

Foreword 

is  directly 

Developing  good  evaluation  practices 
the 
has  become  a  priority  area  for 
relevant 
Commission  and 
to 
the  effective  use  of  Community 
resources. 
Indeed, the sound financial  management 
principles  which  the  Commission  seeks 
to  promote  via  its  SEM2000  (Sound  and 
Efficient  Management)  Programme  rely, 
specifically,  on  the  regular  evaluation  of 
the  Community  activities. 
The  Directorates  General  responsible  for 
structural  policy have considerable expe­
rience  in this area.  Since  1988 they  have 
made  evaluation  an  integral  part  of  each  stage  in  the  process  of 
Community  Structural  intervention.  Thus,  the  programme  development 
stage  includes  a  prospective  evaluation;  the  implementation  stage  a 
strengthening of the monitoring systems and, at the end of the intervention, 
evaluation  establishes  what  has  been  achieved  in  terms  of  programme 
impacts. 
However,  the capacity  of  evaluation to  contribute  to the  effectiveness  and 
transparency  of  structural  interventions  rests  ultimately  on  the  integrity  of 
the  methods  adopted  to  verify  the  credibility  of  results. 
This  is why, four years  ago, on the  initiative for  the  Directorate  General  for 
Regional  Policy  and  Cohesion  (DG  XVI),  the  MEANS  programme  was 
initiated  as a response to the demand for developing  evaluation  methodo­
logy.  This  also  include  practices  combining  both  general  scientific 
requirements  and  the  specific  requirements  of  Community  interventions 
whilst, at the same time,  remaining  accessible to  a large audience. 
One of  the  more tangible  results  of  the  MEANS  programme  has  been  the 
publication of this six volume set by a group of  independent  experts, focu­
sing  on  the  principal  elements  of  socio-economic  programme  evaluation. 
A short  summary of this work can be found at the beginning of the  present 
volume. This collection covers the various methodological approaches  and 
innovations  that  have  taken  place  in  the  context  of  structural  funds  eva­
luation.  It  will  enable  those  responsible  for  evaluation  to  develop  their 
approach  in  a  flexible  and  independent  manner  and  for  the  various 

programme  partners  to  exercise  greater  control  over  the  evaluation  pro­
cess.  Attending  the  conferences  of  evaluation  specialists  and  "clients" 
convened  in  Brussels  (1995),  Berlin  (1996)  and  Seville  (1998),  I  gained 
some  impression of the  interest that  new  evaluation techniques  can  gene­
rate.  I would  stress that  the  development  of  evaluation  needs to  progress 
in  tandem  with  efforts  to  incorporate  evaluation  findings  in  the  decision 
making  process. This  implies  evaluation  as  a  means  of  establishing  whe­
ther  policies  are equitable,  effective  and  relevant to  the  identified  needs. 
In satisfying  these  requirements,  evaluation  should  be seen  as  a valuable 
instrument  affording  both  Member  State  and  Community  authorities  the 
necessary  guarantees  that  Community  funds  are  being  deployed  in  the 
best interests of the EU citizen. This is a cost  worth  paying when the goals 
of  Economic  and  Social  Cohesion  and  the  European  Union  enlargement 
are at  stake. 

Finally,  I wish  to  extend  my  thanks  to  the  members  of  the  Committee  of 
Independent  Experts  established  at  the  launch  of  the  programme.  Their 
expertise and standing  have contributed  greatly to the quality and relevan­
ce  of  the  MEANS  programme.  I therefore  thank:  E.  Chelimsky  (USA),  ex-
President  of  the  "American  Evaluation  Association";  K.  Kennedy  (IRL), 
Director  of  the  "Economic  and  Social  Research  Institute"  (ESRI);  J.  R. 
Cuadrado  Roura  (SP),  President  of  the  "European  Regional  Science 
Association"; C. Seibel (F), Director of the "Statistiques démographiques  et 
sociales de l'Institut  National de la Statistique et des  Etudes  Economiques" 
(INSEE);  L.  Senn  (I),  Director  of  the  Department  of  Economic  Regional  of 
the  University  Bocconi;  E. Stern (U.K.), Director  of the  unit  of evaluation  of 
the  Tavistock  Institute;  A.  Sorber  (t)  (NL),  Head  of  the  "Policy  Analysis 
Department  of the Ministry  of  Finance"; L. I. Strom (SV), Statens  Institut för 
Regionalforsking  (SIR);  L.  Tsoukalis  (GR.),  professor  at  the  University  of 
Athens;  H. Wollmann  (D), professor  at the  University  of  Humboldt  -  Berlin 
and founder  member  of the  "European  Evaluation  Society". 

E.  Landaburu 

The  MEANS Collection: 

"Evaluating socio-economic programmes" 

of  European  Union 

The  evaluation 
structural 
interventions  has  entered  into  a  phase  of  maturity, 
attested  by the extension  of work  carried out  by both the 
Member  States and the Commission. Regulations  based 
on  decisions  by  the  Council  have  largely  contributed  to 
the  development  of  evaluation  practice  and  have 
stimulated  methodological  research. 

initiated  by  the  Commission 

Over  the  past  ten  years,  the  Commission  has  striven  to 
promote  and  improve  methods  for  evaluating  Structural 
Funds. The main  methodological  conclusions  have  been 
debated  and  disseminated  as  part  of  the  MEANS 
in  1991. 
programme 
Achievements  to  date  provide  enough  material  for  a 
complete  and  accessible  publication  intended  for  a 
broad public: a six volume Collection covering all aspects 
of the evaluation  of socio-economic  programmes. These 
Volumes  are  written  with  a  perspective  that  is  broader 
than  the  current  administrative  framework,  for  this  is 
likely to evolve  more rapidly than  professional  standards. 

information  on 

In general, the Collection  is aimed at a readership of non-
specialists  seeking  essential 
the 
evaluation of socio-economic  development  programmes. 
These  programmes  have  multiple  objectives,  a 
partnership  dimension  and  a  multi-sectoral  content  -  all 
characteristics which call for suitable evaluation  methods 
and  solutions  not  yet  provided  by  the 
international 
technical literature. The complete Collection consists of 6 
volumes  designed  to  be  complementary  and  to  provide 
essential 
for 
professionals  wanting  to  independently  develop  their 
own evaluation  practices. Each volume can, however,  be 
read and understood without  having to  refer to the  other 
volumes  in the Collection. 

recommendations 

information  and 

Technical solutions for evaluation within a partnership framework 

The  first  Volume  in  the  MEANS  Collection  entitled 
"Evaluation  design  and  management"  serves  as  an 
introduction  both  to  the  Collection  and to the  evaluation 
of  socio-economic  programmes.  It  enables  readers  to 
understand  the  implications  of  evaluation  and  to  know 
what  they can and cannot  expect from an evaluation at a 
given  time  in  a  particular  context.  It  covers  all  the 
elements  needed to  steer an evaluation  project  from A to 
Z:  sharing  of  responsibilities,  defining  of  the  aims  and 
method  of  an  evaluation,  establishing  a  schedule  and 
budget, assessing the quality of the work  performed and, 
finally,  managing the  use of  conclusions. 

The second Volume focuses on the  "Selection and use of 
indicators  for  monitoring  and  evaluation".  It  deals  with 
the  subject  of  indicators  used  for  monitoring  and 
evaluation.  It  shows  how  indicators  can  be  used  to 
support  policy  decisions,  improve  management  and 
measure  the  overall  performance  of  programmes.  The 
volume  proposes  a  methodological  framework  and  a 
practical  guide for the  use of  indicators  in these  different 
situations. 

The  third  Volume  presents  a  selection  of  "Principal 
evaluation  techniques  and  tools"  applicable  to  the 
evaluation  of  socio-economic  programmes.  These  tools 
are taken from  relevant  scientific  disciplines:  economics, 
sociology,  management, geography, etc. Each one of the 
23 tools  is described  in the form  of  an  information  sheet 
featuring  a  general  description,  the  principles  of  its 
implementation, a list of its strengths and weaknesses for 
the  evaluation  of  Structural  Funds,  an  example  of  its 
application  and a list of  references. 

A number of methods adapted to the evaluation of socio­
economic  programmes  are the  object  of  a more  detailed 
description 
in  the  fourth  Volume  entitled  "Technical 
solutions  for  evaluation  within a partnership  framework". 
Several  pilot  evaluations  were  implemented  within  the 
MEANS programme to design and test the following  four 
innovations,  the  application  of  which  is  described  in 

The MEANS Collection 

detail  in  this  Volume:  impact  mapping,  cross-impact 
matrix, score cards and  multi-criteria  analysis. 

The  fifth  Volume  deals  with  "Transversal  evaluation  of 
impacts  on  the  environment,  employment  and  other 
intervention  priorities"  of  the  European  Union  and 
Member  States.  Among  the  questions  addressed  are 
impacts  in  terms  of  equal  opportunities  between  men 
and  women  and  the  competitiveness  of  SMEs.  The 
Volume  provides the elements  required for an analysis of 
those  impacts  which  are  not  necessarily  part  of  the 
explicit  objectives  of all the actions evaluated. 

Finally,  the  last  Volume  is  a  "Glossary  of  concepts  and 
technical  terms".  The  compilation  of  the  Collection  has 
necessitated  the  rigorous  standardisation  and  definition 
of  the terms  used.  Each  of  the  first  five  volumes  has  an 
index  of terms which, along with  hundreds of  definitions, 
are brought  together  in the glossary. 

Contents 

Introduction 

I  Concept  mapping of  impacts 

A solution for  evaluating  different  measures 

by  means  of  common  criteria 

Instructions  for  use of the  concept  mapping  of  impacts 

Using the  concept  mapping  of  impacts 

II  Constructing  and using scoring  scales 

A solution for  evaluating  different  measures 

in a homogeneous  manner 

Instructions  for  using  scoring  scales 

Using  ratings  for  synthesising  the  evaluation  of  measures 

III  Analysing the  programme  using  a  matrix  of  cross  impacts 

A solution for  analysing  synergy  within  a programme 

Instructions  for  using the  matrix  of  cross  impacts 

Empirically  verifying  the  main  synergies  identified 

IV  Using  multicriteria  analysis within  a  partnership  framework 

A solution for judging  the  overall  success  of  a programme 

Instructions  for  using  multicriteria  analysis 

Using the  analysis to  produce  useful conclusions 

Conclusion 

Appendix 

Index 

9 

15 

17 

29 

43 

47 

49 

55 

67 

71 

73 

81 

87 

91 

93 

99 

115 

117 

125 

129 

Introduction 

Introduction 

Socio-economic  development  programmes  are  often  co-funded  and 
managed  within  a  partnership  framework.  This  is  the  rule  especially  for 
Structural  Fund  programmes.  As  a  result,  the  evaluation  of  these 
programmes  is  subject  to  particular  constraints,  not  only  in  so  far  as  the 
sharing  of decision-making  increases the  number  of stakeholders,  but  also 
because  it  tends  to  multiply  the  number  of  objectives  and to  diversify  the 
domains  of  intervention. 

Until  recently,  evaluation  developed  essentially 
in  the  context  of 
interventions  financed  by  a  single  institution,  in  a  specific  domain. 
Evaluators  consequently  lack  the  know-how  required  in  cases  of  multi-
objective, multi-sectoral  programmes  involving several partners.  Evaluation 
teams  tend  to  view  complex  programmes  as  a  series  of  partial  analyses, 
and  therefore  analyse  each  of  the  components  successively  (axes, 
measures  or  actions).  This  approach  leads  to  difficulties,  however,  when 
synthetic conclusions have to be produced and strategic  recommendations 
made on a programme-wide  scale. With traditional evaluation  methods  it is 
rarely  possible  to  justify  the  reallocation  of  budgets  because  the  different 
components  of  the  programme  cannot  be compared  in terms  of  common 
evaluation  criteria, acceptable to  all the  decision-makers. 

In recent  years,  several  evaluation  teams  working  on the  Structural  Funds 
have  distinguished  themselves  by  introducing  methodological  innovations 
in cases of  "multi-objective  and multi-sector"  evaluations.  Pilot  evaluations 
performed  as  part  of the  MEANS programme  have also  helped to  develop 
several  operational  solutions  applicable  to  such  cases.  By  drawing  on 
experience  acquired  in  recent  years,  this  Volume  describes  and  provides 
detailed  instructions  for  the  use  of  four  new  technical  solutions  now 
available  to  professionals.  Two  of  these  techniques  (concept  mapping  of 
impacts  and  multicriteria  analysis)  are  also  featured  In  Volume  3  of  the 
Collection,  devoted  to  evaluation  tools. The  intention  of  that  Volume  is  to 
assist  the  making  of  methodological  choices;  the  tools  are  therefore 
presented  more synthetically, without  the  detail  provided  in this Volume. 

The  first  technique  presented  here  is  the  concept  mapping  of  impacts. 
This  tool  can  be  used  to  define  the  main  intermediate  impacts  of  a 
programme  in  the  same  way  for  all  the  different  stakeholders.  Concept 
mapping  offers  applicable  cross-sectional  evaluation  criteria  for  an  entire 
multi-objective  and  multi-sector  programme.  It  also  provides  a  precise 
description  of  these  impacts,  which  can  be  used  to  construct  scoring 

Technical solutions for evaluation within a partnership framework 

scales  and  indicators. To date,  no manual  exists for  the  implementation  of 
this technique  in an evaluation  context. 

The  second  technique  presented  is  the  construction  of  scoring  scales. 
Scoring  is  being  increasingly  used  for  project  selection,  particularly  in 
Northern  European  countries. The originality  of  this  Volume  lies in the  fact 
that  scoring  is  applied  to  the  synthesis  of  conclusions  obtained  at  the 
measures  level, and is directly  related to the main evaluation  criteria of  the 
programme. 

Apart  from  the  evaluation  of  each  measure,  the  interaction  between 
measures  must  also  be analysed.  It can  be  expected  that  the  impact  of  a 
coherent  set  of  measures  will  be  greater  than  that  of  the  same  measures 
Implemented separately. The third tool presented in this Volume is an aid for 
the evaluation of these effects of synergy. It is the matrix of cross impacts, 
a key element in the approaches conceived and tested by several  European 
teams whose work  was  included  in the  MEANS  programme. 

Finally,  each  of  the  funding  authorities  would  like  to  see  its  own  criteria 
integrated  into  the  evaluation  report.  Multicriteria  analysis,  the  fourth 
technique  presented, is used as a basis for synthesising conclusions  which 
take  into account  the expectations  of  each partner.  Multicriteria  analysis  is 
typically  used  for  ex  ante  evaluation,  with  a  view  to  classifying  or  sorting 
competing  projects  or the variants of  a single  project.  By using the tool  for 
the  overall  evaluation  of  a  socio-economic  development  programme,  this 
Volume  proposes  an original  use. Among  the  multiple technical  variants  of 
multicriteria  analysis,  it  presents the one which  is most appropriate for  co-
funded  programmes. 

The  four  tools  covered  by  this  Volume  are  presented  with  detailed 
instructions  for  their  use.  For  each  tool,  a  particular  effort  was  made  to 
present  several  variants  which  are  either  very  easy  to  apply  or,  on  the 
contrary,  involve  more  technical  and  rigorous  procedures.  The  reader  will 
note that, in the least complicated variants, the proposals in this Volume are 
little  more than  "organised  common  sense"  and that,  in  many  cases, they 
correspond to  current  practice. 

The  advantage  of  presenting  these  four techniques  in the  same Volume  is 
that they have numerous common points, e.g.: high level of participation  by 
the  partners  in  the  programme;  capacity  to  process  quantitative  and 
qualitative  information;  several  points  of  view  taken  into  account;  etc.  All 
four  techniques  can,  moreover,  be  grouped  together  within  a  single 

Introduction 

evaluation method. The conclusion to this Volume presents two practical 
suggestions in this respect. 

This Volume is intended mainly for evaluation teams (it is the only Volume in 

the Collection devoted primarily to them), but also for evaluators who wish 

to enhance their knowledge on the subject. By extension, it is likely to be of 

interest to those whose work concerns partnership evaluation, outside the 

domain of the Structural Funds. 

This Volume can be read in several ways, as suggested in the following box. 

Box 1 - Possible uses of this Volume 

If you wish to prepare an overall evaluation of a socio-economic programme by 
using all or part of the four techniques: 

>■ Read the Conclusion (see page 115), then refer to the different parts of the 

Volume as and when necessary. 

If you wish to study the coherence of a programme, explore its implicit objectives 
and find transversal evaluation criteria for all the measures: 
>■ Read Part I on the conceptual mapping of Impacts (see page 15) or simply 

its instructions for use (see page 29). 

If you wish to evaluate all the measures of a programme with the same 
homogeneous and transversal logic: 

>■ Read Part II on the construction and use of scoring scales (see page 47) or 

simply its instructions for use (see page 55). 

If you wish to analyse the synergy within a programme and understand how the 
impacts of different measures are mutually reinforcing: 

>· Read Part III on the matrix of cross impacts (see page 71) or simply its 

instructions for use (see page 81). 

If you wish to conclude the overall evaluation of a programme, classify or 
compare the measures of which it is composed in terms of their success, and 
take into account all the criteria which are important for the partners: 
>■ Read the part on multicriteria analysis (see page 91) or simply its instructions 

for use (see page 99). 

I 
Concept mapping of impacts 

The  concept  mapping  of  impacts,  as  presented  in  this  chapter,  is  an 
adaptation  of  the  technique  known  as  "concept-mapping",  used  for  the 
past ten years in evaluation. The tool is used to define the main  intermediate 
impacts  of  a  programme,  by  seeking  the  greatest  possible  degree  of 
transversality between  the  different  measures. A  concept  map  of  impacts 
serves to  clarify  the  main  (explicit  and  implicit)  objectives  of a  programme 
and  to  base  an evaluation  on  clear  criteria  accepted  by  all  the partners.  It 
also provides  a precise  description  of expected  impacts  which  can be used 
operationally  in  field  surveys,  for  example  to  construct  scoring  scales  and 
indicators. 

A solution for evaluating different  measures 
by means of common  criteria 

Overall  evaluation  of  a  programme 

Programmes  financed  by  European  cohesion  policy  are  all  aimed  at 
accelerating  socio-economic  development  in  one  of  its  dimensions  (e.g. 
income,  employment;  quality  of  life).  Thus,  the  main  impact  is  generally 
expected  at  the  macro-economic  or  macro-social  level.  To  obtain  this 
impact,  most  programmes  adopt  multi-sectoral  strategies  consisting  of 
various  kinds  of  measure  (e.g.  facilities,  consultancy,  subsidies,  training, 
communication)  in diverse fields  (e.g. transport,  research, tourism). 

Within the framework  of  European  economic  and social  cohesion  policy, a 
typology  consisting  of  three  main  domains  of  intervention  is  used: 
improvement  of  basic  infrastructure  (transport,  communication,  public 
facilities  in  general);  support  for  the  productive  environment,  including 
support for economic actors (to increase, for example, their ability to control 
quality  and  to  export,  their  integration  into  innovation  networks,  etc.)  and, 
finally, the development  of  human  resources  (essentially  through training). 

This wide  range of heterogeneous fields of  intervention  is frequent  in socio­
economic  programmes.  However,  with  the  Structural  Funds  this  diversity 
has reached a level rarely equalled elsewhere, which could lead to confusion 
if the programmes were not highly structured. The mode of structuring opted 
for  has three main  levels, as well as one or two  intermediate  levels: 

•  The  overall  programme  level  is associated  with  the  global  objective,  for 
example  economic  development  of  employment.  This  level  generally 
comprises  several  priority  axes  which  define  the  main  strategic 
dimensions  of the  global  objective. 

•  The  measures 

level  (between  about  a  dozen  and  several  dozen 
measures)  is  the  basic  unit  of  programme  management,  in  which  each 
measure  has  its  own  management  apparatus.  In  principle,  a  measure 
corresponds to the smallest homogeneous component  of the  programme 
and  groups  together  similar  projects.  However,  in  some  cases  certain 
measures may group together  several categories  of different  projects. An 
additional  subdivision  is then  introduced, called  "action" to  denote  each 
homogeneous  project  category. 

Technical solutions for evaluation within a partnership framework 

• The project level (often a few hundred) is the unit of implementation in the 
field. Each project is the point of articulation between the programme and 
its addressees. 

All the projects in a single measure first produce outputs (in the example in 
Box 2, kilometres of roads improved), then results on direct addressees 
(time and money savings for users of the improved roads). Finally, impacts 
on increasingly distant concerned groups materialise in successive stages 
through multiple, interlinking mechanisms (e.g. easier international travel for 
regional businesses or better access for tourists to tourist sites). The very 
existence of the programme assumes that all these mechanisms converge 
towards the global objective of development. The fact remains that this 
convergence is achieved in numerous and highly complex ways. 

Box 2 ­ Structure of a programme (extracts) 

Programme 
(consisting of 
4 axes) 

Priority axis: 
Communications 
(consisting of 
5 measures) 

Measure: 
Roads 
(consisting of 
several dozen 
projects) 

Project : 
Road between 
A and B 

Reducing the lag 
in development 

Enhanced 
competitiveness 
of enterprises 

Enhanced 
competitiveness 
of enterprises 

Effective time 
savings for the 
transport of 
goods and people 
travelling in 
the region 

φ 

o ë 
culo" 
o 

Ii 
t ., α> 
a. e 
fi 

ç 

tu <υ 
o Í2­

*2 
crt 

αζ 

to 

Q. 
O 

Effective time 
savings for 
travellers and 
goods transported 
on the regional 
road network 

Potential time 
savings for 
vehicles using 
the regional road 
network 

Potential time 
savings for a 
vehicle travelling 
between A and B 

Length of the new 
or improved road 

The evaluation of a programme can be performed in very different ways, 
depending on how impacts are to be analysed. In this Collection, three 
situations have been presented as follows: 

Concept mapping of impacts 

•  Overall  evaluation  which  has  to  examine  all  the  impacts  of  all  the 

measures. 

•  More  in-depth  analysis of  macro-economic  impacts. 

•  More in-depth analysis of a particular  impact, for  example, of a measure. 

This Volume  was  prepared from  the  viewpoint  of  the first  situation,  i.e.  the 
overall evaluation of all the impacts of a multi-sectoral  programme. Volumes 
1 and 3 of the Collection can be referred to for an introduction to  evaluation 
techniques  applicable to the other two  situations. 

Synthesis  of  specific  evaluations 

Each measure is naturally evaluated in terms of its own criteria, for  example 
on  the  basis  of  its  results  and  specific  impacts.  Box  2  suggests  that  a 
measure for improving  roads will  be evaluated, for instance, in terms of the 
time actually or potentially  saved by the vehicles using the road. As is often 
the case, it can be assumed that the evaluation team will estimate the total 
number of hours of transport saved annually by vehicles using the road. The 
number  of  hours  saved  can  be  related  to  the  cost  of  the  measure, so  that 
comparisons  can  be  made  with  road  improvement  programmes  in  other 
similar  regions. 

The  indicators  mentioned  above  are  specific  to  a  particular  domain  of 
intervention  (roads), just  as the  approach  adopted  is that  of  an  evaluation 
specifically  for  the  road  transport  sector.  Specific  evaluations  can  be 
performed  for  all  sorts  of  measures:  railways,  harbours,  airports,  but  also 
for  sectors  of  intervention  as  varied  as  the  training  of  the  long-term 
unemployed,  the  rehabilitation  of  run-down  neighbourhoods  or  the 
establishment  of  innovation  networks. Volume  2  in the  Collection  presents 
numerous  examples  of  interventions, with a description  of their  results and 
specific  impacts  and suggestions for  appropriate  indicators. 

A standard  procedure  in the overall evaluation  of  a programme  consists  of 
performing  specific  evaluations,  measure  by  measure,  and  then  of 
synthesising  the  results  at  the  axis  and  programme  levels,  as  shown  in 
Box 3. 

This  bottom-up  procedure  does  not  involve any  major  problems  when  it  is 
used to evaluate a measure  by synthesising the evaluations of the  projects 
of  which  it  is  composed.  Since  all  the  projects  of  the  same  measure  are 
similar,  it  is  possible  to  proceed  by  way  of  aggregation,  calculation  of 

Technical solutions for evaluation within a partnership  framework 

averages,  comparisons  between  projects,  identification  of  exemplary 
projects,  and so on. 

Box 3 -  Synthesis of specific  evaluations 

Programme 

Axes 

Measures 

Projects 

Evaluation  by  synthesis  can,  in  contrast,  involve  serious  methodological 
shortcomings  if  it  is performed  on  heterogeneous  measures  evaluated  in a 
specific  manner.  Ultimately,  each  specific  evaluation  uses  its  own 
indicators,  analyses  distinct  impacts,  and  applies  different  judgement 
criteria.  In such  situations, the  overall  evaluation  of  a programme  can  only 
be the  juxtaposition  of  partial  evaluations  of  measures. The very  notion  of 
synthesis  therefore  loses  its  meaning.  An  evaluation  report  can  be 
produced  in which a long chapter  is devoted to each  measure, but in which 
the  overall  conclusion  is  nothing  more  than  a  few  lines  with  very  poor  or 
tenuous  contents.  In these circumstances  it is impossible to  make  relevant 
recommendations  for the entire  programme.  For example, it is not  possible 
to  justify  recommendations  for  the  reallocation  of  budgets  between 
measures because there is no argument  showing that one measure is more 
effective  or successful than  another. 

Evaluation  using  transversal  criteria 

To produce  genuine synthetic  conclusions, evaluation  criteria have to  allow 
the  comparison  of  the  value  of  measures  in  relation  to  one  another.  A 

Concept mapping of impacts 

the  speed  at  which 
frequently  used  element  of  comparison 
implementation takes place, reflected, for example, in an absorption  rate or 
completion  rate  indicator.  This  element  of  comparison  plays  an  important 
part in monitoring. It is too often used as the main argument for  reallocating 
budgets, when the evaluation team is unable to  base  its  recommendations 
upon  common  criteria of  effectiveness,  efficiency  or relevance. 

is 

To  enhance  evaluations,  a  comparison  of  measures  in  terms  of  the 
attainment  rate  of  their  specific  objectives  could  be  conducted  (see  the 
following  example). The fact  that  a measure  rapidly  achieves  its  objectives 
may  be  an  argument  in  favour  of  its  effectiveness  or  success,  but  this 
argument  is necessarily weak. If the objectives were set very carefully or, by 
contrast,  very  ambitiously,  the  conclusions  could  be  distorted.  The 
evaluation of the degree to which objectives  have been achieved  relates to 
a  question  of  ex  ante  evaluation:  Can  it  be  said  that  the  objectives  of  all 
measures were set with the same level of ambition? This question relates to 
another  fundamental  question  considered  in this  Volume:  In terms  of  what 
common  criterion  can  the  level  of  ambition  of  the  different  measures  of  a 
programme  be  compared? 

Example: Comparison of the rate of achievement of objectives 
The  aim  of  one  measure  was  to  retrain  2,000  employees  threatened  with 
unemployment, and that of another was to create 500 new international  contacts for 
regional SMEs. When the mid-term evaluation was performed, the first measure had 
achieved 53% of its objectives and the second only 25%. Could it be said, on this basis, 
that  the second measure was a relative failure?  In reality, the first  measure was the 
continuation of  a former intervention of which the objectives  had not really changed, 
while the second measure was innovative and ambitious. Thus, the comparison of the 
two rates cannot be used to compare the effectiveness of the two measures. 

In the context of overall programme evaluation, this Volume tries to  promote 
a genuine synthesis  based on an analysis of the contribution of the  different 
measures to socio-economic  development,  and not  on the speed  at  which 
they  are executed. 

Contribution  to  the  achievement  of  the  global  objective  of 
a  programme 

Each programme  must contribute to the European Union's global  objective 
of  economic  and  social  cohesion. The  global  objective  of  a  programme  is 

Technical solutions for evaluation within a partnership  framework 

itself expressed  in terms  of socio-economic  development,  with a weighting 
that  will  depend  on  a  particular  aspect  of  that  development,  e.g.  sectoral 
diversification, competitiveness, social  integration, sustainability  of  growth, 
protection  of  the  environment,  etc.  As  the  following  example  shows,  the 
global  objective  describes  the  ultimate  impact  that  the  programme  is 
supposed  to  obtain.  In  a  programme  designed  coherently,  each  of  the 
measures  contributes  towards  the  production  of  the  global  impact  of  the 
programme. 

Example: Global  objective  and intermediate  objectives 

The  Belgian  province  of  Hainaut  receives  European  Union  assistance  as  part  of  a 
development  programme the priority of which is its economic  adjustment. The global 
objective  set at the start  of the programme,  in 1994, is the  increase  in the per  capita 
GDP, with a view to attaining  79% of the  Community  average  in 1999. 
The  programme  comprises  four  axes  of  intervention,  each  of  which  is  sub-divided 
into  measures. The  objective  of  the  priority  Axis  1  is  to  boost  economic  activity.  It 
consists  of  stimulating  the  modernisation  of  the  economy  and  the  endogenous 
development  of the region, of increasing  its technological  R&D capacity, of  providing 
a  favourable  environment  for  new  business  creation,  and  of  transforming  the 
economic  growth  resulting  from these efforts  into jobs. The objective  of Axis  2 is to 
enhance the  attractiveness  of the  region. The  Hainaut  province  is still scarred  by  its 
industrial  past.  Making  the  area  attractive,  providing  quality  living  conditions  and 
protecting  the  environment  are  therefore  a  specific  challenge.  Axis  3  is  aimed  at 
completing  the transport  infrastructure within a perspective  of  integration, in relation 
to the other programme measures. The objective of Axis 4 is to open up  opportunities 
for  everyone.  The  labour  market  problems  in  the  Hainaut  province  justify  the 
implementation  of  an employment  policy  aimed at  promoting  the development  of an 
economy  in which  competitiveness  and solidarity  are priorities. 

Although  it  may  be  desirable  to  judge  the  measures  of  a  programme  in 
terms  of  their  contribution  to  the  achievement  of  the  global  objective, 
practical  obstacles  make this  impossible. As Box  4 shows, the  impact  of a 
given  measure  on  economic  and  social  cohesion  involves  too  many 
complex  mechanisms  to  be  the  object  of  reliable  evaluation.  Moreover, 
reasoning.  An 
some  of  these  mechanisms  require  macro-economic 
evaluation  team  may  certainly  provide  useful  knowledge  on  these 
mechanisms,  but  it  will  rarely  be  able  to  analyse  them,  in  a  differentiated 
way,  measure  by  measure,  with  enough  robustness  to  justify  strong 
recommendations  such  as  the  reallocation  of  budgets.  In  practice,  the 
empirical  observation  of  impacts  must  stop  at  an  intermediate  level  which 

Concept mapping of impacts 

often  extends  beyond  direct  addressees,  but  never  to  the  entire  regional 
economy.  Beyond  this  intermediate  level,  extrapolation  coefficients  or 
economic  models  are  used, providing  results  which  are  useful  but  do  not 
allow  measures to  be distinguished  in terms  of their relative  effectiveness. 

Box 4 -  Complexity of impacts  and levels of  objectives 

Measure for 
the amelioration 
of roads 

Time saving 
for users living 
in the region 

Specific impact 

Intermediate  impact 

Growth of 
business activity 
and employment 
in the region 

Global impact 

Better 
economic 
and social 
cohesion 

Identifying  the  main  intermediate  impacts 

for 

the 

techniques  have  been  developed 

Several 
identification  of 
intermediate  causal  relations  between  the  specific  results  of  a  public 
intervention  and  its  more global  impacts. The  most  simple  and  well-known 
is  the  logical  framework.  Originally  designed  for  the  ex  ante  evaluation  of 
projects  in developing  countries, this tool gives a structured  presentation of 
the evaluated  intervention  and  its  expected  effects.  By applying the  logical 
framework  to  the  example  cited  in  Box  2,  a  table  is  obtained  which 
describes expectations concerning outputs (bottom left in Box 5), as well as 
the main logical steps separating  outputs  from the global impact  (top  left). 

With  each  logical  step,  the  evaluation  team  stipulates  which  means  have 
been provided  or planned for  achieving the objectives  (verifiable indexes of 

Technical solutions for evaluation within  a partnership  framework 

success).  It simultaneously  defines  what  must  be considered  a success  or 
failure  (sufficient  /  insufficient  outputs,  results  and  impacts).  By  requiring 
that  this  second  column  of  the  matrix  be  filled  with  precise  qualitative 
indicators or descriptors, the technique often  provides  a useful  clarification 
which  can  then  be  used  to  provide  a  more  precise  description  in the  first 
column. The third  column, entitled  "conditions  and risks", establishes  a list 
of  exogenous  factors  which  can  facilitate  or  hinder  the  advancement  from 
one  step  to  the  next.  More  details  can  be  found  on the  logical  framework 
technique  in Volume 3 of the Collection. 

Box 5 -  Classification  of objectives  by means 

of the  logical  framework 

Road project  between A and  Β - See  Box  2. 

Statement  on 
expectations 

Verifiable 
success 
indexes 

Conditions 
and  risks 

Global 
impact 

Social and  economic 
development 

Specific 
impact 

Result 

Output 

Savings 
for  regional 
businesses 

Time  savings 
for a typical 
journey 

Improved 
roads 

The structure of the logical framework, consisting of only two impact  levels, 
does  not  lend  itself  to  a  representation  of  intermediate  impacts. A  second 
tool,  better  adapted  to  the  complexity  of  socio-economic  programmes,  is 
the objectives tree. As Box 6 shows, this technique is used to determine the 
logical  contribution  of  each  of  the  objectives  at  the  most  specific  level, to 
an  objective  on  a  more  global  level,  and  so  on  right  up  to  the  global 
objective  of  a programme  represented  by the tree trunk. 

Compared  to  the  logical  framework,  the  objectives  tree  provides  two 
interesting  new  elements:  as  many  branches  can  be  created  as  there  are 

Concept mapping of impacts 

measures  in the  programme, and the  intermediate  levels can  be  multiplied 
to account  for the  complexity  of the mechanisms  of  impact  production. 

Box 6 -  Objectives tree 

Airports 

Improvement 
of roads 

Accessibility 
fortravellers 

Harbours 

Training / 
tourism 

Accessibility 
for  freight 

Diversification 

Reduce 
the peripheral 
nature of 
the region 

Economic and 
social 
development 

In practice,  the objectives tree is drawn up in such a way that measures are situated 

on the left and the global objective on the right 

The  objectives  tree  also  reveals  the  intermediate  objectives  common  to 
several  measures.  In  Box  6, the  objective  "reduce  the  peripheral  nature of 
the  region"  can serve as a common  reference forjudging  the  effectiveness 
of the three measures  situated on the top  left-hand  side of the tree. In fact, 
the  criteria  common  to  several  measures  appear  at  each  fork  in  the  tree. 
However,  each  fork  creates  a  possibility  for  comparison  limited  to  the 
branches  which  are  linked to  it. To compare  measures  situated to the  right 
and left of the tree, it would  be necessary to go right down to the trunk,  but 
this  is  not  generally  possible.  Thus,  because  of  its tree-like  structure,  the 
technique  has little  potential for  synthetic  and transversal  evaluation  of the 
multiple measures of  a programme. 

Identifying  cross-sectional  intermediate  impacts 

To  achieve  a  genuine  synthetic  evaluation,  the  image  of  a  tree  has  to  be 
abandoned  and  a  network  of  objectives  has  to  be  considered.  The 

Technical solutions for evaluation within a partnership  framework 

appropriate tool is the logical diagram of  impacts which  is used to  consider 
a  multiplicity  of  logical  relations  between  measures  and  intermediate 
impacts. Thus, in the example in Box 7, the intermediate  impact  "viability of 
enterprises"  is  indirectly  related  to  all  the  programme  measures,  which 
makes  it  possible  to  undertake  comparisons  between  all the  measures  on 
the basis of their  contribution  to the viability  of  enterprises. 

Box 7 -  Logical diagram  of expected  impacts 

Measures 

Intermediate  impacts 

Global  impacts 

R&D 

Tourism 

Industrial 
wasteland 

Enterprises 

Agriculture 

Training 

Opening  up 
to the  outside\ 

Enhanced 
viability of 
entreprises 

Strengthened 
economic 
fabric 

Improved 
employability 

Improved 
productive 
system 

Structural 
balance  of 
employment 

The  logical  diagram  reveals  a  series  of  intermediate  impacts  which  will 
serve  as  a  basis  for  the  overall  evaluation  of  the  programme.  Among  the 
intermediate  impacts,  it  is  necessary  to  look  for  those  which  meet  the 
following  requirements: 

•  they  must  not  be  too  global  to  allow  for  empirical  analysis  based  on 

observations  in the  field; 

•  they  must  be  connected  to  as  many  measures  as  possible  to  allow  for 

comparison  and aggregation  within the  programme. 

The  logical  diagram  of  impacts  can  be  constructed  in  various  ways.  One 
approach  consists  of  constructing  the  objectives  tree,  based  on  programme 
documents,  and  then  of  transforming  the  tree  into  a  network,  based  on 
interviews  with  programme  managers  and  certain  operators.  The  diagram  is 
then presented to the evaluation steering group to be discussed and validated. 

This  approach  is  not  always  satisfactory,  especially  if the  evaluation  team 
identifies  ambiguities  or  inconsistencies  in the  intermediate  objectives.  In 

Concept mapping of impacts 

the  past,  evaluation  teams  would  sometimes  mention  such  problems  to 
argue that the  programme  as a whole was therefore  not  évaluable  and that 
it  wished  to  stop  its  work  at  the  stage  of  the  specific  evaluation  of 
measures. In such situations, where intermediate objectives are not  defined 
clearly  enough, the  evaluation  team  should  not  give  up;  instead,  it  should 
attempt  to  reconstruct  implicit  objectives. 

Box 8 -  Reconstruction  of implicit intermediate  objectives 

Specific 
measures 
and  Impacts 

New  tourist 
products 

Diversification  of 
entreprises 

Development 
of  services 
to  entreprises 

Increased  use  of 
natural  gas 

Creation 
of  technological 
entreprises 

Development  of 
research -enterprise 
contact 

Intermediate 
transversal 
impacts 

Global 
Impact 

Socio-economic 
development 

? 

? 

? 

? 

This  approach  consists  of  constructing  the  logical  diagram  of  impacts 
based,  not  on  the  objectives  tree,  but  on  the  impacts  expected  by  the 
managers  and  operators  of  the  programme.  In  order  to  do  so,  the 
evaluation  team  starts  by  schematically  representing  the  measures,  their 
results, their  specific  impacts  (see the white  boxes  in the  left of  Box  8) and 
the  global  impact  (box  on  the  right).  It  identifies  and  describes  precisely  a 
series  of  expected  intermediate  impacts  (shaded  intermediate  impacts). 
These  should  make  it  possible  to  link  specific  and  global  impacts  through 
the densest  possible  network  of  logical  relations. 

This  approach  was  the object  of  a pilot  evaluation  in the  framework  of  the 
MEANS programme  in 1996. The work  was part of the mid-term  evaluation 

Technical solutions  for evaluation within  a partnership  framework 

of  the  socio-economic  development  programme  In  the  Valencia  region 
(Spain), co-funded  by the regional authorities, the Spanish government  and 
the European Commission. The evaluation team used the concept  mapping 
technique, for  which  instructions  for  use are given  below. 

Instructions for use of the concept mapping of impacts 

The concept  mapping  of  impacts  is an adaptation  of the  technique  known 
as  concept-mapping,  used  for  the  past  ten  years  in  evaluation.  The 
technique was first  used to  help  define the strategies  or the  reorganisation 
of firms, before being applied to  public-policy  making, especially  in the field 
of  social  programmes. The  application  developed  within  the  framework  of 
the  MEANS  programme  was especially  designed for  specifying  the  explicit 
and  implicit  intermediate  objectives  of  a  socio-economic  development 
programme  and for facilitating the choice of  related  indicators. 

The concept  mapping tool is based on collective reflection  and favours the 
diversity  and richness of different  points of view.  Initial ideas are expressed 
and  intuitively  sorted  into  groups  by  the  participants  themselves.  A 
qualitative data processing  program synthesises  individual  representations 
in the form of a collective graphic  image (the concept  map). This technique 
ensures that the evaluation team does  not impose  its own  logical  structure 
which  would  not  necessarily  correspond  to  that  of the  participants. 

Constituting  one  or  more  w o rk  groups 

The technique  may  be  used within the evaluation  steering  group,  provided 
that  the group  has  at  least ten  members.  It  may  also  be  applied  within  the 
framework  of a series of work groups. In this case, it could involve up to two 
hundred  people, as in the following  example. 

Example: Collective  reflection with a  large  number  of  participants 

The  concept  mapping  technique  was  used  in  Northern  Ireland  to  evaluate  a 
programme for reconciliation  between the Catholic  and Protestant  communities. The 
evaluative question  was:  "What  types  of  actions  contribute  best  towards  enhancing 
mutual  understanding  between the communities?". As a result of collective  reflection 
involving  200  people, a large  number  of  proposals  were  made  (e.g.  "support  for  the 
Irish  language  as an integral  part  of our  cultural  heritage"). 

In  a  smaller  meeting,  the  proposals  were  grouped  together  by  theme.  Each 
participant  sorted them  into coherent  piles and gave them a score, between  1 and 5, 
depending  on  their  relevance.  The  groups  thus  formed  by  the  partners  were 
synthesised  in the form of concept  maps. The result was used to group the proposals 
into six categories  which were  named (e.g. "cultural traditions")  and which served to 
articulate the rest of the discussion. 

Technical solutions for evaluation within a partnership framework 

To  constitute  work  groups,  the  participants  can  be  grouped  together  by 
category,  for  example:  a  group  of  programme  managers;  a  group  of 
representatives  of  local  authorities; a group of  company  directors;  etc. 

Example:  Work groups by category of  participant 
The concept  mapping  technique  was  used in the  United  States  in a nursing care 
programme. The aims of the study were to provide managers with information on the 
kinds of nursing care practised, and to review theories and research results on the 
subject. Three distinct groups were formed: a group of nurses, a group of patients, 
and  a group  of  nursing  care  theoreticians.  The  concept  mapping  technique  was 
applied concurrently to each of these groups. 

One  way  of  creating  several  work  groups  is  based  on  the  domains  of 
intervention  of  the  programme.  For  example,  one  group  is  constituted  for 
interventions  concerning 
interventions 
concerning  the  productive  environment,  and a third group for  interventions 
concerning  human  resources. 

infrastructure,  another  one  for 

It  is  preferable  for  each  group  to  have  between  10  and  15  members.  The 
groups  must  be  formed  so  that  their  participants  have  the  same  level  of 
knowledge  about  the  programme,  its  logic  and  its  impacts.  Finally,  the 
participants must  undertake to  be present throughout the group's  meetings. 
Two or  three  meetings  over  a one-  or two-week  period  should  be planned. 
Depending on the circumstance, a part of the work could be organised in the 
form of  virtual  meetings  consisting  of an exchange of  e-mail. 

Identifying  expected  impacts 
in  the  p r o g r a m me  d o c u m e n ts 

The evaluation  team  starts  its  work  by  consulting  all  available  documents 
likely  to  reveal the  potential  impacts of the  programme.  It first  analyses  the 
programming  document, lists the objectives and identifies those  associated 
with  a  description  of  expected  impacts.  The  names  of  measures,  sub-
programmes  or  priority  axes  may  help  it  in this  respect.  Statements,  facts 
or motives  regarding the  programme  rationale may also  inspire the team as 
it attempts to  define what the  programme  is supposed to  produce. 

Apart  from  official  programming  documents,  the  evaluation  team  may 
extend  its analyses to  include the preparatory  work  of the  programme and, 
in  particular,  the  plan drawn  up  by  the  Member  State, the  SWOT  analyses 

Concept mapping of impacts 

and  the justifications  put  forward  for  the  eligibility  of  the  territory  or  target 
group,  in  view  of  the  regulations  governing  the  Structural  Funds.  The 
Community  Support  Framework,  when  relevant,  Is also  a  useful  source  of 
information.  Earlier  evaluation  reports  may  also  be  used,  as  may 
information  from  management  and  monitoring  documents,  such  as 
technical  information  sheets  describing  project  selection  criteria.  Reviews, 
operating  reports,  the  minutes  of  monitoring  committee  meetings,  press 
communiqués  and  official  statements  on the  programme  and  its  expected 
impacts  are similarly  analysed. 

The  analysis  of  these  documents  serves  to  compile  a  list  of  extracts 
describing  or  mentioning  the  effects  that  the  programme  must  or  is 
supposed to produce. If several groups  have been formed to cover  different 
domains  of  intervention,  the  evaluation  team  draws  up  one  list  per  group, 
by  putting  together  all the extracts  likely to  concern  the  different  groups. A 
single extract  may  concern  several  groups. 

Asking for the  programme  actors'  opinions 
on the expected  impacts 

The  second  stage  in  the  evaluation  team's  work  consists  of  a  qualitative 
survey of  between  10 and 20 participants  in the work  groups. The aim  is to 
reveal the implicit objectives of the managers and operators who  implement 
the  programme. The basic  questions  are of the following  kind: 

•  With which  of the  programme  interventions  are you  most  familiar? 

•  For  those  interventions  with  which  you  are  most  familiar,  could  you  say 

which groups  are affected  directly  or  indirectly? 

•  What  is the  nature of the  programme  effects  on these  groups? 

•  Through  which  mechanisms  are these effects  produced? 

•  Should  the effects  produced  be considered  as positive  and  desirable? 

To obtain  the  actors'  opinions  on  programme  effects,  the  evaluation  team 
holds  individual  interviews  either  by  telephone  or  face-to-face.  The  focus 
group technique  may  also  be used. In this  case,  it would  be in the form  of 
an  initial  work  group  meeting.  The  interview  should  not,  however,  be 
structured  nor  make  use of  a questionnaire. The idea is to  enable  as  many 
ideas  as  possible  to  emerge,  by  fully  respecting  the  expressions  of  the 
interviewees  rather than  asking them to fit  into  pre-established  frames. 

Technical solutions for evaluation  within  a partnership  framework 

The  impacts  concerned  are  those  which  the  programme  actors  expect  to 
observe  in the  short  or  medium  term.  These  impacts  may  or  may  not  be 
forecast  in the  programme  documents. They  are generally  positive, but  the 
evaluation  team  must  not  hesitate  to  note  those  which  are  negative.  For 
example,  the  following  two  impacts  could  be  noted:  "assisted  firms 
increase  their  productivity"  and  "jobs  are  lost  in  assisted  firms  due  to 
increased  productivity".  Experience shows that  participants are reluctant  to 
mention  negative  impacts;  evaluation  teams  must  therefore  emphasise  the 
advantages  of  examining 
It  must  also  ensure  that 
these  as  well. 
interviewees do not mention only those impacts which are "easy" to obtain. 

The analysis of  interviews  is concluded  with  a list  of  sentences, taken  from 
the  interviewees'  answers, which  describe the effects  of the  programme as 
expected  by  its  actors.  If  several  groups  have  been  formed,  a  list  of 
sentences  likely to  interest  each  group  is compiled. 

Establishing  an  h o m o g e n e o us  list 
of  statements  on  impacts 

All  the  extracts  of  texts  and  statements  gathered  during  the  two  preceding 
stages  must  be  rewritten  in  an  homogeneous  form  by  the  evaluation  team. 
Thus,  the  team  has  to  reformulate  all  the  elements  collected,  in  short 
sentences  called  "impact  statements". An impact  statement  is always  in one 
of the following forms: a particular group changes  its behaviour in a particular 
way; a particular economic variable is modified in a particular way; a particular 
element  of  the  environment  evolves  in  a  particular  way.  For  example,  if  a 
programme  objective  is to  "reduce the  number  of young  people  leaving  rural 
areas",  the  following  statement  will  be  written  down:  "more  young  people 
settle  in rural towns". The statement  implies that the change  or evolution  is a 
consequence  of the programme,  but this  causal  relation does not  have to  be 
referred  to  each  time.  Thus,  if  an  impact  is  stated  in  the  form  "more  young 
people  settle  in  rural towns",  different  measures  may  have different  levels  of 
success  from  that  point  of  view,  e.g.:  "a significant  number  of  young  people 
settled  and  stayed"  or  "a  few  young  people  settled  exceptionally"  or  "no 
young  people settled". These levels of success  will be discussed  later  in this 
book, where they  are referred to  as "impact  descriptors"  (see page 61). 

A statement always applies to one impact only. For example, if a programme 
objective  is to  "increase the  income of farmers through the diversification  of 
activities",  the  following  two  impact  statements  will  be  made:  (1)  "farmers' 
activities  are diversified" and  (2)  "farmers'  incomes  are rising". 

Concept mapping of impacts 

The evaluation  team  must  bear  in  mind the  aim of  the exercise  which  is to 
record  intermediate  impacts.  At  this  stage,  it  has  to  eliminate  impact 
statements which are too specific and obviously cannot concern  more than 
one  measure.  Similarly,  impacts  which  are  too  global  should  be  deleted 
when  it  appears that they  cannot  be empirically  connected  to  the  different 
measures  of the  programme. 

At  the  end  of  this  stage,  the  evaluation  team  sorts  the  statements  to 
eliminate  those  which  are  obviously  duplications.  It  thus  obtains  a  list  of 
statements  (see the  example  below)  which  must  not  exceed  one  hundred 
items.  If the  list  is too  long,  the  impacts  which  cover  a  particular  measure 
too  specifically  are  put  aside.  If  several  work  groups  have  been formed, a 
list  is compiled  for each group. 

Example: Impacts  stated  by the  managers  of a measure for  direct  aid to  SMEs 

SMEs have better  management  systems 

The qualifications  of craft  enterprise employees  are increasing 

Supply  in the tourism  sector  is becoming  diversified 

A  larger  number  of  firms comply  with environmental  standards 

Firms are replacing  oil-derived  energy  sources  by  natural  gas 

Research centres are developing R&D projects adapted to the needs of regional industry 

Firms are improving  their  production  processes 

Firms are increasing their export  sales 

The financial situation of SMEs  is  improving 

Jobs  are being created  in priority  areas 

Firms are reducing their  production  of  waste 

Validating  the  statements 

The  following  step  is  carried  out  in  a steering  group  meeting  or  in  several 
meetings  if  there  are  several  work  groups.  In the  latter  case,  each  group 
examines the list of  impact  statements  prepared especially for  it. 

The chairperson  of the  meeting  starts  by: 

•  presenting  the  principles  of the  technique 

•  describing the reasoning  behind the grouping together  of the  participants 

•  explaining  what  is  understood  by  an  impact  statement  and  how  these 

statements were  determined 

•  presenting the  list  of  statements. 

Technical solutions for evaluation within  a partnership  framework 

The list  is then discussed  in depth, particularly  around the following  points: 

•  Is  there  a  contradiction  or  not  between  the  statements  drawn  from 
programming, management  and monitoring documents, and those taken 
from  interviews? 

•  Can some statements  be written again so that they  become applicable to 

more  measures? 

•  Are the  statements  understood  correctly  and  in the  same way  by  all  the 

participants?  If  not,  how  can they  be  specified? 

•  Is it possible to combine statements  which  are similar and add nothing in 

terms  of  impacts? 

•  Should  important  statements  be  added? 

Weighting the  statements 

After  this  first  meeting, the evaluation  team  finalises  the  list  of  statements. 
The  statements  are  distributed  randomly  so  that  sequences  of  similar 
statements  are  not  created. The  evaluation  team  prints  a  list  of  numbered 
statements  for  each  participant. 

The work  group  holds  a second  meeting  shortly  after  the  first  one.  If there 
are several work  groups, each of them holds two successive meetings. The 
main  aim  of  these  meetings  is to  grade  the  expected  impacts  in  terms  of 
their  importance  for  the  overall  success  of  the  programme.  The  meetings 
must  be run according  to the following  instructions: 

•  Distribute  the  list of  statements  to  each  participant. 

•  Ask the  participants  to give a score to each statement, depending on the 
the 
importance  of  the  corresponding 
programme  (1  =  relatively  unimportant 
...,  4  =  essential 
impact).  This  scoring  is  also  valid  for  unfavourable  impacts  (4  =  highly 
unfavourable  impact,  ...,  1  =  minor  unfavourable  impact).  The  scores 
must  be  written  on  the  list  of  impact  statements  which  must  then  be 
collected. 

for  the  success  of 

impact, 

impact 

Grouping  together  conceptually  similar  impacts 

The  collective  work  is  continued  with  the  grouping  together  of  impact 
statements  into  several  main  clusters. This  is where  the  concept  mapping 
technique,  in the  strict  sense  of  the  term,  is  applied. The  evaluation  team 

Concept mapping of impacts 

prints, for  each  participant,  a set  of cards with  a statement  and  its  number 
on the same side of  each  card. 

The meeting  is continued  on the  basis  of the following  instructions: 

•  Distribute the complete  series  of  impact  statements to  each  participant. 

Box 9 -  Distribution  of  impact  statements 

•  Ask the participants to sort the cards into several piles, according to their 
similarities.  Each  member  of  the  work  group  will  thus  put  together 
impacts  which, in his or her view, belong to the same cluster  of  impacts. 
The instruction is: "Put those impacts which you think are similar onto the 
same  pile".  Participants  are  free  to  create  piles  as  they  wish. The  only 
constraint  is that  all the cards must  not end up on the same pile and that 
there must  not  be as many  piles as there  are cards. 

•  The participants are asked to explain, on the back of the first card in each 

pile, the  logic  behind the  constitution  of the pile. 

•  The  piles  of  all  the  participants  are  collected,  stapled  together  and  put 

into  an envelope on which  the  name of  the  participant  is written. 

The piles produced by each member of the group are analysed  statistically. 
Clusters of impacts are formed, reflecting  as closely as possible the groups 
constituted  by  the  participants.  This  technique  consists  of  counting  the 
number  of  times  that  two  impacts  are  associated  with  each  other,  and  of 

Technical solutions  for evaluation within  a partnership  framework 

considering  this  number  as  an  index  of  their  conceptual  proximity.  The 
conceptual  distance  is,  in  contrast,  equal  to  the  number  of  times  these 
impacts  were  not  associated. A  square  matrix  is then constructed, with  all 
the  statements  in  lines  and  columns.  Each  cell  shows  the  conceptual 
distance  between 
is  processed  by 
multidimensional  statistical  analysis  which  produces  two  results: 

impacts.  This  matrix 

two 

•  A  concept  map  showing 

impacts  and  reflecting  their  conceptual 

proximity  and distance  as clearly  as  possible  (see Box  10). 

•  A  hierarchical  classification  showing  those  statements  which  are  closest 

together  in conceptual  terms  (see  Box  11). 

The  entire  process  can  be  performed  by  statistical  analysis  software  on  a 
microcomputer.  Several  such  software  packages  exist  on  the  market.  To 
facilitate  the  process,  a  programme  developed  especially  for  concept 
mapping can also  be used (see bibliographic  annex, W.M.K. Trochim, 1989). 

Box 10 -  Concept  map  of  impacts 

<1>SMEs 
management 
systems  have 
improved 

<12> The  financial 
situation  of  SMEs 
is  improving 

<25> Enterprises  are 
reducing their 
production  of  waste 

<7>  Enterprises  are 
replacing  oil-derived 
energy  sources  by 
natural gas 

Based  on  this  map,  the  statistical  analysis  can  be  continued  by  creating 
several  clusters  of  impacts.  Hierarchical  classification  will then  be  used, by 
cutting  the tree at  a given  point.  In the example  in  Box  11, two  possibilities 

Concept mapping of impacts 

are shown: in the first, five impact clusters are created and in the second, six 
impact  clusters  (A,B,C,D,E,F).  In this  example, the division  into five  clusters 
is  preferable  because,  as  the  example  shows,  if  clusters  C  and  D  are 
combined, this group should also be extended to include cluster  E. Once the 
clusters  have  been  defined,  it  is  possible  to  calculate  a  bridging  index  for 
each statement. The bridging index varies between 0 and 1 : a value close to 
one means  that  the  statement  played  an  important  part  in the  construction 
of  the  impact  cluster  to  which  it  belongs; a value  close to  0  means, on  the 
contrary, that  the statement  is not  closely  related to  its  cluster. 

Box  11 -  Hierarchical classification  of  impacts 

Statements 

<FQ 

SMEs  Is improving 

12  The financial  situation of 
37 
1  SMEs management  systems 
15 
have  improved 
28 
45 
18 
6 
8 
40 
39 
4 
25  Enterprises  are  reducing 
7 
Í  0 
19 
23  Enterprises  are  replacing 

their production  of 
waste 

oil-derived  energy 
sources  by  natural gas 

In  the  pilot  evaluation  performed  in  Valencia,  the  "infrastructure"  group 
selected three  impact  clusters, the  "direct  aid to  enterprises"  group  selected 
eight, and the "support for the productive environment" group selected seven. 

If  there  are  few  participants  and  fewer  than  thirty  impact  statements,  it  is 
possible  to  proceed  without  a statistical  analysis  programme,  for  example 
by using a group discussion technique  such as METAPLAN®,  described  in 
Volume 3 of the Collection. This tool serves to  produce a visual grouping  of 
impact  statements, in a context of collective discussion, and a weighting  of 
these statements. This simplified approach was applied in the framework  of 

Technical solutions for evaluation within a partnership  framework 

mid-term  evaluation  of  a  socio-economic  development  programme,  as  the 
following  example  shows. 

Example:  Use  of  the  METAPLAN®  technique  for  creating  impact  clusters 

In the  mid-term  evaluation  of  an  industrial  redeployment  programme  in  France,  the 
first  steering  group  meeting  was  aimed  primarily  at  producing  a  list  of  expected 
impacts. 

The  outside  consultants  responsible  for  drawing  up  the  agenda  and  chairing  the 
meeting  used  the  METAPLAN®  technique.  The  participants  (13  people)  were 
managers  of  measures and  operators. 

The session lasted for about two hours. It revealed the main expected impacts, which 
were then grouped  into five clusters  labelled as follows: 

A -  Reskllling of the  workforce 

Β  - Adaptation  of  training 

C -  Attractiveness  of the  territory 

D  -  Redeployment/Diversification 

E -  Creation of jobs through  local  initiatives. 

Validating the  impact  clusters  obtained 

In  the  concept  mapping  process,  each  work  group  meets  again,  for  a  final 

time,  to  discuss  and  validate  the  impact  clusters  which  appeared  on  the 

map.  The  evaluation  team  would,  in  the  meantime,  have  drawn  up  and 

printed  the  map,  as  well  as  a  list  of  statements  sorted  into  clusters. 

For  each  cluster,  the  participants  proceed  in  the  following  way: 

•  Go  through  the  impacts  in  a  cluster  and  verify  their  meaning. 

•  Reclassify,  where  relevant,  the  impact  statements  which  have  a  low 

bridging  index. 

• 

Identify,  within  the  impact  cluster,  those  statements  which  were  given  the 

greatest  weight  by  the  participants  and  those  which  have  the  highest 

bridging  index. 

•  Discuss  the  logic  behind  the  creation  of  clusters,  if  necessary  by  referring 

to  written  notes  made  by  the  participants  when  they  sorted  the  cards  into 

piles. 

•  Find  a word  or  expression  which  synthesises  all  the  impact  statements  in 

the  cluster,  particularly  those  which  were  given  the  most  weight  and 

which  have  the  highest  bridging  index. 

Concept mapping of impacts 

After  a  discussion  about  each  impact  cluster,  the  work  group  chooses  a 
vaguely  abstract  name,  as  shown  in  Box  12.  Some  participants  may  be 
concerned  about  the fact that,  after  intense collective work, the final  result 
is so  imprecise -  especially  if an effort  has  been made to  clarify  objectives 
considered  vague. In reality, this type  of  criticism  is not  valid.  Whereas  the 
name of a cluster of  impacts  may be abstract, the statements  comprising it 
are very clear  and sufficiently  precise to form  the  basis  of the evaluation. 

Box  12 -  Naming  impact  clusters 

Better  management 

Human  capital 

<1> SMEs  management 
systems  have improved 

<12> The financial 
situation of SMEs is 
improving 

<25> Enterprises 
reducing their 
production of waste 

<7> Enterprises are 
replacing oil-derived 
energy sources by 
natural gas 

Innovation 

Environmental viability 

Work  with  t wo  levels  of  transversality 

An  interesting  variant  was  chosen  in the case  of the Valencia  evaluation. It 
consisted of applying the technique  in two  phases. First, three work  groups 
were  formed,  with  about  ten  local  officials  (operators  and  managers  of 
measures).  A  "transversal"  group  was  then  formed,  with  representatives 
from  the  Commission  and  from  national  and  regional  administrations,  i.e.: 
the vertical  partners  (see Box  13). 

Technical solutions for evaluation within  a partnership  framework 

Box 13 -  Work groups created in Valencia 

Sectoral  groups 

Group A 
"Direct  aid to  enterprise" 

Group  Β 
"Support  for  the  productive 
environment" 

Group  C 
"Infrastructure" 

Transversal  groups 

"Central  authorities"  group 

12 managers from 4  administrative 
institutions  (tourism, trade & industry, 
finance, economy  & enterprise) 

11 managers  from 6  institutions 
(trade, occupational  training, tourism, 
economy,  etc.) 

11 managers  from  5  institutions 
(public  works, transport,  industry, 
environmental  quality, etc.) 

Representatives  of the  Valencia 
Independent  Community, 
representatives  of the Spanish  State 
and desk  officers  from  the  European 
Commission 

For  example,  the  work  group  "Direct  aid  to  enterprise"  consisted  of 
managers  and operators  concerned  by the following  measures: 

•  Investment  support  plan; 

•  Modernisation  of  craft  businesses  and  of their  production  systems; 

•  Programme for  providing  assistance  to  priority  areas; 

•  Venture capital  for  SMEs; 

•  Support  for  energy  savings,  diversification  and/or  the  use  of  alternative 

energy  sources  by firms; 

•  Plan for  promoting  services  to  businesses; 

•  Technological  plan; 

•  Support  for  the  improvement  of  supply  in the tourist  industry; 

•  Support  for  businesses to  comply  with  environmental  standards. 

impact 
Each  of  the  three  sectoral  groups  compiled 
statements.  These  statements  were  systematically  divided  into  two  levels. 
The first  level was  closest  to  the  specific  impacts  of the  measures  and  the 
second  level was closer  to the  global  impact  of the  programme. 

its  own  list  of 

The  lower  level  (closer  to  the  specific  impacts  of  the  measures)  consisted 
of  the  following  types  of  statement:  "the  number  of  entrepreneurial 

Concept mapping of impacts 

initiatives  has  increased",  "new  SMEs  have  improved  management 
industries  have 
systems",  "the  qualifications  of  employees  of  craft 
improved". The higher  level (closer to the global  impact  of the  programme) 
consisted  of  the  following  kinds  of  statement:  "the  number  of  jobs  in 
regional  enterprises  has  increased",  "the  areas which  have  benefited  from 
assistance  have not experienced  population  loss". 

Each  of  the  three  sectoral  groups  worked  only  on  the  impact  statements 
on  the  lower  level.  All  the  impact  statements  on  the  higher  level  were 
grouped  into  a single  list  and  processed  by  the  fourth  group. The  impact 
clusters  thus  formed  were  therefore  arranged  on  two  levels,  as  shown  in 
Box  14. 

Box 14 -  Two levels of  impact  clusters 

Measure A 

Measure  Β 

Measure C 

| Measure J 

I Measure Κ 

Measure  L 

| Measure R 

I Measure S 

Measure Τ 

Impact  cluster 

Using the concept mapping of impacts 

■ Constructing programme evaluation criteria 

The collective reflection on which the concept mapping of impacts is based 
encourages the diversity and richness of different points of view. This 
approach, which takes into account the different partners' opinions, serves 
to produce evaluation criteria readily accepted by the different programme 
partners. The use of computers seems to be a major asset for ensuring that 
the common evaluation reference system is acceptable to all. Thus, the 
technique precludes the evaluation team from imposing its own logical 
structure which might otherwise be foreign to the participants. 

Box 15 - Evaluation criteria common to the different measures 

Impact cluster 

Environmental viability 

Evaluation criterion 

Effectiveness of measures in terms of environmental 
viability 

Evaluative question 

What was the contribution of the different measures to 
the environmental viability of the economic units 
concerned by the programme? 

Example of the 
evaluation method 

Survey of economic units targeted by the different 
measures: businesses, tourist agencies, farms, 
artisans, etc. 

Survey by questionnaire, interviews or case study. 
The questions asked refer to impact statements 
belonging to the environmental viability cluster, i.e.: 
"Has there been a reduction in the production of 
waste?", "Has there been a transfer in energy 
consumption towards natural gas?", etc. 

The main advantage of concept mapping is that it allows for the 
construction of effectiveness criteria which can be applied transversally to 
several measures. For example, by reverting to the step in which the 
clusters are named (see Box 12), the evaluation team could use the cluster 
"environmental viability" as indicated in Box 15. Through this example it can 
be seen that a very simple procedure leads from an impact cluster to an 
evaluation criterion, then to an evaluative question, and finally to a survey 

Technical solutions for evaluation within a partnership  framework 

method  which  will  provide  answers  to  the  questions  in  an  empirical  and 
pragmatic way. Furthermore, it is possible to use the statements  comprising 
each  cluster,  to  construct  scoring  scales. 

The evaluation will check whether the impact  clusters  are really transversal, 
that  is,  whether  they  apply  to  a  large  number  of  measures. The  approach 
can be considered a success if each impact cluster  is potentially  connected 
to  a large  number  of  measures. The  more  connections  there are, the  more 
the evaluation will be able to make comparisons  and aggregations  between 
the  measures,  and  the  more  it  will  be  possible  to  perform  a  synthetic 
evaluation  of  the  programme.  Note  that  the  concept  mapping  technique 
favours  multiple  connections  because  of  the  way  in  which  the  impact 
clusters  are constructed.  It  is  recalled  that  these  clusters  consist  of  all  the 
intermediate  impacts  expected  by the  managers  of the different  measures, 
and that these  impacts  have systematically  been sorted  into groups. 

Construction  of  generic  indicators 

A  generic  indicator  is  an  indicator  which  serves  to  make  a  comparable 
quantification  of  several  different  kinds  of  measures  within  the  same 
programme  (see Volume  2  of  the  Collection).  Such  an  indicator  allows  for 
the  aggregation  of  data  within  the  programme,  in  the  form  of  a  sum  or 
average.  Examples  of  generic  indicators  include  absorption  rate  and 
completion  rate.  Certain  generic  indicators  apply  only  to  a  part  of  the 
programme,  e.g.:  number  of  innovations  (for  all  measures  promoting 
technological  development)  or  proportion  of  SMEs  among  the  addressees 
(for all the  measures  benefiting  business). 

The  more  heterogeneous  the  measures  of  a programme,  the  more  difficult 
it  is  to  define  really  generic  indicators.  If  a  concept  map  is  created,  the 
intermediate  impact  clusters  constitute  an  excellent  support  for  creating 
these  generic  indicators.  A  part  of  the  last  work  group  meeting  can  be 
devoted  to  this  exercise. The discussion  is then  on  each  impact  cluster,  in 
turn,  and follows these  guidelines: 

•  Run through the  impact  statements  comprising  the  cluster. 

•  Ask the  participants  to find  indicators  associated  with these  statements. 

•  Redefine the indicators proposed, so that they are as generic as possible. 
For  example,  the  indicator  "jobs  created  in  assisted  firms"  could  be 
replaced  by  a  more  generic  indicator  such  as  "jobs  created  in  assisted 
economic  units". 

Concept mapping of impacts 

•  Give  priority  to  indicators  associated  with  impacts  which  are  the  most 

heavily weighted. 

•  Select those  indicators that will apply for as many  measures as possible. 

•  2 to  5 indicators  could  be proposed for  each impact  cluster. 

•  Considering  the  importance  of the question  and the time  required for  an 
in-depth  discussion  on  indicators,  it  is  advisable  to  address  the  subject 
after a long  break, or even to  plan a half-day's work  for such discussion. 

Example: Indicators  associated  with  impact  clusters 

Several  Intermediate  clusters  corresponding  to  the  generic  criteria  defined  by  the 
main  group  in  the  MEANS  pilot  evaluation  are  shown  below.  This  evaluation 
concerned  25  measures  of  the  socio-economic  development  programme  in  the 
Valencia  region for  the period  1994-99. 

Impact  cluster 

Generic  indicator 

Information  source 

Programme 

Context 

Better  national 
development 

Use of  urban 
public  transport 

Number  of  users 
per  km of  public 
transport  services 

Transport 
operators 
assisted 

Transport 
operators 
assisted 

Better  productive 
environment 

Improved 
qualifications  of 
the  working 
population 

Operators of 
training  and 
education 
measures 

Density  of 
banking 
institutions 

Statistics  from 
the Transport 
Ministry 

Statistics  from 
the Transport 
Ministry 

National  Statistics 
Institute 

National  Statistics 
Institute 

An  appropriate  technique  for  partnerships 

The concept  mapping  of  impacts  serves to  initiate a group  dynamic  which 
is obviously  an advantage in evaluation. The evaluation will be accepted far 
more readily  by the stakeholders  in the programme  if they  have  collectively 
participated  in defining  a common  evaluation  reference  system.  Moreover, 
interaction  between managers can only enhance their knowledge of actions 
and thus favour the effects  of synergy. The participation  of  managers  in the 

Technical solutions for evaluation within a partnership  framework 

process  favours  a  better  awareness  of  the  effects  which  the  interventions 
are supposed  to  produce,  beyond the outputs  stage. 

The  possibility  of  implementing  the  method  and  obtaining  results  in  a 
relatively  short  lapse  of  time,  such  as  one  week  for  a  series  of  meetings, 
makes  the  concept  mapping  of  impacts  compatible  with  the  evaluation 
schedule.  It does, however, have its drawbacks  which can limit the utility  of 
the technique  under certain  circumstances. 

One of  the drawbacks  of this  method  is that  it mobilises  a large  number  of 
managers  or officials  over  a period of several  days.  Even if some  meetings 
are  replaced  by  an  exchange  of  mail,  this  type  of  project  demands  that 
numerous  officials  devote  time  to  it. Apart  from  practical  problems  related 
to agenda-setting, such  mobilisation  of  public  agents  involves expense  for 
the  community.  It  seems  justified  only  in  so  far  as  significant  stakes 
motivate the evaluation. 

Furthermore  the  creation  of  groups  is  clearly  a  sensitive  issue.  In  case  of 
conflict  or if all the  partners  do  not  agree to  participate  in the process, or if 
it  is not  possible to  achieve a balanced  representation, the  implementation 
of  the  technique 
in  the 
constitution  of  groups  is also  great  when  representatives  of  target  publics 
are  involved  in  the  process.  In  this  case,  identification  and  balanced 
representation  seem  relatively  complex  matters. 

is  compromised.  The  difficulty  encountered 

Finally, the interplay  between actors during group reflection has to  be taken 
into account. The concept  mapping technique  makes  it  possible to  ensure 
that  a  leader  within  the  group  does  not  influence  the  constitution  of  piles 
nor  the  scores  given  to  statements.  This  influence  is,  however,  possible 
during  the  collective  production  of  statements  and  during  discussions  on 
the  impact  map.  It  is then  up to  the  person  chairing the  meeting to  ensure 
that  the  results  are  indeed  those  of  the  group  as  a  whole.  It  is  therefore 
preferable  to  have  a  chairperson  who  is  thoroughly  familiar  with  group 
leadership  techniques,  so  that  the  interplay  between  the  actors  does  not 
influence the results  of the exercise. 

The  fact  that  the  implementation  of  the  technique  requires  adequate 
knowledge  on  the  evaluated  domain  of  intervention  and  on  relevant 
statistics, may cause the project to  be entrusted to a multidisciplinary team. 

Constructing and using scoring scales 

Scoring  systems are being increasingly used for project  selection,  especially 
in  the  United  Kingdom.  Scoring  has  the  advantage  of  taking  into  account, 
with  a  great  degree  of  flexibility,  all  the  quantitative  and  qualitative 
information  needed  to  judge  a  given  criterion.  Its  field  of  application  is 
therefore  wider  than that  of indicators  which  are difficult  to apply  to  certain 
criteria,  due  to  their  quantitative  nature.  The  originality  of  the  approach 
proposed 
in  this  chapter  lies  in  the  fact  that  scoring  is  used  for  the 
evaluation  of measures and not  for the selection  of projects.  This chapter  is 
connected 
to  the  preceding  one  in  so  far  as  a  scoring  scale  can  be 
constructed  for each impact  cluster. It is also related  to Chapter  IV in so far 
as  multicriteria  analysis is  usually  based  on  evaluations  of  impacts  carried 
out  by  means of scoring  scales. 

A solution for evaluating different measures 
in a homogeneous manner 

Qualitative  conclusions  on  the  impacts  of  measures 

Overall evaluation of the measures of a programme aims at examining all the 
impacts  of  all the  measures.  In the  case  of  a socio-economic  programme, 
for  example,  with  about  forty  measures  to  evaluate  in terms  of  five  impact 
clusters  (competitiveness,  employment,  equal  opportunities,  environment, 
and  quality  of  life), the  evaluation  team  will  have to  express  its  opinions  on 
at  least  200  impacts.  This  figure  shows  that  all  these  impacts  cannot  be 
analysed  in  depth.  At  most,  the  overall  evaluation  of  a  programme  can 
include  in-depth  analyses of  a few  particularly  sensitive  impacts. 

In-depth  analysis  of  a  given  impact  requires  observation  in  the  field.  The 
basic  techniques  applied  to  this  type  of  analysis  are,  for  example, 
comparison  groups  and  regression  analysis.  Their  use  requires  time  and 
substantial  resources,  and  the  scope  of  observation  must  be  limited  to  a 
single  category  of  output  and  a single  concerned  group.  In-depth  analysis 
generally focuses on a micro-economic  or micro-social  impact  mechanism. 
An  impact  can  be  analysed  only  at  a  late  enough  stage  for  it  to  have 
materialised  (see Volume  1 of the  Collection). 

All  these  constraints  mean  that  a  genuine  impact  analysis  cannot  be 
extended  to  a  multi-sectoral  programme.  In  such  cases,  pragmatic 
approaches,  such  as  extrapolation  coefficients  for  estimating  impacts, 
must  be  given  precedence.  An  extrapolation  coefficient  will  make  it 
possible to estimate the number of temporary jobs created per million euro 
of  investments  in  infrastructure,  or  the  annual  value-added  generated  per 
million euro of  investments  in SMEs. A coefficient  can  be deduced  from an 
in-depth  survey  carried out  within the framework  of the same evaluation  or 
an  earlier  evaluation.  It  may  also  be  borrowed  from  another  evaluation  or 
research  on  similar  interventions,  provided  that  the  conclusions  are 
considered applicable to other cases. A coefficient  may also be established 
for the  needs of  an evaluation  by one or more experts. 

In situations  of  mid-term  or ex post  evaluation, monitoring  data on  outputs 
can be used as a basis for  extrapolation. Thus, for example, the  number of 
beds  in  a  recently  constructed  tourist  facility  can  be  taken  as  a  basis  for 
estimating  the  number  of  jobs  created,  by  means  of  a  ratio  of  number  of 

Technical solutions for evaluation within a partnership framework 

jobs  per  bed.  In  certain  cases,  the  addressees  are  asked  to  make  a 
statement  on  the  impacts  at  the  time  they  enter  into  contact  with  the 
operators. They  must,  for  example,  indicate the  number  of jobs  created  at 
the  time  the  last  payment  was  made, or  the  number  of  jobs  to  be  created 
is 
when 
recommended  to  use  a  correction  coefficient  to  take  dead-weight  into 
account.  This  coefficient  may  vary  from  0.8,  at  best,  to  less  than  0.3  for 
certain  interventions  where there  is substantial  dead-weight. 

their  application 

for  assistance 

In  this  case, 

is  filed. 

it 

Impacts  can  be  estimated  by  extrapolation  in certain  conditions  only:  (1) it 
is  necessary  to  have  an  extrapolation  base  in  the  form  of  indicators  for 
monitoring  outputs  and  results;  (2)  the  impact  to  be  estimated  must  be 
defined  precisely  and  must  be associated  with an indicator  and (3) existing 
literature  or  previous  evaluations  must  provide  ratios  such  as  "impact  / 
output"  which  can  be  generalised.  These  conditions  are  relatively  severe 
and  are  rarely  met  for  all  impacts  and  outputs. The  most frequent  situation 
is therefore that  in which the evaluation team cannot  make an estimation, in 
figures, of the totality of impacts  of all the measures.  If it is essential for the 
evaluation  to  give  an  overview  of  programme  effects,  it  will  be  able  to 
provide  only qualitative  conclusions  for  at  least  part  of the  impacts. 

Even  when  an  extrapolation  approach  serves  to  produce  quantitative 
conclusions,  estimations  of  impacts  need  to  be  refined  with  more 
qualitative  interpretations.  In  particular,  the  evaluation  team  must  explain 
and  justify  the  extrapolation  coefficients  it  has  chosen,  by  considering  all 
the  context  elements  which  condition 
the  process  of  programme 
implementation,  characterise  the  concerned  groups  and  influence  the 
performance  obtained.  Evaluation  professionals  generally  recognise  that 
indicators  are  only  one  of  the  bases  among  many  enabling  them  to 
formulate  their  conclusions.  In particular,  indicators  must  be  interpreted  in 
light of all the accessible  and  relevant qualitative  information. 

To  conclude  this  rapid  overview  of  impact  evaluation,  it  should  be 
recognised that  it often  produces  qualitative  conclusions. 

Synoptic  presentation  of  qualitative  conclusions 

By writing up the qualitative conclusions on the impacts of a given measure, the 
evaluation team naturally places itself in the perspective and the context of that 
measure. The conclusions  therefore  tend to  be written  within  a logic  and  in a 
vocabulary peculiar to a specific measure, as the following example shows. This 

Constructing and using scoring scales 

creates  problems  in  writing  the  chapter  summarising  the  evaluation  of  the 

programme  as a whole. As a result, evaluation  reports tend to  have  a section  on 

each  measure,  containing  sound  and  well  argued  conclusions,  but  a  final 

synthesis  which  is short,  poor  and  without  solid  foundation. 

Example: Qualitative  heterogeneous  conclusions  for  different  measures 

Conclusion  on the impact  of  Measure A (training for the long-term  unemployed):  "On 
the  basis  of  an  examination  of  the  post-training  trajectories  of  a  few  trainees 
considered  typical,  it  is  estimated  that  employability  has  been  improved  in  the 
majority  of  cases,  with  better  results  in  tourism-related  occupations  and  less 
satisfactory  results in environment-related  occupations (the latter account  for a small 
proportion  only  of  the training  courses).  On the  basis  of  evaluations  of  comparable 
programmes,  the  placement  rate  can  be  estimated  at  between  30  and  50%.  A 
positive  impact  has  been  identified  in terms  of  the  reorientation  of  women  towards 
more secure jobs traditionally  occupied  by  men. Yet a significant  risk of  substitution 
and skimming-off  effects  exists". 

Conclusion  on the  impact  of  Measure  Β (assistance for  new  business  creation):  "On 
the  basis  of  the  evaluation  of  comparable  programmes,  the  survival  rate  can  be 
estimated  between  20  and  30%.  It  seems  that  the  survival  rate  is  higher  for  firms 
created  in  the  environment  sector,  to  which  35%  of  the  assisted  firms  belong. 
Moreover,  it  is  considered  that  the  majority  of  entrepreneurs  who  failed  in  their 
undertaking, substantially  improved their  employability.  It is to  be  noted that  45%  of 
the  new  businesses  assisted  were  created  by  women,  as  opposed  to  a  national 
average of  30%. The difference  can be imputed to the  programme". 

To facilitate  the  preparation  of  a  final  synthesis,  it  is  useful  to  summarise  all 

the  qualitative  conclusions  in  the  form  of  a  synoptic  table  organised  in 

relation  to  the  different  impacts  (see  Box  16). 

Although  the  synoptic  table  is  helpful  for  the  evaluation  team,  a  lot  of  work 

is  still  required  to  produce  genuine  synthetic  conclusions.  In theory,  at  least 

three  approaches  need  to  be  undertaken: 

•  First,  synthetic  conclusions  have  to  be  produced  for  each  impact  cluster. 

For  example,  the  evaluation  team  can  identify  the  measures  which  are 

most  effective  in  terms  of  employment  impacts,  as  well  as  those  which 

have  an  insignificant  or  detrimental  effect  in  this  respect,  and  so  on  for 

the  other  impacts.  This  first  approach  is  developed  in the  following  pages 

devoted  to  scoring  scales. 

•  Secondly, 

the  evaluation 

team  must  analyse  synergy  between 

the 

measures.  For  example,  it  could  be  shown  that  the two  measures  in  Box  16 

Technical solutions for evaluation within  a partnership  framework 

have  impacts  which  mutually  reinforce  one  another  in  the  environmental 

domain. This second  phase  is developed  in  Chapter  III, in connection  with 

the  matrix  of  cross  impacts  . 

Finally, a general synthesis covering the main impacts and synergy  between 

the  measures  must  be made. At the  end of this third  phase, the  evaluation 

team  should  be able to  classify  the  measures  in terms  of  their  contribution 

to the overall  effectiveness  of the  programme. This  point  will  be  developed 

in Chapter  IV, in  relation to the multicriteria  analysis technique. 

Box  16 -  Qualitative  synoptic  table  of  measures  and their  impacts 

Measures 

Impact on 

Impact on 

Impact on 

employment 

equal  opportunities 

the  environment 

Training 

Employability was 

The training  had a 

A small  proportion of 

intended for 

improved in most cases 

positive  impact in 

the training courses 

the  long-term 

unemployed 

terms of reorientation 

resulted in 

On the  basis of the 
evaluation of 

of women towards 
secure jobs 

qualifications in 

occupations in the 

comparable 

traditionally  occupied 

environment sector. 

by  men. 

programmes, the 
placement  rate was 

estimated at between 

30 and  50%. 

There is a considerable 

risk of substitution and 

skimming off effects 

appearing. 

However, very few 

trainees found jobs 

corresponding to 

these qualifications. 

Aid for new 

business 

creation 

On the  basis of the 
evaluation of 

45% of the new 

businesses  assisted 

35% of the 
businesses were 

comparable 

were created by 

created in the 

programmes, the 
survival rate can be 

women, as  opposed 
to a national average 

environment sector. 

estimated at between 

of 30%. The difference 

It seems that their 

20 and  30%. 

can be imputed to the 

programme. 

It is considered that the 

majority of business 

creators who fail, 

nevertheless 

significantly  improve 
their  employability. 

survival  rate is 
higher than the 

national average. 

Etc 

Constructing and using scoring scales 

The present chapter  is devoted to the first of these three approaches, i.e. to 
the writing  up of synthetic  conclusions  on the different  impact clusters. The 
idea  is  to  formulate  comparative  judgements  between  the  different 
measures,  transversally,  throughout  the  entire  programme.  The  proposed 
technique is the scoring scale. It is a technique which produces  conclusions 
that  naturally 
lend  themselves  to  comparison,  aggregation  and  the 
calculation  of  averages.  Although  scoring  scales  are  not  comparable 
between  different  impact  clusters,  the  technique  does  help  directly  in  the 
writing  of the final  synthesis  of the evaluation. 

Instructions for using scoring  scales 

Scoring  scales  are  used  in  many  domains,  often  spontaneously  in  a 
relatively  informal  way.  As  a  result,  the  technique  is  often  criticised  for  its 
subjectivity.  In  some  cases,  however,  rating  is  rigorously  established  and 
plays a part recognised by all. Well-known examples  are international  rating 
agencies  which  provide  assessments  on  creditworthiness,  and  the 
economic  role of ratings in gastronomic  guides. In the domain of education, 
health or social integration policy, hundreds of scoring scales developed for 
research  purposes  are used  in evaluation. 

Scoring  is  still  used  infrequently  in  the  evaluation  of  socio-economic 
development  policies.  A  noteworthy  exception  is,  however,  development 
aid where the technique  has been used  by numerous funding  bodies  since 
the  mid-1980s (see the example on page 59). The following  pages  propose 
instructions  for  using  this  technique  in  the  framework  of  European  Union 
socio-economic  programmes.  Three  variants  of  these  instructions  are 
proposed, depending  on the time the scoring  scale  is established: 

Variant  1:  the  evaluation  team  has  already  formulated  qualitative 
conclusions  for  each  measure;  the  scoring  scale  helps  to  formulate  a 
synthetic  conclusion  at the  programme  level. 

Variant  2:  the  evaluation 
team  has  already  collected  qualitative 
information  on the effects  of the  measures, but  has  not  yet written  up  its 
conclusions;  the  scoring  scale  helps  to  formulate  conclusions,  measure 
by  measure, in an homogeneous  way that  is easy to  synthesise. 

Variant  3:  the  evaluation team  starts  working; the  use of  a scoring  scale 
in 
formulate 
homogeneous  conclusions  that  are easy to  synthétise. 

the  data  collection  process  makes 

it  possible 

to 

Scales  for  synthesising  qualitative  conclusions 

In  the  first  variant,  the  evaluation  team  has  already  formulated 
its 
conclusions. These  are  presented  in the  form  of  the  synoptic  table  in  Box 
16,  in  which  each  column  concerns  an  impact  cluster  and  contains  the 
conclusions  drawn  on  all  related  measures.  Box  17  is  extracted  from  the 
synoptic  table with  regard to  impacts  on  equal  opportunities. 

Are  the  two  impacts  described  in  Box  17  comparable?  Could  one  say, 
when  reading  these  conclusions,  that  Measure  1  is  the  most  effective  in 

Technical solutions for evaluation within a partnership  framework 

terms  of  its  impact  on  equal  opportunities?  At  this  stage  of  its  work,  the 
evaluation  team  notes  that  the  impacts  are  not  directly  comparable  and 
that  it cannot  go  much further,  on its own, towards a synthetic  conclusion. 
A possible way of formulating comparative judgements would be to get the 
steering  group  or an  expert  panel  to  express  an opinion. The  approach  is 
as follows: 

•  Organise  a  half-day  seminar  with  an  expert  panel  or the  steering  group. 
The steering  group should  preferably  be extended, to form  an  evaluation 
committee, as  proposed  in Volume  1 of the  Collection. 

•  Present  the  evaluation  conclusions  to  the  participants,  measure  by 

measure. Verify the credibility  of the conclusions  and validate them. 

•  Present  the  synoptic  table  of  conclusions  (see Box  16). 

•  Examine  the  first  column  of  the  synoptic  table  and  ask  each  participant 
to  intuitively  situate  the  impacts  of  each  measure  on  a scale,  as  in  Box 
18.  Compare  the  participants'  classifications,  discuss  them  and  make 
them converge  as  much  as possible. 

•  Continue the classification  with the  other  columns  in the synoptic  table. 

•  After the seminar, convert the classifications  into scores ranging from  -10 
(maximum  negative impact) to +10 (maximum  positive  impact), through 0 
(neutral  impact).  Convert  the  synoptic  table  in  Box  16  into  a  table  of 
ratings (impact  scoring  matrix). The construction  of scoring scales  allows 
for  comparisons  to  be  made within the  same column  (e.g. one  particular 
measure  has a  better  rating than  another). On the  other  hand, scores  in 
different  columns cannot  be compared  (e.g. a score of  5 for  employment 
is not  comparable  to the  same  score for  an environmental  impact). 

Box 17 -  Synoptic  presentation of conclusions relative to an impact 

1 

2 

Measure 

Description  of the  equal 
opportunities  impact 

Training  intended 
for the  long-term 
unemployed 

The training  had a positive  impact  in terms  of 
orientating  women  towards  secure jobs 
traditionally  occupied  by  men 

Aid for  new  business 
creation 

45%  of the  new  businesses  were created  by 
women,  as compared  to the national  average 
which  is 30%. The difference  can  be imputed  to 
the  programme. 

Other  measures... 

Etc.... 

Constructing and using scoring scales 

Box 18 -  Classification of impacts obtained on an intuitive scale 

Measures 

Impact  in terms 
of equal  opportunities 

Support  for  the 
creation  of  new 
business 

45%  of the  new  businesses  were 
created  by women,  compared 
to the  national average of 30%. 
The difference  can be  imputed 
to the  programme. 

Training  for the 
long-term 
unemployed 

The training  had a positive  impact 
in terms  of the  reorientation  of 
women  towards  more secure jobs 
traditionally  occupied  by men. 

Etc.. 

Maximum 
positive 
impact 

Neutral 
impact 

Maximum 
negative 
impact 

In a pilot  evaluation  in the  MEANS  programme, a technically  more  rigorous 
procedure  was  proposed  and applied. In this variant, the  participants  were 
not expected to intuitively place the impacts on a scoring scale as in Box  18 
above. They were asked rather to express their preferences between all the 
impacts, considered  in pairs. Their opinions were gathered through  a series 
of  questions  such as:  "which  of these two  situations  do  you  prefer: 

•  a situation  in which a positive impact  is observed in terms of  reorientation 

of women  towards  secure jobs  traditionally  occupied  by  men 

or... 

•  a  situation  in  which  it  is  observed  that  a  greater  proportion  of  women 
have created  their  own  business  than that  observed  throughout  the  rest 
of  the  country  (45%  of  the  assisted  new  businesses  were  created  by 
women,  compared  to  30%)....  ?  -  As  far  as  the  situation  you  prefer  is 
concerned,  is your  preference very  strong, strong  or weak?". 

The  questioning  is  continued  for  all  the  impacts  in  the  first  column, 
compared  in  pairs,  and  then  for  the  other  columns.  All  the  answers  are 
processed  by  a computer  program  which  attributes  a  score  between  -10 

Technical solutions for evaluation within a partnership  framework 

and  +10  to  each  impact,  so  as  to  reflect  the  participants'  preferences  as 
closely  as  possible.  This  software,  called  "Macbeth",  was  developed  by  a 
team  at  Lisbon  University  (see the  annexed  bibliographic  references:  Bana 
e Costa,  1994). This  software  is also  mentioned  further  on (see  page  107), 
in connection  with one of the variants  of  multicriteria analysis. 

Whatever  the  procedure  opted  for,  the  final  result  consists  of  a  series  of 
scoring  scales  (one  per  impact  cluster).  Each  measure  is given  a score  on 
each scale. All the scores can be combined  in an impact scoring matrix (see 
Box  19)  which  complements  the  synoptic  table  of  qualitative  conclusions 
(see  Box  16)  and  has  the  same  structure.  The  combination  of  qualitative 
conclusions  and  scores  makes 
transversal 
conclusions  on a programme-wide  scale, for  each  impact  cluster. 

it  possible  to  formulate 

Box 19 -  Impact rating matrix 

Measures 

Employment 
impact 

Impact  on  equal 
opportunities 

Environmental 
impact 

Training  intended  for 
the  long-term 
unemployed 

Aid for  new 
business  creation 

Etc 

+4 

+8 

+3 

+7 

0 

+5 

Scales  for  the  formulation  of  structured  conclusions 

As far as possible, it  is preferable to  construct  scoring  scales a little  earlier 
in the evaluation process, i.e. before the formulation of conclusions for each 
measure.  The  evaluation  team  can  then  use  the  scoring  scales  to  write 
conclusions  which  are  more  structured  and  more  easily  comparable 
between  measures. When  used for  evaluation, scoring scales are generally 
designed  in this  way. 

In  certain  cases  (see  the  example  of  the  World  Bank,  below),  standard 
scoring  scales  are  made  compulsory  in  all  evaluations  concerned  by  a 
particular  policy.  However,  this  approach  is  difficult  to  apply  to  socio­
economic  programmes  financed  by  the  European  Union.  Because  the 
different  programmes  are co-financed  in partnership  with  different  national 
and/or  regional  authorities,  it  hardly  seems  legitimate  to  standardise  the 
ways  in which  their  impacts  are judged. 

Constructing and using scoring scales 

Example: Scoring  scales  used by the World  Bank 

The  World  Bank  performs  about  200  project  evaluations  annually.  All  these 
evaluations  have  qualitative  conclusions  and conclusions  in  the form  of  ratings, 
structured  according  to  a standard  system.  The scoring  scales  used  include  five 
dimensions  of  project  performance:  economic  impact,  sustainability,  impact  on 
financial  performance  of  the  bank,  and  financial 
institutional  development, 
performance  of the borrower. All the projects  are evaluated by their  managers  (self-
evaluation)  at  their  conclusion,  by  means  of  4-tiered  ratings  for  each  dimension 
("highly  satisfactory",  "satisfactory",  "unsatisfactory"  and "highly  unsatisfactory"). In 
many  cases, and particularly  in cases  of doubt  about  the self-evaluation, a second 
rating  is  performed  by the  Bank's  Operations  Evaluation  Department  (OED).  This 
second  rating  is always  discussed  with  the project  managers.  If the judgements do 
not  converge  (and this is very  rarely the  case) an arbitration  commission  is asked to 
decide  on  final  ratings.  The  World  Bank,  unlike  other  organisations,  does  not 
calculate a weighted  global  score to synthesise all the dimensions of performance. 

In  other  cases,  the  evaluation  is  performed  by  an  expert  panel  which 

constructs  its  own  scoring  scales  (see  the  British  example  below).  The 

members  of  the  panel  receive  all the  information  collected  on  the  different 

measures,  and  formulate  qualitative  conclusions  and  conclusions  in  the 

form of scores. In order to do so, they  agree on  one or more scoring  scales 

established  during  the  first  panel  meeting.  Each  panel  constructs  its  own 

scoring  system.  Expert  panels  are  seldom  used  in  the  evaluation  of 

European  Union  socio-economic  programmes. 

Example: Rating  by expert  panels 

Rating by expert panels is an approach that has been used for the past decade in the 
U.K. for evaluating the performance of university research centres. In this country, the 
work of all research centres is monitored quantitatively  and evaluated  periodically by 
expert  panels.  In  their  evaluation,  the  experts  take  into  account  indicators (e.g. 
number of publications) as well as purely qualitative elements (e.g. quality of the four 
best publications of each researcher). They thus  rate research teams (on a scale of 1 
to 5) which  serves as a direct  reference for the allocation of public  funds. 

In the domain  of  European  cohesion  policy,  an interesting  practice  exists  in 
the  U.K.,  in  the  selection  of  projects  for  funding.  The  following  example 
presents  the  technique  used  in  the  Highlands  and  Islands  region.  This 
example  is  interesting  in  so  far  as  the  scoring  scales  are  constructed  in 
partnership  and  are  used  throughout  the  life  of  the  programme.  Many 
British  programmes  use similar  scoring  scales. 

Technical solutions for evaluation within a partnership  framework 

Example  : Scoring  scales for  project  selection 

Within the framework  of the Highlands and Islands regional development  programme, 
two  series  of  calls  for  projects  take  place  each  year.  All the  applications  for  funding 
are first examined in terms of their eligibility. The projects considered eligible are rated 
on  eight  scales  common  to  the  entire  programme  (relationship  with  objectives; 
probable  effectiveness;  respect  for  the  environment;  leverage  effect;  innovation; 
relations  with  other  projects;  intervention  of  the  private  sector;  and  support  for  local 
partners). Other scales  are also used  but are more specific to certain  measures. 
Each  project  is  rated  on  each  of  these  scales.  A  weighting  system  completes  the 
procedure,  so that  a global  score can  be given to  each  project. After  comparing  the 
applications for funds with available resources, an admission  standard  is established 
and  a  list  of  projects  to  finance  is  recommended. The final  decision  is taken  by  the 
programme  managers. 

Since  scoring  scales  are  used  frequently  and  satisfactorily  for  project 

selection,  several  recommendations  have  been  made  to  extend  the  use  of 

this  technique  to  programme  evaluation.  Proposals  generally  suggest  the 

creation  of  scales  concerning  the  main  impacts  of  the  programme,  and  the 

use  of  these  scales  to  rate  measures  rather  than  projects.  This  kind  of 

approach  was  applied  to  the  transversal  evaluation  of  over  one  hundred 

socio-economic  development  measures  in  the  Republic  of  Ireland.  It  was 

used  to  compare  measures  and  to  formulate  recommendations  on  the 

reallocation  of  budgets,  of  which  a  large  number  were  accepted. 

Example: Scoring  scales for  evaluating  the  relevance  of  measures 

The  mid-term  evaluation of  a set  of  socio-economic  programmes  in the  Republic  of 
Ireland  concerned  about  one  hundred  measures.  For  each  measure, the  evaluation 
team tried to  ascertain  whether  its financing  by  public  funds  could  be justified. Four 
categories  of  measures  were  established,  corresponding  to  the  four  forms  of 
justification. For each category the evaluation team created a scoring scale  enabling 
it  to  assess  several  degrees  of  justification.  Each  scale  was  focused  on  a  typical 
measure,  an anchor  measure,  which was  used  as a benchmark  for  other  measures. 
All the  measures were then  rated, mainly  by  means  of the  qualitative  data  collected 
beforehand. The rating made  it possible to  identify  about twenty sensitive measures, 
likely to  be  concerned  by  important  budgetary  decisions. These sensitive  measures 
were evaluated  in depth.  Finally, the evaluation  team  recommended  that  a series  of 
measures be either abandoned or strengthened. In a significant  number of cases, the 
decisions  taken were  in line with  the  recommendations. 

Constructing and using scoring scales 

This  example  applies  to  an  evaluation  of  the  relevance  of  measures.  The 
creation  of  scoring  scales  for  impacts  can  be  inspired  directly  by  it.  The 
procedure leads to the creation of scoring scales such as the one presented 
in Box  20. It is used  in the following  way: 

•  The  evaluation  team  makes  an  initial,  rapid  recapitulation  of  the 
information  collected  and  the  analyses  performed.  It  identifies  a  few 
impact  clusters,  such  as  impacts  on  employment,  equal  opportunities, 
the environment,  etc. 

in  such  a  measure  benefited  from 

•  For  each  impact  cluster,  the  evaluation  team  selects  sentences 
describing  the  different  aspects  of  the  observed  impacts,  such  as:  "the 
women  who  participated 
the 
reorientation  towards  secure  jobs  traditionally  occupied  by  men"  or  "a 
certain  measure  had  a  positive  impact  for  its  addressees  and  the 
proportion  of  women  who  benefited  from  this  impact  was  particularly 
high".  The  evaluation  team  rewrites  these  sentences  as  generally  as 
possible  so that they  can  apply to  most  of the  measures.  Each  sentence 
clearly and precisely describes a situation considered successful in terms 
of  its  impact.  These  sentences  are  called  "impact  statements".  For  a 
more  rigorous  reconstruction  of  impact  statements  and  clusters,  a 
concept  map  of  impacts  can  be used. 

•  Each  statement  describes  a  situation  considered  successful  from  the 
point  of  view  of  its  impact.  In reality,  certain  situations  will  reveal  partial 
success, an absence of impacts or even counter-performance. To be able 
to  treat  all  these  different  possible  cases,  the  evaluation  team  must 
describe  several  possible  levels  of  success.  Scales  of  impacts  with 
several  levels  between  0  (no impact)  and  10 (excellent  impact)  can  thus 
be  constructed. The  sentences  describing  the  different  levels  of  impact 
are called  "descriptors".  For each  impact  statement,  several  descriptors 
for  positive  impact  levels  must  be  written.  Negative  impact  levels  must 
also  be provided  for. 

•  The  evaluation  team  then  groups  together  on  the  same  scale  all  the 
descriptors  concerning  an  impact  cluster,  as  shown  in  the  example  in 
Box  20.  If  the  same  scale  refers  to  several  statements,  the  range  of 
variation  of  each  statement  is  reduced  (for  example,  from  -10/+10  to 
-5/+5),  so that  all the scores  of  the  same  sheet  are always  between  -10 
(case  in which  all the  most  unfavourable  descriptors  are  chosen)  to  +10 
(case where  all the  most favourable  descriptors  are chosen). 

•  A  half-day  seminar  is  then  organised  with  an  expert  panel  or  with  the 

enlarged  steering  group. 

Technical solutions for evaluation within a partnership framework 

The evaluation team presents its work to the participants, starting with the 
first impact cluster and continuing through to the last. For each impact 
cluster, all the statements and descriptors are discussed and improved, 
until they are understood without any ambiguity. The evaluation team 
presents its proposed scoring scale. This scale is discussed by the 
participants and improved until it correctly reflects their opinion. 

Box 20 ­ Scoring scale consisting of impact descriptors 

First Choosing 

impact —»­ a 
statement descriptor 

Reorientation of women towards 
secure jobs traditionally occupied 
by men 

+5 
(maxi) 

Intermediate levels 

Maintenance of women in 
threatened jobs traditionally 
occupied by them 

­5 
(mini) 

Positive impact for 
the addressees; a particularly 
large proportion of women 
benefited from this impact 

+5 
(maxi) 

Second 
impact 
statement 

Choosing 
■a 
descriptor 

Intermediate levels 

Positive impact for 
the addressees but far too 
few women benefited from 
this impact 

­5 
(mini) 

Total score for the impact 

concerning equal opportunities  between ­10 and +10 

After the seminar, the evaluation team once again summarises the 
information collected and analysed. For each measure, it chooses the 
descriptor(s) on the scoring scales which seem(s) to reflect its impacts 
most accurately. This enables it to rate the impacts of the measure (see 
Box 21). The descriptors chosen do not need to be adjusted very 
precisely to fit the conclusions that the evaluation team thinks it has to 
draw from its observations. Apart from the rating, the team also writes up 
complete qualitative conclusions, which gives it full latitude for qualifying 

Constructing and using scoring scales 

its  statements.  The  main  advantage  of  scoring  scales  is  to  provide  a 
framework  for  studying  all the  measures  according  to  an  homogeneous 
structure  which  will facilitate the writing  of  synthesised  conclusions  on  a 
programme-wide  scale. 

Box 21 -Evaluation  of a measure  by means of a scoring  scale 

relative to  an expected  impact 

First 
impact 
statement 

Choose 

-»-  a 

descriptor 

Second 
impact 
statement 

Choose 
a 
descriptor 

Reorientation  of  women 
towards  secure jobs 
traditionally  occupied  by  men 

+5 
maxi) 

Descriptor  chosen 

+2 

Maintenance  of women in 
threatened jobs  traditionally 
occupied  by them 

-5 
(mini) 

Positive impact  for 
the addressees; a  particularly 
large proportion  of  women 
benefited from the  impact 

+5 
(maxi) 

Descriptor  chosen 

+3 

Positive impact  for 
the addressees  but far  too 
few women  benefited 
from the  impact 

-5 
(mini) 

Total score for the  impact 
concerning  equal  opportunities 

+5 

The  evaluation  team  thus  obtains  a  synoptic  table  of  the  qualitative 
conclusions  and  an  impact  scoring  matrix,  in  precisely  the  same  form  as 
those  described  above  (see  Box  16 and  Box  19). The  use of  this  matrix  is 
described  later  in this section. 

From  scoring  scales  to  indicators 

If  multiple  scoring  scales  are  constructed  specifically  for  the  needs  of  the 
overall evaluation  of  a programme  and  are used  only  once, they  should  be 

Technical solutions  for evaluation within  a partnership  framework 

as  light  as possible; that  is to say, there should  be very few  descriptors  per 
scale.  A  reasonable  number  of  descriptors  could  be  considered  to  be 
between four  and  eight. 

The  same  scoring  scale  can  be  used  on  several  occasions,  as  in  the 
following  situations, for  example: 

•  A  scoring  scale  was  created  for  a  mid-term  evaluation  of  measures  and 

was subsequently  to  be  used  in the framework  of  project  selection. 

•  A scoring  scale was created for the mid-term  evaluation  of measures  and 

was subsequently  to  be  used again for ex  post  evaluation. 

•  A  scoring  scale  was  created  for  the evaluation  of  a programme  and  was 

to  be  used again for the  evaluation  of  another  programme. 

If  a  scoring  scale  is  to  be  used  several  times,  it  warrants  more  precision. 
The  scale  in  Box  21  could, for  example,  be  increased  from  four  to  twelve 
descriptors,  to attain the detail  shown  in  Box 22. 

Box 22 -  Refining  and  harmonising  scoring  scales 

What  is the proportion  of women among 
the  participants? 

What  proportion of the women 
participants obtained  a qualification  for 
a job traditionally  occupied  by men? 

What  proportion of the  women 
participants enhanced their employability? 

What  proportion of the  women 
participants found a secure job 
corresponding  to their  qualification, 
whithin one year? 

Many 

0 

Few 

1.5 

None 

Η 

2 

1 

0 

E 

Total score for the impact  concerning 
equal  opportunities 

=  5.5 

An  additional  step  can  be taken  when  the  scoring  scale  has  already  been 
applied at  least two  or three times. At that  stage, it  is possible to validate it, 
to  stabilise  the  definitions  and  to  use  it  in  a  questionnaire  survey  for 
indicators.  It  will  also  allow  for  comparisons 
quantifying  harmonised 
between  different  measures  or  programmes, as shown  in Box 23. 

Constructing and using scoring scales 

Box 23 -  From scoring scales to  indicators 

Evaluated 
measure 

!  Average of 
!  comparable 
measures 

Proportion  of women  participants 

72% 

45% 

Proportion  of women  participants  who  obtained 
a qualification for  a job traditionally  occupied  by 
men 

5% 

23% 

Proportion  of women who  enhanced  their 
employability 

52% 

55% 

Proportion of women  who  found a secure job 
corresponding  to their  qualification, whithin  one 
year 

25% 

29% 

Using ratings for synthesising the evaluation of measures 

Overview  of  programme  impacts 

Box  2 4-  Example  of an  impact  scoring  matrix  in the  case  of  an  urban 

development  programme 

Contribution  of the  measures 
to the production of  expected 
impacts 
(score between  0 and 10) 

Social 
integration 

Employ­
ment 

Urban  development 
(average score for 
measures 1 and 2) 

Infrastructure 
(average score for 
measures 3  to 8) 

Collective social  actions 
(average score  for 
measures 9  to 14) 

Personalised  social  actions 
(average score  for 
measures  15 to 19) 

1 

5 

6 

6 

1 

4 

5 

6 

Balance 
between 
towns in 
a region 

Balance 
between 
neighbour­
hoods 
within 
a town 

1 

4 

3 

2 

2 

3 

3 

2 

The scoring scale technique has the advantage of giving an overview of the 
programme in a summarised form which provides a sound basis for  writing 
synthetic  conclusions.  For  example,  the  mid-term  evaluation  of  an  urban 
development  programme, carried out  by means of case studies,  produced 
large  numbers  of  qualitative  conclusions  concerning 
impacts.  To 
synthesise  these  conclusions,  the  evaluation  team  constructed  and  used 
scoring  scales.  Box  24  shows  how  all  the  qualitative  conclusions  were 
synthesised  in the  form  of  an  impact  scoring  matrix.  In such  a matrix,  the 
measures  (or, here, the  groups  of  measures)  can  be  compared  column  by 
column.  It  could  be  said,  for  example,  that  a  particular  measure  is  more 
successful  than another  one in terms  of  its employment  impact,  but that  it 
is less successful for  another  criterion. On the other  hand, this matrix  does 

Technical solutions for evaluation within a partnership  framework 

not  allow  for  additions  or comparisons  between  columns. A score of 5  in 
the first column cannot  be compared and certainly  cannot  be added to the 
score of 5 in the last column.  For reasoning  or judgements  which  combine 
several columns, it is  necessary  to  use the  multicriteria  analysis  technique 
(see Chapter  IV in this Volume). 

Judging  the  coherence  of p r o g r a m m es  in  view 
of  the  impacts  obtained 

In the example of the urban development  programme  presented  in Box 24, 
the  impact  scoring  matrix  was  also  presented  in the  form of a diagram, to 
visually  show  the  impacts  and their  complementarity.  This diagram,  shown 
in Box  25, is an  important  step forward  compared to the one that  could  be 
drawn up at the start of the programme (see Box 7) in so far as it shows the 
importance  of  impacts  actually  obtained,  as opposed  to  the logic  of 
expected  impacts  only. 

Box 25 -  Contribution  of measures to the  production  of  impacts 

Contribution 
thickness of the arrows 
proportional  to the scores 

LJlJ'.Hii^i-; 
^ — ts 
p 

^ ^ ^* 

^^^^^\s^" 
*s^ 

^^^-~~^~^ 

Balance between 

towns in 
t he region 

^ ^ ^ ~" 

^^ 

Balance  between 

^ V ^ ^ ^ ^ ^ ^ "^ 

^

-* 

in the  town 

Measures 

Measures  1 and 2 
Urban 
renewal 

Measures 3 to 8 
Infrastructure 

Measures 9 to  14 
Collective  social 
actions 

Measures  15 to  19 
Personalised  social 
actions 

-^•^^^ 

^ * * * * * * S^  Social  integration 

This  diagram  has the  advantage of visually  revealing  the  coherence of the 
programme. It can be seen, for  example, that  certain measures  contributed 
very  little towards  the  overall  success of the  programme.  Similarly,  certain 
impacts  are  more  prominent  and  attained  in a complementary  manner by 

Constructing and using scoring scales 

several  measures.  On  the  other  hand,  the  weakness  of  certain  impacts 
suggests  that  several  objectives  are  not  achieved  satisfactorily,  although 
they are considered  important  by several  partners  of the  programme. 

Analysing the programme  using 
a  matrix of cross impacts 

Apart  from  the success  of  each measure,  evaluation  must  also analyse the 
interaction  between  measures.  It  is  to  be  expected  that  the  impact  of  a 
coherent  set  of  measures  will  be  greater  than  that  of  the  same  measures 
implemented  separately. The third  tool presented  in this  Volume, the matrix 
of  cross  impacts,  can be  used  to help  evaluate the effects  of synergy. This 
tool,  based  on  a  synthesis  between  approaches  proposed  by  several 
European  teams,  was  formalised  and  tested  within  the  framework  of  the 
MEANS  programme. 

A solution for analysing synergy within a programme 

Relationship  b e t w e en  programming  and  synergy 

A  programme  is  composed  of  a  set  of  projects  which  all  have  the  same 
objective. Theoretically,  a programme  might  consist  of  the juxtaposition  of 
independent  projects,  of  which  the  total  effect  is  simply  the  sum  of  the 
individual  effects.  In  such  a  situation,  programming  would  be  a  purely 
formal  exercise,  for  the  objective  could  be  achieved  just  as  easily  by 
managing  the  projects  independently.  Programming  adds  value  because 
the projects  reinforce one another, creating  an effect  of synergy. Attempting 
to  achieve such  synergy  is one of the raisons d'être  of  programming. 

With the reform to the Structural Funds in 1988, the notions of  programming 
and synergy appeared to be inseparable. Emphasis was placed on the need 
to  achieve  the  highest  level  of  effectiveness  by  means  of  an  integrated 
programme.  "An  integrated  programme  is  aimed  at  obtaining  an  effect  of 
synergy  between  different  measures,  and  at  guaranteeing  convergence 
between  the  effects  of  the  different  partners  with  different  projects  and 
responsibilities"  (Extract  from the Structural  Fund  regulations,  1988). 

Box 26 - QUBIST, a tool for the  prior assessment  of synergy 

between  projects 

In  1986  the  Commission  requested  Queen's  University  in  Belfast  to  develop  a 
software  package  for  assisting  in  the  selection  of  projects.  This  tool  was 
constructed  in  the  context  of  the  preparation  of  an  integrated  development 
programme  in Northern Ireland. 

The tool  applies to  all  projects  which  could  qualify  for  inclusion  in a  programme. 
For  each  project  examined, the first  step  is to  clarify  its  probable  contribution  to 
the  programme  objectives  (a weight  is  attributed  to  the  objectives,  according  to 
their  importance).  The  probable  relationship  between  the  project  examined  and 
the other  projects  (synergy)  is then shown. 

The  results  of  these  analyses  are  grouped  together  in  a  synergy  matrix  which 
serves  to  calculate  the  synergetic  value of  each  project  (1 point  per  relationship 
identified).  The  projects  are  then  classified  in  terms  of  the  best  relationship 
between their cost  and their  potential  synergy. 

The final  selection  of  the  projects  takes  into  account  this  ranking  established  on 
the basis of criteria of synergy, as well as the potential contribution of the  projects 
to the  programme  objectives. 

Technical solutions for evaluation within  a partnership  framework 

The  characteristics  of  an  integrated  programme  are  twofold.  The  idea  is, 
first,  to  achieve  greater  complementarity  between  projects  and  thus  to 
optimise  the  allocation  of  resources.  It  is  also  to  mutually  reinforce 
interventions  co-funded  by  the  different  institutions  aiming  for  the  same 
goal. 

From the time the principle of programming first appeared, the  Commission 
tried to develop tools for evaluating  synergy. The first  efforts  resulted in the 
development  of  a  technique  called  QUBIST  in  1986  (see  Box  26).  This 
technique  was  designed  for  evaluating  the  potential  synergy  of  projects  at 
the time of  their  selection. It has rarely  been  used, in its original form. 

To  accompany  the  development  of  programming  and  evaluation,  the 
Commission  wanted  to  continue  the  development  of  tools  for  analysing 
synergy, as part  of the  MEANS  programme. A seminar  of  experts  was  held 
in  Brussels  in  1993, for  the  purpose  of  comparing  the  best  cases  in  this 
domain.  The  seminar  produced  a  methodological  report  and  technical 
proposals. 

With  the  ex  post  evaluation  of  Objective  2  programmes  for  the  1989-93 
period,  the  Commission  decided  to  introduce  into  its terms  of  reference  a 
question  relating  to  synergy,  and  to  submit  the  technical  proposals  of  the 
MEANS  programme to  evaluation  teams. 

Several  evaluation  teams  were  thus  able  to  test  the  feasibility  of  these 
technical  proposals.  These  tests  resulted 
in  the  improvements  and 
modifications  discussed  in the following  pages. 

Definition  of  synergy  and  related  notions 

The term  "synergy"  is often  used and  is  part  of the  standard  vocabulary  in 
many  domains. As a result,  its meaning  is sometimes  vague.  For example, 
a firm will enter  into  partnership with  another firm with a view to  developing 
synergy.  In  pharmacology,  synergy  is  said  to  exist  if  two  substances 
administered  simultaneously  produce  an  effect  that  differs from that  which 
would have resulted had they been used separately. It is the latter  definition 
that  applies  in  this  book,  i.e.  there  is  synergy  when  several  public 
interventions  (or several  components  of  an  intervention)  together  produce 
an impact which is greater than the sum of the impacts they would  produce 
alone (e.g. an intervention which finances the extension  of an airport which, 
in turn,  helps to fill tourist  facilities, also financed  by the  intervention). 

Analysing the programme  using a matrix of cross impacts 

Synergy  generally  refers  to  positive  impacts.  However,  phenomena  which 
reinforce  negative  effects,  negative  synergy  or  anti-synergy  may  also  be 
referred to  (e.g. an intervention  subsidises the diversification  of  enterprises 
while  a regional  policy  helps to strengthen  the dominant  activity). 

The fact that synergy can appear within a programme can be explained above 
all by the  notion  of  a critical  mass: a sufficiently  strong  impulse  is  necessary 
to  cross  a  certain  threshold  in  programme  impacts.  Below  this  threshold 
nothing  happens  and  above  it  there  is  a  sequence  of  effects,  not  unlike  a 
snowball  effect.  Box  27  shows  that  synergy  may  result  from  many  other 
mechanisms.  In all cases, synergy  is a dynamic  notion: it  is the  product  of a 
process in which different kinds of investments interact within a given territory. 

Box 27 -  The  dynamic effects  of  programming 

A  Cambridge  Policy  Consultants  study,  entitled  Program  Induced  Dynamic 
Effects,  suggests  a list  of  mechanisms  likely to  produce  synergy  effects: 

•  critical  mass effect:  an  isolated  project  would  have  no impact without  a critical 

mass; 

•  scale effect: reduced costs due to the presence of several  projects in the same 

place; 

•  demonstration  effect:  duplication  of the action through  example; 

•  adjustment  effect:  better  co-ordination  between the different  public  actions; 

•  capacity  effect:  an  isolated  public  action  would  have  no  impact  without  an 

adequate critical  mass; 

•  productivity  effect:  economies  of  scale  of  the  public  action  itself  improve  its 

productivity; 

•  freeing  up  effect:  the  implementation  of  a  project  removes  a  constraint 

weighing  on another  project; 

•  impulsion  effect:  the  implementation  of  one  project  stimulates  that  of  another 

project. 

In  the  case  of  socio-economic  programmes,  synergy  can  be  obtained  by 
different  means:  co-funding;  partnership;  articulation  of  complementary 
projects  within  the  same territory;  convergence  of  several forms  of  support 
towards the same addressees; circulation of information between operators; 
etc.  In all cases, synergy  is the  product  of  complex  mechanisms  which  are 
formed  within  a  system  and  which  materialise  over  the  medium  term. 
Synergy  may,  moreover,  be  internal  or  external,  depending  on  whether  it 

Technical solutions for evaluation within  a partnership  framework 

occurs  within  the  system  constituted  by  the  programme  or  spreads  to  the 
effects  of other  public  interventions. 

The diagram  in  Box  28 represents  the  mechanisms  which  produce  effects 
of synergy.  Internal synergy  is represented  in the centre of the diagram  and 
external  synergy  on  the  right.  The  left-hand  column  represents  an 
imaginary  situation  (a  counterfactual  situation),  in  which  the  programme 
measures  are  independent  from  one  another  and  from  other  public 
interventions. 

Box 28 -  Logical diagram  of effects  of  synergy 

Funding 
authorities 

Joint 
decision­
making 

Joint 
decision-
making 

Programme 
objectives 

Measures 

Sum of 
isolated 
effects 

Integration 
(In terdepen dence 
between  programme 
objectives) 

Complementarity 
(befween  actions 
and  objectives) 

Integration 
(Interdependence 
with  the  other 
policy  priorities) 

Internal 
coordination 

External 
coordination 

Effects  of  internal 
synergy 

Effects  of  external 
synergy 

Socio-economic  development 

Internal  synergy  is the  product  of  internal  coherence  composed  of  three 
elements: 

•  interdependence  of  objectives  (objectives  are linked to one  another); 

•  complementarity  between  measures  and  objectives  (measures  have  the 

same  objectives); 

•  co-ordination  between  measures 

(measures  are  chosen  and 

implemented  so that their  effects  can  meet  in time and  space). 

Analysing the programme  using a matrix of cross impacts 

These  are  not  the  only  elements  needed  to  obtain  effects  of  synergy,  but 
they are essential for an analysis. Internal synergy has many different forms, 
as the following  examples  show: 

•  The  development  of  a tourist  resort  is favoured  by the  construction  of  a 
road  to  the  coast,  and  both  are  subsidised  by  the  programme  (mutual 
reinforcement). 

•  Support  for  university  laboratories  and  incentives  for 

technological 
development  in  enterprise  allowing  the  growth  of  knowledge  networks 
industrial 
innovation 
and  the  promotion  of 
technologies  (combination  of  critical  mass  and snowball  effects). 

in  new 

results 

that 

•  The combination  of assistance to different  levels of an agricultural  branch 

helps to  minimise costs  (effect  of  size). 

The opposite  of this  notion  of  synergy  is what  can  be qualified  as  negative 
synergy, for  example: 

•  The construction  of  an  iron  and steel  plant  and  a pleasure  resort  next  to 

each other on the  coast. 

•  The  granting  of  assistance  to  different  competing  levels  of  the  same 

agricultural  branch  (dispersion  effect). 

The  second  category  of  synergy,  called  external  synergy,  applies  to  the 
mutual  strengthening  or  weakening  of  the  effects  of  the  programme  and 
those of other interventions, e.g. when the aim is to achieve synergy between 
aid for  SMEs granted  by Structural  Fund  programmes  and  loans granted  by 
the  European  Investment  Bank.  Another,  less  positive  example  is that  of  a 
programme  for  the  integration  of  jobless  people, financed  by  the  European 
Union,  which  recruits  a  proportion  of  its  participants  at  the  expense  of 
training  programmes  financed  by  national  authorities. This  type  of  negative 
synergy  has been called a "cannibalism effect"  by certain  evaluation teams. 

The evaluation  budget  does  not  always  allow for  a study  of  external  synergy. 
However,  in  overall  evaluation  it  is  advisable  to  make  a  list  of  the  main 
European, national and regional policies related to the domain of the evaluation 
in  order  to  obtain  an  idea  of  potential  external  synergy.  This  approach  may 
prove highly relevant, without  requiring a time-consuming analysis. 

Evaluating  synergy  in  three  phases 

Synergy  is rarely evaluated because it is not an explicit programme objective, 
despite it being a key mechanism  for achieving objectives. The evaluation of 

Technical solutions for evaluation within a partnership  framework 

synergy  is often  limited, in the framework  of  an a contrario  approach, to  the 
observation of a lack of co-ordination or of wastage related to the duplication 
of interventions. It is also limited by available resources. The time and budget 
available  for  evaluation  are  often  mobilised  by  other  areas  of  intervention, 
considered  priorities, such as employment  or the  environment. 

Certain  very  real  technical  difficulties  exist.  First,  there  are  so  many 
interactions  which  favour  synergy  that  they  cannot  all  be  examined.  It  is 
also  necessary  to  take  into  account  the  absence  of  reliable  data  which 
would  make  it  possible  to  distinguish  the  impact  of  all  the  measures  of  a 
programme  from the  impact  that  would  have  been  produced  by the  same 
measures  had they  been  implemented  separately. 

The solution proposed  in this Volume is based on the use of a tool called the 
matrix of cross impacts. This solution was designed to be applicable without 
substantially  increasing the volume of work devoted to the overall evaluation 
of  a  programme.  It  should  be  seen  essentially  as  an  additional  analysis  of 
information  which  has  inevitably  to  be collected  for  other  purposes. 

Box 29 -  A procedure for evaluating  internal  synergy 

Identification of potential synergies 

Analysis of synergy effects 

2 -4  2 

Empirical verification of main effects 

i 

3 

Empirical  testing  of  this  tool  showed  that  its  application  had  to  remain 
realistic  and flexible. It is for this reason that  its use has to  be broken  down 
into three phases to be adjusted according to the time and budget  available 
for the evaluation: 

•  a phase of  construction  of  the  matrix  for  identifying  potential  synergy; 

•  a phase of  global  analysis  of  effects  of  synergy  by  means  of the  matrix; 

Analysing the programme  using a matrix  of cross impacts 

•  a phase of empirical verification of the main effects, selected on the basis 

of  the  matrix. 

Only  the  first  two  phases  are  absolutely  indispensable  when  the  evaluator 
has  a  very  tight  schedule.  They  offer  a  sound  basis  for  the  analysis  of 
synergy  within  the  programme  under  consideration.  The  third  phase 
provides  information  from  the  field,  making  it  possible  to  confirm  or 
invalidate  the  assumptions  made  by  managers  in  the  second  phase.  It  is 
strongly  recommended  in  cases  where  the  question  of  synergy  must  be 
considered more in-depth, and/or when the first two phases have produced 
no credible  conclusions. 

Instructions for using the matrix of cross impacts 

The  construction  and  validation  of  the  matrix  of  cross  impacts  consists  of 
several  successive  stages:  formulation  of  the  structure  of  the  matrix; 
scoring  the effects  of  synergy; validation  with the  managers;  calculation  of 
coefficients  of  potential  synergy; and, lastly, overall  presentation. 

Choice of the level of analysis of the  synergy 

Depending  on  the  structure  of  the  programme  concerned,  it  will  be  more 
relevant to analyse synergy  between the axes, the measures, the actions or 
the projects. The level of analysis chosen obviously depends on the number 
of  programme  components  at  each  level.  Some  programmes  consist  of 
only  a few  projects,  which  makes  it  possible  to  rapidly  analyse  synergy  at 
their  level.  If  the  number  of  projects  is  very  high,  it  may  be  preferable  to 
analyse  synergy  at the  measures  level. 

Box 30 -Choice  of a  level for analysing  synergy 

in an objectives  tree 

Programme 

Axes 

Measures 

Actions 

Projects 

11 
Training of 
staffin 
SMEs 

L 

12 
Support for 
investments 
by SMEs 

hj 

\/ 

1 
Support for 
productive 
activities 

Growth of 
employ­
ment 

_^ Training courses 

X.Y.Z 

Creation 

-► Enterprises Χ, Υ, Ζ 

1 Modernisation 

-*■ Enterprises Χ, Υ, Ζ 

' Industrial 

- Centres Χ, Υ, Ζ 

Urban 

■ Centres Χ, Υ, Ζ 

-►Centres Χ, Υ, Ζ 

The  lines  in  bold  type  represent  the  most  obvious  potential  synergy. 

Technical solutions for evaluation within a partnership framework 

The choice of a level of analysis can be made by referring back to the 
objectives tree (see Box 6). Those relations of potential synergy which 
appear most obvious can be marked on this diagram of the programme, as 
shown in Box 30. 

In the example of the Catalonian Objective 2 programme, the most 
appropriate level of analysis seems to be the measures level. It is at this 
level that the largest amount of potential synergy was identified. 

■ Interaction between measures 

Once a level of analysis has been chosen, the matrix of cross impacts is 
constructed with as many lines and columns as there are programme 
components at that level. The cells in the matrix allow all identified synergy 
to be marked, as shown in following box. Because the matrix is considered 
symmetrical at this stage, only half of it will be filled in. 

Box 31 ­ Construction of a matrix of cross impacts 

Axes 

I. 

II. 

III. 

IV. 

Creation and 

Protection and 

Invest­

Improvement 

development 

improvement 

ments in 

of the 

of productive 

of the 

R&D and 

telecommuni­

activities 

environment 

training 

cations network 

measures 

11 12 13 

21 22 23 

31 32 

41 42 43 

I. Creation and develop­ 1.1 

ment of productive 1.2 

activities 1.3 

II. Protection and 2.1 

improvement of the 2.2 

environment 2.3 

III. Investments 3.1 

in R&D and training 3.2 

IV. Improvement of the 4.1 

telecommunications 4.2 

network 4.3 

X 

X 

X 

X 

X X 

X 

This approach nevertheless lacks precision and remains purely theoretical. 
Consequently it is necessary to go further, by qualifying in more detail the 
importance of expected synergy and by validating this analysis in interviews 
with the programme managers. 

Analysing the programme  using a matrix  of cross impacts 

The following developments constitute the core of the method: they are the 
minimum required for a reasonable analysis of synergy.  Because this phase 
requires  interviews,  they  should  be  planned  at  an  early  stage  in  the 
evaluation  agenda. 

Rating  effects  of  synergy 

The  evaluation  team  compares  pairs  of  measures  to  identify  any  synergy 
which  may  exist.  When  some  kind  of  synergy  seems  possible,  a value  on 
the following  scale  is chosen  corresponding  to the size of the  effect: 

•2 
•  1 

•  -1 
•  -2 

for  a particularly  strong  effect  of  synergy 
for  a weaker effect  of  synergy 

the  same scale applied to  negative  synergy 
the  same scale applied to  negative  synergy 

Although  the  evaluation  team  expects  varying  degrees  of  synergy,  it  can 
also  estimate  that  certain  effects  are  very  likely  and  others  very  unlikely. 
Unlikely effects  can  be put  in  brackets. 

Box 32 -  Scoring  potential  synergy 

Axes 

I. 
Creation and 

II. 
Protection and 

III. 

Invest­

IV. 
Improvement 

development 

improvement 

ment  in 

of the telecom­

of  productive 

of the 

R&D& 

munications 

activities 

environment 

training 

network 

measures 

11 

12 

13 

21 

22 

23 

31 

32 

.41 

42 

43 

I.  Creation and development  1.1 

of productive 

activities 

II.  Protection and 

improvement of 

the environment 

III. Investment in 

R&D & training 

IV. Improvement 

of the telecom- 

munications  network 

1.2 

1.3 

2.1 

2.2 

2.3 

3.1 

3.2 

4.1 

4.2 

4.3 

(2) 
(2) 

(2) 

2 
(1) 
(D 
1 
(2)  Π) 

(D 
(2) 

(2) 
(2) 

Π) 
(2) 

(1) 
(2) 

(1) 

(2) 

(1) 

-1 
(1) 

(2) 
(2) 

(1) 
(1) 

(1) 

(1) 

At this  stage, the  matrix  of cross impacts  represents  a quantified  synthesis 
of  assumptions  of  synergy,  identified  by  the  evaluation  team. Although  the 

Technical solutions for evaluation  within a partnership framework 

matrix  of  cross  impacts  appears  essentially  quantitative,  it  contains  only 
subjective  assessments.  The  comments  and  remarks  of  the  evaluation 
team are essential  elements  which  are worth saving. 

In some cases the synergy within pairs of measures is unidirectional only. For 
example, a road to the  coast  is essential for the  success  of  a tourist  resort, 
but  the  expected  development  of  tourism  owing  to  the  road  can  be  partly 
achieved  without  building  the  tourist  resort.  This  is  referred  to  as 
asymmetrical  synergy.  This  means  that  a significant  positive  effect  exists  if 
Measure A is added to  Measure  B,  but that the effect  is less if Measure  Β is 
added to Measure A. This characteristic  is noted in the qualitative comments. 

Once the  matrix  has been filled  in, the values  on either side of  the  diagonal 
dividing the diagram are identical  in cases of symmetrical synergy. Only half 
of the  matrix  is filled  in since the other  (shaded)  part  is identical. If cases of 
asymmetrical  synergy  (relationship  of  non-reciprocal  interdependence)  are 
noted, they  are indicated  on the shaded  side  of the diagonal. 

Example:  Case  of  asymmetrical  synergy  in  an  economic  redeployment 
programme in Catalonia 
Among  other  examples  of  asymmetrical  synergy,  the  evaluation  team  noted  that 
"training of unemployed people and workers from the environment and R&D sectors 
has a highly positive effect on efforts to develop and modernise SMEs. On the other 
hand, support for the development and modernisation of SMEs has a comparatively 
weaker  positive  influence on the training  of the  unemployed and workers from the 
environment and R&D sectors". 

Validation  with  the  managers 

Once the matrix  has been filled in, the evaluation team organises  interviews 
or a meeting with the main programme managers. The aim is to discuss and 
validate  assumptions  of  synergy  presented  in the  matrix. 

For  each  question  on synergy,  the  explanations  and  arguments  chosen  by 
the managers  are recorded  in the form of qualitative comments  attached  to 
the  matrix,  with  an  indication  of  the  probability  of  that  synergy  appearing 
(see Box 33). 

The  detailed  commentary  is  very  important  for,  in  cases  where  time  is 
limited,  it  serves to  make a qualitative  analysis which  produces  results  and 
can  be applied to  the  implementation  of the  programme concerned. 

Analysing the programme  using a matrix of cross impacts 

Box  33  -  Comments  on  potential  synergy 

Relations 
between 
measures 

11  &12 

21  &23 

Effects  of 
interactions 

Managers' 
opinions 

Probability 

yes 

strong 

no 

weak 

Better  qualified 
staff  facilitate 
the decision  to 
modernise  a SME 

Training favours 
the choice of non-
polluting 
production 
methods 

Coefficients  of  potential  synergy 

Once the  assumptions  of  synergy  have been validated  with  the  managers, 
the  matrix  of  cross  impacts  can  be finalised  by  calculating  the  coefficients 
of  synergy  (Cs+).  They  are  calculated  for  each  measure  and  for  the 
programme as a whole. The formula  is as follows: 

Cs+  = 

Sum of  positive  scores 
(Number  of  positive  scores) χ 2 

If  all  potential  synergies  between  measures  were to  receive  the  maximum 
score,  the  coefficient  would  be  equal  to  1.00.  The  coefficient  would  be 
equal to 0.00 for a programme with no  positive synergy. If negative  synergy 
exists, the synthetic  coefficient  of  negative synergy  is also calculated (Cs_): 

C  = 

Sum of  negative  scores 
(Number  of  negative  scores)  χ 2 

To facilitate the calculation of coefficients, six columns must  be added to the 
matrix  of  cross  impacts  as  indicated  in  Box  34. The  columns  "s+"  et  "s-" 
sum values of synergy and negative synergy respectively, while the  columns 
"n+"  and  "n-" count  the  number  of  positive and  negative  scores  presented 
in the  matrix.  Columns  Cs+  and  Cs-  calculate  the  synthetic  coefficients  of 
positive and negative  synergy. 

The  calculation  of  synthetic  coefficients  can  be  extended  to  the  entire 
programme  by  calculating  the  average  of  all  the  non-zero  coefficients 

Technical solutions for evaluation within a partnership framework 

(synthetic  coefficient  of  positive  synergy  =  0.7  and  synthetic  coefficient  of 
negative  synergy  =  -0.5).  It  is  important  to  note,  however,  that  the 
advantage of this calculation  is limited to those cases where the same work 
has  been  done  for  other  programmes  by  evaluation  teams  who  used  the 
same  scoring  methods.  Since  that  is  highly  unlikely  to  be  the  case,  the 
coefficients  of  synergy  will  be far  more  useful  as  criteria for  a  comparative 
judgement  of  measures  within the  overall evaluation  of  a programme. 

Box 34 

-  Calculation  of synthetic  coefficients  of 

synergy 

I. 

1.1 

2 

2 

2 

1 

2 

1 

2 

1 

2 

1 

1.  1.1 

1.2 

1.3 

II.  2.1 

2.2 

2.3 

III.  3.1 

3.2 

IV.  4.1 

4.2 

4.3 

T. 

II. 

III. 

IV. 

Tol 

N° 

Coef 

1.2 

1.3 

2.1  2.2 

2.3 

3.1 

3.2  4.1  4.2 

4.3 

s+ 

s- 

n+ 

n-

Cs+  Cs-

2 

2 

1

1

11 

2 

1 

1 

11 

1 

-1 

2 

-1 

1 

2 

1 

12 

1 

1 

2 

11 

11 

1 

2 

2 

1 

1

12 

2 

2 

2 

1 

-1 

-1 

10 

8 

7 

5 
3 

8 

6 

8 

3 

5 

2 

6 

7 

4 

3 

3 

6 

5 

5 

2 
3 

2 

0.83 

0.57 

0.87 

1 

1 

0.83  -0.5 

0.5 

-0.5 

0.66 

0.60 

0.80 

0.75 

0.83 

0.50 

65 

-2 

46 

2 

0.7 

0.5 

Empirically verifying the main synergies identified 

Presenting  the  global  evaluation  of  synergy 

On  the  basis  of  coefficient  calculations  complemented  by  comments,  the 
evaluation  team  proposes  recommendations  for  improving  or  triggering 
effects of  synergy within  the evaluated  programme. 

In  the  example  above,  inspired  by  the  evaluation  of  the  Objective  2 
programme  in Catalonia,  it  appears  that  Measures  1.1, 1.3,  3.2  and 4.2  are 
essential  for  the  success  of  the  programme,  given  their  synergetic  effects. 
By  contrast,  the  disappearance  of  Measure  4.3  should  not  affect  the 
effectiveness  of  the  other  interventions.  The  question  will  be  raised for  1.2 
which  has  a fairly  weak  coefficient  of  synergy  (0.57),  but  which  enters  into 
synergy  with  many  other  measures  (7).  It  has to  be  concluded  in this  case 
that  1.2  has  a  weak  potential  for  synergy  although  having  numerous 
interactions, since these are individually  weak.  Measure 2.1 which  combines 
positive and negative effects of synergy must also be examined in a qualified 
way. In fact, it is always necessary to qualify the interpretation of  coefficients 
in relation to the  calculated  synergies  (columns  "s" and  "n" of the table). 

For  a global  approach  to  a  programme,  positive  and  negative  coefficients 
of  synergy  can  thus  be  used  as  two  evaluation  criteria  among  others.  For 
the  purpose  of  multicriteria  analysis, as described  in the following  chapter, 
these two columns of results can be included  in the multicriteria matrix. The 
scores  concerning  synergy  are  found  alongside  other  criteria  relating  to 
impacts  on the environment,  employment,  etc. 

Identification  of  synergy  to  be  evaluated 

This  phase  needs  only  to  be  undertaken  should  the  synergy  previously 
identified  be  of  major  interest  for  the  continuation  or  success  of  the 
programme. 
It  has  the  advantage  of  transforming  the  assumptions 
formulated during the preceding  phases into a real evaluation  of the  effects 
of  synergy,  as observed  in reality.  It  requires the  extension  of  interviews  to 
include the  addressees  of  the  programme  (the target  public), which  means 
that  the  evaluators  must  be  able  to  identify  and  meet  the  addressees  in 
order  to  obtain  relevant  information.  If  this  phase  is  to  run  smoothly, 
adequate  time  and  resources  must  be  available  for  the  interviews,  before 
the final evaluation  report  is written. 

Technical solutions for evaluation  within a partnership framework 

The  matrix  of  cross  impacts  can  also  serve  to  identify  the  most  critical 
interactions  between  the  measures  taken  two  at  a  time.  An  interaction  is 
considered  critical  if  it  has  been  assumed  that  it  produces  a  high  level  of 
synergy  (whether  positive or negative) or if very different  assumptions  have 
in  the  qualitative 
been  made  by  the  people  questioned  (mentioned 
comments).  Similarly, 
interaction  between  measures  can  also  be 
considered critical  if the managers of those measures coordinate their work 
particularly  well  or  badly. An example  is when  rivalry  between  the  partners 
of  a  programme  impedes  the  success  of  measures  programmed  for  the 
very  purpose  of  creating  synergy. 

When  an  interaction  is  considered  critical,  a  "window"  is  opened  in  the 
matrix  of  cross  impacts.  The  assumptions  of  synergy  are  then  developed 
and  verified  in detail  within  a limited  number  of  windows. 

The  commissioner  of  the  evaluation  is  called  on  to  express  his  or  her 
opinion  on the  number  and  position  of the windows  to  open  because  they 
influence  the  duration  and  cost  of  the  operation.  In  Box  35,  it  was 
considered that the synergy within the (shaded) axes was not a priority. The 
areas with the highest values (a score of 2) and/or the most disputed values 
(scores  in  brackets)  were  given  priority.  This  resulted  in  three  windows 
being opened for the  empirical  verification  of  synergy. 

Box 35  -

Opening  of  "windows' 
of synergy 

for the empirical  verification 

I. 
1.1 

(2) 
(2) 

2 
1 

(2) 

(D 
(2) 

(1) 
(2) 
(D 

I. 

II. 

III. 

IV 

1.1 
1.2 

1.3 

2.1 
2.2 
2.3 

3.1 
3.2 

4.1 
4.2 
4.3 

1.2 

1.3 

II. 
2.1 

2.2 

2.3 

III. 
3.1 

IV 
4.1 

3.2 

4.2 

4.3 

-1 

(2) 

(1) 

(1) 

(1) 

(2) 
(2) 

(2) 

(1) 
(1) 

(D 
(2) 

(2) 
(2) 
(D 

(1) 
(1) 

(D 

(1) 

0 
0 

0 

Analysing the programme  using a matrix  of cross impacts 

Empirical  verification  of  the  effects  of  synergy 

The  empirical  verification  of  synergy  necessitates  interviews  with  the 
addressees  of  the  measures  concerned.  An  addressee  is  an  individual,  a 
firm  or  an  institution  which  has  been  affected  in  some  way  by  the 
programme, either  directly  or  indirectly. 

The  "windows"  opened  during the  preceding  stage may  concern: 

•  individuals, enterprises  or institutions  within the  same target  group; 

•  a set of  individuals  /  firms  /  institutions within the same territory; 

•  or a set of firms which  belong to the same network  (branch of  production 

or  sector  of  activity). 

In  all  instances,  those  addressees  who  have  been  affected  or  who  might 
have been affected  by the measures  studied  must  be identified. 

The  evaluation  team  tries,  in  this  way,  to  compare  the  assumptions  of 
synergy,  made  in co-operation  with  the  managers,  with  that  which  can  be 
observed  empirically.  Depending  on  the  number  of  addressees,  the 
evaluation  team holds  individual  interviews,  a focus  group  or even a series 
of  focus  groups,  when  possible.  Further  information  on  focus  groups  can 
be found  in Volume 3 of this Collection, devoted  to evaluation tools. 

The  interviews  review  assumptions  made  during  the  construction  of  the 
matrix  of  cross  impacts.  Their  contribution  is  qualitative;  it  is  therefore 
essential to  note the  beneficiaries'  arguments. 

Exceptionally,  an  in-depth  analysis  of  the  synergy  between  the  measures 
concerned  by  a  "window"  can  be  made. This  implies  the  multiplication  of 
interviews,  to  compare  information  or  the  carrying  out  of  a  series  of  case 
studies.  This  operation  makes  it  possible  to  understand  the  nature  and 
mechanisms  of  a  particular  effect  of  synergy  in  depth.  It  is expensive  and 
must  be  planned  very  early  in  the  course  of  the  evaluation.  It  can  be 
particularly  useful  if  the  conclusions  obtained  can  be  generalised  to  other 
programmes  (external validity). 

Consolidation  of  the  analysis  and  recommendations 

The  evaluation 
the  managers' 
team  must  systematically  compare 
estimations  and  its  own  observations  in the  field.  This  comparison  will  be 
given  to  the  managers  of  the  measures  and  of  the  programme,  in  a 
qualitative form. 

Technical solutions  for evaluation within a partnership  framework 

The  idea  is to  explain  differences  of  interpretation,  but  also  to  validate  the 
different types of synergy which appear in the programme and to know their 
nature.  In  this  respect,  it  is  important  for  the  managers  to  obtain  this 
information  as early as possible, so that they  can make  adjustments  to the 
implementation  of the  intervention. 

The  matrix  of  cross  impacts  is  adjusted, taking  into  account  the  empirical 
verifications  if  this  is  considered  necessary.  The  calculation  of  synthetic 
coefficients  of  synergy  can then  be  reiterated, measure by  measure. 

Box  36  presents,  for  each  domain,  the  effects  of  synergy  that  were  most 
frequently  recorded  by  the  teams  evaluating  programmes  financed  by  the 
Structural  Funds in areas of  industrial  redeployment. 

Box  36  -  Main  effects  of 

synergy 

observed  in 

evaluations 

Domain 

Aid  to 
enterprise 

Aid to 
enterprise 

Tourism 

Infrastructure 

Training 

Technology 

1 

» 

• 
• 

Tourism 

Infrastructure 

Training 

Technology 

-

) 

> 
) 

-

O 
» 

-

• 

Key : Effects  of  synergy 

•  Strong 

► Average 

O  Weak 

in 
Evaluations  which  addressed  the  question  of  synergy,  especially 
Objective  2  programmes,  revealed  interesting  cases  of  decentralised  co­
ordination. For example, in the Twente and Groningen-Drenthe regions (NL), 
work  groups  were  used  to  promote  integration  between  the  measures  of 
different  axes.  In  the  Bremen  region  (FRG),  a  "Coherence  Fund"  was 
adopted  as an  instrument  for  encouraging  operators  to  co-ordinate.  In the 
Nord-Pas-de-Calais  region  (F), a  "central  observation  office"  has the  same 
mission. 

IV 
Using multicriteria analysis 
within a partnership framework 

Multicriteria  analysis  serves  to  ground  synthetic  conclusions,  taking  into 
account  all  the  criteria  considered  relevant  by  the  evaluation  team.  This 
capacity  is particularly  useful in partnership  evaluation because it allows  the 
expectations  of each decision-maker  to be  taken into  account.  Multicriteria 
analysis is  traditionally  used  as a  decision-making  tool.  There are  multiple 
technical  variants of  this  tool,  the principles  of  which  are  described  in  this 
chapter. By selecting  some of these variants and using the analysis as an aid 
for making judgements,  this  Volume proposes  a new use for the  technique. 
It  consists  of  grounding  conclusions  which  classify  measures  of  a 
programme  in terms  of their success,  according  to a procedure  acceptable 
to all  the partners.  To apply  this  technique,  the  evaluation  team  must  have 
complete  conclusions  on  all  the  measures  and 
impacts, 
summarised,  if possible,  in scoring  scales. 

their  main 

A solution for judging the overall success of a  programme 

Taking  into  account  all  relevant  criteria 
for  grounding  a  j u d g e m e nt 

When  it  concludes  its  work,  the  evaluation  team  must  state  its opinion  on 
the  respective  merits  of  the  different  measures  and,  in  particular,  on  their 
contribution to the overall success of the programme. If the evaluation does 
not  reach  this  stage,  it  cannot  lead  to  recommendations  of  a  sufficiently 
strategic  level in terms of improvement  or reorganisation of the  programme. 
However,  experience  has  shown  that  evaluation  teams  are  reluctant  to 
formulate  such conclusions,  especially  because they  imply judgements  on 
the  relative value of the results  and  impacts  obtained. 

If the evaluation team or steering group are reluctant to compare the  merits 
of  the  different  measures,  this  is  often  through  fear  of  formulating  such  a 
value  judgement  and  especially  through  fear  that  this  judgement  will  be 
considered  as too  subjective.  It  is,  however,  not  unusual  for  an  evaluation 
to formulate value judgements.  Is the word  "evaluation"  itself  not  related  to 
the  word  value?  In  this  perspective,  the  evaluation  team  must  strive  to 
ground  its  judgement  on  explicit  and  accepted  criteria.  As  the  following 
example  shows,  progress  still  needs  to  be  made  in  improving 
the 
justification  of  conclusions. 

Example: Conclusions  containing  an  ill-justified  judgement 

In  the  conclusions  of  an  evaluation  report,  the  following  two  judgements  might  be 
presented: 

"Measure A is greatly  appreciated  by  professionals" 

"Measure  Β has a very  high cost  per job created" 

In this example, the evaluation team ends  its report conclusively, thus  demonstrating 
a  quality  which  is  not  always  present.  Moreover,  its  judgements  are  supported  by 
factual  elements, which  is also a positive  point.  It is regrettable,  however, that  these 
conclusions  are  not  justified  more  fully.  For  example,  as  far  as  Measure  A  is 
concerned, the report does not explain why the professionals'  view  is enough. As for 
Measure  B, it would  be interesting  to  know  why, and  in relation to  what  norm, it  can 
be  concluded  that  the  cost  per  job  created  is  very  high.  It  would  also  be  useful  to 
know whether the measure has other  positive effects  likely to offset the high cost  per 
job  created. 

Technical solutions for evaluation within  a partnership  framework 

If  the  evaluation  team  does  not  adequately  explain  its  judgements,  the 
report  as  a  whole  is fragile.  The  entire  credibility  of  the  evaluation  is  then 
based  on  the  trust  that  is  placed  in  the  evaluation  team's  impartiality  -
something  which  can  always  be  doubted  (if  necessary  with  a  bit  of  bad 
faith).  To  avoid  this,  but  without  abandoning  the  idea  of  producing  real 
conclusions,  the  evaluation  team  can  help  itself  by  means  of  several 
techniques  enabling  it to  explain  its judgements  and  protecting  it from  the 
strongest criticism. Box 37 presents four of these techniques and their  main 
characteristics. 

Box  37  -  Aids  for  making  a  judgement 

Number  of 
effectiveness  criteria 

Who decides  on the 
respective  importance 
of the  criteria 

Cost-
effectiveness 
analysis 

Satisfaction 
survey 

Cost-
benefit 
analysis 

Multicriteria 
analysis 

One only 

Many 

Many 

Many 

(not 
necessary) 

The 
addressees 

The 
evaluation 
steering 
group 

The 
evaluation 
team, with 
the  opinions 
of the winners 
and  losers 

Are the criteria  explicit? 

Yes 

No 

Not  entirely 

Yes 

The  simplest  technique  consists  of  deciding  whether  the  judgement  will 
refer  to  a  priority  impact,  generally  related  to  the  global  objective  of  the 
programme. Two  criteria  are then  applied: the  achievement  of  the  priority 
impact,  and  the  inputs  mobilised.  In such  cases  the  evaluation  takes  the 
form  of a cost-effectiveness  or value-for-money  evaluation. For example, it 
compares the different  programme  measures  in relation to the cost  per job 
created. 
framework  of  European  Union  socio-economic 
programmes,  it  is difficult  to  be satisfied  with  such  a simplified  technique. 
In  reality,  the  contribution  of  all  the  measures  to  the  achievement  of  a 
global objective (e.g. socio-economic  development)  is not easy to  estimate 
because  such  an  objective  is  too  vague  and  distant  (see  page  21).  A 
situation  in which  the  merits  of  a measure  are judged  in relation to  several 
criteria  (e.g.  competitiveness,  diversification,  employment,  environment, 
etc.)  is  more  likely. The  evaluation  team  must  be  careful  not  to  forget  any 

the 

In 

Using multicriteria  analysis within  a partnership framework 

relevant  criteria, or  it could  make the conclusions  unacceptable  for  one  of 
the readers of the report.  It is therefore  necessary to use techniques  which 
allow  for judgements  to  be made  by taking  several  criteria  into  account. 

identified  all 

One  type  of  multicriteria  judgement,  often  chosen  implicitly,  consists  of 
having  the  addressees  judge  the  measure.  When  an  evaluation  team 
concludes, after  a survey  on the  assisted firms, that  "the  measure  is highly 
appreciated  by  the  professionals",  that  amounts  to  saying  that  the 
professionals  questioned  had 
the  advantages  and 
disadvantages  of  the  measure and  synthesised  them  when  they  answered 
the  interviewer's  questions.  After  extrapolation  from  the  survey  sample,  a 
multicriteria judgement  is obtained, but its main drawback  is that it is totally 
opaque.  It  is  impossible  to  find  any  trace  of  what  counted  in  individual 
it  is  based  only  on  direct 
judgements.  Its  other  drawback 
addressees'  opinions,  considering  that  their  judgement  may  be  biased  by 
an accommodating  attitude towards the administration which granted  them 
assistance. 

is  that 

For  structuring  judgements  well  and  taking  all  impacts  into  account,  the 
evaluation  manuals also suggest  cost-benefit  analysis. With this technique, 
each  impact  is  identified  and  estimated  quantitatively  by  the  evaluation 
team.  It  is  then  weighted  with  a  monetary  value  (in  some  cases  called  a 
fictive  price).  This  value  represents  what  society  or,  rather,  the  people 
concerned  are  prepared to  pay to  obtain the  impact.  Negative  impacts  are 
also  given  a value in terms of  the willingness to  pay to  avoid them. 

Cost-benefit  analysis  is  a  method  which  theoretically  results  in  a  clear 
judgement  in  the  form  of  a  "yes  or  no",  without  requiring  any  form  of 
comparison. The judgement  is  positive  if the  estimated  advantages  for  the 
groups who win are greater than the estimated disadvantages for the groups 
who  lose.  In  practice, this  simple  equation  is  rarely  applicable  because  so 
many projects or interventions pass the cost-benefit  test that they cannot all 
receive public funding. A positive cost-benefit  balance sheet is therefore  not 
enough for  making  a final judgement  on an isolated intervention. 

The  use  of  cost-benefit  analysis  is often  tricky  because  many  impacts  are 
difficult 
to  translate  into  incontestable  monetary  terms  (particularly 
environmental impacts). Moreover, the technique is extremely demanding in 
terms  of  impact  quantification,  which  makes  it  difficult  to  apply  to  a  very 
large  socio-economic  development  programme.  In  practice,  its  use  is 
recommended  mainly for  certain ex ante evaluations  of  large  projects. 

Technical solutions for evaluation within a partnership  framework 

Unlike  the  preceding 

three  approaches  (cost-effectiveness  analysis, 

surveys  and  cost-benefit  analysis),  multicriteria  analysis  is  an  approach 

which  requires the steering  group  of the evaluation  to clarify  the  judgement 

criteria. This  is often  an  advantage  in a partnership  context.  The  technique 

is not  particularly  demanding  in terms  of  the  quantification  of  impacts,  and 

qualitative  conclusions  can  easily  be taken  into  account  if they  have  been 

structured  by  means of  scoring  scales. 

Box 38 -A  European  workshop  on evaluation techniques  adapted 

to  a partnership  context 

Evaluations  of European  Union  socio-economic  programmes  are organised and 
performed  within  a  context  of  partnership  between  the  Commission  and the 
national  and/or  regional  authorities.  As  part  of  the  MEANS  programme,  three 
workshops  attended  by about  ten European  experts  and practitioners  were  held 
for the  purpose of identifying  difficulties  of partnership  evaluation  and proposing 
technical  solutions. 

The  participants  recognised  that  differences  of  interests  frequently  appeared 
when  objectives  were  clarified  jointly,  when  ad hoc indicators  were  constructed 
and when  benchmarks  were established. 

Many  techniques  were  mentioned  during  the  three  workshops:  the  logical 
diagram  of  impacts;  scoring  scales;  multicriteria  analysis;  SWOT;  METAPLAN®; 
and  the colour  vote.  These  techniques  are generally  so simple  that  some  have 
been  accused  of being  childish. Their  advantage  is, however,  that  they  structure 
the  partners'  reflections  and  help to deal with disagreements  without  threatening 
working  relations. 

Partnership  evaluation  is not free  from  power  play  and the fact  that  participants 
tend  to  talk  the  official  language  of  their  institutions.  If  the steering  group  is 
broadened to include the main partners of the programme, it is often the victim of 
a  sort of  "ritual of courtesy"  which  papers  over  differences  in points  of view and 
censures  certain  sensitive  questions.  Techniques  such  as  multicriteria  analysis 
favour  the expression  of  disagreement  within  the steering  group,  which  avoids 
phenomena  of  self-censorship  and enhances  the relevance  of  conclusions. On 
the  other  hand, the technique  channels  discussions  within the steering  group by 
avoiding  risks of conflict. 

Evaluation  criteria:  definition  a nd  e x a m p l es 

In order to justify  its conclusions, the evaluation team clarifies  its judgement 

references. That  is to  say,  it  describes  its  main  criteria,  each  of  which  may 

Using multicriteria analysis within a partnership framework 

be associated  with a measurement  unit and a norm to  meet, as symbolised 
in the  example  of the thermometer  in Box 39. 

Box 39 -  Reference judgement,  norm  and  criteria 

degrees Celsius or 
% of trainees employed 
after 12 months 

measurement  unit 

Judgement 
reference 

criterion 

O 

temperature or 
placement rate 

The  main  evaluation  criteria  are  relevance,  effectiveness,  efficiency  and 
utility. Relevance judges whether the objectives correspond to real needs of 
the economy  and society. Efficiency judges whether the results and  impacts 
have  been obtained  at a reasonable  cost  or whether  they  could  have  been 
obtained  at  a  lower  cost.  Effectiveness  judges  whether  the  results  and 
impacts  obtained  will  make  it  possible to  achieve the explicit  objectives. In 
the following  pages, the  notion of  effectiveness  will  be extended to  implicit 
objectives,  which  gives  the  term  a  meaning  close  to  that  of  the  fourth 
criterion,  utility.  Utility judges  all the  results and  impacts  obtained, whether 
expected  or unexpected, in relation to socio-economic  needs. 

Multicriteria  analysis  is  used  below  with  a strong focus  on  effectiveness  in 
the broad sense of the term, i.e. the achievement  of expected  impacts. The 
analysis  is  known  as  "multicriteria"  because  the  partners  expect  several 
impacts,  each of  which  is considered  a success  criterion  for  the analysis. 

Instructions for using multicriteria analysis 

Multicriteria  analysis  appeared  in  the  1960s  as  a  decision-making  tool. 
Initially  it  was  developed  by  mathematicians  specialised  in  operational 
research.  This  development  was  guided  for  a  long  time  by  two  separate 
"schools",  one  based  in  France  and  the  other  in the  United  States.  These 
two  currents  are  now tending to  merge. 

Multicriteria  analysis  serves to  illuminate a complex  decision, by  helping  to 
compare  all  the  solutions  envisaged  and  by  taking  into  account  all  the 
points  of view  considered  relevant. As a decision-making  tool,  multicriteria 
analysis  is  currently  used  in  the  private  sector,  for  example  when  a  firm 
chooses  a  site  for  location,  or  in  the  public  sector,  for  example  when  a 
community  chooses  a way  of  stocking  its waste. 

In  the  domain  of  evaluation  in  the  strict  sense  of  the  term,  multicriteria 
analysis  is seldom  used  for  purposes  other  than  those  closely  resembling 
decision-making  aid  and,  in  particular,  the  ex  ante  evaluation  of  transport 
infrastructure  projects.  A  way  of  using  the  technique  is  suggested  below 
which departs from the context of decision-making, instead taking the form 
of an aid for  making  a judgement. Within the framework  of  socio-economic 
development  programmes,  it  concerns  a judgement  on the  success  of  the 
different  measures, for  the  purpose  of  drawing  synthetic  conclusions.  This 
judgement  takes  into  account  the  main  relevant  criteria  for  the  steering 
group. 

The  use  of  multicriteria  analysis  in  such  situations  is  an  innovation.  The 
following  instructions for  use can, however,  be considered operational  in so 
far  as  they  take  into  account  the  results  of  several  full-scale  tests.  The 
procedure  is described  in five steps, some  of  which  can  be varied. 

Structuring  the  analysis 

Choice  of  objects  to  be  compared 

Theoretically,  the  procedure  described  below  serves  to  compare  the 
measures  of  a  programme.  It  is  worth  investigating,  however,  whether  it 
could  be applied  at  another  level, by  breaking  down  certain  measures (i.e. 
the  level of what  is often  called an  "action")  or by  grouping together  similar 
measures.  It is not  advisable to compare  too  few  objects  (e.g. two  or three 

Technical solutions for evaluation within a partnership  framework 

groups  of  measures),  for  multicriteria  formalisation  cannot  be  justified  in 
such  cases.  By  contrast,  a  large  number  of  measures  can  be  compared 
since there is no  limit  in this  respect  (e.g. 50  measures  or more). 

The  number  of  measures  analysed  could  nevertheless  be  limited  by 
constraints  related not to the technique but to data collection. As described 
above,  the  analysis  is  based  on  an  impact  scoring  matrix  which  can  be 
simplified  if  only  the  most  sensitive  measures  are  to  be  analysed.  The 
decision  that  must  be  taken  is  thus  related  to  cost  /  utility.  If  a  decision 
concerning  the  reallocation  of  budgets  or  a  new  programme  is  to  be 
clarified,  it  is  logical  to  focus  primarily  on  the  most  important  measures  in 
budgetary  terms,  the  most  innovative  measures  or  the  measures  whose 
effectiveness  is doubted. 

Choosing  criteria 

In the  case  of  European  Union  socio-economic  programmes, the  success 
of  a  measure  is  normally  judged  in  terms  of  its  contribution  to  the 
achievement  of  the  intermediate  objectives  stated  in  the  programming 
documents.  The  main  European  Union  policy  priorities  (e.g.  environment, 
equal opportunities) are also judgement  criteria. A variant consists of relying 
instead  on  the  implicit  objectives  of  the  programme,  reconstructed  by  the 
steering  group  or  extended  work  groups,  e.g. with  the  aid  of  the  concept 
mapping  of  impacts  (Chapter  1). 

The  latter  option  is  presented  below.  By  means  of  a  logical  diagram  of 
impacts  (see  Box  7),  and  with  the  steering  group's  agreement,  the 
evaluation team defines the  list of  criteria. Each criterion  is derived from  an 
intermediate  impact. At this stage, the evaluation team  must check that the 
criteria chosen are logically  independent  from one another. Thus, in Box 40, 
the  impact  entitled  "improved  viability  of  enterprises"  is logically  related  to 
the  preceding  impact  entitled  "attractive  environment".  The  multicriteria 
analysis  must  therefore  opt  for  one  of  these  criteria  for  judging  the 
measures;  it  cannot  use  both.  On  the  other  hand, the diagram  shows  that 
"attractive  environment"  and  "employability"  are  considered 
to  be 
independent  impacts.  Both  can therefore  be chosen  as judgement  criteria. 

To  facilitate  the  formulation  and  comprehension  of  conclusions,  it  is  also 
important  to  avoid  mixing  criteria  that  are  too  different.  If  the  evaluation 
primarily  concerns  the  effectiveness  of  the  programme,  multicriteria 
analysis  must  focus  on  the  results  and  impacts  obtained  by  the  different 

Using multicriteria analysis within a partnership framework 

measures (that is the option chosen in this chapter). If the evaluation 
focuses primarily on the relevance of the programme, a multicriteria 
analysis can also be performed but it muet concentrate on the main 
strengths and weaknesses of the regional economy and the way in which 
the different measures build on strengths or offset weaknesses. 

Box 40 ­ Independence and interdependence of evaluation criteria 

Measures 

Intermediate impacts 

R&D 

\ jr 

Tourism 

Industrial 
wasteland 

Entreprises 

Training 

Opening up 
to the outside 

Attractive­
ness of the 
environment 

Sectoral 
diversification 

Proliferation 
of SMEs 

Improved 

r viability of 

entreprises 

Improved 
employability 

—*■ 

! 
1 

Interdependent impacts 

Independent 'mpacts 

The synergy between the impacts of the different measures is a particular 
case. It is logical for this synergy to be a judgement criterion in its own right, 
considering that certain measures are essential for the success of others. If 
the analysis of the effects of synergy has been performed in the spirit of 
Chapter III of this volume, it is possible to use a matrix of cross impacts 
and, in particular, coefficients of synergy for taking this criterion into 
account in the formulation of a synthesised judgement on the measures. 

Unlike the number of measures to be compared, which can be very large, 
the number of criteria must not exceed a reasonable limit. Experience has 
shown that the participants of a steering group or a work group cannot 

Technical solutions for evaluation within  a partnership  framework 

express  valid  opinions  on  more  than  eight  criteria.  This  is  the  number 

chosen  in  the  following  example. 

Example: Criteria  chosen  for the  evaluation  of  a socio-economic  programme  in 
Belgium 

As part of a mid-term  evaluation, a multicriteria analysis was performed, involving an 
extended  steering  group  of  about  ten  people.  Eight  criteria  were  chosen  and  were 
given  the following  names  and  meanings  by the steering  group: 

•  Sectoral  diversification  (the  measure  orients  the  region's  economic  activity 

towards  growth  markets). 

•  Strengthening  SMEs (the measure  improves the share of  small and  medium-sized 

firms  in the economic  activity  of the region). 

•  Opening  up to  the outside  (the  measure  increases exports  out  of the  region). 

•  Improving  services  to  enterprises  (the  measure  enables  more  businesses  to  use 

services  in the region). 

•  Better  environmental  integration  (the  measure  reduces  the  number  of  regional 
firms  whose  development  or survival  is hindered  by  environmental  problems). 

•  Better  territorial  distribution  (the  measure  orients  the  economic  activity  of  the 

region towards the  most  disadvantaged  areas). 

•  Better  employability  (the  measure  improves  the  employability  of  people  in  the 

region who  have work  or are looking for jobs). 

•  Synergy  (the  measure  contributes  to  the  success  of  other  measures  in  the 

programme). 

Multicriteria  evaluation  matrix 

Once  the  judgement  criteria  and  measures  (or groups  or  parts  of  measures) 

to  be  evaluated  have  been  chosen,  the  evaluation  team  constructs  a 

multicriteria  evaluation  matrix.  This  matrix  is  a  table  with  as  many  columns 

as  there  are  criteria  and  as  many 

lines  as  there  are  measures  to  be 

compared.  Each  cell  represents  the  evaluation  of  one  measure  for  one 

criterion.  Multicriteria  analysis  requires  an  evaluation  of  all the  measures  for 

all  the  criteria  (no  cell  must  remain  empty),  but  does  not  require  that  all  the 

evaluations  take  the  same  form.  As  Box  41  shows,  the  technique  can 

support  a  mix  of  quantitative  criteria  expressed  by  indicators,  qualitative 

criteria  expressed  by  descriptors,  and  intermediate  criteria  expressed  by 

scores. 

The  most  pragmatic  way  of  filling  in  the  multicriteria  evaluation  matrix  is  to 

structure  all  the  evaluation  conclusions,  whether  quantitative  or  qualitative, 

by  means  of  scoring  scales  created  for  that  purpose  by  the  evaluation  team 

Using multicriteria  analysis within  a partnership  framework 

(see  page  55). The  multicriteria  evaluation  matrix  is then  equivalent  to  the 
impact  scoring  matrix described  in Box  19. 

Box 41 -  Combination of qualitative and quantitative criteria 

Criterion 

Modalities 

Diversification 

Employability 

Environment 

(%  of  assisted 

(impact  descriptors) 

(impact  rating 

for  evaluating 

businesses  active 

the  criterion 

in growth  sectors) 

Measures 

Investment  aid 

In-house  training 

7 2% 

2 1% 

between  0 and 10) 

3 

1 

Neutral 

impact 

Significant 

increase 

in  the 

employability 

of  trainees  already 

qualified;  neutral 

impact 

for 

the  others 

Industrial 

wasteland 

5 2% 

Neutral 

impact 

8 

B ox  42  -  C a se  of  e v a l u a t i on  c r i t e r ia  w h i ch  a re  t oo  s p e c i f ic 

Criterion 

Modalities 

for  evaluating 

the  criteria 

Measures 

Investment  aid 

In-house  training 

Industrial  wasteland 

Diversification 

Employability 

Environment 

(impact  rating 

(impact  rating 

(impact  rating 

between 

Oand  10) 

between 

Oand  10) 

between 

Oand  10) 

7 

0 

0 

0 

5 

0 

0 

0 

8 

Before  continuing  with  the  multicriteria  analysis, the  evaluation  team  must 
check  whether  its  initial  conclusions  will  allow  it  to  compare  measures 
satisfactorily. In choosing the criteria, the team should already have ensured 
that  they  apply  to  as  many  measures  as  possible.  The  majority  of  these 
measures must  have produced  impacts  related to the  majority  of criteria. In 
other  words,  the  impact  scoring  matrix  must  not  have  too  many  neutral, 
absent  or  insignificant  impacts.  If  the  only  scores  which  are  not  equal  to 
zero are situated  in the diagonal  (as in Box 42), it  means that the  measures 

Technical solutions for evaluation within a partnership  framework 

evaluated  have  nothing  in  common  and  that  their  evaluation  criteria  are 
specific.  In that  case, multicriteria  analysis cannot  be performed. 

Choice  of  a  procedure  for  analysis 

Two  main  possibilities  exist  for  the  evaluation  team,  for  comparing  the 
merits  of the  different  measures:  multicriteria  analysis  by  compensation  or 
multicriteria  analysis  based  on  outranking. The  most  well-known  variant  is 
the  first.  It  consists  of  attributing  a  weight  to  each  criterion  and  then  of 
calculating  a  global  score  for  each  measure,  in  the  form  of  a  weighted 
arithmetic  average  of the scores attributed to that  measure for the  different 
criteria.  This  variant  is  called  "compensatory"  because  the  calculation  of 
the  weighted  average  makes  it  possible  to  compensate  between  criteria. 
For  example, a  measure  which  had  a very  bad  impact  on the  environment 
could still obtain a good global weighted score if its impact on employability 
were considered  excellent. 

No global score  is calculated  in the  outranking  variant  because the  criteria 
are  not  all  considered  commensurable.  The  analysis  is  based  on  multiple 
comparisons  of  the  type:  "does  Measure  A  outrank  Measure  Β  from  the 
point  of  view  of  the  environment  criterion?",  "does  Measure  A  outrank 
Measure Β from the point of view of the employability  criterion?", etc. These 
questions  can  be  answered  yes  or  no  (franc  criterion)  or  be  qualified,  in 
which  case  the  notions  of  a weak  preference  and  a threshold  criterion  are 
introduced.  The  analysis  makes  all  possible  comparisons  and  presents  a 
synthesis  of  the  type:  "Measure  A  is  at  least  as  good  as  Measure  B,  in 
relation  to  a  majority  of  criteria  (case  of  agreement),  without  being 
altogether  too  bad  in relation to  the  other  criteria  (case of  disagreement)". 

This  type  of  analysis  often  includes  a  form  of  protection  to  avoid  a 
favourable  judgement  for  a  measure  which  would  be  disastrous  from  the 
point  of  view  of  the  given  criterion.  For  this  purpose,  a  veto  threshold  is 
determined  for  each  criterion.  Just  as  qualified  preferences  can  be 
established  between  criteria,  so  too  it  is  possible  to  qualify  the  veto 
threshold  by  introducing  a  progressive  veto.  The  introduction  of  a  veto 
threshold  strongly  differentiates  the  logic  of  outranking  from  the  logic  of 
compensation.  If  there  were  a  veto  threshold,  a  very  bad  impact  on  the 
environment  would  make it impossible to consider the measure good, even 
if  its  impact  on  employability  were  considered  excellent. 

Using multicriteria analysis within a partnership framework 

Outranking  has the advantage  of  reflecting the  nature  of  relations  between 
public  institutions  better. When  it is said that  "Measure A outranks  Measure 
Β for the majority  of criteria, and that  it is not excessively  outranked for  any 
of  the  criteria",  a  mode  of  reasoning  is  adopted  that  is  very  familiar  to 
policy-makers.  Everyone  knows  that  a public  decision  is easier  to  make  if 
the  majority  of  stakeholders  agree  and  if  there  is  not  a  radically  opposed 
pressure  group.  Moreover,  there  is  often  a  correspondence  between 
evaluation  criteria and evaluation  stakeholders. In cases where the  steering 
group  is extended to about ten  partners, it is not unusual for  participants to 
identify  themselves  strongly  with  the  "environment"  or  "employment" 
criteria. In this situation the outranking variant will probably  better reflect the 
collective  process  of formulating  a judgement  within the  steering  group. 

From  a  technical  point  of  view,  the  compensation  variant  is  very  easy  to 
implement with any spreadsheet. The outranking variant, on the other hand, 
requires  the  use of  specialised  software  which  differ  according  to  the  type 
of  analysis, of  which  the  most  well-known  are presented  in Box 43. 

Box 43 -  Variants  of multicriteria  analysis  using  outranking 

The main variants of multicriteria analysis which use outranking are: 
ELECTRE I -  This variant functions with an agreement index and a disagreement 
index,  presented  in  the  form  of  scores.  A  disagreement  threshold  (a  veto)  is 
introduced for all the criteria. The outranking and veto thresholds are of the franc 
type. The software  processes  a situation in which the best  measure(s)  must be 
chosen, for example a situation in which the aim is to identify best practice. 
ELECTRE TRI - This variant serves to sort measures into different  categories, for 
example,  the  most  successful  measures,  measures  which  have  no  significant 
impact and intermediate measures. 
ELECTRE II produces  a ranking  of  measures, from the  most  successful  to the 
least successful. Outranking and veto thresholds are of the franc type. 
ELECTRE  III  also  performs  a  classification,  but  introduces  vague  outranking 
relationships. 
PROMETHEE  uses  only  an  index  of  agreement  and  introduces  progressive 
outranking. 
For more information, see the annexed bibliography: Vincke 1989. 

Outranking  does  not  always  produce  clear  conclusions. The  analysis  may 
result,  for  example,  in  a  classification  which  considers  that  some  good 
measures  clearly  outrank  the  others,  that  other  measures  pose  the  same 

Technical solutions for evaluation within a partnership framework 

problem because they are frequently outranked, and that a third group of 
measures cannot be ranked. 

By contrast, if the analysis is based on compensation it is always 
conclusive. Throughout the rest of this chapter, the compensation variant 
has been opted for because of its technical simplicity and its capacity to 
produce clear conclusions. In a logic of overall evaluation of a programme 
and in a perspective of budgetary reallocation, it is considered that the 
readers of the evaluation report require conclusions on all the measures 
evaluated. The formula of the weighted global score is appropriate, 
provided that each member of the steering group has confirmed that there 
is no major problem which might justify the use of the veto system. 

■ Choice of assessors 

Multicriteria analysis is an aid for helping the evaluation team to make 
judgements. It enables the team to explain its judgements and to write final 
conclusions without being accused of subjectivity. In reality, the technique 
produces a judgement based on factual and objective elements concerning 
impacts, but also on the impartial integration of the points of view and 
preferences of the main partners. 

It is a technique which cannot be implemented without the effective 
participation of the programme partners. When using multicriteria analysis, 
the evaluation team has to hold individual interviews or focus group 
interviews with a number of people whose points of view are considered 
most relevant for judging the programme measures. These people are 
referred to below as assessors. 

The first option consists of considering the members of the evaluation 
steering group as assessors. In order to do this, the steering group must be 
large enough to reflect the main points of view. Even if the evaluation has 
almost always been commissioned by one of the partners, it would be 
dangerous to consider evaluation as a unilateral exercise controlled by a 
single decision-maker for his or her exclusive use. For this reason a steering 
group which allows for the expression of several points of view is often 
instituted in practice. The extension of the steering group to about ten 
stakeholders has many advantages: the people invited into the group voice 
the expectations and fears expressed at their level and improve the relevance 
of the questions asked. Their presence also makes the evaluation more 
credible. Greater diversity within the steering group brings together more 

Using multicriteria analysis within a partnership framework 

points  of  view, facilitates  their  confrontation,  and  helps to  raise the  level  of 
consensus.  Each  participant  contributes  his  or  her  own  questions,  value 
system, judgement criteria and norms. Provided that it can take advantage of 
this  plurality,  the  evaluation  reaches  richer,  more  credible  and  useful 
conclusions. The taking into account of several partners' expectations allows 
for  better  acceptance of results and increases the interest that will be shown 
in them. Volume 1 of the Collection features a series of recommendations  for 
constituting  and managing  an extended  steering group. 

If the steering  group  has been  extended  to  include  more than ten  persons, 
not  all  of  them  will  be  able  to  participate  effectively  in  the  multicriteria 
analysis.  A  group  of  six  to  ten  assessors  will  then  be  chosen  within  the 
steering group. As shown  below, the evaluation team will have to work  with 
the assessors, either collectively  or through  individual  interviews. 

a  Expression of  preferences  between  criteria 

The expression  of  preferences  is a step intended  to establish the  weighting 
of  the  different  criteria. The  assessors'  preferences  are taken  into  account 
according to  one of the following  variants. 

o  Direct  expression  of the  assessors'  preferences 

Example: Creation of a rating-weighting  system in England 
In the framework  of  a socio-development  programme  in the  North-East  region of 
England, for the period 1991-93, a small group of ten persons developed a project 
selection system. The work group met several times during a three-month period. Its 
members were representatives of the main authorities responsible for the programme 
and its implementation. The group established a list of criteria and the weight of each 
criterion, as well as the scoring scales for the different criteria. 
Some  meetings  were  conflictual  and  the  work  went  through  a crisis  period. The 
members  of  the  group  nevertheless  managed  to  reach  an  agreement,  given  the 
excellent  spirit  of partnership in the region and unanimous support for the regional 
development  policy. After  the work group had reached agreement, all the agencies 
involved in the  programme were consulted  and the authorities  responsible for the 
programme validated the system. The system was then applied to the selection of all 
the projects and served as a model for most other British programmes. 

A first  variant  consists  of  bringing  together  the assessors  and asking  them 
to  express  their  preferences  in  the  form  of  a  weighting  attributed  to  each 

Technical solutions for evaluation within a partnership framevi/ork 

criterion. This can be done by means of a vote through the distribution of 
points. The discussion can also be conducted by means of several 
successive meetings, as in the example above. 

Revealing preferences by classification of profiles 

Another variant consists of presenting the assessors with "profiles" of 
measures or projects described in such a way that they reveal preferences 
between criteria. For example, the assessors may be asked to compare two 
"theoretical" measures A and Β with the same impacts represented 
schematically, as in Box 44. 

Box 44 ­ Profiles of two measures which favour two criteria 

** / f t 

Α*·° J? ·& ¿Ρ 
& jr j? ¿f 

cf ¿r </ 

r~\_r^---

G o od i m p a ct 

/ \ 

<6 é~<x 

f V Neutral i m p a ct 

Profile of Measure A 

­ Profile of Measure B 

(KJ~ 

o 

The theoretical measure A will be described as a measure of which the 
impact is neutral, that is, nil, for all the criteria except the "environmental 
integration" criterion, for which its impact is good. In other words, the 
measure systematically helped to improve the environmental viability of the 
addressees' activities. The second theoretical measure Β will be described 
as a measure of which the impact is neutral, that is, nil, for all the criteria 
except the "strengthening of SMEs" criterion for which its impact is good. 

Using multicriteria  analysis within a partnership framework 

In  other  words,  the  measure  affected  a  large  proportion  of  SMEs  and 

contributed  decisively  to  the  improvement  of  their  competitiveness. 

The  assessors  have  to  choose  one  of  these  t wo  profiles  and,  if  possible, 

must  state  whether  their  preference  is weak,  average,  strong  or  very  strong. 

The  exercise  is  repeated  for  all  the  pairs  of  profiles.  All  the  answers  are 

processed  by  means  of  a  software  package  which  attributes  a  weight  to 

each  impact,  expressed  as  a  percentage  so  that  the  weightings  add  up  to 

100%.  The  software  used  is  presented  in the  chapter  on  scoring  scales  (see 

page  57). 

Revealing  preferences  through  the  ranking  of  real  projects 

Example: Classification  of  real  projects  by  assessors 

A  French  urban  development  programme,  co-financed  at  the  regional  and  national 
levels,  was  evaluated  mid-term.  Eight  evaluation  criteria  were  selected.  The 
evaluation team conducted  a case study  survey of about twenty completed  projects. 
Each  project  was  the subject  of  a monograph  describing  its  impacts  in terms  of  the 
different  criteria. The evaluation  team then had individual  and confidential  interviews 
lasting  about  two  hours,  with  six  assessors  appointed  by the steering  group. These 
interviews  took  place as follows: 

•  The  evaluation  team  showed  the  assessor  several  monographs  of  projects  with 
which  he  was  thoroughly  familiar.  It  checked  whether  the  monographs  closely 
reflected  reality  in the assessors'  view. 

•  The assessor was then Informed about three or four other  monographs of  projects 
unknown  to  him.  The  projects  were  chosen  so  that  some  were  more  successful 
from the  point  of  view  of  certain  criteria,  while others  were  more  successful  from 
the  point  of  view  of  other  criteria.  The  monographs  were  generally  considered 
sufficiently  precise to  make it  possible to judge the  projects. 

•  The  assessor  was  then  invited  to  classify  these  three  or  four  projects  intuitively, 
sorting  them  from  the  best,  i.e.  the  most  worthy  of  programme  funding,  to  the 
"worst",  i.e. the  least  worthy. 

•  The evaluation team and the assessor  discussed the  reasons for  his classification. 
They  deduced  a weighting  of  criteria  which  was  discussed  and  validated  by  the 
assessor.  It was confirmed  that  the assessors  had very  different  points  of view  on 
the  importance  to  be given to the  different  criteria. 

The  choice  offered  to  the  assessors  in the  preceding  variant  could  have  the 

drawback  of  seeming  artificial. To avoid  this  problem,  it  is  preferable  to  ask 

Technical solutions for evaluation within a partnership  framework 

the assessors to state their  preferences  between  real projects. This  variant 
was  applied  in the  evaluation  of  a  French  urban  development  programme 
(see Box  24). It  is the  one which  should  be chosen  by the evaluation  team 
whenever  possible. The  procedure  is presented  in the  example  above. 

•  Different  points  of  view  of  assessors 

Assessors'  preferences  can  be  revealed  in  meetings  or  in  individual 
interviews.  In  the  first  case,  a  single  weighting  system  for  criteria  can  be 
deduced. In the second, the evaluation team and steering group can decide 
to  establish  average  weightings,  which  has  the  effect  of  effacing  different 
points  of  view  among the  assessors. 

By  contrast,  a variant  consists  of  maintaining  the  individual  weightings  of 
each  assessor.  In  this  case,  the  anonymity  of  the  assessors  must  be 
respected  when  the  weightings  carried  out  individually  by  them  are 
processed.  It  may then  be interesting to  present the weightings  separately, 
In  the  example  of  the  urban 
for  a  particular  category  of  assessor. 
development  programme  cited  above,  the  evaluation  team  presented 
weightings,  as  indicated  in  Box  45,  which  depended  on  whether  the 
assessors  claimed to  identify  more with  national  or regional  concerns. 

Box 45 -  Differences  of weighting  between  assessors 

weak 

weighting 

- strong 

Criterion 1 

Criterion  2 

Criterion  3 

Criterion  4 

3  Assessors close to national concerns 
ã]  assessors close to regional concerns 

Using multicriteria analysis within a partnership framework 

Multicriteria  classification  of  measures 

Global  score  and  classification  of  measures 

At this  stage  of  its work, the  evaluation  team  has all the  elements  it  needs 
to  calculate  global  weighted  scores  for  the  different  measures.  It  is 
considered  that  the  results  and  impacts  of  each  measure  have  been 
evaluated  in  relation  to  the  same  criteria,  that  all  these  evaluations  have 
been presented  in the form of  scores  in an impact  scoring  matrix, and that 
there  is  a  weighting  system  which  expresses  the  average  preferences  of 
assessors for  a particular  criterion. 

The global  score  is calculated  by  multiplying  each elementary  score  by  its 
weighting and by adding the elementary weighted scores, as in the example 
in Box  46 which  uses the  impact  scoring  matrix  in Box 24. 

Box  46  -Calculation  of  the  global  scores  of  measures 

Contribution  of 
measures to  the 
production  of 
expected  impacts 
(score between 
0 and 10) 

Urban  development 
(average scores 
for measures 
1 and 2) 

Infrastructure 
(average score 
for measures 3 to 8) 

Collective  social 
actions 
(average score for 
measures 9 to 14) 

Personalised  social 
actions 
(average score  for 
measures  15 to 19) 

Weighting  coefficient 
for  criteria 

Social 
integration 

Employ­
ment 

Balance 
between 
neighbour­
hoods  in a 
town 

Balance 
between 
towns  in 
the  region 

Average 
weighted 
score 

1 

5 

6 

6 

1 

4 

5 

6 

1 

4 

3 

2 

1.2 

3 

3 

4.1 

4.4 

2 

2 

4.2 

35% 

20% 

20% 

25% 

100% 

Technical solutions for evaluation within a partnership  framework 

Based  on  weighted  average  scores,  the  evaluation  team  can  classify 
measures by order of contribution  to the overall success of the programme. 
This  synthesised  judgement  on  the  effectiveness  of  measures  can  be 
considered  sound  and  impartial  provided  that  the  evaluation  team  has 
taken all the  necessary  precautions  presented  below. 

•  Have the evaluation  criteria  validated  by the  steering  group. 

•  Present the  conclusions  on the  impacts  of  each  measure, as well  as the 

impact  scoring  matrix  summarising  them, and  have them validated. 

•  Establish  the  weighting  coefficients  for  criteria,  with  the  help  of  the 

assessors  chosen  with the agreement  of the steering  group. 

•  When  veto  thresholds  have  been  applied  to  certain  criteria,  check  with 
the  steering  committee  that  this  is  preferable  to  using  an  outranking 
approach. 

If,  on  the  contrary,  the  evaluation  team  chooses  and  weights  the  criteria 
itself, without  any interaction with  its partners, the impartiality  of the  results 
will suffer  and the  multicriteria analysis will  be  less useful. 

Multicriteria-multijudge  analysis 

If preferences among criteria are revealed through individual interviews  with 
the  assessors,  and  if  these  interviews  show  strongly  divergent  points  of 
view  between  them, it  is possible  to  establish  several classifications  of  the 
measures.  On  the  same  impact  scoring  matrix,  the  evaluation  team  can 
apply  different  systems  of  weighting  (a set  of  different  weightings  for  each 
assessor).  Differences  between  the  weighted  global  scores  and  hence 
differences  in ranking  will  result,  since  each  measure  can  be  considered  a 
success from the  point  of  view of one assessor  and a failure from the  point 
of  view  of  another.  Box  47 shows  how  twenty  measures  were  rated  by  six 
different  assessors,  based on the  "multicriteria-multijudge  analysis  model". 

In this  example, the  graph  shows  areas  of  consensus  and  dissension.  For 
certain  measures, there  is consensus on the fact  that they were  successful 
(e.g.  2  and  12)  or  that  they  were  not  (e.g.  19  and  20).  On the  other  hand, 
certain measures are the subject  of heterogeneous judgements  in the group 
of  assessors  (e.g. 1, 4 and  11). 

Multicriteria evaluation can therefore  be organised with a view to  producing 
a single synthetic  conclusion at the end of the evaluation or, on the contrary, 
with  a  view  to  producing  conclusions  adapted  to  the  preferences  and 

Using multicriteria  analysis within a partnership framework 

priorities  of several  different  partners. In the case of  European  Union socio­
economic  programmes,  the  different 
levels  of  partnership  (European, 
national and  regional)  may be concerned. Each of these levels  is  legitimate 
in  establishing  its  own  priorities  and  expressing  its  own  preferences 
between  criteria. 

Box 47 

-Multicriteria-multijudge  ranking  of  measures  according 
to their  contribution to the success  of the  programme 

Assessors 

O 
CD 

12 

3 

4 

5 

6 

9  10  11  12  13  14  15  16  17  18  19  20 

Measures 

The objective  of  multijudge  analysis  is therefore to take  into  account  these 
different  priorities  in the  production  of  evaluation  results.  The technique  is 
well suited to the  way  in which  partnerships function  in so far as it  outlines 
areas of consensus  in which the partners agree on the ranking of measures, 
and areas of  dissension  which  reveal the  measures  considered  successful 
for  some  and  unsuccessful 
for  others.  Experience  has  shown  that 
consensual  conclusions  are generally  in the majority. This can be explained 
by  the  fact  that  the  different  weightings  apply  to  the  same  impact  scoring 
matrix. Thus, a measure which  has a low score for  all the  criteria will  never 
have  a  high  weighted  global  score,  irrespective  of  the  differences  of 
priorities  between  partners.  The  different  points  of  view  of  the  partners 
the  conclusions  resulting  from  empirical 
cannot  strongly  contradict 

Technical solutions for evaluation within a partnership  framework 

observation  if these conclusions  show that certain  measures are really  part 
of  best  practice  and  that  others  pose  real  problems  of  effectiveness. 
Experience  also  shows  that  the  partners  are far  more willing  to  accept  the 
conclusions  of the report  if the evaluation team  has recorded their  opinions 
carefully  and  taken  the  trouble  to  take  their  preferences  into  account  in 
presenting  its  conclusions. 

Using the analysis to produce useful conclusions 

Recommendations  on the  reallocation  of  budgets 

In  the  context  of  socio-economic  programmes,  evaluation  can  influence 
decisions  on  the  reallocation  of  budgets,  either  while  the  programme  is 
underway  or during the  preparation  of the following  programme. The  main 
decisions  in this respect  are taken at the  measures  level. Measures  judged 
to  be  the  least  successful  must  be  re-examined  with  a  view  to  either 
reducing 
their 
effectiveness.  Where  relevant,  recommendations  can  also  be  made  to 
increase the  budgets  of those  measures  ranked  as being  the  best. 

their  budgets  or 

re-organising 

to  enhance 

them 

Diffusion  of  best  practice 

Multicriteria  analysis  serves  to  identify  the  areas  of  success  and  the  most 
the  programme.  The  communication  and 
effective  measures  of 
dissemination  of  these  conclusions  among  the  programme  actors  is  very 
useful. Those measures judged as being the most successful (best practice) 
can  be  presented  during  seminars,  and  brochures  or  monographs 
describing  them  can  be  disseminated.  This  diffusion  may  also  include  the 
media,  if  the  authorities  running  the  programme  wish  to  show  the  public 
how  the funds of the  programme  were  spent. 

It  is  also  possible  to  publish  concrete  examples  of  successful  measures 
with  a  view  to  informing  the  managers  of  similar  measures  financed 
elsewhere. The exchange of information  on practices  identified as being the 
best  is  an  excellent  way  of  enhancing  the  performance  of  European 
cohesion  policy  in the  long run. 

Feedback  on  project  selection  methods 

The choice of  evaluation  criteria, their  precise definition  and their  weighting 
constitute  a useful contribution  to  multicriteria analysis. This work  makes it 
possible  to  formulate  a  clear,  complete  and  coherent  description  of  the 
intentions  and  priorities  of  the  programme  partners.  It  is  then  possible  to 
use  these  results  to  spread  clear  messages  to  the  managers  of  the 
measures  and the  operators. 

Technical solutions for evaluation within a partnership  framework 

^ 

The  results  of  multicriteria  analysis  can  also  serve  to  enhance  the  project 
selection  process. It is relatively easy to transfer criteria, scoring  scales and 
weightings  to the  project  selection  system  if this  system  is also  organised 
on the  basis of  scoring-weighting  (see the example  described  on page 60). 
By  basing  the  selection  of  projects  on  the  same  logic  as the  evaluation  of 
measures, the chances of stimulating and funding projects which contribute 
effectively  to  the  programme  priorities  are  increased.  To  be  globally 
successful  a programme  must  select its projects on criteria which  resemble 
its own effectiveness  criteria as closely  as  possible. 

In-depth  analysis 

If the  "multijudge"  variant  of  multicriteria  analysis  has been  performed, it is 
also important to know how to use its non-conclusions,  i.e. those  measures 
judged  favourable  from  certain  points  of  view  and  unfavourable  from 
others.  In this case, the  steering  group  can recommend that the  authorities 
responsible  for  the  programme  launch  a  more  in-depth  evaluation  to 
ascertain  why  there  was  disagreement  on  these  measures.  Was  the 
disagreement  due  to  inadequate  definition  of  the  criteria?  Did  the  criteria 
take the policy objectives  into account  adequately?  Did it take the points of 
view  of  all the  partners  into account  in a fair  and  balanced  way? 

Conclusion 

four  alternative  techniques 

for  the  overall 
This  book  has  presented 
evaluation  of  a  socio-economic  development  programme.  All 
four 
techniques are constructed  in the same spirit.  They all stress the importance 
of interaction  between  the evaluation team and  the partners represented in 
the  extended  steering  group.  They are  structured  and  formalised  without 
being  too  demanding  in  terms  of  quantitative  data,  and  can  easily  be 
combined,  as shown  in the  conclusion. 

Combination  of the four  alternative  techniques 

To  sum  up  the  four  techniques,  the  evaluation  procedure  can  be  broken 
down  into  four  standard  steps  as  shown  in  Box  48. The  evaluation  team 
starts  by  examining  the  programming  documents  to  identify  expected 
impacts.  If necessary, it clarifies these  impacts with the help of the steering 
group  (step  1).  It  then  has  to  choose  quantitative  indicators  or  qualitative 
descriptors to  study the impacts  (step 2). The team uses these  instruments 
to  observe  results  and  impacts  in  the  field  (step  3).  The  evaluation  is 
concluded  with  a  synthetic  judgement  about  the  programme.  This 
judgement  takes  into  account  the  different  impacts  obtained  and  the 
relative  importance  of these impacts for the  different  partners  (step 4). 

In  the  case  of  an  ex  ante  evaluation,  step  3  is  replaced  by  a  forecast  of 
impacts,  based  on available  knowledge. 

Box  48  -  Sequence  of  the  four  technical  solutions 

1 

2 

3 

4 

Clarify the main 

Choose indicators 

Gather  information 

Formulate a 

expected  impacts 

and descriptors 

I  and analyse it to 

synthetic 

for  each  impact 

estimate impacts 

judgement on 

the  different 

impacts obtained 

Draw  up the 

concept  map 

of impacts 

Construct scoring 

Evaluate 

Apply  multicriteria 

analysis 

scales 

measures by 

empirical 

observation of 

results and 

impacts 

in the field 

Analyse the 

Conclude the 

synergy  between 

evaluation and 

make 

recommendations 

measures by 

means of a 

matrix  of cross 

impacts 

Technical solutions for evaluation within a partnership  framework 

Box 49 -  Empirical  study  of impacts  by  means 

of  parallel  case  studies 

A  case  study  examines  an  individual  project  (the  case)  and  concludes  with  a 
monograph  describing  it  in  a  few  pages.  The  researcher  starts  by  gathering  all 
available documents on the project and its management.  He or she then visits the 
site and  questions  the  manager  of  the  measure to  which the  project  belongs, as 
well  as the  operator  responsible  for  the  project.  He or  she  also  holds  individual 
interviews  with  several  direct  and  indirect  addressees.  Each  case  study  involves 
high  costs  (between  2,000  and  5,000  euro)  but  the  number  of  case  studies  is 
sufficiently  low to contain the total cost of the evaluation within a reasonable limit. 
A  typical  survey  of  parallel  case  studies  concerns  between  ten  and  thirty 
projects,  observed  and  analysed  with  the  same  analytical  grid.  This  type  of 
survey  enables  information  to  be  obtained  that  can  be  generalised,  on  the 
addressees'  motives,  their  real  needs,  the  reasons  for  their  participation  in  the 
programme, their way  of judging  the  quality  of the  programme, the  mechanisms 
of the appearance  of impacts, the existence  of unexpected effects, etc.  Because 
the  case  study  survey  is  essentially  qualitative,  its  main  function  is  to  better 
understand  the functioning  of  the  programme. 

Case studies  can also  produce quantitative  or  qualitative estimations  of  impacts, 
or  estimations  structured  by  means  of  a  scoring  scale.  In  this  case,  each 
monograph  produces  conclusions  on the impacts  of the  project  studied and that 
project only. The limited number of case studies does  not allow for  extrapolations 
of  a statistical  nature. 

It  is  nevertheless  possible  to  use  a  case  study  survey  to  produce  systematic 
conclusions  about  the results and  impacts  of the  measures of a  socio-economic 
programme.  For  example,  one  or  more  cases  considered  "typical"  of  each 
measure could be selected. The conclusions  obtained would  be used to  estimate 
the size of the impacts, measure  by measure. A more useful approach consists of 
studying  two  projects  per  measure, one  project  considered  highly  successful  by 
the  managers  of  the  measure,  and  a  project  considered  a  relative failure.  In this 
way  a range of  effects  can  be  estimated,  measure  by  measure. The  conclusions 
formulated  by  the  evaluation  team  can  then  usefully  be  based  on  a  survey  of 
parallel  case  studies,  on  the  understanding  that  other  sources  of  information 
cannot  be neglected (monitoring data on outputs, opinions of the managers of the 
measures, analysis  of the  impacts  of comparable  measures, etc.). 
For  a  more  complete  presentation  of  the  case  study  technique,  the  reader  is 
referred to Volume  3 of  the MEANS Collection. 

For  undertaking  an  overall  evaluation  of  a  programme  in  a  context  of 

partnership,  it  is  possible  to  articulate  the  different  techniques  presented  in 

this  Volume,  as  indicated  in  Box  48. This  diagram  shows,  however,  that  the 

Conclusion 

application  of  the  four  techniques  does  not  constitute  a  complete 
evaluation. In particular, they do not concern the core of the evaluation, that 
is,  the  empirical  observation  of  the  results  and  impacts  produced  by  the 
measures  in  the  field  (step  framed  with  a  bold  line  in  the  diagram).  No 
evaluation  has  meaning  if  this  function  has  not  been  fulfilled.  Different 
strategies  can  be  adopted  for  the  empirical  observation  of  results  and 
impacts,  given  that  the  main  difficulty  here  consists  of  observing  several 
impact  clusters  for  many  measures  which  are  not  intended  for  the  same 
publics.  Each evaluation team  must  choose the  best approach  possible, in 
relation  to  the  characteristics  of  the  programme  and  its  publics.  Box  49 
shows  a possible solution, based  on the  principle  of  parallel  case studies. 

A global  approach  which  is above  all  quick 

There  are  several  ways  of  combining  the  alternative  techniques  presented 
in  this  Volume.  The  first  approach  presented  here  facilitates 
the 
implementation  of  an evaluation. This  approach  may  proceed as follows: 

•  Create  an extended  steering  group  (see pages 29 and 106). 

•  Identify expected  impacts in the programming documents  (see page  30). 

•  Define  between  five  and  seven  impact  clusters,  using  the  METAPLAN® 
technique,  according  to  the  procedure  presented  on  page 23. This  step 
requires  an  initial  meeting  with  the  steering  group,  possibly  combined 
with the validation  of the evaluation  plan. 

•  Evaluate the impacts  of the measures (this step is important  and  difficult, 

but  is common to  all types  of  evaluation). 

•  Combine  all the  qualitative  and  quantitative  conclusions  concerning  the 
impacts  of the  measures, in the form of  a synoptic  table (see Box  16). 

•  Transform  the  synoptic  table  of  conclusions  into  an  impact  scoring 
matrix, according to the simplified  procedure  presented on page 55. This 
step  necessitates  a second  steering  group  meeting,  possibly  combined 
with the  presentation  of the surveys carried out  by the evaluation team. 

•  Construct  the  matrix  of cross  impacts, analyse all the  potential effects  of 
synergy  and  produce  coefficients  of synergy  (see page 81 onwards). 

•  Ask  the  steering  group  to  weight  the  criteria, according  to  the  simplified 
procedure  presented  on  page  107.  This  step  necessitates  a  steering 
group  meeting,  possibly  combined  with  the  presentation,  by  the 
evaluation team, of  an initial series of  conclusions. 

Technical solutions for evaluation within a partnership  framework 

•  Calculate weighted global scores for all the measures (see page 111) and 
use them as a basis for global conclusions  and recommendations  for  the 
re-allocation  of  inputs. 

As Box 50 shows, this entire procedure  hardly  requires  more work than any 
good quality  evaluation. 

Box 50 -Time required for implementing a simplified approach 

Required time 
(depending on the complexity  of 
the  programme) 

Constitution  of extended  steering  group 

Identification  of expected  impacts  and 
constitution  of  clusters 

Evaluation  of  measures 

Synoptic  table of  conclusions  and 
impact scoring  matrix 

Identification  of  potential  synergy  and 
global analysis  of  effects  of  synergy 

Weighting  of criteria  and  calculation 
of weighted  scores 

Production  of evaluation  report, 
synthesised conclusions  and 
recommendations 

Additional time  compared 
to another  evaluation 

Min. no. of days 

Max.  no. of  days 

(not included  in 
the evaluation) 

(not  included  in 
the  evaluation) 

4 

15 

(as in any  other 
evaluation) 

(as in any other 
evaluation) 

3 

4 

3 

6 

15 

6 

(as in any  other 
evaluation) 

(as in any other 
evaluation) 

14 

42 

•  A global  approach  which  is above  all technically  rigorous 

In the second approach proposed, all four techniques are used in their  most 
rigorous  form.  The  approach,  which  may  be  applied  to  a  mid-term  or  ex 
post  evaluation, consists  of the following  steps: 

•  Constitute  several sectoral work  groups, as well  as an extended  steering 
group  which  will  function  as  a transversal  group  (see  pages  29,  38  and 
106). 

•  Draw the concept  map  of  impacts  (see pages  30 to 39). 

Conclusion 

•  For each impact  cluster, define a structured scoring scale (see Boxes 20 to 
22) based on the  impact statements  comprising  that  cluster  (see Box  12). 

•  Construct the matrix of cross impacts and study the synergy  between the 
measures  (see  pages  81 to  85) and  identify  synergy  to  studied  more in-
depth. 

•  Select  one  or  more  typical  projects  for  each  measure,  as  well  as  the 

projects  which  produce the most  potential  synergy. 

•  Conduct  a  survey  of  parallel  case  studies  of  all  the  selected  projects. 
Present  the  conclusions  of  each  case  study  in  the  form  of  a  series  of 
scores, using scoring  scales  constructed for the evaluation  (see Box  20) 

•  Write up qualitative conclusions, measure by measure, based on the case 
studies  and  on  all  the  other  available  and  relevant 
information. 
Summarise  all  the  conclusions  in  the  form  of  an  impact  scoring  matrix 
(see Box 19). 

•  Perform  the  multijudge  variant  of  multicriteria  analysis  (see  pages  99  to 

114). 

•  Conclude  the  evaluation  and  proceed  to  recommendations  on  the 

reallocation  of  budgets  or changes  in strategy  (see page 115). 

It  is not  realistic  to  give  indications  of  costs  for  the  implementation  of  this 
more technical approach  because it has to be finely adjusted to the context 
and  nature  of the  evaluated  programme.  It should  be noted, however,  that 
it  involves  a series of  meetings  with the  programme  partners  in the form of 
group  work,  steering  group  meetings  or  individual  interviews  with  the 
assessors  chosen for the multicriteria analysis. The approach also  includes 
empirical  observation  of  the  effects  of  the  measures.  It  therefore  involves 
visits  in the field and  interviews  with the  addressees. 

In  short,  because  of  the  concern  for  technical  quality,  the  evaluation  is 
scheduled  to  last  at  least  eight  months.  It  is  therefore  clear  that  this 
approach  is  more  cumbersome  than  those  adopted  until  now  by  many 
evaluations.  This  additional  investment  in evaluation  can  be justified  if  the 
return on investments is sufficient.  It should be remembered that the results 
which  can  be  expected  from  evaluations,  and  have  generally  not  been 
achieved satisfactorily  to  date, are: 

•  Conclusions  and  recommendations  which propose and justify  budgetary 
reallocations  and  changes  of  priorities  between  the  measures  of  a 
programme. 

Technical solutions for evaluation within a partnership  framework 

Conclusions  and  recommendations  based  not  on  absorption  rates  and 
completion  rates,  but  on  a  real  examination  of  impacts  and  on  the 
contribution  of the  measures to the overall success  of the  programme. 

Conclusions  and  recommendations  which  impartially  take  into  account 
all the opposing  points  of view,  in the framework  of  partnership. 

Appendix 

Bibliographie references 

Bana e Costa, CA. and Vansnick, J.C. (1994) 'MACBETH ­ An interactive 
path towards the construction of the cardinal value functions', in 
International Transactions in Operations Research, 1(4), 489­500. 

Guba, E.G., Lincoln, Y.S. (1989) Fourth Generation Evaluation, Newbury 

Park: Sage Publications. 

Hemerson, M.E., Morris, L.L., Fitz­Gibbon, G.T. (1987) How to Measure 

attitudes?, London: Sage publications. 

Knox, C. (1995) 'Concept mapping in policy evaluation: a research review 

of Community relations in Northern Ireland', in Evaluation, the 
International Journal of Theory, Research and Practice 1(1), 65­79. 

Miller, D. C. (1991) 'Guide for the selection and construction of social scales 

and indices', in Handbook of Research Design and Social 
Measurement, London: Sage publications, 174­188. 

Monnier, E, (1992) Evaluations de l'action des pouvoirs publics, Paris: 

Economica. 

OECD ­ DAC Expert Group on Aid evaluation (1996) The Role of Rating 
Systems in Aid Management Experiences of DAC Member 
Organisations, Paris: OECD (unpublished). 

Schärlig, Α., (1985) Décider sur plusieurs critères, Lausanne: Presses 

polytechniques et universitaires romande. 

Roy, B. (1997) 'Un chaînon manquant en Recherche Opérationnelle ­ Aide 

à la Décision : les conclusions robustes', Cahiers du LAMSADE, 
144, Paris: Université Dauphine. 

Toulemonde, J., Fontaine, O, Laudren, E., Vincke, Ph. (1998) 'Evaluation ¡n 

partnership: Practical suggestions for improving their quality', in 
Evaluation, the International Journal of Theory, Research and 
Practice, 4(2). 

Trochim, W.M.K. (1989), 'An introduction to concept mapping for planning 

and evaluation', in Evaluation and Program Planning, 12, 1­16. 

Vincke, Ph., (1989) L'aide multicritère à la décision, Bruxelles: Editions de 

l'Université de Bruxelles,. English translation (1992), Multicriteria 
Decision­aid, Chichester: Wiley. 

Technical solutions for evaluation within a partnership framework 

Yin,  R.K.,  Kaftarian,  S.J., Jacobs,  N.F.  (1996)  'Empowerment  Evaluation  at 
Federal  and  Local  Levels:  Dealing  with  quality',  Fettermann  &  al. 
Empowerment  evaluation,  London: Sage, 188-207. 

Yoon,  K.  P.  (1995)  Multiple  Attribute  Decision  Making,  London:  Sage 

publications. 

Young, F.W.W., Harris,  D.G. (1994)  'Multidimensional  Scaling', Norusis  M.J. 

SPSS Professional  Statistics:  SPSS  Inc. 

Index 

Index 

action  17,  8 1,  99 
addressee  18,  23,  50,  87,  89 
axis  17,  22,  8 1,  88 

Β 

best  practice  105,  114,  115 

cannibalism  effect  77 
casestudy  43,  67,  120,  123 
colour  vote  96 
commissioner  88 
comparability  64 
comparison  group  49 
compensation  104 
complementarity  68,  74,  76 
concept  mapping  of  impacts  11,  29, 

100,  122 

concerned  group  50 
co-ordination  76 
cost-benefit  analysis  95 
cost-effectiveness  analysis  94 
counterfactual  situation  76 
credibility  56,  94 
criteria  20,  96 
critical  mass  75 

dead-weight  50 
descriptor  6 1,  62,  64,  102,  119 
direct  addressee  95 

economic  and social cohesion  17,  21 
effectiveness  97 
efficiency  97 

evaluation  committee  56 
evaluation  team  23,  3 1, 49,  58,  103, 

110 

ex  ante  evaluation  12,  21,23,  99 
ex  post  evaluation  49 
expert  panel  56,  59,  61 
extrapolation  coefficient  49 

focus  group  31 
funding  authority  12 

global  impact  23,  27 
global  objective  17,  2 1,  24,  94 

Η 

harmonisation  64 

I 

impact  18 
impact  scoring  matrix  56, 58, 63,  67, 

100,  111 

impact  statement  32,  61 
impartiality  94,  112 
implicit  objective  27,  3 1,  97,.  100 
in-depth  evaluation  49 
indicator  19,  20,  2 1, 24,  44,  50,  64, 

96,  102,  119 

individual  interview  3 1, 89,  106,  107, 

112 

integrated  programme  73 
interdependence  76 
intermediate  impact  26,  100 
internal  coherence  76 
internal  synergy  75 

Technical solutions for evaluation within a partnership  framework 

logical  diagram  of  impacts  68 
logical  framework  23,  24 

M 

manager  26,  27,  3 1,  39,  79,  84,  89 
matrix  of  cross  impacts  12,  52,  8 1, 

82,  123 
measure  17,  43 
METAPLAN®  37,  96,  121 
mid-term  evaluation  2 1,  27,  49 
monitoring  2 1,  49 
multicriteria  analysis  12,  52,  96,  99, 

123 

multicriteria-multijudge  analysis  112 

N 

needs  97 
negative  synergy 
norm  97 

77 

objectives  tree  24,  26,  27,  82 
operator  26,  27,  3 1,  39,  50 
output  18 
outranking  104 
overall  evaluation  12,  13,  19,  20, 2 1, 

4 9 , 7 8,  117 

partnership  evaluation  96 
performance  50 
policy  priority  100 
programme  17 

project  18 
project  selection  59,  64 

qualitative  data  29 
quantification  95,  96 
questionnaire  survey  43 

ranking  73,  105,  112,  113 
rating  59,  107 
regression  analysis  49 
relevance  97 
result  18 
robustness  22 

sample  95 
scoring  1 2 , 3 4 , 5 1 , 5 5 , 5 8 , 8 1 , 8 6 , 9 6, 

107,  109,  123 

specific  impact  19 
specific  objective  21 
stakeholder  11 
steering  group  29,  56,  6 1,  96,  99, 

100,  105,  106,  116,  121,122 

success  24,  34,  112,  115 
SWOT  30,  96 
synergy  5 1,  73,  74,  79,  83,  87,  88 
synoptic  table  of  conclusions  5 1, 55, 

58,63 

w 
weighted  global  score  59 
weighting  37,  60,  73,  104,  107,  109, 

110,  111, 112,  115 

Acknowledgements 

The preparation of this Volume took  place over several years from  1993 to  1998. The 
preparatory  work  included  the following  steps: 

•  Several workshops  and meetings  allowed the points  of view of experts  in different 
themes and the transversal (or multi-thematic) perspective of those responsible for 
the Structural  Funds to  come  together. 

•  A series of  pilot  evaluations  allowed the technical  solutions  proposed  in this  book 

to  be tested  and stabilised. 

The production of this Volume was coordinated  by Miguel Benito Alonso and Andrea 
Mairate (D.G.  XVI.G.2  : Coordination of evaluation) and involved several Directorate-
Generals  of  the  European  Commission,  notably  D.G.  II, V, VI,  XIV  and  XVI  and  in 
particular  Philippe Goybet  and Anastassios  Bougas. 

This  Volume  has  been  written  by  the  Centre  for  European  Evaluation  Expertise, 
notably  with  the  participation  of  C.  Diaz-Gomez  and  Céline  Fontaine.  It  was 
translated  into  English by  Liz  Libbrecht. 

The  authors  wish  to  thank  all  those  who  contributed,  either  directly  or  indirectly,  to 
the  production  of this Volume, and in particular  the following  experts: J.Aldas  (IVIE  -
Valencia), J. Anselmo (Université Bocconi -  Milan), C. Bana e Costa (CESUR, Lisboa), 
J.Bignè  (IVIE  -  Valencia),  J.Cuadrado  (QUASAR  -  Madrid),  C.Gillo  (Ministère  de 
l'Equipement,  Paris),  S.Green  (Department  of  Environment,  London),  C.  Knox 
(University of  Ulster), J.Malone  (Ernst & Young, London), S Martin (Warwick  Business 
School,  United  Kingdom),  C.  Mouilleron  (RIDER  -  University  of  Louvain,  Belgium), 
M.Rey  (CEAT, Lausanne,  Switzerland),  P. Tyler  (Cambridge  Policy  Consultants,  UK), 
Ph.Vincke  (Université  Libre de  Bruxelles), RViveret  (Cour des Comptes,  Paris). 

The authors  wish also to thank  everyone from  regional  institutions  who  co-operated 
very  openly  to  the  pilot  evaluations,  and  more  particularly:  Javier  Quesada-lbanez 
(Communauté  autonome  de  Valencia,  Spain),  Luc  Vandendorpe  (Région  VVallonne, 
Belgium),Manuel  Vaquero  et  Delphine  Vincent  (Région  Poitou-Charentes  et 
Préfecture de  Région, France). 

European  Commission 

Evaluating  socio-economic  programmes 
Technical solutions for evaluation  within a  partnership  framework 

Luxembourg: Office for  Official  Publications  of the  European Communities,  1999 

1999—  133  pp. —  14.8x21  cm 

ISBN  92-828-6626-2 

Venta  ·  Salg  ·  Verkauf  ·  Πωλήσεις  ·  Sales  ·  Vente  ·  Vendita  ·  Verkoop  ·  Venda  *  Myynti  · Försäljning 

BELG1QUE7BELG1É 

ÖSTERREICH 

J un De Lannoy 
Avenue du Rol 202/Kcningsiaan 202 
B­1190 Bruxelles/B russel 
T«. (32-2)538 43 06 
Fax (32­2) 538 08 41 
E­mail fean de lannoy Omfoboard be 
URL: http "www ¡ean­do­lannoy bo 

La librairie européenne/ 
De Europese Boekhandel 
Rue de la Loi 244/Wetslraat 244 
B­I040 Bruxelles/Brussel 
Tél. (32­2) 295 26 39 
Fox (32­2) 735 08 60 
E­mail: mailOhbeurop be 
URL: http "www.iibeurop be 

Moniteur belge/Belgisch Staatsblad 
Rue de Louvain 40­4 2/Le uve rise weg 40­42 
B­1000 Bnjxelles/Brusser 
Tel. (32­2) 552 22 11 
Fax (32­2) 511 0184 

DANMARK 

J. H. Schultz Information A/S 
Hersledvang 10­12 
DK­2620 Albertslund 
TU. (45)43 63 23 00 
Fax (45) 43 63 19 69 
E­mail: scbultzO schnitz dk 
U RL http //www schutti.dk 

DEUTSCHLAND 

B undosa n ie I ger Verlag GmbH 
V e π r ιβΟ sa b ι e 11 ung 
Amsterdamer Straße 192 
D­50735 Köln 
Tel (49­221) 97 66 BO 
Fax (49­221 ) 97 66 B2 78 
E­Mail: vertrieb O bundesanzeigef de 
U R I: httpy/www­bundesanzetger de 

ΕΛΛΑΔΑ/GREECE 

G. C. EleflheroudakJs SA 
Inlemat onal Bookstore 
Panepisümiou 17 
GR­10564 Athina 
Tet(30­1)331 41 8071/2/3/4/5 
Fax (30­1) 323 98 21 
E­mail: elebooksOnetor.gr 

ESPAÑA 

Boletín Oficial del Estado 
Trafalgar. 27 
E­29071 Madrid 
Tel (34)915 38 21 11 (Libros). 
913 84 17 i 5 l S u s c n p) 
Fax (34) 915 38 21 2t (Lx&ros). 
913 84 17 14(Suscnp) 
E­mail: Chentes O com boe es 
URL: http­Jnffww boe es 

Mundi Prensa U b r o s, SA 
Castello. 37 
E­28001 Madrid 
Tel (34)914 36 37 00 
Fax (34)91575 39 98 
E­mail: lxxeriaOmundiprensa.es 
URL: http y/www. mundxxensa com 

FRANCE 

Journal otticici 
Service des pubhcabons des CE 
26. rue Desa« 
F­75727 Paris Cedex 15 
Tel (33)140 58 77 31 
Fax (33) 140 58 77 00 
URL: httpJ/www journal­officiel gouv Ir 

IRELAND 

Government Supplies Agency 
Pubbcalions Section 
4­5 Harcourt Road 
Dublin 2 
Trt (353­1)661 31 11 
Fax (353­1) 475 27 60 

ITALIA 

Licosa SpA 
Via Duca di Calabria. 1/1 
Casella postale 552 
1­50125 Firenze 
Tel (39)055 64 83 1 
Fax (39)055 64 12 57 
E­mail licosaOftbccit 
URL: hitp www ttbcc rl1>cosa 

LUXEMBOURG 

Messageries du livre SAHL 
5, rue Rarrleisen 
L­2411 Luxembourg 
Tel.(352)40 10 20 
Fax (352) 49 06 61 
E­mail: maHOmdllu 
URL http//www moi lu 

NEDERLAND 

SDU Servicecentrum Uitgevers 
Chnstoflel Plantx/islraat 2 
Postbus 20014 
2500 EA Den Haag 
Tel. (31­70) 378 98 80 
Fax (31­70) 378 97 83 
E­rnaü" sduOsdu nl 
URL: http /«ww sdu nl 

Universi ta t sbuchhendlung GmbH 
Kohlmarkt 16 
A­1014 Wien 
Tel. ( 4 3 ­ 1 ) 5 3 1 6 1 1 00 
Fax(43­1)53 16 1l 67 
E ­ M ail : beste ι len Ο ma n ζ co. a I 
URL: http J/www.manz.al/index htm 

PORTUGAL 

Distribuidora de Livros Bertrand Ld." 
Grupo Bertrand. SA 
Rua das Terras dos Vales. 4­A 
Apartado 60037 
P­2700 Amadora 
Tel. (351­1)495 90 50 
Fax(35l­1,496 02 55 

Imprenta Nacional­Casa da Moeda, EP 
Rua Marques Sa da Bandeira, 16­A 
P­1050 Lisboa Codex 
Tel (351­1)353 03 99 
Fax (351­1) 353 02 94 
E­mail: oal.incmOrruil.telepac.pt 
URL: hrtp^/www incm.pt 

SUOMI/FINLAND 

Akateeminen Kirjakauppa/ 
Akademiska Bokhandeln 
Keskuskatu 1/Centralgatan 1 
PL/PB 128 
FIN­00101 Helsinki/Helsingfors 
Ρ ./tfn (358­9) 121 44 18 
F/lax (358­9) 121 44 35 
Sähköposti: akaiitausOakaleemnencom 
URL: httpJ/www akateeminen com 

SVERIGE 

B T J AB 
Traktorvägen 11 
S 221 82 Lund 
TIn (46­46) 18 00 00 
Fax (46-46) 30 79 47 
E­post: bt)eu­pubObt| se 
URL: httpJ/www bt| se 

UNITED KINGDOM 

The Stationery Office Ltd 
International Sales Agency 
51 Nine Elms Lane 
London SW8 5DR 
Tel (44­171)873 90 90 
Fax (44­171)873 84 63 
E­mail: ipa enquiries O theso.couk 
URL: http^/www Ihe­stabonery­otfice co uk 

ISLAND 

Boka bud Larusar Blondel 
SkOlavôrdustig. 2 
IS­101 Reykjavik 
Tel. (354)551 56 50 
Fax(354)552 55 60 

NOflGE 

S wet β Norge AS 
Ostenjoveien 18 
Boks 6512 Etterstad 
N­0606 Oslo 
Tel. (47­22) 97 45 00 
Fax (47­22) 97 45 45 

SCHWEIZ/SU ISSE/S VIZZE ΠΑ 

Euro Info Center Schweb: 
c/o OSEC 
Stamplenbachsiraße 85 
PF 492 
CH­8035 Zurich 
Tet (41­11365 53 15 
Fax ¡41­1) 365 54 11 
Ε­maa: eicsOosec eh 
URL: tinp ,'.www osoc ctveica 

BALGARUA 

Europres« Euromedia Ltd 
59. btvd Vitosha 
BG­1000 Sol­a 
Tel (359­2)980 37 66 
Fai (359­2) 980 42 30 
E­mail: MilenaOmbox.at.bg 

CESKAREPUBLIKA 

ÚSIS 
NIS­piodejna 
Havelkova 22 
CM 30 00 Praha 3 
Tel (420­2) 24 23 14 86 
Fax (420­2) 24 23 11 14 
E­mail: nkpospOdec nis cz 
URL http /.usiscr cz 

CYPRUS 

Cyprus Chamber of Commerce and Industry 
PO Sox 1455 
CY­1509 Nicosia 
Tel (357-2)66 95 00 
Fax (357­2)661044 
E­mail oametiapOccci org cy 

HRVATSKA 

Medlalrada Lid 
Pavia Hatza 1 
HR­10000 Zagreb 
Tel. (385­1)481 94 1 
Fax (385­1)481 94 1 

MAGYARORSZÂG 

Euro Info Servie· 
Europa Haz 
Margitsziget 
PO Box 475 
H­1396 Budapest 62 
Tel (36­1)350 60 25 
Fax (36­1)350 90 32 
E­mail: euroinloOmail.matav.hu 
URL: hup .'Vwww euiomlo hu'mdox him 

MALTA 

Miller Distributor» Ltd 
Matta International Airport 
PO Box 25 
Luqa LOA 05 
Tel.(356)66 44 88 
Fax (356) 67 67 99 
E­mail: gwirthOusa nel 

POLSKA 

Ars Polona 
K ra k ow s k ie Priedmiescie 7 
Skr. pocztowa 1001 
PL­OO­950 Warszawa 
Tel (48­22)826 12 01 
Fax (48­22) 826 62 40 
E­mail ars_potObevy hsn com pi 

ROMANIA 

Euromedia 
Str. G­ral Benhelot Nr 41 
RO­70749 Bucuresti 
Tel (40­1)315 44 03 
Fax (40­1 )314 22 86 

ROSSIYA 

CCEC 
60­letrya Oktyabrya Av. 9 
117312 Moscow 
Tel. (7­095) 135 52 27 
Fax (7­095) 135 52 27 

SLOVAKIA 

Centrum VTl SR 
Nâm.Slobody. 19 
SK­81223 Bratislava 
Tel. (421­7) 54 41 83 64 
Fax (421­7) 54 41 83 64 
E­mail: europOtbbl sltkstuba.sk 
URL· http^www.sttk sluba sk 

SLOVENUÀ 

Gosp­xiarskl Vestnlk 
Dunajska cesta 5 
SLO­l00OL|ubl|ana 
Tel ( 3 8 6 ) 6 1 3 0 9 1 6 40 
Fax (386) 6 1 3 0 9 1 6 45 
E­mail europOgvestnik si 
URL: httpJnvww rjveslrMk si 

TURKIYE 

Oünya Intotel AS 
100. Yd Mahallessi 34440 
TR­80050 BaocUar­lslanbul 
Tel. (90­212)629 46 89 
Fax (90­212) 629 46 27 
E­mail infotelOdunya­gazate.com tr 

AUSTRALIA 

Hunter Publications 
PO Box 404 
3067 Abbotsford. Victoria 
Tel. (61­3)94 1 7 5 3 61 
Fax (61­3) 94 19 71 54 
E­mail : jpda vies O oz ema il com. au 

CANADA 

Les editions La Liberté Inc. 
3020. chemin Samte­ Foy 
Gl X 3V Sainte­Foy. Québec 
Tel, (1­418)658 37 63 
Fax (1­800) 567 54 49 
E­mail libetleOmediomqc ca 

Renouf Publishing Co. Ltd 
5369 Chemin Canotek Road Unit 1 
KU 9J3 Ottawa. Onlario 
Tel. (1­613) 745 26 65 
Fax (1­613) 745 76 60 
E­mail: order.deptOrenoufbookscom 
URL: http7/www renoufbooks.com 

EGYPT 

The Middle East Observer 
41 Sher.i Streel 
Cairo 
Tel. (20­2)392 69 19 
Fax (20­2) 393 97 32 
E­mal mafouoaOmeobservef com eg 
URL: h tip .//www meob server com eg 

EESTI 

INDIA 

Eesti Kaubandus­Tooatuekoda {Estonian 
Chamber ol Commerce and Industry) 
Toom­Koc* 17 
EE­0001 Tan­nn 
Tel (372) 646 02 44 
Fax(372)646 02 45 
E­mail: einfo O koda ee 
URL: htlp : www koda ee 

EBIC India 
3rd Floor. Υ. Β Chavan Centre 
Gen J, Bhosale Marg 
400 021 Mumbai 
Tel (91-22)282 60 64 
Fax (91 ­22) 285 45 84 
E­mail: ebicOçaasOmOl .vsnl.net.in 
URL· httpJ/www ebKindia com 

ROY International 

41. Mishmar Hayarden Street 
PO Box 13056 
61130 Tel Aviv 
Tel. (972­3)649 94 69 
Fax (972­3) 848 60 39 
E­mail: royilOnetvision.net.il 
URL· httpJ/www.royint.co it 

Sub­agent lor the Palestinian Authority: 

Index Information Services 

PO Box 19502 
Jerusalem 
Tel. (972­2) 6 2 7 1 6 34 
Fax (9722)627 12 19 

JAPAN 

PSI­Japan 

Asani Sanbancho Plaza 1206 
7­1 Sanbancho. Chiyoda­ku 
Tokyo 102 
Tel. (81­3)32 34 69 21 
Fax (81­3)32 34 69 15 
E­mail: booksOpsi­)apan.co jp 
URL: http .'/www psi­|apan.com 

EBIC Malaysia 

Level 7, Wisma Hong Leong 
18 Jalan Perak 
50450 Kuala Lumpur 
Tel. (60­3) 262 62 9B 
Fax (60­3) 262 61 98 
E­mail: ebic­klOmol.nelrny 

Mundi Prensa Mexico, SA de CV 

Río Panuco No 141 
Colonia Cuauhtemoc 
MX­06500 Mexico. DF 
Tel (52-5) 533 56 58 
Fax (52­5) 514 67 99 
E­mail 101545 2361 Ocompuserve.com 

EBIC Philippines 

I DIN Floor, PS Bank Tower 
Sen. Gil J. Puyal Ave cor Tindalo St. 
Maketi City 
Metro Manilla 
Tel. (63-2) 759 66 80 
Fax (63-21 759 66 90 
E­mail: eccpcomQglooe.com ph 
URL: httpjAvww.eccp.com 

EBIC Sri Lanka 

Trans Asia Hotel 
115 Sir chittampalam 
A Gardiner Mawatha 
Colombo2 
Tel. (94­1)074 71 50 78 
Fax (94­1)44 87 79 
E ­ mail : ebicsl O Umin com 

EBIC Thalland 

29 Vanisse Building, Sth Floor 
SoiCh.dlom 
Ptoenchil 
10330 Bangkok 
Tel (66­2)655 06 27 
Fax (66­2) 655 06 28 

• E­mail: eblcbkkOksclS.th.com 
URL: http /www.ebÆbkk org 

UNITED STATES OF AMERICA 

Bornen Associates 

4611­F Assembly Dnve 
U n h am MD207Ô6 
Tel (1­800) 274 44 47 (lollfree telephone) 
Fax (1­800) 865 34 50 (loll free lax) 
E­mail: query O beman com 
URL: hftpj/vnvw bernan.com 

Bitte wenden Sie sich an «In Büro Ihrer 
Wahl/ Please coniaci Ih« salts office 
of your choice/ Veuille* vous adresser 
su bureau de vente de votre choix 

Office for Officiel Publications 
of the European Communities 
2. rue Mercier 
L­2985 Luxembourg 
Tel (352) 29 29­42455 
Fax (352) 29 29­42758 
E­mail: info infoOopoce cec.be 
URL· http://eur­opeu.lnt 

The  MEANS collection: 
"Evaluating  socio-economic  programmes" 

This  six-volume  set  gives  wider  public  access  to  the  lessons  of  the  MEANS 
programme  which the  European  Commission  introduced  with a view to  Impro­
ving  and  promoting  evaluation  methods.  Its  publication  meets  a  growing 
need for  reliable  benchmarks  to  deal  with  the  problems  of  evaluation  encoun­
tered when  implementing  public  measures  in general and Structural  Fund ope­
rations  in  particular. 

In  this  context,  the  set  constitutes  a  unique  methodological  guide  that  can 
solve  the  technical  and  organisational  problems  arising  in  the  course  of 
evaluation  work  and  for  which  the  existing  literature  provides  little  help. 

The  collection  is  aimed  not  only  at  professional  evaluators,  for  whom  it  will 
provide  a valuable handbook,  but  also at  non-specialists,  in particular  manag­
ing  authorities  and  structural  programme  Monitoring  Committees. 

Evaluation  design  and  management 

Each  volume  is  designed  to  be  independent  of  the  others,  yet  they  are  also 
complementary.  Together, they deal with  all aspects  of the evaluation  of  socio­
economic  programmes: 
I. 
II.  Selection  and  use  of  indicators  for  monitoring  and  evaluation 
III.  Principal  evaluation  techniques  and  tools 
IV.  Technical  solutions  for  evaluation  within  a  partnership  framework 
V.  Transversal  evaluation  of  impacts  on  the  environment,  employment  and 

other  intervention  priorities 

VI.  Glossary  of  300  concepts  and  technical  terms 

OFFICE FOR OFFICIAL PUBLICATIONS 
OF THE EUROPEAN COMMUNITIES 

L-2985 Luxembourg 

EC Structural Funds 

iluating socio­economic programmes 

Transversal evaluation of impacts on the 

¡bther* 
■¡ties 

MEANS 
Collection 

Volume 5 

opean Commission 

EC Structural Funds 

Evaluating socio-economic programmes 

Transversal  evaluation  of  impacts 
on the  environment,  employment 
and  other  intervention  priorities 

Volume 5 

European Commission 

MEANS is a programme  of  the European Commission. 
It is aimed at improving  Methods  of Evaluating Structural  Policies. 
The MEANS Programme and  Collection  was supervised by 
Phillipe  Goybet and Miguel Angel Benito  Alonso, 
Unit for  the Coordination  of  Evaluation in the Directorate  for  Coordination 
and Evaluation of  Operations of DG XVI 
Coordination-evaluation@fmb.dg16.cec.be 

The realisation of  the Programme  was entrusted  to the 
Centre for European Evaluation Expertise  (C3E) 
under  the direction  of Eric Monnier and Jacques Toulemonde. 
Responsibility  for  the opinions  expressed in this 
book  lies solely with  the authors of  the text: 
C3E  -  13 bis Place Jules Ferry  -  F  -  69006 Lyon  -  France  -  c3e@c3e.fr 

A great deal of additional information on the European Union is  available 
on the Internet. 
It can be accessed through the Europa server (http://europa.eu.int) and 
on the Inforegio Website (http://inforegio.cec.eu.int). 

Cataloguing data can be found at the end of this  publication. 

Luxembourg: Office for Official Publications of the European 
Communities, 1999 

ISBN  92-828-6626-2 

©  European Communities, 1999 
Reproduction is authorized, except for commercial purposes,  provided 
the source is acknowledged. 

PRINTED ON WHITE CHLORINE-FREE PAPER 

Printed in Italy 

Foreword 

Developing  good  evaluation  practices 
has  become  a  priority  area  for 
the 
is  directly  relevant 
Commission  and 
to 
the  effective  use  of  Community 
resources. 
Indeed, the sound financial  management 
principles  which  the  Commission  seeks 
to  promote  via  its  SEM2000  (Sound  and 
Efficient  Management)  Programme  rely, 
specifically,  on  the  regular  evaluation  of 
the Community  activities. 
The  Directorates  General  responsible  for 
structural policy have considerable expe­
rience in this area.  Since  1988 they  have 
made  evaluation  an  integral  part  of  each  stage  in  the  process  of 
Community  Structural  intervention.  Thus,  the  programme  development 
stage  includes  a  prospective  evaluation;  the  implementation  stage  a 
strengthening of the monitoring systemsand, at the end of the intervention, 
evaluation  establishes  what  has  been  achieved  in  terms  of  programme 
impacts. 
However, the  capacity  of  evaluation  to  contribute  to the effectiveness  and 
transparency  of  structural  interventions  rests  ultimately  on the  integrity  of 
the  methods  adopted  to verify the credibility  of  results. 
This  is why, four  years  ago, on the initiative for  the  Directorate  General  for 
Regional  Policy  and  Cohesion  (DG  XVI),  the  MEANS  programme  was 
initiated as a response to the demand for  developing  evaluation  methodo­
logy.  This  also  include  practices  combining  both  general  scientific 
requirements  and  the  specific  requirements  of  Community  interventions 
whilst, at the same time,  remaining  accessible  to  a large audience. 
One of  the  more tangible  results  of  the  MEANS  programme  has  been  the 
publication of this six volume set  by a group  of independent  experts, focu­
sing  on the  principal  elements  of  socio-economic  programme  evaluation. 
A short summary  of this work can be found at the beginning  of the  present 
volume. This collection covers the various methodological  approaches  and 
innovations  that  have  taken  place  in  the  context  of  structural  funds  eva­
luation.  It  will  enable  those  responsible  for  evaluation  to  develop  their 
approach  in  a  flexible  and  independent  manner  and  for  the  various 

programme  partners  to  exercise  greater  control  over  the  evaluation  pro­
cess.  Attending  the  conferences  of  evaluation  specialists  and  "clients" 
convened  in  Brussels  (1995),  Berlin  (1996)  and  Seville  (1998),  I  gained 
some  impression of the  interest that  new evaluation techniques  can gene­
rate.  I would  stress that  the  development  of  evaluation  needs to  progress 
in  tandem  with  efforts  to  incorporate  evaluation  findings  in  the  decision 
making  process. This  implies  evaluation  as a  means  of  establishing  whe­
ther  policies  are equitable,  effective and  relevant to the  identified  needs. 
In satisfying  these  requirements,  evaluation  should  be seen  as  a  valuable 
instrument  affording  both  Member  State  and  Community  authorities  the 
necessary  guarantees  that  Community  funds  are  being  deployed  in  the 
best  interests of the EU citizen. This is a cost worth  paying when the goals 
of  Economic  and  Social  Cohesion  and  the  European  Union  enlargement 
are at stake. 

Finally,  I wish  to  extend  my  thanks  to  the  members  of  the  Committee  of 
Independent  Experts  established  at  the  launch  of  the  programme.  Their 
expertise and standing have contributed  greatly to the quality and relevan­
ce  of  the  MEANS  programme.  I therefore  thank:  E.  Chelimsky  (USA),  ex-
President  of  the  "American  Evaluation  Association";  K.  Kennedy  (IRL), 
Director  of  the  "Economic  and  Social  Research  Institute"  (ESRI);  J.  R. 
Cuadrado  Roura  (SP),  President  of  the  "European  Regional  Science 
Association"; C. Seibel (F), Director of the "Statistiques démographiques et 
sociales de l'Institut  National de la Statistique et des Etudes  Economiques" 
(INSEE);  L.  Senn  (I),  Director  of  the  Department  of  Economic  Regional  of 
the  University  Bocconi;  E. Stern  (U.K.), Director of the  unit of evaluation  of 
the  Tavistock  Institute;  A.  Sorber  (t)  (NL),  Head  of  the  "Policy  Analysis 
Department of the Ministry of  Finance"; L. I. Strom (SV), Statens  Institut för 
Regionalforsking  (SIR);  L.  Tsoukalis  (GR.),  professor  at  the  University  of 
Athens;  H. Wollmann  (D), professor  at the  University  of  Humboldt  -  Berlin 
and founder  member  of the "European  Evaluation  Society". 

E. Landaburu 

The  MEANS  Collection: 
"Evaluating socio-economic programmes" 

The  evaluation  of  European  Union  structural  interventions  has 
entered into a phase of maturity, attested  by the extension  of  work 
carried out by both the Member States and the Commission. Regu­
lations based on decisions  by the Council  have largely  contributed 
to  the  development  of  evaluation  practice  and  have  stimulated 
methodological research. 

Over the past ten years, the Commission has striven to promote and 
improve methods for evaluating Structural Funds. The main metho­
dological conclusions have been debated and disseminated as part 
of  the  MEANS  programme  initiated  by the  Commission  in 1991. 
Achievements  to date  provide  enough  material for a complete  and 
accessible  publication  intended  for  a broad  public:  a six  volume 
Collection covering all aspects of the evaluation of  socio-economic 
programmes. These Volumes are written  with  a perspective that  is 
broader  than the current  administrative  framework,  for  this  is  likely 
to evolve more rapidly than professional standards. 

In general, the Collection Is aimed at a readership of  non-specialists 
seeking  essential  information on the  evaluation  of  socio-economic 
development  programmes. These programmes have multiple objec­
tives, a partnership dimension and a multi-sectoral content - all cha­
racteristics which call for suitable evaluation methods and solutions 
not  yet  provided  by  the  international  technical  literature.  The 
complete Collection consists of 6 volumes  designed to  be comple­
mentary and to provide essential information and  recommendations 
for professionals wanting to independently develop their own evalua­
tion practices.  Each volume can, however,  be read and  understood 
without having to refer to the other volumes in the Collection. 

The  first  Volume  in  the  MEANS  Collection  entitled  "Evaluation 
design and management"  serves as an introduction both to the Col­
lection  and  to  the  evaluation  of  socio-economic  programmes.  It 
enables  readers to understand the implications of evaluation and to 
know  what  they  can  and  cannot  expect  from  an  evaluation  at a 
given time in a particular context.  It covers  all the elements  needed 
to steer an evaluation project from A to Z: sharing of responsibilities, 
defining  of  the  aims  and  method  of  an evaluation,  establishing  a 

schedule  and  budget,  assessing  the  quality  of  the work  performed 

and, finally, managing the use of conclusions. 

The second Volume focuses on the  "Selection  and use of  indicators 

for monitoring  and evaluation".  It deals with the subject  of  indicators 

used for monitoring  and evaluation. It shows  how indicators can  be 

used  to  support  policy  decisions,  improve  management  and 

measure  the  overall  performance  of  programmes.  The  volume 

proposes a methodological framework  and a practical  guide for  the 

use of indicators in these different situations. 

The third Volume presents a selection of  "Principal  evaluation  tech 

niques  and  tools"  applicable  to  the  evaluation  of  socio-economic 

programmes.  These  tools  are taken  from  relevant  scientific  disci­

plines:  economics,  sociology,  management,  geography,  etc.  Each 

one of the  23 tools  Is described  in the form of an information  sheet 

featuring a general description, the principles of  its implementation, 

a list of its strengths and weaknesses for the evaluation of Structural 

Funds, an example of its application and a list of references. 

A number  of methods adapted to the evaluation  of  socio-economic 

programmes  are the  object  of  a  more  detailed  description  in  the 

fourth  Volume  entitled  "Technical  solutions  for  evaluation  within  a 

partnership  framework".  Several  pilot  evaluations  were  implemen­

ted  within the  MEANS  programme  to design  and test  the  following 

four  innovations,  the  application  of  which  is  described  in  detail  in 

this Volume: impact  mapping, cross-impact  matrix, score cards and 

multi-criteria analysis. 

The fifth Volume deals with  "Transversal evaluation of impacts  on the 

environment,  employment  and  other  intervention  priorities"  of  the 

European  Union  and  Member  States.  Among  the  questions 

addressed are impacts  in terms of equal opportunities between  men 

and women and the competitiveness of SMEs. The Volume provides 

the elements required for an analysis of those impacts which are not 

necessarily  part of the explicit objectives of all the actions evaluated. 

Finally,  the  last  Volume  is  a  "Glossary  of  concepts  and 

technical 

terms".  The  compilation  of  the  Collection  has  necessitated  the 

rigorous  standardisation  and  definition  of  the  terms  used.  Each  of 

the  first  five  volumes  has  an  index  of  terms  which,  along  with 

hundreds of definitions, are brought together  in the glossary. 

Contents 

Introduction 

I  Technical considerations relating to the assessment 

of thematic impacts 

II  Thematic analysis of impacts on political priorities 

Evaluating the environmental impact of structural interventions 
Evaluating the impact of structural  interventions 
on the competitiveness of Small and Medium-Sized Enterprises 
Evaluating the impact of structural interventions on employment 
Evaluating the impact of structural interventions 
on equal opportunities 

III  Thematic evaluations of domains of interventions 

Evaluating impacts arising from interventions 
in the domain of Research and Technological Development 
Evaluating impacts arising from interventions 
in the domains of transport, telecommunications and tourism 

IV  Framework for performing thematic evaluations 

A step-by-step approach for thematic analysis 
Cross programme thematic evaluation 

Conclusion 

Appendix  1  - Evaluation tools and corresponding themes 
Appendix 2 - Extracts from a Terms of Reference 
for a programme evaluation incorporating thematic elements 
Appendix 3 - An example of terms of Reference 
for a transnational thematic evaluation 
Appendix 4 - List of boxes 

Index 

7 

13 

27 

29 

59 
77 

107 

129 

131 

151 

167 

169 
179 

189 

195 

197 

199 
209 

213 

Introduction 

Economic  and  social  cohesion  is  the  global  objective  of  programmes 
financed by the Structural Funds. The objective has three elements:  catching 
up of lesser-developed regions, social and  economic  reconversion of urban 
and  rural areas and  reintegration  of social groups  in difficulty.  Interventions 
implemented by way of these main objectives must try to satisfy four political 
priorities common to the European Union and the Member States (see Box 1). 

These priorities  are  most  often  recalled  in  the preamble  of  programmes. 
However, axes entirely devoted  to them within a programme  are not neces 
sarily found. A  sub-programme,  an environmental  priority,  or even  specific 
measures  concerning  the  competitiveness  of  SMEs are sometimes  to  be 
found.  But  countries  or regions have rarely programmed  specific  measures 
focusing on equal  opportunities. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Box  1 -  Priorities, domains  and  themes 

The political priorities defined for the 1994-99  programming  period of the 
Structural Funds comprise: 
•  the fight against unemployment and the creation of new jobs; 
•  increasing the competitiveness  of  productive structures, more particu­
larly SMEs; 
•  protecting the environment; 
•  and finally, reducing inequality between men and women. 
The notion of  "domain" refers to specific  domains in which  interventions 
take  place and, therefore, outputs  are programmed. Thus, for  example, 
impacts  in terms  of  "competitiveness"  are the consequence of  interven­
tions  in the  domains  of  research  and  development,  transport,  telecom­
munications, etc. 
The notion of "theme" is used in this book to cover both political  priorities 
and  domains  of  intervention.  A  thematic  evaluation  might  therefore 
examine if a priority  has been respected in all of the domains of interven­
tion. On the other hand, another thematic  evaluation might  be  interested 
in a given domain and examine all of the possible impacts (notably for the 
four priorities mentioned above). 

The four  priorities,  which  differ  in  importance  according  to  the  context,  are 
transversal  in relation to the programmes  financed  by the  Structural  Funds. 
All managers and operators are supposed to apply them in the implementa­
tion of  measures, in project  selection, in the  definition of target  publics, etc. 

Most  often, the  evaluation  of  a programme  favours  an analysis  of  impacts, 
which  are  explicitly  described  by  the  principal  objectives  of  priorities  and 
measures. It tends to  neglect  transversal  priorities, which  concern, in a way 
that  is  less  precise,  the  entirety  of  programme  measures.  Progress  is 
therefore necessary  if reports are to contain  reliable thematic  conclusions  at 
a reasonable cost. These conclusions  must  be presented  in a  standardised 
and sufficiently  synthesised form in order that the Commission Services  can 
themselves use them to produce a European-level synthesis. 

Moreover,  in accordance  with  the  principle  of  subsidiarity, the  Commission 
has to periodically  evaluate the different  modes of intervention applied in the 
Member  States  in order to  identify  best  practices  and  organise the transfer 

Introduction 

of know­how in Europe. This is done through evaluations performed at the 
European level, with a domain of intervention (transport infrastructure, 
support for research activities, professional training, etc.) being subject to an 
in­depth analysis. These domains of intervention are generally found, to 
greater or lesser degree depending on the case, in the majority of regional 
programmes. 

■ A Volume for Community and programme level 

commissioners of evaluations 

This Volume is destined for those who, at the programme or European Com­
mission level, want to undertake a transversal examination of impacts cor­
responding with a priority or domain of intervention. 

These "thematic" evaluations are designed differently according to whether 
they are performed at the programme level itself or the European level. At the 
programme level, thematic evaluation covers both the direct impacts of 
measures devoted to a given priority (a measure in favour of the environ­
ment, for example) and the indirect consequences of other measures (envi­
ronmental consequences of road building, for example). 

In relation to a given theme, this first type of evaluation is a synthesis of the 
entirety of possible impacts, both direct and indirect, and generally verifies 
the overall coherence (and the mutual reinforcement) of the different 
measures contained in a programme. The conclusions of this type of evalua­
tion are often placed in a series of chapters dedicated to each of the themes 
at the end of an evaluation report. 

The second type of thematic evaluation, for which the Commission Services 
are responsible, is to produce conclusions relevant for a number of regions 
or countries of the European Union. These evaluations commence with a 
"trans­programme "analysis of the impacts of similar interventions in order 
to identify the most efficient, and to distinguish, in the case of a successful 
intervention, between what is proper to the economic, administrative or ins­
titutional context and lessons that can be applied to other territorial, regional 
or national units. This type of evaluation highlights the relative efficiency of 
interventions and plays a central role in the identification of best practices. 

This Volume contains advice and solutions targeted at public officials who 
want to include these themes in their terms of reference and evaluation 

Transversal evaluation of impacts on intervention priorities 

teams wanting to design a work programme (see examples of terms of 
reference in the appendix). 

In order to take on the challenge of obtaining reliable thematic conclusions at 
a reasonable cost, in the following pages analytical grids and methodologi­
cal solutions can be found for: 
• identifying potential impacts of a programme in relation to a theme ; 

• transversally taking into account impacts within the same programme or 
within group regional or national programmes; 

• identifying and defining suitable indicators for monitoring and evaluating 
these impacts; 

• collecting and processing appropriate information; 

• and finally, presenting a synthesis of the conclusions of an evaluation. 

■ T he structure of t he Volume 

This Volume does not claim to deal with specialist concepts and methods 
relating to the themes tackled. It could easily be imagined, for example, that 
experts in employment policy would consider that the subject has not been 
sufficiently developed, or that environmental specialists would judge that the 
analysis of ecological impacts is highly superficial. Indeed, the raison d'être 
of this volume arises from the unrealistic demands of experts of different 
themes concerned by structural programmes. 

The challenge for programme evaluation is to analyse a programme's merits 
in respect to its own objectives and in relation to the four political priorities 
common to the European Union and the Member States. For reasons of time 
and budget, an approach which simplifies each theme has be taken. In order 
to do this, concepts and solutions specific to each area of expertise have to 
be selected, and a trade­off has to be made between reliability and feasibility. 

The text is organised in four parts. General technical considerations appli­
cable to all thematic evaluations are brought together in the first part of the 
Volume, so as to highlight the points common to all themes. It should be 
noted that the development of certain points is limited and these points are 
more fully developed in the first and third Volumes of the Collection. Volume 
1 of the MEANS Collection describes the entire evaluation procedure and 
Volume 3 presents a set of tools for the collection and processing of informa­
tion. 

Introduction 

The second part of the book is given over to the principle political priorities: 
environment,  competitiveness  of  SMEs, employment  and equal opportuni­
ties.  Methodological  requirements  vary from  one  priority  to  another. Thus, 
the taking  of  environmental  impacts  into  account  in an evaluation  is  based 
on methods adapted to the estimation of impacts that are diffuse and hete­
rogeneous.  On the  other  hand, employment  impacts  are  by  nature  more 
homogeneous  since they  relate  to  people and  enterprises.  For  example, a 
survey  of  direct  addressees  is suitable for  studying  employment  impacts, 
but not environmental impacts. 

The  creation  and  maintenance  of  employment  has, since the  start  of  the 
1980s,  been  one  of  the  preoccupations  of  the  European  Union. This  is 
concretely  demonstrated  by the  numerous  measures  devoted  to  employ­
ment  within  programmes  financed  by the  Structural  Funds.  Moreover,  a 
large amount of conceptual and technical work has been undertaken on this 
theme by  research centres, allowing a substantial  quantity  of  know-how  to 
be  accumulated.  Within  the  framework  of  the  MEANS  Programme, 
numerous  workshops  and  reports  have  been  given  over  to  the  theme  of 
employment. This Volume emphasises the selection of realistic and uncom­
plicated solutions within the framework of thematic  evaluations and cannot 
claim to be situated at the same level of sophistication  as an actual evalua­
tion of employment policy 

A theme such as equality of opportunity between men and women has given 
rise to  important  conceptual  advances, for  example, the  notion  of  gender 
mainstreaming (Box 2). 

Box 2 -  Gender  mainstreaming: a  definition 

Gender  mainstreaming  is the  systematic  inclusion  of the  priorities  and 
needs  specific  to  women  and  to  men  in  all  Community  policies  and 
actions, with the unique aim of achieving equality. This is to be achieved 
by actively and visibly taking account, at the planning stage, of the effect 
of interventions on the specific situations of women and men during their 
implementation,  follow-up  and  evaluation.  Gender  mainstreaming 
requires the development of measures aimed at adapting the way society 
is  organised  to  ensure  the  more  equal  distribution  of  roles  between 
women and men. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

However,  evaluations that tackle this theme are still few,  reflecting the  fact 
that  this  preoccupation  is relatively  new  for the  majority  of  countries. This 
type  of  thematic  evaluation  is  more  often  ex  ante  than  ex  post.  The 
examples  of  evaluations  presented  are therefore  aimed  at  prospective 
rather  than  retrospective  analysis.  In order  to  prepare  this  chapter  it  has 
therefore been necessary to go beyond existing practices. 

The  third  part  of  the  text  deals  with  thematic  evaluations  of  domains  of 
intervention. Four domains have been selected not for any strategic reason, 
but  more  prosaically  because  they  have  recently  been  the  object  of 
European level thematic  evaluations within the framework  of the  Structural 
Funds. The domains  of research  and  development, transport,  telecommu­
nications and tourism are examined. The first of these domains is tackled in 
some  detail, while the  others  are  more  briefly  examined. The  Volume  2 of 
the  Collection  provides  a  more  complete  view  of  all the  domains  of  inter­
vention from the  point  of  view  of  indicators  for  monitoring  and evaluation. 

Part  IV of the Volume focuses  on  methodological  issues which  cut  across 
thematic  boundaries  and  which  are  specific  to  evaluations  conducted  at 
the  programme  or  EU  level.  Firstly,  a standardised  procedure  is  provided 
which  is designed to structure the implementation of programme  level eva­
luations in such a way as to ensure that conclusions can be aggregated and 
synthesised  at  a  supra-national  level.  Secondly,  the  basic  principles  of 
transnational  thematic  evaluation, organisational  issues  associated  with 
this  type  of  evaluation  and  its  use  in  highlighting  best  practice  are 
examined.  Finally,  a  possible  future  direction  for  transnational  evaluation 
practice is mapped out. 

I 
Technical considerations relating 
to the assessment of thematic impacts 

The evaluation of the impacts  of Structural  Fund interventions  is  structured 
around a number of basic, but very important,  tasks which need to be under 
taken, whether the evaluation is focused on a theme, a political  priority  or an 
entire programme.  Some of these key tasks include:  the clarifying of explicit 
and  implicit  objectives,  selecting  indicators,  gathering  data, and  estimating 
net impacts.  The objective  of this chapter is to present  how  these four tasks 
can be approached.  This presentation  is not theme specific  and is illustrated 
by  examples  drawn  from  the  individual  themes  presented  further  in  this 
book.  The issues tackled  are, of  course,  applicable  to  themes  that  are not 
presented  here,  but  which  may  be  taken  into  consideration  in  or  be  the 
object of future evaluations. 

Technical considerations 

■ Clarifying the explicit and implicit objectives 

of t h e me focused interventions 

An early step in an evaluation is the clarification of a programme's objectives. 
During the ex ante evaluation of a programme it is important to clarify not 
only its central or global objectives, but also those objectives related to 
specific priorities, such as those mentioned in this book. In intermediate eva­
luations, additional objectives that might have emerged in the course of the 
programme should be clarified, along with the contribution of projects to 
attaining objectives. 

Even if there is a single overall priority objective (e.g. the promotion of envi­
ronmentally sustainable development), attaining that objective is likely to 
depend on attaining specific and intermediate objectives that may be 
expressed differently. It is therefore necessary to draw up a clear hierarchy of 
the objectives and the relationships between them. 

In order to draw up the hierarchy of objectives an "objectives tree" can be 
constructed. This tool allows the degree of internal coherence between the 
measures of a programme in relation to overall priority objectives to be 
examined (both vertically ­ between different levels of objectives ­ and hori­
zontally ­ between the objectives of different components of the programme 
which are at the same level). It also facilitates the examination of the external 
coherence of a programme with other public policies. For example, Structu­
ral Fund intervention in favour of SMEs might target the development of 
innovatory capacity, increased networking and technology transfer, while 
many Member states are seeking to create a general business environment 
which is conducive to the development of small firms. An examination of 
external coherence would focus on whether the objectives of these interven­
tions were compatible or not. 

While the hierarchy of objectives is usually constructed on the basis of 
official documents, in the case of thematic evaluations it can also be useful 
to incorporate implicit objectives into the analysis. An objectives tree 
constructed in this way can therefore be considered as a thematic objectives 
tree. Although the incorporation of implicit objectives into an analysis allows 
a better understanding of the logic of a programme, it is important to be 
careful when using implicit objectives as evaluation criteria. When the work 
of programme managers and operators has to be judged, this should only be 
done in relation to explicit objectives of an intervention. 

Transversal evaluation of impacts on intervention priorities 

The principle of a thematic objectives tree is illustrated in Box 3. Actions 
financed by the Structural Funds are designed to attain the explicit global 
objective of social and economic cohesion. A hierarchy of the explicit 
objectives linking actions to the global objective can be constructed in the 
form of an objectives tree. At the same time the promotion of equality between 
men and women is a priority concern for Structural Fund programmes, but 
objectives related to this political priority are largely implicit. While global 
objectives are stated in official texts, working documents and studies, this is 
usually not the case for specific and intermediate objectives. These objectives 
are often constructed implicitly by the various actors of the programme and 
serve to regulate and direct their operational decisions. The hierarchy of 
implicit objectives is presented in a thematic objectives tree which allows the 
relations between actions and the global objectives of a political priority to be 
structured. 

It should be remembered that the objectives trees presented in the following 
chapters are only illustrative, objectives trees should be constructed by the 
evaluation team in consultation with the steering committee of the evalua­
tion on the basis of the specific programme to be evaluated. Moreover, an 

Box 3 ­ Using objectives trees for clarifying implicit and explicit 

objectives related to a given theme 

Global objectives 

Intermediate and specitic objectives 

Actions 

■* 

Objectives tree 
(of a programme) 

Social and 
economic 
cohesion 

Balanced 
development 
for men 
and women 

Thematic objectives 
tree 

Explicit objectives 

Implicit objectives 

Technical considerations 

objectives tree is neither exhaustive nor definitive and it will have to be 
modified over time to take into account the evolution of political priorities 
and the socio­economic context. 

For more information on the clarification of objectives using an objectives 
tree, see Volumes 1 and 4 of the MEANS Collection. 

■ Selecting indicators for thematic evaluations 

Volume 2 of the MEANS Collection is devoted to the subject of indicators for 
monitoring and evaluating Structural Fund interventions. Only essential 
elements are developed hereafter. 

In order to assess the impacts related to a given theme, indicators need to 
be defined which provide an estimation of the phenomenon to be measured. 
The selection of several complementary indicators may be necessary to 
estimate an impact that is considered as important. For example, in order to 
estimate air quality, a number of indicators relating to different types of 
atmospheric emissions would have to be employed. 

Given the multiplicity of objectives, the number of components of a 
programme (sub­programmes, measures, and actions) and the need to take 
into account transversal political priorities, it is difficult to imagine an opera­
tional system of indicators which would include indicators for all compo­
nents at all levels. It is therefore necessary to select indicators that meet the 
most important needs of managers and officials and can be collected at a 
reasonable cost. In the case of thematic evaluations, since the role of indica­
tors is to allow an intervention to be judged in relation to the political priori­
ties to be taken into account, they should be related to the most important 
impacts of the theme concerned. The indicators retained should be reliable, 
giving the same indication irrespective of the person quantifying them and 
providing up­to­date information on the effects being measured. 

The construction of a system of indicators for performing thematic evalua­
tions requires, first of all, that the relevant evaluatory questions are establi­
shed. These questions will usually encompass the following issues: "To what 
extent have changes taken place?", "To what extent are these changes attri­
butable to the programme?" "Are the results and impacts that have occurred 
satisfactory?" Secondly, those evaluation criteria by which an intervention 
will be judged are agreed. Thirdly, that context indicators, which provide 
information on the reference situation in which an intervention occurs, and 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

programme  indicators, which  focus  on  the  direct  and  indirect  effects  of  an 
intervention  its  addressees,  are  defined  in  the  light  of  the  evaluatory 
questions and criteria selected. 

The  in-depth  analysis  of  impacts  also  requires  that  exogenous  factors  are 
taken  into  account,  i.e.  those  factors  over  which  the  programme  has  no 
control  and  which  have  an  influence  on  impacts.  It  is therefore  useful  to 
identify the main effects of confounding factors by means of indicators. 

A  system  of  indicators  is designed  to  allow  the tracking  of  an  intervention 
from the resources employed during  its implementation  to  its impacts as far 
as possible along the chain of causality. In order to do this indicators furnish 
information on the human and financial resources employed in implementing 
an  intervention  (resource  indicators),  the  material  and  immaterial  outputs 
produced  (output  indicators),  the  immediate  impacts  on  the  addressees 
resulting  from  an  intervention  (result  indicators),  and  the  subsequent 
impacts on direct and indirect addressees  (impact  indicators). 

When  referring to  different  types of  indicators  and the concepts  associated 
with them, it is highly desirable that a standardised terminology  be used. It is 
therefore strongly  recommended  that the  definitions figuring  in Volume 2 of 
the MEANS Collection are systematically employed. 

Example:  Indicators  for  measuring  the  effects  of  interventions  for  improving 
water quality and  resources 

In order to improve and safeguard the quality of local water supplies, a programme of 
investments  is  planned  with  the  objective  of  increasing  the  amount  of  waste  water 
treated and reducing the amount  of treated water  lost through  leakage. At the  output 
level indicators would  be chosen that measure the physical achievements of the inter­
ventions, for example: kilometres  of  mains supply  pipes  modernised, the capacity  of 
new waste water treatment  plants constructed and the capacity of waste water  plants 
to  undertake  primary  /  secondary  treatment.  An  indicator  at  the  result  level  is  the 
number of households or businesses  connected to the plant. At the impact  level indi­
cators  would  include  the  volume  of  water  consumed  via  the  modernised  system, 
number  of days of  when  supply  is restricted  and volume  of  water  lost through  leaks. 
Impact indicators also reflect improvements  in the quality and quantity of  underground 
water  resources and river quality, for example: withdrawal rate from sources, withdra­
wal  rate as a  %  of  annual  availability,  %  of  waste  water  samples  complying  with  EC 
directives, % of length of rivers monitored subject to slight or moderate pollution. 

It  is  necessary  to  take  the  timescale  of  effects  into  account,  otherwise 
judging  them  will  be  distorted  by  incomplete  information. This will  result  in 

Technical considerations 

the underestimation of expected effects that have yet to occur and / or the 
overestimation of temporary effects that have not come to an end. 

Example: The timing of employment effects arising from the promotion of 
research collaboration 
A project to improve collaboration between SMEs and public research bodies would 
produce different employment effects at different time scales. In the short­term, 
employment effects would be mainly temporary with employment being maintained or 
created during implementation (e.g. advisors, consultants, etc.). In the more medium­
term the recruitment of technicians, researchers and scientists to take part In the 
project would lead to permanent employment of individuals associated with the per­
forming of research. In a longer­term perspective employment Impacts will occur 
within the SMEs which are involved in research projects as the resulting innovations 
are marketed and levels of production increase. 

Four basic features of the indicators should not be forgotten in the interpre­
tation of data. First, the information provided might require additional pro­
cessing, since it is not always usable or communicable in its initial state. 
Secondly, the changes measured may not be entirely caused by the inter­
vention being evaluated and in any event, the analysis of an indicator is not 
enough to establish relations of cause and effect. 

Thirdly, when comparing information provided by an indicator, it is important 
to ensure that the comparison is relevant (i.e. that the context of the 
reference situation is the same). Finally, indicators should also provide infor­
mation on confounding factors which influence the evolution of the reference 
situation (e.g. when evaluating the impact of a job creation programme on 
unemployment, changes in the size of the working population should be 
taken into account). 

■ Gathering data about a t h e me 
When the evaluation of a theme is performed within the framework of a 
programme, it is recommended that this is based on documentary research 
and existing data (e.g. monitoring data, statistical surveys and existing eva­
luation reports) supplemented by minimal case studies, interviews and focus 
groups (see the "step by step", approach presented in Part IV). 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Box 4  - 

In-depth  analysis  of impacts: appropriate  tools  and 
constructing  an  evaluation  method 

When  an  in-depth  analysis  of  impacts  is  required,  it  is  necessary  to 
construct  an  evaluation  method  composed  of  a  number  of  steps  (a 
"technical chain") which  includes  a combination of  interdependent  tools 
for primary data collection and analysis. The "step-by-step"  approach for 
thematic evaluations, proposed in Part IV of this Volume, while remaining 
faithful to the "technical chain" concept favours the use of a limited range 
of tools that facilitate a "quick and dirty" approach to primary data collec­
tion.  Some  tools,  more  often  associated  with  in-depth  analysis,  are, 
however,  better  suited  to  certain  types  of  thematic  evaluations  than 
others. It is with the objective of highlighting the suitability of certain eva­
luation  tools  to  different  themes  that  tools  for  in-depth  analysis  are 
presented in each of the chapters devoted to specific themes. 

Volume  3 of the  MEANS  Collection  provides  a full  discussion  about  the 
choice and use of evaluation tools within the framework of the evaluation 
of Structural Fund interventions. 

The  monitoring  of  the  implementation  of  interventions  makes  most  use  of 
resource,  output  and  result  indicators.  These  are  largely  quantified  by 
programme  managers  and their  services  which  issue  payments  and  create 
or collect information on: number of outputs and / or level of physical output; 
expenditure;  descriptions  of  the  projects;  reports  on  activities  of  manage­
ment  services.  Several  services  are  usually  involved  in this  activity,  so  it  is 
generally  necessary  to  centralise  these  data  in advance,  concentrating  on 
the most  useful data for  managing and evaluating interventions.  If this  basic 
information is to be suitable and complete, the collection of monitoring  infor­
mation should be planned from the outset, bearing evaluation needs in mind. 

It  is  not  always  possible,  however,  to  supply  data  for  indicators  through 
existing  information  systems. This  is, in fact,  the  case  for  the  majority  of 
indicators  focusing  on the short  or  medium term  impacts  of  structural  inter­
ventions. 

The use of statistical time series data, while relatively cheap and quick,  has a 
number  of  major  drawbacks  which  limit  their  use  in evaluation: the  time-lag 
inherent in the production of data for standard statistical surveys; the absence 
of a direct  link between an evaluation criterion and available statistics, the dif-

Technical  considerations 

ference between the geographical area covered by an intervention and the ter­
ritories  or groups of territories  used to  compile  statistics. Regional  statistical 
observatories have also been created but, even when statistics are presented 
at  the  level  of  territorial  sub-divisions  of  the  Member  States, these  sub­
divisions do not always correspond with zones eligible for Structural Funds. In 
these conditions, the trends highlighted by the statistics may not allow direct 
conclusions to be drawn on the effects of a particular programme. 

Statistical  information  systems  are, however,  being  developed  to  remedy 
this  shortcoming.  Eurostat  is co-ordinating  the  collection  of  data  used  to 
provide information for indicators of the activity of 35 industrial sectors in all 
the areas eligible for support to promote development, structural adjustment 
and  industrial  reconversion. These data are available for the  period  1985-
1993 and will be available for 1994-1999. 

The  different  information  sources  have various  strengths  and  weaknesses 
that are summarised in Box 5. 

Box 5 -  Typical strengths and weaknesses  of  sources 

of existing  information 

Strengths 

Weaknesses 

Monitoring  systems 

Produced at frequent  intervals 
Relatively  up to date 
Generally  readily  available 
Significance  Is usually  apparent 
Relatively  low  cost 

Permanent  statistical 
surveys 

Low  cost 
Reliability 
Richness 
Comparability 
Ease with which  information  Is obtained 

Other research and 
evaluations 

Low  cost 
Data already  analysed 

Sometimes  unreliable  statements 
Rarely available for all 
measures /  actions 
Data on physical  outputs  and  results 
of various components  not  easily 
aggregated 
Not easily  comparable 

Often  not directly  linked to  evaluation 
criteria 
Geographical  level not always suitable 
Information  not up to  date 

Information not up to  date 
ften too  restricted to  provide a reliable 
answer or one that  can be generalised 

Given their strengths and weaknesses, the sources of information are com­
plementary  rather  than  alternative.  For  example,  permanent  statistical 
surveys  provide  data  for  comparison  (e.g.  evolution  of  employment  at 
regional  and  national  level), and  indications  of the  evolution  of the  context 

Transversal evaluation of impacts on intervention priorities 

(e.g. economic situation, evolution of workforce, growth and decline of 
certain sectors) which allow for a more in­depth analysis of impact indica­
tors, and in some cases, relevant impact indicators at a reasonable cost. On 
the other hand, evaluations and existing research provide elements for com­
parison (effects of similar interventions), elements for estimating deadweight 
or displacement coefficients allowing the calculation of net impacts, and 
hypotheses to be postulated on ways in which effects are produced. 

■ Estimating net impacts 

The need to estimate the net impact of an intervention applies to all evalua­
tion themes, however, Indicators are only capable of providing information on 
the gross impact of an intervention (i.e. the observed change in the reference 
situation or the situation of direct and indirect addressees). The gross impact 
of an intervention is not a proxy for its net impact and, as a consequence, the 
latter has to be estimated for the evaluation to provide accurate information. 
The estimation of gross impacts is therefore only an initial step in the evalua­
tion of impacts that can be genuinely attributed to an intervention. While the 
factors that have to be taken into account when calculating net impacts 
depend to some extent on the theme of the evaluation and the indicators 
employed, the following convention represents a minimal definition: 

Net impacts = 
Gross impacts ­ Deadweight ­ Displacement (Substitution)* 

('Unintended or unwanted) 

Deadweight is the term given to the change that would have been experien­
ced even in the absence of an intervention. Assessing displacement is 
relevant for interventions targeted at geographical zones. On the other hand, 
assessing substitution is relevant in the case of social group­targeted inter­
ventions. Only when displacement and substitution are not objectives of an 
intervention are they subtracted from gross impacts. 

In order to assess the significance of deadweight, it is necessary to construct 
the counterfactual situation or policy­off scenario based on detailed projec­
tions of the likely evolution of key variables if no intervention had occurred. 
The reconstruction of the counterfactual situation is of critical importance for 
the accurate assessment of the impacts of structural interventions. It is, 
however, a difficult task that becomes even more complex at programme 

Technical  considerations 

level  when  a variety  of  inter-related  interventions  has to  be considered. In 
practice, there are very few occasions when the counterfactual situation can 
be reconstructed  with  certainty;  an open, reasoned and  informed  effort  to 
make a best estimate is frequently the best that can be done. 

An  illustration  is  given  in  Box  6  of  the  principle  of  deadweight.  In  this 
example the change in the number of jobs during the period of an interven­
tion  (from  baseline to  the  moment  of the  evaluation)  is j0n less jo, in  other 
words  the  gross  employment  impact.  During  this  period,  however,  it  is 
estimated  that  in the  absence  of  the  intervention, the  policy-off  situation, 
employment would have increased to j0ff- The impact of the intervention net 
of displacement  is in the difference between j0n and j0ff. 

Box 6 - Estimating impacts net of deadweight 

1  on 

J  off 

Pollcy-on 

</ 

A 
, 

A 

;/ 

^^-^^ 

Policy-off 

Impact 
net of 
deadweight 

y 
k 

Deadweight 

Change of 
situation 
(gross 
impact) 

γ 
τ 

y 

Baseline 

Evaluation 

Τ 

When  displacement  occurs,  impacts  on  addressees  are accompanied  by 
effects on non-addressees within the programme area. An example of this is 
when  a  reduction  in  traffic-related  atmospheric  emissions  due  to  the 
construction  of  a new  road  leads to the  unintended  displacement  of  road 
traffic and exhaust emissions to another part of the programme area. In this 
instance  the  effect  is  unwanted  and  displacement  should  be  subtracted 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

from  gross  impacts.  If,  on  the  other  hand, displacement  is  sought  as  an 
impact  of  an intervention  (e.g. the  movement  of  small-scale  industrial  firms 
causing  noise and atmospheric  pollution from a heavily  populated  city  area 
to a greenfield site), displacement  is not subtracted from gross  impact. 

When an intervention  leads to the unintended  substitution of addressees for 
non-addressees, the effect  should  be considered  as negative  and  subtrac­
ted from gross impacts (e.g. the retraining of unemployed  males who  replace 
female  workers  in local enterprises)  If,  however,  these  effects  are  expected 
or  sought  (e.g. the  insertion  of  long-term  unemployed  which  occurs  at  the 
expense  of  the  short-term  employed), the  effect  should  not  be  subtracted 
from gross impacts. 

Displacement  should  not  be put on an equal footing with deadweight  since 
the  latter  is much  more crucial  to the estimation  of the  impact  of  Structural 
Fund interventions. When deadweight  is substantial, the  impact  of an  inter­
vention  is correspondingly  reduced.  On the  other  hand, displacement  only 
implies  that  impacts  have  occurred  in one  part  of  the  assisted  area  at  the 
expense of another  part. 

The homogeneity of impacts has repercussions on how deadweight, displace­
ment and substitution are assessed. In the case of employment,  impacts are 
homogenous and the estimation of deadweight and displacement focuses on 
a single evaluation criterion: the number of jobs created or maintained. 

Environmental  impacts  are,  however,  heterogeneous  (e.g. air  quality,  water 
quality, bio-diversity, etc.) and the net impact  of an intervention may have to 
be  estimated  using  a  number  of  relevant  indicators.  Similarly,  impacts  of 
interventions  on  SME  competitiveness  (e.g.  in  terms  of  innovations, 
products,  turnover,  value  added,  etc.)  can  be  estimated  using  an  array  of 
indicators  and the  estimation  of  deadweight  and displacement  will  depend 
on the impacts. 

The leverage effect, defined as the amount of private sector finance invested 
in an assisted  project  that  can  be directly  attributed  to  Structural  Fund  co-
financing  of  projects  (see  Box  7). This  is  another  effect  that  should  be 
estimated  whenever  an  objective  of  interventions  is  to  encourage  the 
financial participation of the private sector. 

In addition to these effects, there are a number effects which, depending  on 
the theme of the evaluation, will need to  be taken  into account.  In the case, 
for  example,  of  interventions  in favour  of  employment,  income  multiplier 
effect  (i.e. the employment  created or maintained indirectly  by the additional 

Technical  considerations 

income generated by the direct employment effects of an intervention) and 
supplier effects (i.e. the additional employment created or maintained  indi­
rectly  in suppliers to  assisted firms)  may  need to  be estimated  in order  to 
have a more accurate picture of the employment  impacts. 

It  is  important  that  these  effects  some  times  referred  to  as  additional 
effects, are not confused with the notion of additionally  as it is used by the 
European Commission. This concept  is illustrated in Box 7. If the  principle 
of additionality  is fully respected, Structural Fund expenditure is fully addi­
tional  to  national  government  expenditure  (lower  left-hand  figure). The 
opposite case is presented in the lower right-hand figure. This concept  is a 
concern  for  financial  monitoring  at  a  national  level  rather  than  for 
programme evaluation. 

Box 7 - Additionality, deadweight  and  leverage  effects 

Programme  level 

SF 
expenditure 

Private 
expenditure 

expenditure 

Govt 
expenditure 

Govt 
expenditure 

100% deadweight  effect 

Leverage  effect 

National  level 

SF 
expenditure 

Private 
expenditure 

Private 
expenditure 

Private 
expenditure 

Govt 
expenditure 

Govt 
expenditure 

Govt 
expenditure 

SF 
expenditure 

Private 
expenditure 

Govt 
expenditure 

100%  additionality 

0%  additionality 

Il 
Thematic analysis of impacts 
on political priorities 

Structural  Fund interventions are intended  to promote  economic  and social 
cohesion and are complementary  to the policy  efforts of individual  member 
states.  To achieve this objective  they selectively address the political  priori 
ties  of  the  European  Union  and  the  major  preoccupations  of  European 
citizens.  There are currently  four political  priorities  which  are  specifically 
addressed  by  interventions:  environmental  protection, 
competitiveness, 
employment and the promotion  of equality between men and women. 

The evaluation of political  priorities  can  take place  within  a programme  or 
across programmes  (transnational or  transregional  thematic  evaluation). In 
the  first  instance  the  objective  is  to provide  conclusions  on  the  extent  to 
which  a programme  impacts  on one or more  policy  priorities.  The second 
scenario is that of an evaluation that is specifically centred  on a given theme 
covering a number  of different  regions across a range of countries. A major 
objective  of this type of evaluation is to highlight  examples of best  practice. 

The following  chapters  are  structured  around  the  key  points  that  were 
presented  in Part I,  these key points  being addressed and developed  within 
the context  of each of the four policy priorities.  In addition  to these points,  a 
number  of evaluation tools are presented  that can be employed  when an in-
depth assessments of impacts is required. 

Evaluating the environmental impact of structural 
interventions 

■ Why do environmental impacts need to be evaluated? 

• The regulatory framework 

For at least the past three decades growing international concern with the 
state of the environment has resulted in increasing provision for environmen­
tal matters in legislation and public policy. The European Union's response 
to this concern can be traced back to the Paris Declaration on the Environ­
ment in 1972. The Commission published its First Action Programme on the 
Environment in the following year and has commissioned research on envi­
ronmental impact assessment (EIA) since 1975. In 1977 work was begun on 
an EIA Directive which, after extensive consultation, was approved by 
Council in 1980. 

In 1985, Directive 85 / 337 / EC made it compulsory to assess the effects of a 
wide range of public and private projects, on the basis that "the best envi­
ronmental policy consists in preventing the creation of pollution or nuisances 
at source, rather than subsequently trying to counteract their effects". 

In 1992, the Treaty on European Union reflected an important shift towards a 
more active approach by requiring that environmental considerations be 
integrated into the formulation and implementation of all types of policy. 

A strategy for achieving sustainable development (i.e. development that 
meets the needs of the present without compromising the ability of future 
generations to meet their needs) in such an integrated fashion was spelt out 
in further detail in the Fifth Action Programme on the Environment which was 
adopted by Council in 1993. In this document it is stated that "The word 
'sustainable' is intended to reflect a policy and strategy for continued 
economic and social development without detriment to the environment and 
the natural resources on which continued human activity and further deve­
lopment depend." 

The economic development policies of the European Union thus increasin­
gly recognise the connection of development to the environment and ack­
nowledge that increasing the cohesion of the Union will demand a path that 
is sustainable in both economic and environmental terms. Since the reform 
of the Structural Funds in 1993, it has been required that explicit considera­

Transversal  evaluation  of  impacts  on  intervention 

priorities 

tion be given to the environmental  impact of all the activities assisted by the 
Funds: plans submitted  by Member States seeking assistance under  objec­
tives  1, 2  or  5b  have  since  been  expected  to  include  "an  appraisal  of  the 
environmental  situation of the area concerned and an evaluation of the envi­
ronmental  impact  of  the  strategy  and  operations..."1. This  requirement  was 
underlined  with the  creation, in  1993, of  the Cohesion  Fund that  devotes a 
substantial  portion  of  its resources to  projects  with  primarily  environmental 
objectives. However, the primary objective of many interventions  is not  envi­
ronmental  and there is often  a tension  between  the type of  activity  which  is 
typically  deemed  to  be  necessary  to  promote  economic  development,  on 
the one hand, and environmental concerns, on the other. 

•  The relationship between the environment and cohesion 

There  is,  of  course,  no  immediate  or  automatic  correlation  between  a 
region's  level  of  economic  development  and  the  state  of  its  environment. 
Both economically  developed  and  underdeveloped  regions can either  have 
a healthy  environment  or experience  severe environmental  problems. In the 
past, controlling environmental  impact was widely viewed as entailing  costs 
and therefore  as  inevitably  having  a constraining  effect  on  economic  deve­
lopment. 

High  growth,  low  environmental  protection  strategies  are  not  sustainable 
over the medium to long-term. They need, therefore, to be replaced by more 
sustainable  strategies  that  take  advantage  of  a  high  quality  environment  in 
order to attract investment and use activities related to environmental impro­
vement  as a vehicle for economic  growth. Responding  to  relatively  specific 
environmental  problems  may  also  offer  short  to  medium-term  employment 
opportunities  and furnish  necessary  elements for wider  economic  develop­
ment strategies. 

The growth  of  "eco-industries"  provides  what  is perhaps the  most  obvious 
example of the complementarity  of environmental  protection and  economic 
development. These industries  have developed to  meet  increasing  demand 
from consumers  for environmentally  friendly  products. They  have also  been 
a  response  to  increased  controls  on  pollution  and  the  generally  enhanced 
awareness of the importance of the environmental  impact of human activity. 

1. Council Regulation No 2081 /93, part III. 

Evaluating  environmental 

impacts 

•  The need for evaluation 

Since cohesion can develop only through activity that is sustainable In both 
economic  and environmental terms, monitoring and evaluation  need syste­
matically  to  take  account  of  both  economic  and  environmental  factors. 
Indeed, the Commission  is now obliged to respect environmental  concerns 
in the design, monitoring and evaluation of structural interventions. 

In the case of  structural  interventions  in lagging  regions  and  rural zones, a 
sizeable  number  of  measures  have clear  environmental  objectives  and  the 
implications of supported  activities for the environment  are relatively direct, 
even if the measurement of the actual effects of such activity may be techni­
cally very difficult.  In the  case of  interventions  in other  regions the environ­
mental  consequences  of  supported  activity  are often  less direct  and  more 
difficult to evaluate. 

While techniques  for the  evaluation  of  environmental  impact  have  become 
increasingly sophisticated over the past few decades, for the most  part they 
have concentrated  on providing ex ante projections  of  likely  impact  so that 
informed choices might be made between alternative  policies and projects. 
Ex ante environmental impact  assessment  is an increasingly significant  and 
systematic  element  in the formulation  and  planning  of  structural  interven­
tions. However, evaluation of the actual impact of these interventions  (either 
individually or in total) is not yet widespread. 

Nevertheless, the Commission  needs to ensure that  methods  are  available 
for  evaluating the actual  environmental  impact  of the heterogeneous  set of 
activities supported by the Structural Funds for three main reasons: 

•  first, the  overall  orientation  of  the  structural  policies  and  the  degree  to 
which  the  efforts  they  devote  to  promoting  cohesion  through  sustainable 
development  have borne fruit, need to  be checked  by obtaining some indi­
cation of the environmental impact of structural policies as a whole; 

•  second, whether  the  environmental  priorities, which  are often  set  out  in 
supported programmes, have in fact been achieved needs to be verified; 

•  finally, in relation to  individual projects, project  selection criteria related to 
the  environment  need to  be validated  by  comparing  predicted  with  actual 
outcomes. 

The EU's structural  policies are delivered through fairly complex and hetero­
geneous  programmes.  Because  an appreciation  of  the  overall  impact  of 

Transversal evaluation of impacts on intervention priorities 

these policies is required, it is necessary to evaluate not only the cumulative 
consequences of particular projects on the environment with explicit envi­
ronmental objectives, but also the environmental implications of both 
projects and broader, macro­level policies which may have neither explicit 
environmental objectives or direct environmental impacts. 

Unfortunately, established methodologies for the evaluation of environmen­
tal impact have been developed almost exclusively for application to 
specific projects rather than to programmes or broader policies. It is true 
that, in recent years, strategic environmental assessment (SEA) has 
emerged as an approach to the evaluation of the environmental impact of 
broader policies. However, it appears that this approach is yet to be fully 
established technically and is, in any event, most useful in relation to ex ante 
evaluation. 

The approach presented here is by necessity less ambitious than a full eva­
luation of environmental impacts. This chapter is therefore oriented towards 
the evaluation of direct environmental impacts, focusing on those tech­
niques that in a simplified and operational form seem to be particularly well 
suited to ex post evaluation of environmental impacts. 

■ Defining what needs to be evaluated 

• The environment and its influences. 

The concept of the environment can cover a vast range of topics that are inter­
related in complex ways. It is therefore important to clearly identify the main 
features of the environment and the most fundamental mechanisms shaping it. 

Natural elements may simply be consumed directly as resources. This, in 
itself, has an impact on the environment and can cause dramatic changes in 
eco­systems ­ changes which may be viewed as either positive or negative. 
In practice, human intervention in the environment always goes beyond 
simply consuming natural resources, to entail the establishment of infra­
structures and settlements, the use of energy, the transport of people and 
material, and the output of waste from each of these activities ­ all of which 
impact on the environment. All of these elements are interconnected and 
changes which occur may be either positive or negative. 

Evaluating  environmental 

impacts 

Example: The environmental effects of agricultural activity 
Agricultural  activity  is  highly  varied  in terms  of  both  products  produced  and  tech­
niques  used.  In the  case  of  the  use  of  intensive  farming  techniques,  it  can  have 
negative environmental effects leading, for example, to a deterioration in the quality of 
soil, water  and  air, as well as the  loss of  habitats  and bio-diversity.  But  agricultural 
activity  can  also  have  positive  effects  with  crops  acting  as a sink  for  greenhouse 
gases, and with  appropriate  land  management  contributing  to the conservation  and 
enhancement  of  bio-diversity  and the  country  landscape,  and to  the  prevention  of 
flooding and landslides. 

The basic determinants of environmental change are set out schematically in 
an adaptation  of  the  "Driving  Force-State-Response  (DF-S-R)  framework 
(See  Box 8). This  model  represents  the environment  as comprising essen­
tially  non-human elements that evolve in interaction with each other to form 
complex  and  delicately  balanced  eco-systems.  It  is one  of  a  number  of 
models  that  have been developed  to  explain the  determinants  of  environ­
mental  change  (other  more  recent  examples  include  the  "Driving  Force-
State-Impact-Response"  model of the European Environmental Agency and 
the  OECD's  adaptation  of  the  DF-S-R  framework  to  include  agri-environ­
mental linkages and sustainable agriculture). 

Box 8 -  An adaptation  of the  Driving  Force-State-Response 
(DF-S-R)  model of sustainable  development 

Economy 

1 
transport 

energy 

industry 

agriculture 

trade 

tourism 

t_ 

State of the 
environment 

\ 
Wealth and welfare 

(economic) 
pressures  or 
driving  forces 

(economic) 
use of 
resources 

air 

water 

land 

wildlife 

natural 
resources 

responses 

(public  and 
private) 
welfare 

(public 
action) 
responses 

Actors 

4 

administrations 

households 

enterprises 

international 
agencies 

Source: Adapted from Department  of the Environment. (1993), Environmental Appraisal of Deve 
lopment  Plans, A good practice  guide,  London: HMSO. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

The environment  can be  broadly  defined  as the set of  natural elements  that 
surround  an  individual  (human, animal  or  vegetal)  or  its  species. Within  the 
framework  of the DF-S-R  model this set of natural elements is comprised of 
air, water, land, wildlife and natural resources. 

This framework allows environmental change to be situated in the context of 
development  in  specific  economic  sectors  to  identify  more  accurately  in 
terms of: 
•  those  driving  forces,  which  can  be  defined  as  the  elements  causing 
changes  (both  positive and  negative)  in the state  of the environment,  these 
include:  natural  environmental  process  and  factors,  biophysical  inputs  and 
outputs, and socio-economic  pressures; 
•  the  precise  changes  in  the  state,  which  reflects  the  changes  in  the 
condition  of  the  environment,  which  are  attributable  to  the  various  driving 
forces; 
•  the  policy  and  behavioural  responses  which  are  being  made  to  those 
changes in the state of the environment. 

While this framework allows the issues involved to be reviewed in a clear and 
systematic  way,  and  fosters  a  balanced  consideration  of  environmental, 
economic  and  policy considerations,  it does  not  help with the ultimate  need 
to  reconcile  different  types  of  criteria. This  is  particularly  necessary  in  the 
matter  of  sustainable  development  since  there  is  an  element  of  tension 
between economic activity and environmental concerns. The very concept of 
sustainable  development  is  based  on  the  premise  that  economic  growth 
should,  at  the  very  least,  not  entail  unacceptable  environmental  cost.  Fre­
quently  it is difficult  to define criteria which allow us to make judgements that 
are equally  valid  in relation  to  both economic  and  environmental  considera­
tions. 

•  Environmental impacts arising from the pursuit of  non-environmental 

objectives 

Whilst the  DF-S-R framework  is unquestionably a useful tool for  conceptua­
lising the  determinants  of  environmental  change,  it  is structured  around  a 
chain of causality which  is circular  in nature and therefore of limited use as a 
basis  for  evaluating  the  impacts  of  structural  interventions  on the  environ­
ment.  A  more  operational  approach  is to  construct  a typology  of  environ­
mental  impacts,  which  combines  socio-economic  and  environmental 
concerns and is generally relevant across the European Union. 

Evaluating  environmental 

impacts 

For example, such a typology might consist of the following categories: 

•  Water quality and resources 
•  Atmosphere and material flows 
•  Environmental management and innovation in enterprises 
•  Land use and landscapes 
•  Flora and fauna 
•  Environmental behaviour of individuals 

The  categories  of  this  sort  of  typology  are  not  intended  to  be  mutually 
exclusive. For example, an action to increase public transport, which  might 
have a direct  impact on the  "environmental  behaviour of individuals",  could 
also  have the  indirect  impact  on  "atmosphere  and  material flows"  through 
the diminished use of private vehicles leading to reduced exhaust emissions 
and demand for non-renewable energy resources. Likewise, impacts that fall 
within the category of  "environmental management and innovation  in enter­
prises" will tend to be upstream of impacts in other categories. However, it is 
unlikely  that  all  of these  levels  of  impact  could  be accurately  traced. This 
type of approach allows the most direct impact to be specified while leaving 
open the potential to identify indirect impacts. It also allows specific themes 
to be analysed and reported on separately. 

Examples  of  effects  (impacts  and  results)  are classified  according  to  the 
typology of impacts (see Box 9). 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Box 9  -  Detailed typology  of environmental  impacts 

of structural  interventions 

Categories of impact 

Examples of impacts and results 

Water quality and resources 

•  Increased proportion of Industrial and toxic waste water treated 
•  Increased proportion of population connected to treatment facilities 
•  Reduced use of nitrates in agriculture 
•  Improvement in the quality of underground, river and marine water 
•  Reduced leakage of treated water from mains system 
•  Increased recycling of used water 

Atmosphere and material flows 

•  Improved collection, treatment and recycling of solid industrial 

waste 

•  Increased re-use, recovery and recycling of urban waste 
• Reduced size and number of landfill sites, and reduced pollution of 

underground water 

• Reduced Industrial emissions of CO2, sulphur and nitrogen oxides 

and unburned hydrocarbons 

Environmental management 
and innovation in enterprises 

•  New patents in the field of environment related technologies 
•  New enterprises specialising in environmental products and services 
•  Increased investment in "clean technologies" 

Land use and landscapes 

• Treatment and reclamation of derelict industrial sites and abando­

ned urban areas 

• Increased number of areas designated for their landscape signifi­

cance 

• Natural areas protected from urban and industrial development, and 

development projects 

• Protection of areas and buildings of outstanding historical and archi­

tectural interest 

Flora and fauna 

•  Reduced use of pesticides and herbicides in agriculture 
•  Maintenance of bio-diversity 
• Cleaning up coastal areas, beach and  rivers 
•  Better management of timber and agricultural resources 

Environmental behaviour 
of individuals 

•  Increased proportion of commuters using public transport 
•  Increased proportion of tourists visiting off-peak 
•  Reduction in use of private vehicles 

•  Clarifying environmental  objectives 

The object of evaluating the environmental  impact  of structural  interventions 
is to  obtain  a  measure  of  the  effects  of  a  mix  of  activities,  all of  which  are 
intended to promote economic  and social cohesion, but only some of  which 
have environmental objectives. When environmental objectives are explicitly 
incorporated  into  an  intervention  they  are obviously  designed  to  result  in a 
positive impact. However, other environmental impacts that result from  inter­
ventions may be either positive or negative. Consequently, a complete  envi­
ronmental  impact  assessment  of  these  activities  would  need  to  take  into 
account the full range of these effects. 

Evaluating  environmental 

impacts 

Since the objectives  of  each component  of  a programme  and the  connec­
tions between different levels of a programme are rarely articulated as clearly 
and as explicitly  as they might  be, and since the environmental  implications 
of  particular  interventions  are frequently  unclear,  it  is usually  necessary  to 
begin  by clarifying  the  objectives.  One approach  to  the clarification  of  the 
objectives  of Structural  Fund Interventions  is to develop an objectives tree. 

Box 10 -  Environmental objectives of Structural Fund Interventions 

presented  in the form of a thematic  objectives  tree 

Global  objectives 

Examples of specific  objectives 

< 

Promote 
environmentally 
sustainable 
development 

Improved water  quality 

-  and  resource 
management 

-  Cleaner 

atmosphere 
and  reduced 
material  flows 

< 

- 

I 

•  Increase  the number  of  households 
and industrial  plants connected to  waste 
water  facilities 

> 

•  Improve the collection, treatment  and 
recycling of industrial  waste 

* - - -. 

1 

I 
1 

Better  environmental 
management 
and  innovation 
in  enterprises 

•  Increase  environmentally  oriented  R &D 

•  Increase  energy  efficiency  of SMEs 

Better  land  use and 
landscape  protection 

Better  conditions 
for flora and fauna 

| 

. 
<-, 

I 

1 

•  Make  more  industrial  sites  available 
in developed  areas 
•  Improve  protection for  areas  designated 
for their  landscape  significance 

•  Improve  protection  of  marine  areas  and 
coastal zones 

More  responsible 
behaviour  in  respect 
to the  environment 

•  Facilitating  the use of  public  transport 
•  Promote  less environmentally  damaging 
tourist  activity 

By way of an illustration, the typology of potential impacts has been integra­
ted  into  an objectives tree specific  to environmental  impacts  resulting  from 
structural  interventions. The global  objective, which  is to  promote environ­
mentally  sustainable  development,  is linked, via intermediate  objectives,  to 
examples  of  specific  objectives  relating  to  possible  measures  of  a 
programme and other interventions. This example is largely based on explicit 
environmental  objectives  since the  implicit  objectives  of  a programme  only 
become  apparent  when  programme  managers  and  decision-makers  are 
interviewed. Relations between intermediate objectives are indicated by their 
relative position and the links between them (see Box 10). 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

The drawing up of an objectives tree facilitates: the specification  of environ­
mental objectives  of a programme, the clarification of  relationships  and any 
priorities  that  may  exist  between  them,  and  better  judgement  of  both  the 
internal  coherence  of  elements  of  a  programme  in  relation  to  the  environ­
ment and their external coherence with other public  policies. 

•  Gross and net environmental  impacts 

The environment  is subject to a complex  combination of pressures arising from 
economic activity and the effects of individual and collective behaviour, as well 
as the effects of a range of public policy interventions. As a result, an observed 
environmental  impact  may  not  be wholly  attributable  to  an  intervention.  It is 
therefore  important  to  calculate the deadweight  of  intervention, so allowing  its 
net  effects  to  be estimated. A  definition  of  deadweight  in this context  is the 
environmental  change  which  may  reflect the  environmental  objectives  of  an 
intervention but which would have occurred even without the intervention. 

Displacement  in this  context  refers  to  the  geographical  shifting  of  negative 
environmental  impacts  rather than to their decline. This  has emerged  as an 
issue of  concern  in relation  to the environment,  for example,  with  regard  to 
practices  such  as exporting  toxic  and  radioactive  waste  from  their  sites  of 
production  to  other  locations  that  are  no  better  equipped  to  cope  with  the 
presence of such substances. 

Example: Estimating the net Impacts of a town by-pass on the environment 
Assuming that after the completion of a town by-pass project, vehicle exhaust emissions 
in the town centre are estimated as having fallen by around 20% (the gross impact of the 
intervention). Yet, over the same period, the growth of private car use in the rest of the 
country  has led to a 5%  increase  in exhaust  emissions  (counterfactual  situation). 
However, although there is an obvious positive effect, there might be negative impacts 
on a number of residential areas on the outskirts of the town that are located near to the 
by-pass because of an increase in traffic levels as travel patterns change. The resulting 
increase in vehicle emissions, corresponding with the displacement effect of the inter­
vention, might be estimated at 10%. To estimate the gross impact of this hypothetical 
intervention the deadweight effect (counterfactual situation minus baseline) and the dis­
placement effect would have to be subtracted from the gross effect. 

It  is  also  important  to  verify  that  none  of  the  measures  contained  in  a 
programme  merely duplicate  existing  regional policies. A case of this  would 
be if existing support  mechanisms for general R&D projects were capable of 

Evaluating environmental impacts 

increasing environmentally oriented R&D. In that case, the environmental 
impact of environmentally oriented R&D may be positive, but it would not be 
entirely attributable to the measure being evaluated. A proportion of the 
impact observed would therefore have to be subtracted from gross impacts. 

■ Examples of impacts by domain of intervention 

Three broad types of intervention are commonly supported by the Structural 
Funds: the provision of basic infrastructure, support for the productive envi­
ronment, and the development of human resources. All these types of inter­
vention may have possible environmental impacts and an example of each is 
dealt with separately below. 

• Basic infrastructure 

Support for the provision of basic infrastructure is usually intended to indi­
rectly promote cohesion by relaxing specific constraints on economic 
activity (e.g. by providing a high quality water supply which is suitable for 
specific production processes) or generally reducing production costs (e.g. 
by upgrading the road network). Infrastructure projects may have explicit 
environmental objectives and, whether or not this is the case, significant 
environmental impacts. In either case, evaluation of the economic impact of 
an infrastructure project needs to be complemented by an evaluation of its 
environmental impact if the sustainability of development based on this type 
of intervention is to be assessed. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Example:  Light  rail s c h e m es  in t he  United  Kingdom 

The  S o u th  Yorkshire  Supertram  a nd  Midland  Metro  were  b o th  f u n d ed  under  objective 

2  in  the  United  K i n g d o m.  Both  of  these  projects  should  result  in  the  creation  of  new 

e m p l o y m e nt  not  only  in  the  construction  and  operation  of  the  new  infrastructure  but 

also  b e c a u se  of  the  Increased  competitiveness  of  the  region  d ue  to  its  improved  links 

to  transport  corridors  and  decreased  journey  times.  Both  projects  should  also  lead  to 

reductions  in  noise  a nd  air  pollution  and  enhanced  values  of  investments  in  b o th  resi­

dential  a nd  c o m m e r c i al  properties. 

Outputs  ΐ> 

Construction  of 
a  new  light 
rail  system 

Decreased 
volume  of  private 
vehicle  use 

Effects 

Reduced  noise 
levels 

> 

Reduced  exhaust 
emissions 

Improved 
air  quality 

Improved 
public  transport 
links 

Reduced 
journey  times 

Improved 
competitiveness 

Priority 

•  Productive  environment 

Various  types  of  investment  subsidy  have  traditionally  been  a  major 
component  of  most  regional  development  strategies,  supporting  the deve­
lopment  of  increased  capacity  for  production,  increased  productivity,  other 
types of direct support to individual enterprises (e.g. in relation to marketing, 
R&D or  specifically  environmentally  friendly  business  practices),  or  indirect 
support  by  promoting  complementary  activities which may  include environ­
mental services. Whether  the environmental  impact  of this type of  interven­
tion  is positive  or  negative, any  individual case will depend  on the  nature of 
the supported  activity. 

Evaluating  environmental 

impacts 

Example: Energy audits and  advice for S M Es  (United  Kingdom) 

In  an  effort  to  e n h a n ce  t he  c o m p e t i t i v e n e ss  of  SMEs  in  the  Knowsley  area  of 

Liverpool,  the  Powersave  project  supported  experts  from  the  local  electricity  supply 

c o m p a ny  in  visiting  c o m p a n i e s,  assessing  their  energy  efficiency  and  p r o p o s i ng 

improvement  plans.  Putting  these  plans  into  action  was  c o - f u n d ed  equally  by  the 

ERDF  and  the  electricity  company.  For  a total  investment  of  3.7  MECU,  cost  savings 

of  over  1  MECU  were  achieved  annually  and  C 02  emissions  were  cut  substantially. 

Outputs 

^>  [ 

Effects 

O 

Energy 
management 
plan for  SMEs 
TJ 

Reduced fuel 
consumption 

Lower  emissions 
of  CO, 

Improved air 
quality 

Reduced 
demand  for 
electric  power 

Reduced 
production  costs 

Improved 
competitiveness 

Priority 

•  Human resources 

Interventions  in support  of  human  resource  development  are intended  to 
strengthen  and  enhance  the  skills  and  abilities  of  beneficiaries  and  so 
improve their employability. The objective is also to make the local economy 
in which  they  work  more  competitive, owing  to  improved  adaptability  and 
productivity  levels. To the  extent that  such  interventions  increase the  envi­
ronmental awareness of beneficiaries or their capacity to engage in activities 
which are not detrimental to the environment  or which have a positive envi­
ronmental  impact, they  increase the  potential sustainability  of the  resulting 
development.  A  positive  environmental  impact  is  likely  to  result  from  the 
supported activity if the human resource development  itself, or the employ­
ment that its beneficiaries eventually obtain, has an environmental objective. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Example: Developing recycling activities In Germany 
Assistance was  provided from the Structural  Funds to a training company  in Geissen 
(Germany)  which  aimed  to  provide jobs  and  qualifications  for the  local  unemployed, 
training  for  local  employees  and  services  with  a  positive  environmental  impact.  The 
company  was established  in 1988. By 1996 it employed 370 people, was training  150 
and had recycled thousands of domestic  appliances (refrigerators, washing  machines 
and  dryers).  This  had  also  led  to  a  significant  reduction  of  CFC  emissions  and  oil 
wastage. 

Outputs  C> 

Effects 

ΐ> 

Provision  of 
training  places 

Recycling  of  domestic 
appliances 

Increased  employment 

Increased  skill  levels 
in the  area of  recycling 

Reduced  CFC 
emissions 

Reduced  oil 
wastage 

Diversification  of 
economy  into 
eco-industries 

Priority 

It is generally difficult to say whether  economic  development will be accom­
panied  by  positive  environmental  spin-offs  and  consequently  follow  a sus­
tainable pattern over the longer term. However, it is important to bear in mind 
that  a project that  gives rise to negative environmental  effects  is not  neces­
sarily a case of environmentally  unsustainable economic  development.  Miti­
gation measures (i.e. measures that reduce the adverse effects of an activity 
on  the environment)  can  be  introduced  which  avoid, reduce  or  remedy  the 
negative  environmental  impact  of  an  intervention.  These  measures  may 
succeed  in stopping  the  deterioration  of the  environment  or  even  lead to  a 
net improvement, therefore achieving  sustainability. 

Defining  a nd  selecting  indicators 

•  The need for and limitations of indicators 

Volume 2 of the MEANS Collection  is devoted to the subject of  indicators for 
monitoring  and  evaluating  Structural  Fund  interventions. The above  Volume 
includes an information sheet on indicators for the priority of the environment. 

Evaluating  environmental 

impacts 

The overall  environmental  impacts  of  the  EU's structural  policies  are  the 
cumulative consequence of a wide range of different types of activity taking 
place  in a similarly  diverse  set of  local  environmental  contexts. To be  in a 
position  to  obtain  an  accurate  overview  of  the  nature  and  scale  of  these 
impacts, access must be available to information that accurately reflects the 
most important aspects of change in the environment over time. In order for 
environmental  change to  be  monitored, indicators  should  be able  to  show 
trends by incorporating a description of conditions which exist at a specified 
point  in time  (a baseline  value) against  which  subsequent  changes  can  be 
measured  through  monitoring.  The  number  of  indicators  selected  and 
defined should be limited enough to be both feasibly attainable and focused 
on the  most  relevant  issues, but  also  comprehensive  enough to  reflect  the 
most significant developments. For example, it is probable that many general 
levels of atmospheric or water contaminants would not be noticeably influen­
ced for a considerable time by even relatively substantial individual  projects. 
In this  case, for  point  sources  of  contamination  (specific  industries,  major 
outlets)  a simple  measure of  decreased  output  of  specified  pollutants  from 
the source  in question  is more likely to  be useful, while for  diffuse  sources 
(such as sectors of production - e.g. levels of use of artificial fertiliser, or broad 
patterns of behaviour - e.g. levels of private vehicle usage) a reasoned calcu­
lation of expected reductions in such outputs would be acceptable. 

•  Recent efforts to develop indicators of sustainable development 

Considerable  efforts  to  develop  indicators  of  sustainable  development  on 
the  international  level  have been made since  1992, notably  by the  UN, the 
OECD, and Eurostat. The United Nations Commission on Sustainable Deve­
lopment has produced individual methodological  specifications for the defi­
nition, collection and interpretation of 132 different  indicators. These indica­
tors  are  developed  within  the  DF-S-R  framework  for  four  categories  of 
information:  economic,  social,  environmental  and  institutional.  Eurostat 
produces  a variety of publications which supply information on the environ­
ment (often prepared jointly with the OECD) and two on-line databases (New 
Cronos and Envstat). Eurostat has also been engaged  in a pressure indices 
project  which  is  intended  to  both  establish  a conceptual  framework  for 
measuring the impact of economic  activity on the environment, and develop 
basic  indicators  for  this  purpose.  The  OECD  has  been  engaged  in  a 
programme  to  develop  policy-relevant  environmental  indicators  and  has 
working groups, for example, on environmental indicators for agriculture and 
for forestry.  Finally, the  European  Environment  Agency  was  established  in 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

1993 to produce information on the environment, including a comprehensive 
triennial  state  of  the  environment  report  on  Europe, the  first  of  which  was 
published in 1995. 

The  European  Commission  is also  developing  environmental  indicators, 
including  the  establishment  of  a  European  System  of  Environmental 
Pressure  Indices,  comprising  50-100  physical  pressure  indicators  which 
show  trends  for  ten  key  "policy  fields".  Once  these  data  series  have  been 
established, they will be to combine them into indices by attributing  different 
weights to them, making the information they provide more manageable and 
better tailored to particular policy needs. Such indices are essentially  proxies 
for more direct measurements of the impact of policies which are less readily 
available.  For  example,  in the  case of  deforestation,  an  index would  ideally 
not  simply  be  an  indicator  of  the  rate  of  deforestation  but  something  that 
combines  this  with  an  indicator  of  incentives  for  forest  clearing.  Similarly, 
instead of simply reporting the weight of herbicides  used each year, a useful 
index would reflect the subject of eco-toxicity more directly. 

It is also possible to construct  ad hoc regional level indicators for  evaluating 
environmental impacts, as the following example shows. 

Example: The use of ad hoc indicators  (France) 
The evaluation of the  Burgundy  objective 5b programme to support  the  development 
of  rural zones  required the  construction  of  regional  level  indicators  of  environmental 
impact.  The  percentage  of  hedgerows  destroyed  was  used  as  an  environmental 
impact indicator of land regrouping operations. Furthermore, this measure was refined 
by  categorising  the  destroyed  hedgerows  in terms  of  quality.  In the  area  of  public 
works, impacts were measured by the number of new beneficiaries of water treatment 
facilities, and  in terms of the  increase  in the equivalent  number  of  inhabitants  having 
access to the facilities. The impact  of waste management  projects was measured  by 
the increase In tonne /  months of waste collected at disposal sites and the increase in 
tonne / months of selective waste collected. 

•  Selecting indicators for a given programme 

Indicators are useful only if they reflect the changes which are to be evaluated. 
To an extent, the  nature of these changes  may  be self-evident.  Nonetheless, 
ensuring  that  they  are adequately  identified  and  interpreted  requires  some 
reflection. Regardless of whether the indicators of environmental impact which 
can  be obtained  relate to changes  in the state of the environment  itself or  to 
changes  in  the  forces  acting  on  the  environment,  evaluators  must  be 
conscious of the variety of ways in which the environment  may be influenced. 

Evaluating  environmental 

impacts 

They may also be both the direct consequences of assisted activity (e.g. the 
damage caused by the construction of a major infrastructure project such as 
a motorway) and the indirect consequences of these or other actions (e.g. an 
action which may even have had a positive direct impact on the environment 
- the establishment of a country park - but which has ultimately led to negative 
impacts - the pressures on the environment resulting from visitors travelling to 
and using the park). Environmental  impacts  may also be of differing  duration 
and be either discrete or cumulative. It is important to ensure that all of these 
types of environmental impact are considered in any evaluation. 

Once relevant changes in the environment have been identified, the interpre­
tation of how these changes were caused may be approached from two di­
rections. First, the outputs,  results and direct  impacts  of the  intervention  in 
question can be monitored  and reasonable  inferences drawn  regarding  the 
impact  that  the  outputs  have  on  the  overall  state  of  the  environment. 
Second, the state of the  environment  can be  monitored  and  any  changes 
noted  may  be  attributed  to  any  one  or  to  a  combination  of  a  range  of 
possible influences. 

Since the  environment  is open to  such  a variety  of  influences,  it  is  often 
impractical  to  establish  an  empirical  connection  between  programme 
outputs, on the one hand, and impacts which can  be attributed to them, on 
the other (at least if impacts in terms of the state of the environment are to be 
measured). In such circumstances, the first is the most practical one to take, 
although it should be crosschecked  when possible with the results of  some 
efforts that follow the second approach. 

Indicators of the state of the environment  have become increasingly sophis­
ticated in recent years. Examples of such indicators include: 

•  an index of evapo-transpiration which provides a powerful indication of the 
state and character of the vegetation of a region; 

•  an  index  of  the total  appropriation  of  net  primary  production  of  a  region 
which provides an equally powerful indication of the degree to which vegeta­
tion is being exploited. 

Yet it is almost  inconceivable that extensive practical use could  be made of 
such  indicators  in  evaluating  the  diverse  and  fragmented  operations 
assisted  by the  Structural  Funds. The strength  of  indicators  of this  kind  in 
reflecting the environment is largely due to their independence from individual 
influences, but in terms of Structural Fund interventions this type of indicator 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

is too all embracing. Consequently, the impacts of supported  projects  would 
be lost amidst the numerous influences arising from other sources. 

For this  reason, a  "bottom-up"  approach  is appropriate  for the  selection  of 
indicators  for the  ex-post  evaluation  of  environmental  impacts  arising  from 
Structural  Fund  Interventions.  This  approach  consists  of  establishing  a 
system of indicators, which allows the impacts of interventions to be tracked 
from the micro level towards the macro level. 

Evaluations  of  environmental  impacts  seek  to  answer  two  fundamentally 
different types of question requiring the use of different types of information. 
First,  in  relation  to  the  components  of  an  individual  programme  which  had 
explicit  environmental  objectives,  how  successful  were  they  in  reaching 
those objectives?  Second, in relation to all types of intervention, what  is the 
potential environmental  impact at the programme  level or in terms of groups 
of programmes (e.g. those aimed at a particular global objective of the struc­
tural policies or those in a particular  Member State)? 

In the first  instance, it  is often  difficult  to definitively  attribute changes  in the 
overall  state  of  even  particular  aspects  of  the  environment,  to  specific 
projects.  Reasonable  assumptions  have  to  be  made  on the  likely  impact  of 
assisted  activity  on the  basis  of  information  that  does  not  precisely  reflect 
the state of the environment.  In some cases this is perfectly satisfactory (e.g. 
in the case of clear  reductions  of discharges from  point  sources of water  or 
air  pollution).  However, the  more  indirect  the  relationship  between  interven­
tions and changes  in the state of the environment  (e.g. in the case of  human 
resource  measures  for  training  in environmentally  relevant  skills),  the  more 
difficult  it is to develop a convincing  measure of  impact. 

This  problem  is less taxing than it might  be due to the fact that the environ­
mental  objectives  of  a  structural  intervention  are  themselves  often 
expressed  in terms  of  change  in  driving  force  variables.  These  objectives 
make assumptions  about  the  relationships  of  such  variables  to  the state  of 
the environment.  Provided such assumptions are credible, it is best to  focus 
the evaluation of the environmental  impact of the programme in question  on 
the analysis of the driving forces concerned. 

In the second  instance,  it might  be expected  that  because the  scale  of  the 
overall effort  is larger,  impact  is easier to  identify  in terms of state variables. 
This  may  be  the  case  in  some  instances  but  usually  information  on  the 

Evaluating  environmental 

impacts 

impacts  of the overall  effort  is itself  also  available, at  best, only  in terms of 
driving-force variables. 

Moreover,  evaluation  of  impact  at this  level  is complicated  by the  need to 
aggregate a variety of different  measures of impact which, even if they refer 
to  the  same  variable,  may  have  been  expressed  for  good  reason  on  the 
programme  level in ways which are context-specific.  For this reason, when 
evaluating environmental impact on the multi-programme /  policy  level, it is 
necessary to  refer  more systematically  to  established  data-series  on  state 
variables, even if it is impossible to establish a direct link between these and 
the information which is available on the actual impacts of interventions. 

By way of an example, a system of indicators which could be used to track 
the  production  of  impacts, both  environmental  and socio-economic  arising 
from the construction of waste water treatment plants is presented in Box  11. 

Box 11 -  Example of a system  of indicators for the  construction 

of waste-water  treatment  plants 

Programme  Indicators 
(Related to the intervention and Its effects) 

Context  Indicators 
(Related to the assisted area) 

Output  Indicators 

Progress 

Capacity 

Result  Indicators 

Compliance  with  project  duration 
Rate of  completion 
Number of new waste water  treatment 
plants  constructed 
Maximum annual throughput (m1) 

Treatment  in relation  % of waste-water  undergoing 
to demand  and pro- 
cessing  standards 

!  primary  treatment 
% of waste  water 
undergoing  secondary  treatment 

Impact  indicators 

Consumption 

Indirect  economic 
effect 

Volume of waste-water  treated 
Number of households /  businesses 
connected to waste-water  treatment 
facilities  by  programme  after one year 

%  of waste water  treated 
%  of households /  businesses 
connected to waste water  treatment 
facilities 

Number  of economic  units (firms, 
farms, etc.) declaring  that the new 
water supply  system  has released a 
major 
constraint  for their  development 

Environment 

%  of waste water samples from  point 
sources which show a specified  level of 
decline of  Identified  pollutants 

%  of  km of rivers monitored on which 
pollution  overcome a specified 
standard 

Source: Volume 2 of the MEANS  Collection 

Transversal evaluation of impacts on intervention priorities 

Although the "top­down" approach to measuring environmental impacts is 
dominated by the use of quantitative data (e.g. the indices constructed by 
Eurostat), a "bottom­up" approach can be based on qualitative data. Fur­
thermore, this qualitative data can later be transformed into a quantitative 
indicator, should this be necessary. 

Example: Constructing an environmental impact indicator on qualitative data 

A French evaluation team had to evaluate the environmental impact of a motorway. 
Their method consisted of sending a group of people to photograph what they consi­
dered to be landscape blackspots in the area of the motorway before corrective 
measures had been undertaken. The same photographic survey was then carried out 
after the corrective measures had been undertaken. The change in the number of 
environmental blackspots was then recorded. 

To summarise, indicators will have to be selected which are neither simply 
the standard type of socio­economic indicators traditionally used in the eva­
luation of structural policies, nor the specialist technical indicators which are 
being developed to monitor change in the state of the environment. These 
indicators will tend to resemble the traditional systems of indicators used for 
the evaluation of structural interventions in relation to resource, output and 
result indicators. However, the impact indicators will be largely made up of 
indicators of changing driving forces on (rather than the changing state of) 
the environment. 

■ Principal sources of information 

Each programming document should contain the following information on 
the environment: 

• a description of the main environmental characteristics of the geographical 
area covered by the programme; 

• an identification of any sub­areas of particular environmental significance; 

• the nature, cause and location of any significant pollution or other environ­
mental driving force problems; 

• a description of existing national / regional environmental policies; 

• information on national / regional environmental authorities and procedures; 

• a projection of the expected environmental impact of the programme as a 
whole and of each constituent priority / measure; 

• details of any steps which are proposed to mitigate any negative environ­
mental impacts of the programme or its components. 

Evaluating environmental impacts 

Although all of these details are not always found in a programming 
document, these documents are still the obvious starting point for obtaining 
information. 

Programme monitoring systems should provide data on resource expendi­
ture and outputs needed to answer many evaluation questions, but are 
unlikely to provide data on the evolution of the environment itself in the 
region concerned. However, in cases where the preparation of an interven­
tion has shown that gaps exist in the availability of specific context data that 
are of particular importance in evaluating its impacts, provision may be 
made to obtain this information within the monitoring system. 

Established statistical time series of environmental data provide an increa­
singly rich source of information that can be used to complement the infor­
mation provided in programming documents. 

Information from programming documents and monitoring systems can be 
expected to provide very specific micro­level information on the allocation of 
resources and their impacts in terms of driving forces on the environment. 
More established data series tend to concentrate on macro­level state 
variables. One of the tasks of evaluation is to try to draw connections 
between these two levels of information, and in carrying out this task it may 
also need to draw on information from previous research and evaluations or 
engage in original data collection and / or production. 

■ S o me examples of current practice in the European Union 

While a more detailed treatment of evaluation tools is provided in Volume 3 
of the MEANS Collection, a number of examples in­depth analyses of envi­
ronmental impacts are presented below. 

A variety of tools has been used in the identification and analysis of the envi­
ronmental impact of public policy. Selecting the most appropriate evaluation 
method or technique to use in any instance depends on the particular eva­
luation requirements that prevail. It is important to note, however, that each 
technique has its own strengths and limitations and, even within the European 
Union, the choice made is likely to be influenced by relevant national legislation 
and established national practices in relation to: project planning; the series of 
pre­feasibility, feasibility and technical studies which are typically required; and, 
lastly, established implementation, monitoring and audit procedures. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

An  assessment  of  the  impact  of  interventions  on  the  environment  can  be 
undertaken  using the  "step-by-step"  approach  presented  in  Part  IV of  this 
Volume. The  choice  of  tools  used  in this  instance  would  focus  on  minimal 
case studies, individual  interviews and focus groups. On the other hand, for 
an  in-depth  evaluation  of  environmental  impact  analysis,  other  evaluation 
tools have to be employed. 

•  Strategic environmental assessment of programmes in the Netherlands 

Strategic  environmental  assessment  (SEA)  refers  to  the  environmental 
assessment  of  programmes  and  policies  as opposed  to  individual  projects 
(the  latter  is the  traditional  function  of  environmental  impact  assessment  -
EIA). SEA developed as a result of the realisation that EIA may occur too late 
in the planning process to allow adequate consideration to be given to  alter­
natives. SEA is a formalisation  of the  process  in which the other  techniques 
mentioned  in this  section  are  applied,  and  an  SEA  system  is expected  to 
include formal  guidelines  on the  use of  scenario techniques,  screening  and 
scoping, case studies, etc. 

SEA takes  place  in the  context  of  a tiered  forward  planning  process  where 
policies at the international /  national  level are seen as shaping  programmes 
at  the  regional  level  and  projects  at  the  local  level.  Within  the  EU  this 
approach is adopted with  increasing frequency  in relation to waste manage­
ment, land-use and energy policies. 

Within the EU, the Netherlands  is the country where the practice of  strategic 
environmental  assessment  (SEA) is most  developed. The Dutch  experience 
in this area dates  back to the early  1980s and SEA is now legally required for 
various activities  including electricity  and water  supply, nature  conservation 
and outdoor recreation, and provincial waste management  proposals. 

A  National  Programme  on  Waste  Management  has  been  in  place  in  the 
Netherlands since 1992. This programme is revised every three years and an 
integrated  set  of  ElAs  as  well  as  a  SEA  have  been  implemented  on  a 
voluntary  basis  by the Waste  Management  Council  -  a partnership  of local, 
regional and national authorities. The approach works on three levels: 

•  Decisions  are made  on the  national  level concerning  the  waste  treatment 
options /  technologies  that  might  be  used. These decisions  are subject to a 
SEA before a selection is made. 

•  Decisions are made on the regional level regarding the possible  location of 
treatment facilities; these are also subject to a SEA. 

Evaluating  environmental 

impacts 

•  Finally, individual  project  alternatives  are selected for  each of the  chosen 
locations; these are subject to ElAs. 

The SEA considered two alternative scenarios. First, a situation in which the 
objectives  of the  National  Environmental  Policy  Plan on  recycling  and  the 
production  of  waste  were  met.  Second, a situation  in which  these  broad 
policy  objectives  might  not  be  attained  but  individual  programmes  and 
measures  were still expected  to  be effective  in reaching  their  more  limited 
objectives.  Based  on  this  analysis,  a  strategy  was  formulated  for  the 
medium-term  development  of  waste  disposal  facilities.  Regular  meetings 
with  provincial  authorities  (responsible  for  implementing  the  policy)  and 
interest groups were an integral part of the development of the SEA. 

The policy was then implemented by each of the 12 provincial authorities, with 
each one developing  four-year  programmes.  In each case global  objectives 
were set (e.g. 400 factories should have a waste prevention plan by the end of 
the programme), needs calculated, the potential for improvement in relation to 
specific indicators estimated and project selection criteria established. 

Within the provincial programmes project selection is usually carried out on the 
basis of ElAs. In view of the fact the EIA method is more established, it is not 
surprising  that  work  on the  project  level  has  proved to  be  more  evidently 
methodical, with  a clear  focus  on  assessing  alternative  processing  tech­
niques, and  a definite choice of  indicators. The relationship  of  output  from 
this  project  level activity  to  the  SEA  is not  clear-cut  and  it  seems that  the 
latter exercise has an inherently poorer quality of information. With regard to 
intermediate  or  ex-post  evaluation, even though  Dutch  legislation  is  more 
demanding  in this  respect  than that  of  many  other  countries,  there  is  only 
advice (rather than a requirement) that this  be implemented. Even if there is 
no obligation to adopt technically sophisticated evaluation techniques within 
SEAs, experience  has shown that the formalisation  of the process itself has 
brought  a number  of  benefits to the design and  implementation  of  policies 
with an environmental  impact. The following advantages  of the formal  inte­
grated  approach  offered  by  SEAs  have  been  identified  by  administrators: 
•  environmental  issues  are given  a greater  degree  of  influence  on  general 
policy formulation; 
•  decision makers routinely  use those results of the SEA process which are 
clearly well designed; 
•  although test  case experiences  may  not  - strictly  speaking  -  be generali­
sed, an impression of their relevance is more readily obtained in the context 
of a SEA and their results are more likely to influence decision-making. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Further potential benefits of adopting a SEA approach are: 
•  it  encourages  consideration  of  alternatives  which  might  otherwise  be 
ignored 
•  it allows more effective analysis of the cumulative effects of  projects 
•  it facilitates both the planning and analysis of synergistic  effects 
•  it is better able to evaluate ancillary / secondary effects of  actions 
•  it is capable of taking delayed impacts into account 
•  it  permits  the  development  of  standard,  cross-measure  mitigation 
provision 
•  it  can  enhance  both  communication  between  those  responsible  and  the 
involvement of the public 
•  it provides a framework  in which cumulative and synergistic  environmental 
impacts can betaken  into account  in a systematic fashion. 

•  The screening and scoping of a programme in the UK 

The concept of screening refers to two different types of  process  depending 
on the level at which  it  is applied. Within a measure, screening  involves the 
selection of projects for assistance through the application  of a standard set 
of project selection criteria. An example of this process is supplied below. At 
the  measure  level  itself,  screening  refers  to  the  initial  examination  of 
measures  to  determine  the  degree  of  environmental  impact  assessment 
which is required before approval is given for their implementation. 

The  process  of  scoping,  on  the other  hand, is concerned  with  determining 
which  potential  impacts  of  alternative  projects  deserve  consideration  in an 
EIA. In practice, this  is a considerably  more complex  process as there  may 
not  be  agreement  among  interested  parties  as  to  precisely  which  impacts 
are relevant. 

Under  UK  legislation, specific  types  of  physical  project  are either  classified 
as  being subject  to  mandatory  environmental  impact  assessment  (typically 
very  large  capital  projects  with  a clear  and  direct  environmental  impact)  or 
checked against  a number of criteria (such as the  location, scale,  proposed 
method of dealing with waste, etc.) of the individual project to judge  whether 
a full EIA is needed. Scoping is carried out  in voluntary discussions  between 
project  developers  and  local planning  authorities  prior to the submission  of 
environmental statements on the proposed  projects. 

The  region  of  Merseyside  in  England  is  the  subject  of  a  programme  of 
assisted activity for the period  1994-1999  under objective  1 of the structural 

Evaluating  environmental 

impacts 

policies. Each project seeking assistance under this programme is screened 
in a two-stage process: 
•  an initial stage which verifies that the proposed project meets UK statutory 
controls; 
•  a second  stage  which  scores  proposals  against  environmental  criteria 
which have been separately established for each of the programme's priori­
ties  and  measures  and  also  takes  account  of the  precise  location  of  the 
proposed  project. 

A summarised  example of an environmental  screening form for  a particular 
measure of the Merseyside programme is provided in Box 12. This approach 
is of  particular  interest  as  an example  of  combining  economic  objectives 
with the other  objectives  pursued  under  a specific  programme  assisted  by 
the Structural Funds. 

Box 12 - Example of an environmental  screening form 

Priority  1: Measure  1: Quality  sites and premises for industrial  and commercial investors. 

Objective: To ensure an adequate  provision of quality  industrial  and commercial sites and  premises 
encourage  indigenous growth and Increased employment  opportunities. 

to 

Project: 

Reference: 

Yes 

N/A 

No 

1.  If the development  Is on a brownfield site, will It reduce the area of 
derelict  land or return buildings to productive  use? 

2. Does the development  respect designations  regarding  the natural 
environment  or built  heritage? 

3. Does the development  incorporate good  practice for energy conservation, 
water recycling  or waste reduction or involve innovative  environmental 
measures? 

4. Will the site have an environmental  management  plan or are measures 
In place for post-development  environmental  management? 

5. Have any  measures been taken to ensure that  the site is easily 
accessible by  public  transport? 

6. Is the development  designed to  Improve the local built  environment, 
for example by minimising the effects  of crime, vandalism  or litter? 

7. Does the development  project  contain  proposals for  environmental 
Improvements such as  landscaping? 

1 

1 

1 

1 

1 

0.5 

0.5 

0.5 

0.5 

0.5 

0.75 

0.38 

0.75 

0.38 

Total score: 

If the project  Is In a priority  area multiply  this  score by  1.5  and round to the nearest 1. 

0 

0 

0 

0 

0 

0 

0 

In those sub-areas of the Merseyside region which are judged to experience 
particularly  severe environmental  problems  or where environmental  impro­
vement could be expected to have an unusually  strong impact  on develop­
ment,  the  total  score  of  environmental  criteria  is  positively  weighted  in 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

relation  to  other  project  selection  criteria. These  particular  sub-areas  were 
themselves defined according to a number of criteria, including: the visibility 
and reputation of the site in question, its proximity to strategic  transportation 
corridors or locations regarded as assets, and whether it already qualified for 
assistance from other sources of funding. 

Council  Directive 85 / 337 /  EEC specifies all projects for which  assessment 
is mandatory  and  others  to  which  particular  thresholds  and  criteria  apply. 
However,  in the context  of  programmes  supported  by the Structural  Funds, 
Member  States  are  also  required  to  ensure that  any  individual  operation  is 
compatible  with  EU environmental  policy  and  legislation.  Some  form  of 
screening is therefore required of all the components of any prograrhme, but 
much  depends  on  the  quality  of  the  specific  criteria  developed  and  the 
design of the process adopted in each instance. 

Screening is obviously necessary at the ex ante phase and should at least be 
confirmed during subsequent  phases of evaluation. It may also be useful for 
ex-post  evaluations  to  adopt  a screening  technique  to  focus  analysis  on 
those measures likely to produce the greatest  impact. 

•  Case studies of projects in Scotland 

It is often useful to select  individual examples of a phenomenon for  in-depth 
study, for example: 
•  when  exploratory  fieldwork  is  required  due  to  the  absence  of  adequate 
information; 
•  to provide a detailed analysis of a particular process of environmental impact; 
•  to test a hypothesis or assertion about a particular  project. 

Case studies may be individual or multiple and in the latter case may involve 
sophisticated  techniques  to  ensure  the  replication  of  design  in  different 
contexts. The  approach  has four stages: design and  selection, data  collec­
tion, data analysis and report  preparation. Selection of the  case(s) for  study 
is particularly important for the relevance of findings. Cases may be selected 
on the basis of being either typical or representative of a situation which is of 
interest for other reasons (e.g. best / worst cases, particular  circumstances). 
Data collection may involve the use of surveys and interviews. 

In  an  attempt  to  develop  a  useful  method  for  appraising  environmental 
projects  intended  to  restore  or  protect  specific  sites,  the  Scottish  Office 
engaged  evaluators  to  review  methods  of  valuing  environmental  impacts. 

Evaluating  environmental 

impacts 

On the basis of the first stage of the study and in view of the type of  projects 
being considered, it was decided  that the contingent  valuation method (the 
various impacts are valued by the beneficiaries and losers through question­
naires), in particular, warranted further investigation. The second stage of the 
work  involved  applying  this  method  to  seven  projects. The cases  studied 
include the improvement  of an old industrial site, the restoration of an urban 
area,  the  protection  of  a mountain  area  and  woods,  and  support  for  an 
industrial heritage centre. In each case members of the public were intervie­
wed  or surveyed. The  results of this  case study  confirmed  the value of the 
technique  examined  and  gave  a clearer  indication  of  the circumstances  in 
which it could usefully be applied. 

Multiple case studies can be used to analyse the effects of interventions in a 
number  of contexts  across the programme  area. Provided  adequate provi­
sions were made for design replication, they could be used to better  identify 
impacts by comparing situations in which a measure was applied to those in 
which it was not. Another subject, which appears to offer particular scope for 
the  application  of  this  technique,  is the  relationship  between  changes  in 
pressure  variables  and  state  variables.  Coefficients  might  be  developed 
from case studies that would allow calculations of impact on the state of the 
environment to  be made  on the  basis of  indicators of  change in pressures. 

While the nature of effects can be determined inductively by using the case-
study  technique,  this  technique  cannot,  strictly  speaking,  demonstrate 
causality.  However,  since the  evaluation  of  environmental  impact  generally 
needs to  make an  inductive  leap  of faith  from  knowledge  about  change  in 
driving force variables to related changes in environmental state variables, a 
case-study approach is at no disadvantage in this respect. 

•  Combining cost-benefit & multi-criteria analysis to assess projects in 
Ireland 

Cost-benefit analysis attempts to measure in monetary terms all of the costs 
and benefits of a project  over the course of its lifetime. It is commonly  used 
at the ex ante stage of  project  evaluation, to choose  between a number  of 
alternative  projects  and to judge for which, if any, the calculated  economic 
benefits exceed the economic costs involved. 

In a more sophisticated version of cost-benefit  analysis, instead of attemp­
ting to express all benefits in terms of a single monetary value, a number of 
explicit  evaluation  criteria  are  established  which  can  be  weighted  by 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

decision  makers  to  allow  for  a  ranking  of  alternatives.  This  approach  is 
known  as  multi-criteria  analysis  (MCA)  and,  in  relation  to  our  present 
concerns, offers the advantage of permitting systematic account to be taken 
in the decision-making  process  of  both  environmental  costs  and  economic 
costs,  rather  than  attempting  to  translate  all  environmental  criteria  into 
monetary  terms.  In  Ireland, the  Department  of  the  Environment  engaged 
consultants  to  undertake evaluations  both to  meet  EU requirements  for  the 
selection of specific  environmental  infrastructure  projects  (co-funded  in this 
case  by  the  Cohesion  Fund)  and  to  establish  a  methodology  for  future 
decision-making.  Traditional  cost-benefit  analysis  was  rejected  by  the 
consultants  who  recognised  that  the  quantification  of  benefits  in  monetary 
terms  posed  too  many  problems  in this  case  (water  supply,  wastewater 
treatment  and water quality management  projects). 

One  potential  weakness  of  adopting  a  multi-criteria  approach  is that  the 
weightings  used  are  essentially  subjective  and  scores  are  sometimes 
arbitrary.  For this  reason, the  approach  adopted  combined  the  techniques 
for  identifying  criteria  used  in  MCA,  with  a  common  denominator  of 
monetary values derived from  relevant studies. In each case the basis of the 
weightings  used  was  explained. This  approach  was  applied  with  variations 
for each of the three types of project to be evaluated. 
In the  case  of  water  supply  projects,  the  approach  had  four  components: 
•  identification of all costs associated with each alternative project; 
•  quantification of all direct and indirect  benefits; 
•  ascribing values to benefits for each project; 
•  calculation of benefit / costs ratios. 

Information  was  collected  on  pre-  and  post-project  demand  levels  broken 
down  by sector,  population  served  and the  design  demand  and  population 
for  each  project.  Standard  operating  costs  were assumed  to  apply  to  each 
alternative and three types of benefit were identified: 
•  quality  improvements 
•  extension of supply to new  consumers 
•  wastage elimination and control. 

Each  of  these  three  benefits  was  assigned  a  specific  unit  value  and  a 
weighting  derived  from  analysis  of  relevant  studies.  The  basis  of  these 
values was  identified  and  both the  reasons for  using them  and the  limits  to 
their  applicability  were  explained. Competing  projects  were  then  analysed 
using  a  number  of  different  scenarios  of  likely  demand  evolution  and  a 
benefit / cost ratio was calculated for each one. A ranking of the projects was 

Evaluating environmental impacts 

made and the results of this ranking were clearly explained so that the impli­
cations for decision­makers were obvious. 

All of the variants on this approach are essentially tools for decision­making 
in the selection of alternative projects. Much depends on the professiona­
lism of evaluators in clearly indicating either the assumptions involved in 
establishing single indices for ranking projects or (in the case of MCA) the 
need for decision­makers to themselves arbitrate between criteria. 

To the extent that these techniques are applied rigorously, they can be a 
useful means not only of appraising the relative strengths and weaknesses 
of alternative projects but also of providing a synthetic score for their sustai­
nability which can then be checked against actual performance. 

■ Summary 

The quality of the environment and how it can be maintained in the face of 
economic pressure is an issue that has risen rapidly to the top of the political 
agenda. At the same time, there has been an increasing acknowledgement 
that economic development does not have to involve a trade­off in environ­
mental terms and that the two can, in fact, be mutually reinforcing. However, 
ensuring that economic development and cohesion are sustainable requires 
that both economic and environmental factors are taken into account when 
evaluating structural interventions. For evaluation, this means that interme­
diate and ex post evaluation of environmental impacts of projects and pro­
grammes need to be undertaken, in addition to the more common practice 
of assessing projects for their environmental effects. 

Although the standard DF­S­R framework may be a useful tool in other cir­
cumstances, it is of little use for the evaluation of the effects of Structural 
Fund interventions on the environment. A "bottom­up" approach is therefore 
proposed which, at the impact level, focuses on driving force variables, 
combining both socio­economic and environmental concerns. 

The predominance of ex ante project assessment means that there are few 
examples of good practice in the use of tools in intermediate and ex post 
evaluation. Nearly all the examples presented are thus of tools that have 
been used in circumstances of ex ante project assessment, but which can 
generally be adapted to other evaluation situations. 

Transversal evaluation of impacts on intervention priorities 

■ For m o re information see... 

Department of the Environment. (1993), Environmental Appraisal of Develop­
ment Plans, A good practice guide, London: HMSO. 

Ecotec Research and Consulting Ltd. (1997), Encouraging Sustainable 
Development through Objective 2 Programmes: Guidance for Desk Officers, 
Note of the European Commission (DGXVI). 

European Commission (1998), A Handbook on Environmental Assessment 
of Regional Development Plans and EU Structural Funds Programmes, 
European Commission, DG XVI. 

European Commission (1999), The Thematic Evaluation of the Impact of the 
Structural Funds on the Environment, European Commission, DG XVI. 

European Environment Agency. (1996), Guidelines for Data Collection for 
The Dobris+3 Report, Technical Report, Copenhagen: EEA. 

European Environment Agency. (1995), Environment in the European Union, 
Report for the Fifth Environmental Action Programme, Copenhagen: EEA. 

Fehily, Timoney, Weston. (1995), The Economic Appraisal of Environmental 
Projects Supported by the EU Cohesion Fund, Report for the Department of 
the Environment, Dublin. 

OCDE. (1996), La Cohérence dans l'Evaluation Environnementale, Orienta­
tions générales pratiques pour les projets de coopération pour le développe­
ment, Paris: OCDE. 

OECD. (1997), Environmental Indicators for Agriculture, Paris: Publications 
Service, OECD. 

Sadler B. (1996), Environmental Assessment in a Changing World : Evalua­
ting Practice to Improve Performance, International Study of the Effective­
ness of Environmental Assessment, Canadian Environmental Assessment 
Agency, Minister of Supply and Services Canada. 

Statistical Office of the European Community. (1997), Indicators of Sustai­
nable Development, Luxembourg: Office for Official Publications of the 
European Communities. 

Wood C. (1995), Environmental Impact Assessment, A comparative review, 
Harlow: Addison Wesley Longman Ltd. 

Evaluating the impact of structural interventions on the 
competitiveness of Small and Medium­Sized Enterprises 
SMES 

■ Why evaluate impacts on Small and Medium­Sized 

Enterprises? 

Small and Medium­Sized Enterprises (SMEs) make an important contribu­
tion to job creation, competitiveness and growth. Net job creation, particu­
larly in enterprises with fewer than 100 employees, more than compensated 
for job losses in large enterprises during the period 1988 to 1995. Despite 
this apparent dynamism, employment created by new enterprises tends to 
be somewhat unstable, with only half of new businesses created still in 
existence after five years. 

Box 13 ­ Definition of SMEs 

The European Commission has adopted a Recommendation concerning 
a standard definition of SMEs: Medium­sized enterprises have less than 
250 employees, an annual turnover of less than 40 MECU or balance 
sheet assets of less than 27 MECU and must be independent, which 
means having less than 25% of their capital owned by a large enterprise. 
Small enterprises have to have less than 50 employees, a turnover of less 
than 7 MECU or assets of less than 5 MECU and must be independent. 
Micro­enterprises are defined as firms with less than 10 employees (The 
recommendation on the definition of SMEs of 3 April 1996, can be found 
in the JOL 107 of 30.4.1996). 

The Madrid SME Policy Paper2 stresses that European SMEs' full potential 
for growth and employment creation has not yet been properly realised for a 
number of reasons. Firstly, it has been argued that the complex legal, fiscal 
and administrative environment faced by European enterprises has a relati­
vely higher impact on SMEs than on large companies. Secondly, SMEs often 
have problems accessing bank loans at reasonable rates or raising equity on 
the markets. In particular, small enterprises have difficulties in gaining 
access to parts of the market due to a lack of comprehensive market intelli­
gence or problems in identifying suitable partners. Others have argued that 
the business environment in Europe (in particular, the lack of sufficient risk 
capital, the existence of significant regulatory barriers and an aversion to risk 
taking) is not as conducive to the development of SMEs and, in particular, of 

2. Commission Communication of 29.11.1995 entitled Small and Medium Sized Enterprises 
­ A dynamic source of employment, growth and competitiveness in the European Union, 
Document CSE (95)2087. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

New  Technology  Based  Firms  (a restrictive  definition  of  New  Technology 
Based  Firms  -  NTBFs  -  is that  of  firms  active  in the  developing  industrial 
sectors  based  on  emerging  technological  paradigms  such  as  bio-techno­
logy or  laser technology).  For the purpose of this document,  the wider  term 
of  high-tech  firm  covering  all  enterprises  active  in  high-tech  sectors  or 
producing  high-tech  products  will  be  used  (a  typology  of  sectors  and 
products  has  been developed  by the  OECD). Where SMEs are  not  techno­
logy intensive, better  use needs to be made of readily available technologies 
in order to remain competitive.  However, very few  SMEs search patent  data 
banks in order to find out what technology  is available  "off-the-shelf". 

In  less  favoured  rural  areas  SMEs  account  for  a  relatively  high  share  of 
employees  in the  local  economy  but  also  face  particular  problems.  Whilst 
difficulties  vary  between  regions,  the  quality  of  infrastructure  and  the 
distance  from  main  markets  can  sometimes  be  a  problem,  as  can  poor 
access  to  information, poorly  developed  local  industrial  base and  a  lack  of 
suitably  qualified  workers.  Although  these  factors  pose  problems  for  the 
development  of  SMEs  in  general,  in  some  regions  self-employment  and 
entrepreneurship  are  values  that  are  recognised  and  encouraged.  Thus, 
some less favoured rural areas belong to the most dynamic regions, in terms 
of  new  and small  business development,  and  generate  more new jobs  than 
the national average. 

•  The Community  context 

Recent  European  Councils  have  given  a  positive  political  signal  towards 
SMEs,  notably  in Amsterdam  (June  1997),  Luxembourg  (November  1997) 
and  in Cardiff  (June  1998). As a result  of  the Amsterdam  Council, the  BEST 
task force was created and  presented  its report to the Commission  in 1998. 
This report  includes a number  of recommendations  to  improve the  business 
environment,  some  of  which  should  help  SMEs  to  grasp  the  opportunities 
offered  by  Community  programmes. The  Luxembourg  Special  Employment 
European  Council  recognised  the  promotion  of  entrepreneurship  as  a 
common guideline for Member States' employment policies within the Union. 
As  a follow  up, the  Commission  has  adopted  a  Communication  entitled 
"Fostering Entrepreneurship  in Europe - Priorities for the Future" setting out a 
number of priority  issues both at European and national levels. Moreover, the 
conclusions  of the Job Summit gave a positive signal towards SMEs by sup­
porting the further development of financial instruments and cross border part­
nerships. The Cardiff European Council has asked the Commission to present a 
report in Vienna on the follow up of the BEST recommendations. 

Evaluating  impacts  on  the  competitiveness  of SMEs 

Since SMEs play a primary role in balanced  local and  regional  development, 
and are a driving force behind  effective  adaptation  to  structural change and 
innovation, the  European  Commission  has recommended  that, within  the 
overall framework of interventions in industry and services, it was necessary to 
take the needs of SMEs systematically  into account  in the implementation of 
programmes. For this purpose, the Commission has developed the Integrated 
Programme, a global framework for  co-ordination  of  all activities for SMEs. 
This includes the Third  Multi-Annual  Programme for SMEs (1997-2000), the 
contributions that  other policies make to the development  of SMEs, and the 
Concerted  Actions  which  have the goal  of  promoting  the  co-ordination  of 
Member States' SME policies through the exchange of best practices. 

To accomplish this aim, the following five priorities are set out in the Integra­
ted Programme: 

•  simplify  and  improve the administrative  and  regulatory  business environ­
ment; 

•  improve the financial environment for enterprises; 

•  help SMEs to Européanise and internationalise their strategies, in particu­
lar through better information and services; 

•  enhance  SMEs'  competitiveness  and  improve  their  access  to  research, 
innovation and training; 

•  promote  entrepreneurship  and  support  special  target  groups  such  as 
women and young entrepreneurs. 

Structural  Funds  contribute  to  achieving  the  objectives  of  SME  policy 
through  their  support  for  the  creation  and  development  of  SMEs  in  the 
various  operational  programmes. The  European  Investment  Bank  and  the 
European Investment  Fund are also involved in providing support for SMEs. 

Although  the  main  objectives  of  Structural  Fund  interventions  in favour  of 
SMEs are similar across the EU, there are differences  in the specific  objec­
tives  from  one  region  to  another. Thus, in objective  1 regions  there  is  an 
emphasis  on  strengthening  the support  structure  for  SMEs.  In objective 2 
regions  where  large-scale  traditional  industries  have declined,  measures 
focus on promoting entrepreneurship and creating alternative job opportuni­
ties. In objective 5b regions the emphasis is on supporting the diversification 
and  modernisation  of  rural  SMEs and  in objective  6 areas the  priority  is to 
help  SMEs  in overcoming  problems  caused  by their  distance  from  major 
population centres. 

Transversal evaluation of impacts on intervention priorities 

The development of the potential of European SMEs in general, and in 
relation to the role they play in social and economic cohesion in particular, 
represents a major strategic aim of the European Union. The level of support 
for SMEs in terms of resources allocated from the Structural Funds for the 
period 1994­1999 reflects the importance that the Commission attaches to 
small and medium­sized firms. For instance, on average 10% of ERDF 
resources are devoted specifically to measures dealing with the promotion of 
SMEs and the improvement of the framework conditions in which they 
operate (Framework conditions correspond with the business environment in 
the widest sense, incorporating: the overall macroeconomic conditions, tax 
system, regulatory environment, bankruptcy laws, competition policy, etc.). 
This is almost double the level of support provided in the period 1989 ­1993. 

The purpose of evaluating the impacts of Structural Fund interventions with 
respect to the priority of SME competitiveness is defined by the European 
Commission as being to "provide a thorough and systematic analysis of the 
contribution and impact of Structural Funds' support to SMEs in the EU 
Member States and regions". Thus, evaluation has a major role to play in 
assessing the extent to which policy objectives have been or are being 
achieved, in providing insights into the impacts of Structural Funds on 
SMEs, in guiding the process of policy making and, consequently, in 
enhancing the effectiveness of future programmes in this domain. 

■ Defining w h at n e e ds to be evaluated 

Evaluations of Structural Fund interventions in favour of SMEs need to look 
at the strategies underpinning the implementation of programmes (i.e. how 
well­founded the strategies are in relation to the strengths and weaknesses 
of the assisted regions, to other public policies related to SMEs and to 
current ideas on SME support). They also need to examine the implementa­
tion of measures in favour of SMEs in order to assess them in terms of effi­
ciency and effectiveness. Impacts of measures not specifically targeted at 
SMEs should also be assessed to see if interventions are benefiting SMEs 
while meeting their stated objectives (i.e. increasing employment or growth). 

• Guidelines and key issues in SME evaluations 

The effectiveness of Structural Fund interventions in favour of SMEs 
depends as much on how efficiently support is delivered as on their design. 
For this reason it is important to examine the implementation of pro­
grammes. The following key issues can be used as guidelines: 

Evaluating  impacts  on  the  competitiveness  of SMEs 

•  How are support measures being delivered? Has the fragmentation of SME 
support  structures and schemes that exists in some countries been reduced 
by  the  introduction  of  innovations  such  as  "one  stop  shops"  (a  function 
performed,  for  example,  by  Chambers  of  Commerce  in  Germany  and 
Business  Links  in the  UK)? Are addressees  less  confused  about  support 
schemes?  Who  manages  SME  programmes  and  to  what  extent  are they 
implemented  through  partnerships  between  public  and  private  bodies?  In 
regions where institutional structures are relatively weak, to what extent have 
SME support  programme  delivery  mechanisms  been strengthened  through 
institutional  restructuring, capacity  building  or the  involvement  of  private 
agencies? 
•  How  are the different  types  of  SME  programmes  themselves  appraised? 
How  is  the  implementation  of  measures  monitored  and  evaluated  by 
different national and regional authorities? How will the results of the evalua­
tion be taken  into account  in the formulation  of future  policy  developments 
or the reorientation of existing programmes? 

There are a number of key issues related to the impact on regional develop­
ment of Structural Fund interventions in favour of SMEs: 
•  Interventions aim at both encouraging start-up of new businesses (e.g. by 
providing seed capital) and supporting existing SMEs. Issues of survival and 
performance of assisted enterprises are therefore both relevant. 
•  Interventions  also  aim  at  promoting the exchange  of  information  and the 
diffusion  of  best  practices  between  SMEs situated  in different  regions. The 
extent of transborder networking is therefore an important issue for evaluation. 
• To a certain extent, the creation of employment and the creation of wealth at 
the micro-level involve a trade-off  between these objectives. It may therefore 
be important  to examine to what  extent assisted enterprises  reinvest  profits 
to assure their survival over the medium and long-term, and the  potential of 
these investment  to engender job creation. Furthermore, when  employment 
creation or  maintenance  can be attributed  to the  contribution  of  Structural 
Fund interventions, the cost should, whenever possible, be estimated. 
•  Many  small firms  produce  a narrow  range of  products  and services. This 
means that when diversification  is the  objective  of  an intervention, it  is the 
emergence of new firms specialising in new market niches, rather than the reo­
rientation of existing ones, that will tend to be the main motor of this process. 
•  SMEs may also have a very limited client base, and in some cases they can 
find  themselves  in  a  position  where  they  have  only  one  buyer  of  their 
products  or  services.  In this  situation  close  relations  with  larger firms  can 
have  a positive  effect  as subcontractors  adopt  technological  and  perfor-

Transversal  evaluation  of  impacts  on  intervention 

priorities 

mance standards of the contracting  firm, or benefit from technology  transfer, 
technical training and assistance. On the other hand, the competitive  position 
of such firms is likely to be weak, and the economic fabric of the region will, as 
a consequence, be fragile. The place of assisted enterprises in supply  chains 
and clusters may in certain cases represent a key evaluation issue. 

When comparing different  programmes at the national or regional level, top-
down  techniques  can  be  employed  to  gather  relevant  contextual  informa­
tion. On the other hand, the use of bottom-up techniques allows data collec­
tion  about  programme  outputs  and  results,  such  as the  number  of  SMEs 
assisted,  the  number  of  new  businesses  assisted,  and  the  efficiency  and 
effectiveness  of  the  implementation  of  interventions. The  estimation  of  the 
impacts of  interventions  at the project  and measure  level also lends itself to 
the bottom-up approach. 

It  is  important,  when  looking  at  measures,  to  distinguish  between  those 
which are directly targeted at SMEs and which are easily identified  in Opera­
tional  Programmes,  and  those from  which  SMEs  benefit  only  indirectly  and 
which are therefore much more difficult to trace. 

Finally, it is essential to distinguish  between addressees in terms of their size 
by using the classification of SMEs, including micro-enterprises,  established 
by the Community. 

•  Clarifying the objectives of SME  interventions 

An early step in the evaluation of SME interventions is to develop a hierarchy 
of objectives  in order to be able to examine the strategic aims of the Structu­
ral Fund programmes at various  levels. This can be done by constructing  an 
objectives  tree, which  allows  specific  SME  objectives  to  be  linked  up  to 
more  global  socio-economic  objectives  envisaged  within  the  Structural 
Fund framework. An example  of  such an objectives  tree for Structural  Fund 
interventions  in the  domain  of  SMEs  is  presented  in  Box  14. The  interme­
diate objectives of increasing competitiveness and promoting entrepreneurship 
and, to a lesser extent, Europeanisation and internationalisation of SME opera­
tions, are those most closely associated with Structural Fund interventions. 

By integrating objectives that are not necessarily uniquely those of Structural 
Fund  interventions  (e.g. simplifying  start-up  procedures  and the  legislative 
framework, etc.), the objectives tree allows the coherence of objectives to be 
examined  and the existence of synergies  to  be explicitly taken into conside­
ration. 

Evaluating impacts on the competitiveness of SMEs 

Box 14 ­ Example of a thematic objectives tree for SME 

interventions 

Global objectives 

Examples of specific objectives 

Maximise the 
contribution of 
SMEs to social 
and economic 
cohesion 

Improve the business 
environment 

Européanise and 
internationalise the 
operations of SMEs 

Increase the 
competitiveness 
of SMEs 

Promote entrepreneurship 
within special target |­
groups 

Improve access to venture capital 
Reduce administrative burdens 

Increase the supply of equity capital 
Simplify start­up procedures 
Simplify the legislative framework 

Reduce costs In trading internationally 
Improve Information and co­operation strategies 

Reduce trade barriers to third countries 

• Improve access to R & TD Programmes 
• Adapt training programmes to the needs 

of SMEs 

Encourage entrepreneurship culture 
Increase the development of new SMEs 
Increase the number of women and 
young entrepreneurs 

Links between intermediate objectives <­­

Objectives outside the Structural Funds  Π 

• Net impacts arising from interventions in favour of SMEs 

Whenever possible, attempts should be made to assess the impacts of SME 
programmes on a net basis. This means that deadweight effects should be 
estimated so that the counterfactual situation can be established (e.g. the 
level of turnover, the amount of investment, the number of innovations, etc. 
that would have occurred in a policy­off situation). Furthermore, displace­
ment effects may have to be taken into account. Substitution effects may 
also occur and may even, in certain instances, be sought (e.g. an increase in 
the proportion of women and young entrepreneurs, diversification of the 
local productive activities towards more dynamic sectors). 

Transversal evaluation of impacts on intervention priorities 

Example: From gross to net impacts in the case of support for new business 
creation 
A scheme, part financed by the Structural Funds, to provide seed capital to finance 
the setting up of new businesses, has led to 60 new micro­enterprises being launched 
with a combined turnover of 1,36 million euro after the first year. Surveys of the 
addressees of the scheme found that 15 new enterprises with a turnover of 0,38 
million euro would have been launched in the absence of the scheme (deadweight). It 
was also found that two addressees had been running undeclared businesses with a 
combined annual turnover of 32 000 euro. However, because this was considered as a 
positive effect of the intervention (i.e. the substitution of registered actives for unde­
clared activities) it was not subtracted from the gross effects. The net effect of the 
intervention was thus estimated at 0,98 million euro. 

In the case of support for investments (e.g. in physical infrastructure, capital 
goods, research and development, etc.) the leverage of Structural Fund 
expenditure is often explicitly sought. The leverage effect, defined as the 
amount of private sector finance invested in an assisted project that can be 
directly attributed to Structural Fund co­financing of the project, should be 
estimated when relevant. 

■ Examples of i m p a c ts by d o m a in of intervention 

SMEs are directly affected by interventions aimed at supporting productive 
investment. Furthermore, interventions targeting the provision of basic infra­
structure and the development of human resources have direct and indirect 
effects on SMEs. 

Evaluating 

impacts  on  the  competitiveness 

of  SMEs 

•  Basic  infrastructure 

Infrastructure  investments  in  favour  of  SMEs  are  directly  targeted  at  the 
needs  of  businesses, for  example,  interventions  supporting  the  construction 
of  facilities  specially  designed  to  nurture small  high-tech  enterprises  or  new 
businesses. This type  of infrastructure  is often an integral element  of  broader 
interventions  in  favour  of  SMEs.  Other  infrastructure  interventions  have  an 
indirect  impact  on  industry,  usually  by  reducing  environmental  and  accessi­
bility  constraints  (transport,  energy supplies,  information). 

Example: A road improvement project in Ireland 
Within the framework  of Structural  Fund interventions  in the area of  basic  infrastruc­
ture, one of the prime objectives is to increase mobility within regions. The Athlone By-
Pass  project  involved  the construction  of  a  10  km dual  carriageway  by-pass  and a 
river  bridge. This  project  did  not  include  specific  measures  to  support  industry  or 
SMEs,  but  by  allowing  road freight  to  avoid  the town  centre, constraints  on  local 
industrial  development  and  access  to  ports,  and therefore to  export  markets,  was 
improved. In addition to these indirect effects, during the period of construction  local 
SMEs were directly  involved  in subcontracting to the main  contractor  of the  project. 

Outputs  > 

Effects 

> 

Creation of temporary 
employment 

Jobs created 
or maintained 

Construction of 
a new dual 
carriageway 

Reduction in 
delivery times 
to major markets 

Improved road 
accessibility for local SMEs 

Priority 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

•  Productive environment 

Interventions  in the  area  of  the  productive  environment  are those  with  the 
most direct effect on SMEs. These may include interventions  in the domains 
of  rural  development  and  agriculture,  tourism, fisheries,  etc.  which,  whilst 
not specifically aimed at SMEs, are intended to have an impact on entrepre­
neurship  and  small  businesses.  Other  interventions  explicitly  target  SMEs 
(e.g.  information  services,  provision  of  tailored  facilities  and  on-site  advice 
services, support for venture capital schemes, etc.). 

Example: T he  Ouest  D o u ze  Initiative  (France) 

The  Ouest  Douze  Initiative,  located  in  C a p d e n a c,  A u b in  et  Decazeville,  in  southern 

France,  is  c o - f i n a n c ed  by  the  ERDF,  the  French  g o v e r n m e n t,  local  b o d i es  a nd  the 

private  sector.  It  ¡s  a  non-profit  organisation  offering  technical  advice  a nd  financial 

s u p p o rt  (interest-free  loans)  to  potential  local  entrepreneurs.  It  also  runs  a  s c h e me 

w h e re  entrepreneurs  c an  be  t w i n n ed  w i th  e s t a b l i s h ed  c o m p a n i es  a nd  w h e re  past 

addressees  return  to  pass  on  their  experience  to  their  successors  on  the  s c h e m e. 

Outputs 

J^> 

Effects 

Support  and  advice 
sessions  with  potential 
entrepreneurs 

Arrangements  for 
soft  loans  for 
the  launching 
new  enterprises 

New  businesses 
formed 

ΐ> 

Increased 
entrepreneurial 
activity  in the  region 

Strengthening  of  SME 
base  in the  region 

Priority 

Q 

Evaluating impacts on the competitiveness of SMEs 

• Human resources 

Interventions concerning education, training and research in this domain 
may either be directed at industry as a whole or explicitly oriented towards 
supporting SMEs (e.g. training for individuals wishing to become self­
employed, assistance for research and development projects, brokering of 
research partnerships with other firms or institutes, etc.) 

Example: Furniture Technology and Design Project (Finland) 
This project, based in Päijät­Häme in the south of Finland, aims to help increase the 
competitiveness of businesses in the furniture industry. To do this it supports co­
operation and networking between manufacturers, suppliers and training organisa­
tions. Vocational training schemes are tailored to the needs of individual companies 
and advice is given on training strategies and on anticipating future training require­
ments. The project has succeeded in increasing the level of technology and design in 
participating firms, leading to increased competitiveness both at home and abroad. 

Outputs 

> 

Employees attending 
courses and improving 
their skills 

Effects 

\ t 

> 

Participating SMEs 
increasing export sales 

Training courses 
designed specifically 
for SMEs 

SMEs participating in 
the project and 
improving the quality 
of their workforce 

Participating SMEs 
expanding into new 
overseas markets 

Priority I I 

■ Defining and selecting indicators 

A more detailed treatment of indicators is provided in Volume 2 of the 
MEANS Collection devoted to indicators for monitoring and evaluation. The 
above Volume includes an information sheet on indicators for the priority of 
competitiveness of SMEs and enterprises. 

Interventions may be directly targeted at SMEs, others will affect SMEs indi­
rectly. The definition and selection of indicators will, to some extent, depend 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

on the type of intervention being studied. For interventions that do not focus 
uniquely on SMEs, but on the productive sector as a whole, a system of  indi­
cators should systematically  incorporate  the percentage of resources  being 
allocated to SMEs, % of  projects concerning SMEs, etc. This approach  cor­
responds  with  that  of  mainstreaming  used  to  systematically  highlight  the 
extent to which the promotion of equality  between men and women is being 
taken into account  in programmes (see Box 2). 

Demand  for  many  SME  support  services  depends  to  a  large extent  on  the 
economic  context.  Financial  consumption  will  consequently  tend  to 
fluctuate over time, depending  on the  position  of  national  economies  in the 
business  cycle.  It  is  also  interesting  to  examine  the  percentage  of  support 
being taken up by projects in the more dynamic or high-tech sectors. 

Output  indicators concern the quantity  of  physical  infrastructure  or  support 
services financed. In the case of  business support  services, which are of  an 
intangible  nature  (e.g. information  management,  training  or  consultancy), 
output  indicators  focus  on  the  scale  of  activity  actually  taking  place and  of 
services financed by the programme (e.g. hours of advice received by addres­
sees). They also focus on the number of addressees or projects assisted. 

Result  indicators  inform  on  how  the  supply  structure  of  a  region  has  been 
modified  by the  intervention  by, for  example,  looking  at the number  of  new 
business  start-ups. Other  indicators  may  be  more sophisticated; they  may, 
for example, look at the leverage effect that funding generates. 

Impact  indicators  of  interventions  in this  domain  reflect the  heterogeneous 
nature of  impacts  on  SMEs. They  generally  focus on the  increased  in value 
added  arising from  an intervention,  knock-on  effects  on suppliers  and sub­
sequent employment  impacts that eventually arise. In the case of new firms, 
indicators  focus  on  the  survival  rate  of  new  start-ups.  It  is,  however, 
important  to  be realistic  about the  impacts that  can  be observed during  the 
timescale  of  an  intervention.  Whilst  some  direct  impacts  on  employment 
may  manifest  themselves  in  a  relatively  short  period, indirect  employment 
impacts, knock-on  effects, and  impacts on product  and process  innovation 
usually  occur  only  over  the  medium  to  long-term. The  estimation  of  such 
impacts usually has to be based on the expectations of addressees. 

By way of an example, a system of indicators that could be used to track the 
production of impacts arising from a project  providing informational  support 
to promote exports by SMEs is presented in Boxi 5. 

Evaluating impacts on the competitiveness of SMEs 

Box 15 ­ Example of a system of indicators for an intervention 

providing informational support for SMEs to promote exports 

Programme Indicators 
(Related to the Intervention and its effects) 

Context Indicators 
(Related to the assisted area) 

Output Indicators 

Number of addresses 

Number of assisted SMEs 

Number of SMEs in the eligible area 

Result Indicators 

Satisfaction rate 

Geographical 
diversification 

Impact indicators 

Exports 

% of addressees who are satisfied or 
very satisfied with the support 
services provided 

Number of SMEs becoming 
new exporters 
Number of SMEs exporting 
to new markets 

% of exports sales in the turn over 
of assisted SMEs after 18 months 

Regional exports of SMEs 
related to GDP 

Value added 

Increase In value added generated 
after 18 months 

Average value added by employee 
in the region 

Direct employment 

Number of net jobs created / 
maintained (FTEs of which held by 
women) In firms In relation to export 
sales after 18 months 

Source: Volume 2 of the MEANS Collection 

■ Obtaining useful information 

Total number of 
unemployed in the assisted area 

The extent to which an intended approach to the evaluation of SME interven­
tions can be adopted will clearly depend on the quality and availability of data 
and the extent to which statistical data on regional economies can be disag­
gregated to the level of assisted regions. It needs to be emphasised that some 
impacts are not possible to quantify, and in this case a purely qualitative 
approach has to be adopted. The evaluation of Structural Fund interventions in 
the domain of SMEs will, to a large extent, be strongly dependent upon primary 
data, complemented by established statistical sources. 

One particular problem which arises when using the established definition 
based on employment levels of small (0 to 50 employees) and medium 
enterprises (51 to 250 employees) is that a medium­sized enterprise consis­
ting of 250 employees has very little in common with one of 50 employees. 
Similarly a small enterprise of 50 employees is usually very different to a firm 
of 10 or fewer employees. When primary data is to be collected, the hetero­

Transversal  evaluation  of  impacts  on  intervention 

priorities 

geneity within the overall category of SMEs can be dealt with through careful 
sampling  and  choice  of  case  studies.  However,  established  data  sources 
often do not collect data on enterprises of less than 20 employees - a popu­
lation which for  certain  types  of  intervention  (i.e. financial  support  aimed  at 
assisting small firms to break through the 20 employee barrier or the creation 
of  micro-enterprises)  may  be  of  particular  interest.  Another  problem  with 
secondary  data  is that  while the  "birth"  of  a new  enterprise  is  immediately 
registered  when the owner  goes through  the  relevant  administrative  proce­
dures, the  "death" of a firm may not  be recorded until a long time after  it has 
terminated  its  activity.  This  may  lead  to  an  over-estimation  of  overall  SME 
activity and to the inclusion of non-existent firms in analyses. 

Many  context  indicators, such  as business  start-up,  may  be available  from 
European  and  national  administrative  records,  statistics  and  surveys  (e.g. 
those  collected  for  taxation  purposes).  Eurostat  has  developed  a  special 
regional  database,  called  Regio,  of  regional  industrial  statistics  with  annual 
Europe-wide comparable data dating back to 1994. 

In conjunction  with  DG XXIII,  Eurostat  publishes  the  report  "Enterprises  in 
Europe"  at  regular  intervals. This  report  contains  detailed  information  by 
sector of economic  activity  and by payroll size classes on all the  enterprises 
in  the  European  Union  (on  the  basis  of  their  number,  workforce  and 
turnover),  as  well  as  thematic  analyses  (the  demography  of  enterprises, 
innovation in SMEs, labour costs and productivity  in SMEs, etc.). 

An  annual  publication  of  the  European  Observatory  for  SMEs  gives  an 
overview of the performance and perspectives of SMEs in all Member  States 
of  the  European  Economic  Area and  Switzerland  since  1993, and  provides 
structured data on SMEs in both quantitative and qualitative terms as well as 
a number of detailed thematic studies (tourism, environment, etc.). 

The BACH database, based on detailed balance sheet data and covering 23 
manufacturing  sectors  in  1987-1995  in  12  EU  countries,  contains  data  on 
SMEs. Data is broken down  by different size groups based on a harmonised 
definition.  The  information  is  provided  by  central  banks  and  statistical 
agencies. The database is particularly well suited to financial analysis. 

Evaluating impacts on the competitiveness of SMEs 

■ Appropriate tools for analysing impacts on SMEs 

While a detailed treatment of evaluation tools is provided in Volume 3 of the 
MEANS Collection, a number of tools particularly suited to evaluating 
impacts on SMEs are presented below. 

An assessment of the impact of interventions on SMEs can be undertaken 
using the "step­by­step" approach presented in Part IV of this Volume. 

Face­to­face interviews can be used to examine issues concerning the 
strategic aspects of Structural Fund interventions. They might be carried out 
with key actors in national authorities and managers who have overall res­
ponsibility for SME support programmes in the Member States. A disadvan­
tage of this technique is that it can be very time consuming when a large 
number of interviews have to be organised. When a large number of people 
sharing a collective experience (e.g. in the implementation of interventions) 
have to be interviewed, the use of focus groups may be less time consuming 
and more useful. However, it should be kept in mind that whilst the use of 
this tool may allow useful qualitative information about implementing inter­
ventions to be gathered, it is not suited to all situations (e.g. where a detailed 
description of processes is required, individual interviews are more likely to 
produce more useful information). 

Questionnaire surveys allow a basic assessment of the impacts of measures 
and give insight into the contribution of different schemes to SME competiti­
veness and growth as well as the implementation of programmes (i.e. the 
accessibility and delivery of EU­funded schemes). When a questionnaire is 
well designed, it will allow reasonably definite judgements to be made on the 
extent to which improved performance can be linked to assistance from EU­
funded schemes. 

Whilst it is less expensive and time consuming to rely on monitoring data for 
constituting samples, this presents a problem of sampling bias. Monitoring 
data can effectively include non­addressee firms (i.e. firms which have made 
contact with the operators but which have not subsequently become addres­
sees). However, using information on these enterprises to construct the coun­
terfactual situation ignores the behaviour of non­addressee firms that, for a 
number of reasons have not approached the authorities with regard to the 
programme (e.g. they may be unaware of a programme's existence, they may 
perceive that the cost of applying for assistance outweighs the benefits, etc.). 

Factor analysis is particularly useful for analysing the characteristics of large 
homogenous populations. Because SMEs are highly differentiated, this 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

technique can be used to generate clusters of firms according to their key cha­
racteristics. Impacts can then be analysed in relation to these characteristics. 
A  factor  analysis  is  usually  fed  with  data  from  questionnaire  surveys.  This 
technique  was successfully  used  in Spain when  evaluating the  employment 
impacts  of job  subsidies  in the Catalonian  co-operative  sector  (see Box  16). 

Box 16 - Example of a factor analysis of the Catalan co-operative sector 

The  use  of  factor  analysis  can  be  illustrated  with  respect  to  an  ex  post 
evaluation  project,  oriented  towards  the  employment  effects  (creation 
and maintenance  of jobs)  of  subsidies  in the sector of Catalan  co-opera­
tives. Evaluation in this case was complicated  by the diversity  of co-ope­
ratives and the consequent  heterogeneity of the population. The  problem 
was twofold: first, the  relevance  of  measures  taken  by the  region  had  to 
be  determined  in  view  of  this  situation;  secondly,  the  importance  of 
impacts  on  those  co-operatives  which  received  aid  had to  be assessed. 

Given the variety,  as well as the  lack  of  knowledge  of the population  and 
of  impacts,  factor  analysis  was  chosen  for  its  ability  to  reveal  the  main 
characteristics  of  the  population  under  consideration  (about  3,000  co­
operatives). 

The first step in the evaluation consisted of sending a questionnaire to all 
the  co-operatives.  This  was  completed  by  interviews.  The  data  thus 
gathered  were first  analysed  using traditional  statistical  methods  (search 
for  the  main  characteristics  of  the  population, drawing  up  of  cross-refe­
renced tables). The second step in the data analysis consisted of a factor 
analysis  with  a view  to  isolating  variables  likely to  explain  variations  in 
employment  in the  co-operatives:  including  the  size  of  the  entity,  age, 
sector of activity, technological level, etc. 

Source: Volume 3 of the MEANS Collection 

Surveys  can  be  prepared  or  complemented  by  a  number  of  case  studies 
which  are  able to  uncover  impact  causation  of  SME  measures  and to  test, 
justify  or qualify  results  obtained  by  surveys  or  other  methods. They  might 
focus on particular types of Structural Fund interventions (e.g. financial engi­
neering, human  resources,  business support  services, innovation and  tech­
nology, etc.). It is important, however, to carefully select case studies to allow 
useful in-depth analysis and comparison between  individual  projects. 

Although  the  selection  is  strongly  dependant  upon  the  willingness  of 
sponsors and partner  organisations  responsible for the action to  participate 
in the research, and on the availability of quality information (including moni-

Evaluating  impacts  on  employment 

toring  data),  reasonable  coverage  of  the  various  types  of  actions  is 
desirable.  The  selection  of  case  studies  can  be  guided  by  taking  into 
account  the SME initiatives that  involve  major  expenditure, and the  impor­
tance of  different  SME actions  from the  point  of  view  of  regional  develop­
ment strategies. For an example of the use of case studies see Box 17. 

Box 17 -  Example  of the  use of the case study technique  in the 

evaluation of the  NordTek programme,  Denmark 

This programme, intended for small and medium-sized firms, consisted of 
about  100 projects offering services to firms in the areas of training, consul­
tancy,  access  to  modern  computer  equipment,  etc.  It  was  therefore  a 
programme  that  could  be qualified  as  "soft",  as opposed to  "hard"  pro­
grammes  aimed  at  improving  infrastructures.  The  results  of  such  pro­
grammes  are difficult  to  evaluate  precisely  because they are not tangible. 

Six  large  projects  were selected  as cases to  be studied. In terms  of  the 
selection  criteria, the  projects  had to  be  representative  of  the  diversity 
existing  in intervention  strategies  and  locality. The selected  projects  also 
had to  have  benefited  from  significant  funding  in the framework  of  the 
NordTek Programme and be oriented exclusively towards firms (rather than 
research  institutions  or administrations). The six  projects  evaluated repre­
sented such a large portion of the total Programme that the analysis of the 
impacts  of these  interventions  could  be used  to  cover  all the  important 
phenomena. It was recognised without ambiguity that the selected projects 
represented the "best cases" of the programme. 

The data were collected at two operational levels: at the level of the project 
as such, and at that of the firms. As far as the  project was concerned, the 
evaluators studied each of the cases  in the field four  or five times  over a 
period of two  years. Each time the  managers were questioned  and each 
project  had to  keep  a record, in a  "register"  designed  especially  for  the 
purpose, of the services proposed and the firms concerned. The analysis of 
the  documents  thus  obtained  was  handed  to  the  project  leaders  and 
discussed  during visits. As for  the firms, the  data  were collected  during 
interviews with a sample of 20 SMEs which  had used the services offered. 
Finally, 40 SMEs who had not used the proposed services were questioned 
in the final stage of the evaluation. 

Source: Volume 3 of the MEANS  Collection 

Transversal evaluation of impacts on intervention priorities 

■ S u m m a ry 
The important role of small and medium­sized enterprises (SMEs) in suppor­
ting growth and job creation in the European Union has been brought into 
sharper focus over the past few years. The flexibility and innovative capacity 
of SMEs enable them to respond quickly to structural changes and adapt to 
changing demand. However, there are a number of factors that hamper the 
competitiveness of European SMEs and basic market and policy imperfec­
tions that inhibit growth and job creation. The evaluation of Structural Fund 
interventions in favour of SMEs therefore has a major role to play in 
assessing the extent to which stated policy objectives have contributed to 
overcoming these problems. 

Interventions may focus directly on SMEs or on SMEs as a sub­population of 
industry. SMEs are also implicitly targeted by interventions in other domains, 
especially those where diversification of productive structures is an 
objective (i.e. rural development, fisheries and tourism). 

As far as suitable empirical techniques for evaluating Structural Fund inter­
ventions in favour of SMEs are concerned, experience of their application to 
this domain is limited, but it is likely to accumulate rapidly as evaluations of 
this type of intervention become more commonplace. 

■ For m o re information see... 

EIM Small Business Research Consultancy (1997), European Observatory 
for SMEs: Fifth Annual Report, The Netherlands: EIM. 
European Commission. (1995), Activities in favour of SMEs and the craft 
sector, Luxembourg: Office for Official Publications of the European 
Community. 

European Commission. (1997), Report from the Commission to the 
European Parliament, the Council, and the Economic and Social Committee 
and the Committee of the Regions on the Coordination of Activities to Assist 
Small and Medium­sized Enterprises (SMEs) and the Craft Sector. 
Grant Thornton International Business Strategies Ltd. (1998), European 
Business Survey: Prospects and Issues for SMEs, Spring (6). 
OCDE. (1993), Les Petites et Moyennes Entreprises: technologie et compéti­
tivité, Paris: OCDE. 

OECD. (1998), Besf Practice Policies for Small and Medium­sized Enter­
prises, Paris: OECD. 

OECD, (unpublished), Assessment of SME Policies: An Analytical 
Framework, Working Document, DST/ IND/ PME (98)2. 

Evaluating the impact of structural interventions on 
employment 

■ Why do employment impacts need to be evaluated? 

• The Community context 

Structural unemployment is a persistent problem with which the European 
Union has to contend, even during periods of strong economic growth. 
Those who are unemployed experience a loss which is more than financial 
and the wider community has to bear the burden of supporting an economi­
cally inactive section of the population, the costs of increasingly established 
pockets of long­term unemployment and the associated deprivation which 
tends to be concentrated in both social and geographic terms. 

The response of the European Union to this situation is based on the 
premise that a Union­wide effort is needed to establish a both a micro and 
macro­economic context that favours the creation of employment. This 
effort must simultaneously co­ordinate and work in partnership with initia­
tives and action taken on a national, regional and local level. 

The Structural Funds are used to co­finance policy interventions which aim 
not only to promote a better economic and social balance across the Union 
by reducing regional and national disparities (cohesion), but also to assist 
the overall development of the Union's structural fabric, including that of the 
labour market. They are intended to stimulate economic growth and the 
creation of employment. Although some of the objectives of structural 
policies have always been explicitly related to employment, in recent years a 
growing concern with producing substantial employment effects has 
become increasingly evident both in principle and in practice. As a major ins­
trument for influencing economic development patterns and a major vehicle 
for the delivery of resources to local partnerships, it is evident that the use of 
the Structural Funds needs to reflect the increased priority which the Union 
has given to employment. 

The types of employment impacts likely to be produced by interventions 
assisted by the Structural Funds are critically influenced by the particular 
instruments used in each case. The choice of such instruments is influenced 
in turn by the objectives of the intervention in question. An intervention in the 
area of the productive environment or basic infrastructure may aim to 
generate employment effects by creating new jobs or maintaining existing 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

ones.  On  the  other  hand,  interventions  in the  area  of  human  resources 
focusing on training are targeted  at enabling  individuals to meet employabi­
lity  (i.e. the  ability  of  an  individual  to  meet  the  technical  and  organisational 
requirements  of employment  available at a given time and in a given  place). 
The  addressees  of  these  interventions  may  be for  people  already  in  work 
who  are adapting to changing work conditions, or the unemployed, particu­
larly  special  target  groups  such  as women, the  long term  unemployed  and 
young  people,  who  need  to  improve  their  employability  by  enhancing 
existing skills or developing  new ones. 

A  defining  characteristic  of  structural  interventions  is that  they  are concen­
trated  on  a limited  number  of  global  objectives,  and  each  programme  of 
interventions  assisted  by the Structural  Funds  is concentrated  on  achieving 
one of these objectives. Global objectives concern both specific regions and 
specific social groups and in either case they may be defined in relation to a 
specified  economic  sector.  The  relationship  between  the  different  global 
objectives  and  the  priority  of  promoting  employment  varies  considerably. 

•  Accurate evaluations of the effectiveness  of Structural Fund 
interventions 

The  potential  benefits arising from  effective  Structural  Fund  interventions  in 
the  area  of  employment  are  substantial:  a  reduction  in  the  poverty  and 
misery that  unemployment  causes, the creation of  labour supply  conditions 
that  promote  growth  and  competitiveness  in the  weaker  regions,  and  an 
increase  in social  and  economic  cohesion.  Ultimately  these  benefits  could 
also  include  increased  European-wide  competitiveness  and the  underpin­
ning of the European social model. However, if Structural Fund  interventions 
are  to  be  shown  to  be  effective,  evaluations  will  need  to  be  sufficiently 
sophisticated to measure employment  impacts accurately. 

If interventions  are to  be effective, the ex ante and ex-post  measurement  of 
employment  effects  has to  be as accurate as possible. On a more  strategic 
level, accurate  information  allows the  measurement  of the overall  impact  of 
interventions  on employment,  permitting  informed decisions to  be made by 
the  European Commission  and  Parliament  on the continuation  or  modifica­
tion of Structural Fund interventions. 

Since Structural Fund interventions combine  programmes of  measures  with 
a  variety  of  objectives,  there  is  a particular  need  to  evaluate  not  only  the 
overall  employment  effects  of  these  programmes,  but  also  the  impacts  of 

Evaluating  impacts  on  employment 

each  component  measure.  Moreover,  in  many  cases  it  is  necessary  to 
evaluate both the general impacts on employment  (e.g. how many new jobs 
have  been  created?)  and  impacts  related  to  more specific  objectives  (e.g. 
how many jobs are now concentrated in a particular region or held by young 
people? - regardless of whether the total number of jobs has increased). 

To fully appreciate the effectiveness of interventions it is of the utmost  impor­
tance that  net  impacts  are  estimated.  Consequently,  this  means  that  the 
creation or maintenance  of employment that would  have taken place in the 
absence  of  an  intervention  has to  be taken  into  account.  Similarly,  allo­
wances  must  be made for  employment  that  is displaced  elsewhere  in the 
programme area and for  individuals who are eligible for  assistance who are 
substituted for non-addressees. Finally, the wider repercussions of an inter­
vention on suppliers  of recipient firms and the increased purchasing  power 
of assisted individuals should be estimated when this is feasible. 

Continuous  interaction  between labour  markets and other types of  markets 
(i.e. goods, services  or financial  markets)  or  markets  outside the  defined 
geographical  area  also  has  to  be  taken  into  account.  For  example,  an 
increase in exports due to favourable economic  conditions overseas  would 
have a positive effect on employment creation. Another example could be a 
period  of  relatively  low  interest  rates  that  may  have  positive  effects  on 
investment  and  exports  with  consequent  benefits  for  employment.  These 
confounding factors should  be taken into account  if the effectiveness  of an 
intervention is to be accurately measured. 

Furthermore,  numerous  interventions  often  occur  at  a national  level. As a 
result, synergies may often be generated between Structural  Fund interven­
tions  and  national  initiatives. It  is therefore  useful  to  be able  to  identify  the 
existence of such synergies. 

Certain  interventions are specifically  focused on social groups and, in parti­
cular,  the  employability  of  individuals  in  employment  or  unemployed. 
Employability  is closely  linked to  the evolution  of the  skills of  individuals  to 
reflect  the  changing  needs  of  the  economy.  The  problem  of  maintaining 
employability  is particularly acute in former industrial regions where profound 
changes  in the  organisation  of  production  and corporate  strategies  have 
occurred  as a result  of  structural  change,  leading to the  obsolescence  of 
workers' skills. Box  18 presents several forms of skill obsolescence and cor­
responding training strategies for increasing the employability  of  individuals. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Box 18 - Different types of skill  obsolescence 

Form of 
obsolescence 

Related to 
the individual 

Related to 
the  conditions 
of the post 

Related to 
changes in 
the post 

Related to 
the firm 

Related to 
to the market 

Description 

Age, Illness, 
injury 

Insufficient 
use of skills 

Demands 
related to the 
type of job 
change 

Appearance 
of new compe­
titors undermi­
nes a business 

Employment 
in a given 
sector declines 

Response in terms 
of training 

Revision of 
skills already 
acquired 
or total 
re-quallfication 

Revision of 
skills already 
acquired 

Training for new 
requirements 
of the job 

Development 
of existing 
skills or total 
re-qualification 

Training for 
similar jobs in 
In a new sector 
or total 
re-quallfication 

In order to  be clear about the type of employment  effects  under discussion, 
reference will be made throughout this chapter either to job creation or main­
tenance effects  produced  by interventions  in non-human  resource  domains 
or to  employability  effects  produced  by  human  resource type  interventions 
(mainly  concerning  training).  The  essential  difference  between  these  two 
types  of  effect  is that  in the  first  instance  employment  effects  concern  the 
creation  or  maintenance  of  a  post  which  may  be  occupied  over  time  by  a 
number  of  individuals. In the second, employment  effects  concern  maintai­
ning  or  improving  the  employability  of  individuals  so  that  they  can  either 
remain in a post at a certain level of qualification (i.e. in the case of training of 
workers  in  employment),  or  have  a  better  chance  of  matching  the  skills 
requirements of  potential employers and therefore of finding a job (i.e. in the 
case  of  training  for  the  unemployed).  The  characteristics  of  these  different 
types of employment effect are summarised  in Box 19. 

Box  19 -  Breakdown  of different types  of employment  effects 

Employment  effects 

Job  creation  /  maintenance 
effects 
(Related  to a  post) 

Employment  created  or  maintained 
by  the  process  of  implementation 
of  SF interventions  or  as a  consequence 
of  interventions  where job  creation 
or  maintenance  is an  explicit 
or  implicit  objective 

Employability  e f f e c ts 

(Related to  an  individual) 

Employment  maintained  (existing 
employees)  or found  (unemployed 
persons)  as a consequence  of  SF 
interventions  targeted  specifically  at 
maintaining  or  improving  the  employability 
of  individuals  through  the  development 
of  skills  and  qualifications 

Evaluating impacts on employment 

■ Defining what needs to be measured 

How employment impacts are produced 

Many interventions are not explicitly targeted at the creation of employment 
effects; they are therefore unlikely to have a substantial direct effect on 
employment. Other interventions are targeted at the development of human 
resources, including training and research which produce only limited direct 
employment effects in the short term. Consequently, the most significant 
employment effects resulting from the use of the Structural Funds are often 
indirect and tend to arise in the medium to longer term. They can therefore 
be relatively difficult to isolate and measure. This situation is further compli­
cated by the fact that certain interventions may well create negative employ­
ment effects initially, before any positive longer term impacts occur. This is 
particularly the case with interventions targeted at increasing productivity 
and competitiveness. 

The ways in which structural interventions may influence employment 
patterns are listed in Box 20. While it may be argued that the key analytical 
distinction to be made in relation to employment effects is whether they 
result from changes on the supply­side or the demand­side of the economy, 
the types of effects listed serve as a more practical reference to ensure that 
no effects are overlooked. 

The employment effects presented in Box 20 are situated at different points 
along the chain of causality, it is important to understand the complexity of 
the causal mechanisms that link interventions with their eventual employ­
ment impacts. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Box 20 - Production of employment effects through structural interventions 

•  As  a direct  consequence  of the  implementation  of  an intervention (e.g. 
construction, administration and training jobs), 
•  As  a  direct  consequence  of  assistance  provided  i.e.  through  job 
creation  or  maintenance  in  companies  which  have  been  established  or 
which have grown, 
•  As a consequence of training  measures targeted  at the employability  of 
existing workers and the unemployed, 
•  As  a  consequence  of  interventions,  which  may  or  may  not  have  an 
explicit  and  immediate  employment  objective,  either  through  furnishing 
an intervention with goods and services (e.g. providing building materials, 
office  supplies,  etc.)  or  as  a  result  of  improved  efficiency  in the  local 
economy  (e.g. hauliers employed to ship goods, the export of which  have 
been  made  competitive  due to  decreased  transport  costs  resulting  from 
the use of new  infrastructure), 
•  In response to increased demand generated by any increase in income 
resulting  from  the  intervention  ("income multiplier  effects"-  e.g.  increase 
in  local  retail  employment  or  local  house  construction  to  meet  the 
increased demand from the newly employed). 

These mechanisms  are presented  in Box  21 and detailed  in the  resume that 
follows it. 

Box 21 - Structural Fund interventions and employment effect mechanisms 

Low complexity of causal mechanisms 

High complexity of causal mechanisms 

(a) 
Construction and 
implementation 

(b) 
Employment 
subsidies 

(c) 
Training 

(d) 
Support for 
productive 
activities 

H 
Infrastructure and 
increased 
competitiveness 

(a) Construction and implementation leading to job creation 
Almost  all  structural  interventions  will  have  employment  as  a side  effect  in 
terms  of  construction jobs  (e.g. workers  employed for the construction  of a 
motorway)  or  implementation  jobs  (e.g. teachers  recruited  for  training  pro­
grammes). These effects may range from the almost  insignificant (i.e. certain 
financial  support  measures), to the  major  part of effects  (i.e. construction  of 
physical  infrastructure).  In this  case the  link  between  the  interventions  and 
the employment effect is also fairly direct. 

(b) Employment  subsidies leading to job creation 
Employment  subsidies for the long-term  or young  unemployed  are a typical 

Evaluating  impacts  on  employment 

example of policies producing employment effects as an immediate result. It 
is quite straightforward  to  connect  the  recruitment  of  a new  worker  to  the 
subsidy given to the employer. 

(c) Employment effects as a consequence of the improvement of skills 

Training  policies  are an example of  measures that  can have a complex  link 
with employment  effects. The rationale for training  policies  is that there is a 
mismatch between the skills of the available labour force and those needed 
by sectors with good growth prospects. Training is then supposed to ensure 
the employability  of the unemployed  or of workers whose skills are likely to 
become outmoded. In the case of the training for existing workers, employ­
ment effects are relatively direct. Less directly, this may lead to an improve­
ment in the quality of the workforce causing both productivity and the supply 
of trained workers to increase, moderating pressure on wages and inducing 
firms to eventually increase their level of activity and employment. 

(d) Induced or facilitated employment effects through support for  productive 
activities 

This  category  encompasses  all  jobs  created  by  companies  or  other 
economic actors in response to the policy, through the creation or growth of 
productive activities supported directly (with investment grants or subsidies) 
or  indirectly  (with  services,  provision  of  sites  etc.).  It  is therefore  the  most 
important channel through which employment effects occur. However, along 
with  other  types  of  interventions  that  create  employment  effects  through 
complex  causal  mechanisms,  impacts  on  employment  are  difficult  to 
attribute and measure. This is partly due to the fact that as the impact  chain 
from intervention to employment effects becomes more complex, numerous 
factors, other than the support  scheme, will be considered  by an economic 
actor before taking a decision on employment. 

A significant  example to  consider  here is the employment  impact  of  invest­
ment  subsidies  to firms. The resulting  increase  in the  level  of  investment 
should  lead to  an increase  in the economic  activities  of the  firm  and  to  a 
consequent  increase  in  employment.  However,  if  demand  is  static  or 
declining, there might well be a loss of employment. This will happen because 
any  measure  directed  to  the  better  use of  capital  equipment  will tend  to 
increase the productivity  of fixed  capital, making  labour  less competitive. In 
conditions  of  static  demand  this  will  lead to  the  substitution  of  labour  by 
capital. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Support  for  productive  activities  also  has  indirect  effects  (income  multiplier 
and  supplier  effects)  with  consequent  employment  effects.  Further  steps 
have  to  be  taken  along  the  impact  chain  and  it  is thus  more  complex  to 
disentangle such effects. 

(e)  Employment  effects  as  a  consequence  of  increased  competitiveness 
arising from new  infrastructure 

It  is also  possible  that  employment  effects  might  arise from  the  increased 
competitiveness  of the assisted economy  owing to  material (e.g. roads, tele­
communications  etc.)  and  immaterial  (e.g. business  services,  research  and 
development  etc.)  infrastructure.  These  long-term  effects  are,  however, 
produced  by a complex and indirect  mechanism, situated close to the end of 
the impact chain, and are difficult to identify and connect to a specific interven­
tion. 

Not only do the Structural Funds produce employment  effects  in a variety of 
ways, but the types of employment  effect  produced  also vary. In addition  to 
the distinction  between job creation or maintenance  effects and employabi­
lity effects, two other important distinctions can be made. 

When carrying out  an evaluation  it is necessary to distinguish  between sus­
tainable  (or  durable)  employment  (i.e. jobs  resulting  from  an  intervention 
which will continue in the absence of public support) and employment  which 
are likely to  be of a transitory  nature (i.e. jobs which cease to exist when  the 
funding  stops).  Moreover,  while  employment  in the  implementation  of  an 
intervention  is usually temporary, experience  has shown that  only a  propor­
tion  of  so-called  permanent  jobs  can  be  expected  to  last,  even  in  the 
medium-term  (e.g. a significant  proportion  of jobs  created  in newly establi­
shed  enterprises  cannot  be expected  to  survive for  long). When  measuring 
employment  effects, it is important  to specify the  nature of the jobs  created 
or maintained and  not to give a false impression of the long-term  security of 
any jobs  which  are a consequence  of  the  intervention. Some jobs  are  only 
created for a short  duration  (temporary  employment),  and are often  related 
to  the  implementation  of  the  intervention.  If aggregated  figures  are  to  be 
produced  then,  rather  than  simply  counting  the  number  of  jobs  that  are 
created by an intervention, it is more accurate to use expected job-years as 
a unit of measurement. Time discounting  of expected job years can be used 
to estimate their present value. 

Evaluating  impacts  on  employment 

Furthermore, the  employment  effects  of  an  intervention  may  include  both 
part-time and full-time jobs. Once again this  distinction  needs to  be made. 
Consequently, two different types of figure are useful: first, full-time equiva­
lents (FTEs) which  is the  aggregation  of  part-time  employment  so that  it is 
expressed as the equivalent  number of full-time positions (e.g. one half-time 
and two  quarter-time jobs  together  being  considered  as the  equivalent  of 
one full-time job)  and, secondly, the simple number of jobs  generated (e.g. 
one half-time  and two  quarter-time jobs together  being  considered  as the 
equivalent  of  three jobs)  which  better  reflects  the  objective  of  combating 
unemployment. 

•  Clarifying employment objectives 

When evaluating  the employment  effects  of  interventions,  reference  needs 
to  be made to the objectives  of the  intervention  in question  and  how  they 
relate to employment.  In practice, neither the objectives of each component 
of  a  programme  nor  the  connections  between  different  levels  of  a 
programme are always articulated as clearly and explicitly as they might be. 
The  need  for  clarifying  the  multiple  levels  of  objectives  in  a  programme 
supported  by  the  Structural  Funds  is  all  the  more  necessary  when  the 
uncertain implications for employment of individual types of intervention are 
considered. 

For  example,  improvements  in  transport  infrastructure  serving  remote 
regions  might  lower  the  costs  for  exporters  from  such  regions  and  allow 
easier access to support services, but they might equally expose the regions 
concerned  to  even  stronger  competition  and  facilitate  the  seasonal  or 
permanent  emigration  of  the  labour  force.  Similarly,  assisted  training 
schemes should generally increase workforce qualification  levels, producti­
vity and the attractiveness of a particular location for inward investment, but 
in some circumstances they might lead to increased emigration of the newly 
qualified in search of better career prospects. 

If the nature of the employment effects expected from an intervention are to 
be assessed and appropriate techniques for their measurement applied, it is 
important  for the Commission to  work  closely with those partners  involved 
on the  national,  regional  and  local  level. Together  they  should  make  the 
specific  objectives  relating  to  employment  effects  at  all  levels  of  each 
programme as explicit and clear as possible. This task is of particular  impor­
tance during the ex ante phase of programming. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

However, it is commonly found, especially in the cases of programmes  which 
run  over  a  relatively  long  period,  that  absorption  problems  or  changing 
economic  situations  can  lead  to  a  need  for  refocusing  in order  to  better 
achieve their  desired  impact  on employment,  or for that  matter on any  other 
priority, and to refine objectives originally specified in individual  programmes. 
Objectives related to employment must therefore not only be clearly specified 
at the outset  of  programmes, they must also be open to  modification  in light 
of new detailed political priorities or changing economic  conditions. 

The construction  of an objectives tree, preferably  at the ex ante phase, can 
be used to clarify the employment  objectives  of a programme. By way of an 
example, a tree of employment  objectives has been constructed to  illustrate 
the  use of  this  tool  in the  context  of  interventions  to  promote  employment 
(see Box 22). 

Box 22 -  Example  of a  thematic 

ctives  tree for  the 

promotion  of  employmc 

abje 
int 

Global  objectives 

Specific  objectives 

Reduce 
disparities  in 
unemployment 
between 
the  regions 

Increased  employment 
potential  of  SMEs 

Increased  employment 
opportunities 

•  Promote  new  business  creation 
•  Promote  investment  by SMEs 
•  Promote  innovation  by SMEs 
•  Promote collaboration  between  SMEs 
and research  institutes 
•  Improve consultancy  services  for  SMEs 

•  Diversify  the  range of products  and  services 
of  regional  firms 
•  Increase  export opportunities  for 
regional  firms 
•  Attract  Inward  investing  enterprises 

Increased  employability 
of workers  already 
In  employment 

•  Adapt  existing  employees  skills  and 
qualifications 
•  Develop  professional training  activities 

Increased  integration  of 
special groups  into 
the  labour  market 

•  Retrain the  long-term  unemployed 
•  Improve  the  qualifications  of  young  people 
•  Improve training  opportunities  for  women 

Evaluating  impacts  on  employment 

In this  example, which  is in fact  a composite  of  different  types  of  interven­
tions not normally found within the context of a single programme, interven­
tions  are targeted  at  a  range  of  specific  objectives  which  themselves  are 
subordinate to four  intermediate objectives: increasing employment  oppor­
tunities,  increasing  the  employment  potential  of  SMEs  (interventions 
targeted  at the productive environment), integrating special groups  into the 
labour  market  (interventions  targeted  at training  for the  unemployed),  and 
increasing the  employability  of  existing  workers. The global  objective  is to 
reduce regional disparities in unemployment. 

•  Estimating net employment impacts of an intervention 

It is important that only those impacts that genuinely arise from an interven­
tion are attributed to it. To do this, the net employment impact of an interven­
tion should be estimated. 

A first step  in estimating  net employment  impacts  is to construct the coun­
terfactual situation  in order to estimate the proportion of gross  employment 
impacts  which  would  have  occurred  in the absence  of  an  intervention (i.e. 
deadweight). 

The  gross  employment  impact  of  an  intervention  is  defined  as the  total 
number  of  jobs  created  or  maintained  or  the  number  of  people  passing 
through training courses, counselling  sessions, etc. who  remain  in employ­
ment  or  find  a  job.  In  practice,  data  on  gross  employment  impacts  is 
collected  by monitoring  systems  and tends to be  used as the basis for  this 
figure. 

When  interventions  focus  on training  either  existing  workers  or the  unem­
ployed, the  deadweight  effect  is defined  as the  number  of  people  passing 
through a scheme who would  have remained in employment  or found a job 
without  receiving  the  training. The  counterfactual  situation  in this  case  is 
usually  estimated  by establishing  a control group. The profile of  addressee 
employees is important  in establishing the extent of this type of  deadweight 
effect.  For example, the training of  management  or other  highly  skilled  staff 
is more likely to occur without the support of interventions. Similarly, the size 
of the firm may be a determining factor, since larger firms tend to  undertake 
more training activities than SMEs. Deadweight  is subtracted from the gross 
impacts of an intervention. 

Displacement  can occur  in the case  of  area-based  interventions, when  the 
creation or maintenance of jobs takes place at the expense of jobs elsewhere 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

in  the  programme  area.  An  example  of  displacement  is  of  companies 
assisted  by an intervention  expanding  their  business and their workforce  by 
taking business from other companies  within the programme area. Displace­
ment should be subtracted from gross impacts when the effect is unintended. 

When interventions are targeted at particular groups of addressees (e.g. long 
term  unemployed, working  mothers, young  people, etc.), the substitution  of 
addressees  for  non-addressees  may  in  fact  explicitly  be  sought,  for 
example, with better trained addressees finding employment  at the expense 
of non-addressees on the labour market. Similarly, interventions which expli­
citly  aim to  create  or  maintain  employment  by  supporting  the  development 
of  a  sector  at  the  expense  of  another  can  be  considered  as  substitution. 
When substitution  is the  explicit  objective  of  interventions,  it should  not  be 
subtracted  from  gross  impacts.  If,  however, the  intervention  was  not  expli­
citly  aimed  at  creating  this  effect,  substitution  should  be  subtracted  from 
gross impacts. 

Example: A training scheme for business creation: from gross to net effects 
A training scheme for  business  creation  might  lead to the creation  of some 400  busi­
nesses of which only 250 businesses were still active two years later, with 50 of  these 
employing the owner  half time only. Assuming that  surviving  businesses created  350 
additional jobs (since the aim of the intervention was to create sustainable jobs, it was 
considered  that  two  years  was  an adequate  time  scale  to  judge  sustainability),  the 
gross number of jobs  at the end of this period would  amount to 300 (employees)  plus 
250 (business owners), a total of 550 gross jobs. However, because  100 jobs are half-
time,  the  number  of  gross jobs  created  amounts  to  525  FTEs  (200  + 50 /  2  +  250  + 
100/2 = 525). 

If,  however,  deadweight  is estimated  to  be 20%, and the displacement  of  businesses 
because  of  the  support  given  to  recipient  businesses  is estimated  at  30%, the  net 
effect  of  the  intervention  will  be  262.5  net  jobs  in terms  of  FTEs  (525  -  [525x0.3]  -
[525x0.2] = 262.5) 

The differences between these concepts as they relate to job creation or mainte­
nance effects or employability effects are summarised in Box 23. 

In addition to these different  effects  employment  may be generated  through 
income multiplier effects (i.e. employment which is created / maintained  indi­
rectly  by the  fact  that  the  intervention  results  in additional  income  for  local 
consumers) or supplier effects (i.e. employment which is created or maintai­
ned as a consequence  of  backward  linkages between addressees  and their 
suppliers) which tend to  be separately  identified. However, these effects  are 
virtually impossible to quantify for individual regional  interventions. 

Evaluating  impacts  on  employment 

Box 23 -  Comparison  of deadweight,  displacement  and 

substitution  effects 

Definition for job creation 
/ maintenance 

Jobs which would have been 
created or maintained in the 
absence of the Intervention 

Jobs created or maintained by a 
company in the area targeted at the 
expense of other companies 
In the programme area 

Deadweight 

Displacement 
(area-based 
interventions) 

Substitution 
(social group-based 
Interventions) 

Definition for 
employability 

Retrained individuals who would have 
remained in employment or found a job 
In the absence of the intervention 

Companies employing or recruiting 
retrained individuals which take 
business and reduce employment In 
other companies in the programme area 
which have not retrained their employees 
or recruited retrained people 

Retrained individuals who remain 
In existing posts instead being 
replaced by previously better qualified 
unemployed persons 
Retrained unemployed individuals 
belonging to specific target groups who 
are substituted for non-addressees 

The impact  of  an intervention, either  positive or  negative will not always  be 
directly  translated  into  an effect  on  unemployment  of  a similar  magnitude. 
This  is  because  any  changes  in  the  number  of  people  actively  seeking 
employment  need to be taken into account. 

One such  effect  is the  impact  on  activity  rates  that  is  induced  by  new  job 
opportunities,  the  so-called  attraction  effect  (or added  worker  effect).  For 
example, new part-time jobs may attract women or retired people who were 
not registered as unemployed before the intervention occurred. 

Another effect on the supply of labour is that of migration: new job opportuni­
ties may have an impact on both immigration (by attracting some  individuals 
to  move into the target area to obtain work) and emigration  (by reducing the 
number who would otherwise leave the target area in search of work). 

In this context, some effects on travel-to-work  patterns may be considered as 
migration effects and these may be even more significant than more permanent 
migration flows since greater numbers of people outside the target area may be 
encouraged  by new job opportunities  to simply  commute  into the area. The 
smaller the target area, the more likely it is that new jobs will go to  individuals 
who were not part of the local supply of labour before the intervention. 

Transversal evaluation of impacts on intervention priorities 

■ Examples of impacts by area of intervention 
In order to illustrate how interventions can lead to employment impacts, 
examples are presented below which correspond with the three broad areas 
of intervention of Structural Funds. 

• Basic Infrastructure 

The aim of interventions in this area is largely to relax constraints on economic 
development resulting from the absence or poor quality of basic infrastruc­
ture. Such interventions do not, therefore, have explicit employment objec­
tives. Job creation and maintenance effects are either of a temporary nature 
during implementation, or indirect due to increased economic activity. 

Example: T he upgrading of analogue telephone e x c h a n g es (Greece) 

The public s w i t c h ed t e l e c om network in Greece w as almost 1 0 0% analogue at the 

beginning of the 1990s. As a result, the network suffered from low quality and reliabi­

lity. One of the interventions designed to improve the network has been the progres­

sive replacement of existing analogue e x c h a n g es w i th new digital e x c h a n g e s. T he 

objective of this p r o g r a m me is to reach a level of 6 8 ­ 7 0% digitisation of e x c h a n g es by 

the year 2 0 0 0. As e x c h a n g es b e c o me d i g i t i s e d, f i r ms a nd h o u s e h o l ds are able to 

benefit g l o b a l ly f r om i m p r o v e m e n ts in t he n e t w o r k. T he m o d e r n i s ed n e t w o rk will 

p r o v i de f i r ms w i th t he o p p o r t u n i ty to t a ke a d v a n t a ge of t he n ew p o s s i b i l i t i es for 

business offered by t e l e m a t i c s. While m o st e m p l o y m e nt effects will be t e m p o r a r y, 

occurring during i m p l e m e n t a t i o n, over the longer term it is possible that permanent 

e m p l o y m e nt opportunities will arise as firms take advantage of the market possibilities 

that the improved infrastructure offers. 

Outputs ~^> 

Effects 

Telephone exchanges 
re­equipped 

Gross temporary jobs 
created or maintained 

Gross jobs created 
or maintained 

Reduction In 
network faults 

Firms potentially 
served by 
modernised 
exchanges 

Firms using telematics to 
expand into new markets 

Increased 
production, 
sales, etc. 

Priority 

Evaluating  impacts  on  employment 

•  Productive  environment 

For  some  regions  the  aim  of  Structural  Fund  interventions  is to  assist  the 

reconversion  or diversification  of the economy  in order to combat  structural 

unemployment.  In  rural  regions,  diversification  of  productive  structures  to 

reduce  dependence  on  agriculture  and  to  encourage  tourism,  a  highly 

labour-intensive  activity,  is an  essential  element  in  a strategy  of  economic 

development. 

Example: Developing tourism in the Republic of Ireland 
In Ireland, an Operational Programme in Tourism was implemented from 1990 with the 
generation  of employment  effects  (the creation  of an additional  25 000  jobs)  as its 
global  objective.  One of the  specific  objectives  of interventions  was to diversify the 
range of tourism  products. One of the  product themes  chosen as a suitable target of 
intervention was the development  of conference  travel to identified tourism  centres. 
During  the 1989 to  1993 period  twelve  new facilities  for conferences  were co-
financed. The construction of conference  facilities  produced  temporary  employment 
effects  and an eventual impact on the growth and seasonality of the activities of local 
businesses such as hotels, restaurants, conference services, etc. 

Outputs 

> 

Effects 

Gross temporary  jobs 
created or maintained 

Construction 
of  conference 
centres 

Organisation of 
conferences and 
events 

Increase in 
non-seasonal and 
bus mess  tourism 

1 

New  activity  generated 
for the  region's  businesses 

t 

Priority 

J\ 
,  > 
1/ 

Gross jobs 
created or 
maintained 

k 

, 

Transversal evaluation of impacts on intervention priorities 

• Human resources 

Interventions in the area of human resources focus on maintaining or 
improving the employability of individuals through enhanced skills and 
abilities, and on supporting Research and Technological Development. 

Example: Support for vocational training in t he North East of England 

The North East of England, o n ce a major industrial region of the UK, has lost over 200 

000 j o bs in traditional sectors (coal mining, shipbuilding, steel making and heavy e n g i­

neering) since 1970. At the s a me t i m e, high t e c h n o l o gy activities in the region have 

d e v e l o p ed in such s e c t o rs as pharmaceuticals, biotechnology, information t e c h n o­

logy and electronics. One of the factors contributing t o w a r ds the emergence of new 

specialisations is the strong links b e t w e en industry and the region's universities and 

o t h er higher e d u c a t i on i n s t i t u t i o n s. S o me of t he m e a s u r es in f a v o ur of industrial 

reconversion of the North East (CSF 1989­91 and 1992­93) are aimed at vocational 

training. These measures include the provision of facilities for training in industrial 

sectors with a high g r o w th potential and especially in technologies having an indige­

nous base in the region. 

Outputs 

> 

Effects 

C> 

Gross temporary jobs 
created or maintained 

Gross jobs created or 
maintained 

/ 

Training projects 
prepared in 
partnership 
with local industry 

Unemployed trainees 
J being offered employment 

during the course 

Trainees successfully 
completing the course 

^ Unemployed trainees 
finding employment 

λ 

* Recipients promoted 

. Employees remaining 

in employment 

Priority 

■ Defining a nd selecting indicators 

Volume 2 of the MEANS Collection is devoted to the subject of indicators for 
monitoring and evaluating Structural Fund interventions. This Volume 
includes an information sheet on indicators for the priority of employment. 

Whilst evaluations are not limited to an analysis of indicators alone, they are 
greatly facilitated by the existence of relevant indicators. The subject of indi­

Evaluating  impacts  on  employment 

cators  and  their  quality  needs  to  be considered  at two  different  levels  in 
relation to employment effects: 
•  Since the promotion of employment is one of the general objectives of the 
Structural Funds, the ongoing development of a system of indicators should 
result in a strengthening of information on the relative employment effects of 
the different components of a programme. 

•  The issue of  indicators  also arises  in relation to more detailed  analyses of 
effects. Any evaluation which focuses on the subject  of employment  effects 
requires  ad-hoc  employment  indicators  to  be  defined  and  selected  in 
relation to the specific evaluation questions being asked. 

In order to evaluate the employment effects of the Structural Funds, it is conse­
quently useful, first, to ensure that some basic and reliable indicators of employ­
ment are included in a programme's system of indicators and secondly, in the 
context of any particular evaluation exercise, to define those central indicators 
which respond to the specific evaluation questions being posed. 

When  measuring  employment  effects,  account  should  be  taken  of  the 
temporary  jobs  needed to  implement  the  interventions  (e.g. jobs  related to 
the  building  of  infrastructure, advisory  services for  businesses  or training). 

Whilst  output  indicators  do  not  focus  directly  on  employment,  they  are 
usually  essential for  an analysis  of the  production  of  employment  effects. 
This  is  particularly  the  case  when  interventions  result  in the  provision  of 
services such as training, information or advice which, due to their intangible 
nature, can be difficult to measure. 

Effort  should be concentrated  on the establishment  of those indicators that 
provide  information  on the  objectives  which  are most  significant  as far  as 
employment  is concerned. These indicators  refer for the most part to  gross 
direct effects. 

The objectives  of  employment-related  structural  interventions  are  diverse 
and the  definition  of  systems  of  indicators  must  incorporate  this  diversity. 
There is no one universal indicator  of employment  effects  which is  relevant 
regardless of the type of evaluation being carried out. For example, if the aim 
of  an intervention  is to  promote the  return to work  of those who are unem­
ployed in an urban area that is being redeveloped, the number employed  in 
this urban area will not in itself  be sufficient as an indicator. Since it does not 
enable the proportion of jobs occupied  by residents of the area to be deter­
mined, the effectiveness of the intervention cannot be judged. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Finally, selected indicators must be precisely defined. For example, the defi­
nition should  state whether a job created or maintained  includes all types of 
job,  or  are  part-time  and  full-time  jobs  distinguished?  How  is  a  job  is 
qualified as sustainable? Is it a job that is expected to remain in existence for 
more  than  two  years,  more  than  five  years,  etc.?  How  are  specific  social 
groups  targeted  by  interventions  defined?  For  example,  the  information 
"30%  of the jobs created by the project  benefited  marginalised people" is of 
little use if the concept of marginalisation  is not clearly defined. 

•  Indicators for a more in-depth analysis of employment  effects 

Apart from the central employment  indicators  in a system  of  indicators,  it is 
often relevant during an evaluation to use ad hoc indicators in order to  make 
a  more  in-depth  analysis,  particularly  of  impacts. Any  such  use of  supple­
mentary  indicators  needs to  be  consistent  with  the  questions  that  the eva­
luation is required to answer, the  particularities  of interventions and the cha­
racteristics  of  the  region  being  evaluated. Supplementary  indicators  might 
therefore focus on: 

•  the proportion  of jobs  created  or maintained  which are unskilled / skilled  / 
highly skilled; 

•  the  remuneration  of jobs  created  or  maintained  (e.g. the effect  will  not  be 
the same if 10 jobs remunerated  at the legal minimum rate or 10 higher  level 
posts are created); 

•  the personal earnings of addressees; 

•  the  profile  of  the  addressees  (even  when  the  objectives  do  not  refer  to 
target  groups,  it  may  be  interesting  to  know  who  is  benefiting  from  job 
creation). An assessment  of compliance with the principle of equal opportu­
nities is relevant at this level; 

•  the career paths of addressees. In addition to knowing that they have been 
re-employed,  it  may  be  interesting  to  know,  for  example,  how  long  it  took 
them to get a job, how appropriate the job is in relation to their qualifications, 
or even what  effect the  intervention  had on their approach to seeking  work; 

•  the type  of the jobs  created  or  maintained. Making  a distinction  between 
part-time  and  full-time  jobs  is  one  way  of  making  inferences  about  the 
quality of jobs created or maintained. However, such inferences have to take 
into  account  the  fact  that  some  addressees  may  prefer  to  work  part-time 
rather than full-time.  Differences  in national  employment  and social  security 
legislation are also key determinants  of the quality of  part-time and full-time 
employment.  It  may  also  be  useful to  look  at the  proportion  of  addressees 

Evaluating  impacts  on  employment 

working  for  temporary  employment  agencies, the  rate  of  unionisation  in 
firms where jobs are created and make comparisons  of wages in respect to 
sectoral / industrial and regional averages. 

Because numerous measures co-financed by the Structural Funds have only 
a long-term effect on employment, apart from the temporary jobs which are 
needed  to  implement  them  in the first  place,  results  and  intermediate  or 
specific impact indicators tend not to reflect employment effects. This is par­
ticularly the case of  interventions  in the areas of  infrastructure and  produc­
tive environment.  For these types  of  intervention  the  more immediate non-
employment  impacts can be converted  into  predicted  employment  effects. 
For example, by dividing the turnover resulting from the intervention, by the 
average additional turnover  needed to create a full-time job, it is possible to 
estimate the maximum level of jobs indirectly created. This method is useful 
in ex ante evaluation to quantify the expected effects of interventions. It may 
also  be  useful  in  intermediate  evaluation  to  assess  indirect  long-term 
employment effects from established intermediate impacts. The conversion 
coefficients  used in such an exercise are determined  in relation to  previous 
experience,  research  or  econometric  models. Whenever  such  coefficients 
are used, the basis of their definition should be clearly indicated. 

When interventions  in the area of  human resources focus on maintaining or 
improving the employability of individuals, employment impacts occur  much 
earlier in the causal chain. In this case the choice of  indicators depends on 
the  definition  of  employability.  An  operational  approach  is to  associate 
employability  with the maintenance of employment  of individuals already in 
work or the recruitment of individuals out of work. 

When  studying  the  maintenance  of  employment  of  existing  workers,  it  is 
necessary  not only to take  into  account  the fact  that  employment  may  be 
maintained  within  a firm  or  sector  (in the case  of  an  industry  level training 
programme), but also that  it may  be maintained  in a wider  inter-firm or inter-
sector perspective (i.e. employment maintained at a comparable level of quali­
fication outside the firm or industry under consideration). The wider the pers­
pective  of  employment  maintenance, the  more problematical  its  evaluation 
since the development  of an individual's situation  is more difficult  to  monitor 
outside  a firm or area of  activity.  Furthermore, the  links between the level of 
qualifications, jobs and salaries can vary greatly, making comparison  difficult. 

The in-depth  analysis  of  employment  impacts  also  requires  an analysis  of 
confounding  factors,  i.e. those factors  over  which  the  programme  has  no 
control  and  which  have  an  influence  -  whether  positive  or  negative  -  on 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

employment  or unemployment. The main effects of confounding factors can 
be identified by means of indicators. 

Indicators  of  adverse  effects  on  employment  may  also  be established (e.g. 
number  of  redundancies  in the  agricultural  sector  due  to  rationalisation  of 
farming  enterprises).  These  serve  as  warnings,  or  to  enhance  knowledge 
about the harmful effects on employment  of the measures for which there is 
no stated employment  objective. 

Indicators  may  also  be  used to  make  a judgement  in terms  of  efficiency,  in 
which  case  they  relate  result  or  impact  indicators  to  the  resources  used, 
producing  a cost /  effectiveness  ratio which  may  be compared  to  reference 
data.  Examples  of  this  type  of  indicator  include  the  average  cost  per  job 
created or maintained. 

•  Precautions to take when using and interpreting indicators of 
employment  effects 

The  accurate  evaluation  of  employment  effects  requires  that  the  time 
needed for the effect to be produced should be taken into account, so as to 
avoid producing invalid information which might be based on the under-esti-
mation  of  expected  effects  (i.e. not  taking  full  account  of  effects  which  will 
subsequently  occur) and / or their over-estimation  (i.e. not taking account of 
the limited duration of temporary  effects). 

In the case of intermediate evaluations, employment  indicators are meaning­
ful above  all for  counting  the  number  of temporary  jobs  created  or maintai­
ned  by  the  implementation.  Impact  indicators  must  take  account  of  the 
delay  that  occurs  before  employment  impacts  are  produced, which  varies 
depending on the type of intervention being studied. 

A single indicator may be used to measure several levels of effects  if used at 
different  points in time. In the case of an intervention aimed at attracting new 
businesses to the area under redevelopment, the  number  of jobs created or 
maintained  might  include: an  indicator  of the jobs  required for the  develop­
ment being undertaken to attract  businesses to the area, an indicator of the 
result of the intervention  if it measures the number of jobs  provided by firms 
when they  set  up, and  an  impact  indicator  if  it  counts  the  number  of  direct 
and indirect jobs created in the area. 

Finally, when comparing  information  provided by an indicator, it is important 
to ensure that the comparison  is relevant. For example, a comparison on the 

Evaluating impacts on employment 

basis of "cost per job created" is only relevant if the context of the reference 
situation is the same. 

By way of an example, a system of indicators that could be used to track the 
production of employment impacts generated by a training programme for 
the unemployed is presented in Box 24. 

Box 24 - Example of a system of indicators for a return-to-work 

programme 

Programme Indicators Context Indicators 

(Related to the Intervention and its effects) (Related to the assisted area) 

Output Indicators 

Number of 
addressees 

Result indicators 

Rate of Immediate 
transition 

Impact indicators 

Gross placement rate 

Net placement rate 

Quality of jobs 

Number of long term unemployed 
participating In the programme 
(of which women) 

Number of addressees who are either 
hired or In a better social situation at the 
end of their participation in the 
programme (of which women) 

Number of addressees hired after 
12 months (of which women) 

Number of long-term 
unemployed in the region 

Gross rate less estimated deadweight 
(trainees who would have been hired 
without assistance) less unacceptable 
substitution (trainees whose hiring has led 
another worker to long-term unemployment) 

% of hired addressees (of which women) 
who's job Is full-time and permanent after 
12 months 

% of jobs In the region that 
are full-time and permanent 

Source: Volume 2 of the MEANS Collection 

■ Sources of useful information 

Numerous series of statistics have been established at the European, 
national and regional levels. At the Community level, a Labour Force Survey 
is conducted annually throughout the Union. The current series of surveys 
was launched in 1983 and a survey is carried out annually. The structure of 
the survey is such that the data presented is disaggregated by gender, 
allowing the equal opportunities dimension of employment effects to be 
integrated into an evaluation. Many other data are also available through this 
survey such as: educational levels, number of hours worked per work, the 
presence of young children, etc. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Category 

Disaggregation by : 

Unemployment 

Jobs 

• Gender 
•  Age group 

• Gender 
•  Professional status 
•  Field of activity 

Groups of job seekers 

• Gender 
•  Duration of unemployment 
•  Mode of job-seeking 

The  Regio  regional  database  covers  unemployment  and  labour  force 
surveys, and the results are disaggregated  by region. Statistics are available 
on  total  employment,  salaried  employment,  active  population,  long-term 
unemployment  and  remuneration  of  employees.  Data  are  available  from 
series which go back to  1970, 1975 or  1983. The reporting  period is annual, 
with  the  exception  of  certain  unemployment  statistics  which  are  monthly3. 
The statistical observatories  MISEP (Mutual Information System on  Employ­
ment  Policies)  and  SYSDEM  (System  of  Documentation  on  Employment) 
provide information on the different  national employment situations. 

Among the drawbacks of using statistical surveys is the problem of  non-cor­
respondence  between  place  of  work  and  place  of  residence  which  may 
create  particular  difficulties  in  the  interpretation  of  statistics  when  an 
analysis is focused on unemployment.  For example, due to commuting, the 
jobs  recorded  in a  region  are  not  all  occupied  by  residents  of  that  region. 
Statistics  may  therefore  indicate  a falling  trend  in  unemployment  levels, 
without  the  majority of the addressees  of that  improvement  being  residents 
of the supported area targeted by the intervention. This phenomenon  is par­
ticularly important when the assisted area is small (e.g. urban areas in indus­
trial  decline  that  are eligible  for  support  to  promote  reconversion)  or  when 
border regions are concerned. 

Permanent  statistical  surveys  on  employment  are  also  available  from 
national statistics institutes, labour and employment  administrations  (statis­
tics on unemployment  databases on human resources, etc.). However, a dif­
ficulty with this sort of data is the time  delay  between the collection  of data, 
its processing and eventual publication. 

Databases  such  as those  compiled  for  the  LEADER  Community  Initiative 
have  proved  to  be  equally  useful. The  LEADER  Observatory  gathers  data 

3. Labour force survey and data from the Regio data base can be obtained from Eurostat. 

Evaluating impacts on employment 

concerning the objectives of projects (including employment) and elements 
concerning counterfactual situations. 

There has been a surge of interest over the last few years in evaluating 
employment effects and evaluations in this field are rapidly evolving. For this 
reason an inventory of existing research reports and previous evaluations 
represents a valuable stock of knowledge on which evaluations can be built. 

The employment data provided by the evaluations of a programming cycle 
provide an extremely useful basis for estimations of effects of the next cycle. 
Estimations of employment effects which feature in programmes and ex ante 
evaluations should also be used in intermediate and ex post evaluations. 

Information produced by a monitoring system includes data that are 
necessary to evaluate employment effects. When the monitoring system 
comprises an entire set of indicators on resources and on physical and 
financial outputs, the extent to which the operational objectives of interven­
tions are being progressively attained and the associated employment of 
human resources during the implementation of interventions can be shown. 
However, it is important to bear in mind that monitoring can only, at the most, 
focus on addressees and would­be addressees (i.e. rejected applicants). 
This means that an analysis which focuses uniquely on monitoring data ­ for 
example, the construction of the counterfactual situation ­ will be biased. 
Furthermore, data gathering which is restricted to addressees and would­be 
addressees will be the subject of respondent bias. Nonetheless, monitoring 
data can be a useful source of information, since monitoring systems of 
most types of intervention include some information on employment. 
Examples of indicators that can be directly informed from the monitoring 
system include the employment generated by businesses attracted into the 
target area and the number of participants in a training programme who have 
obtained a qualification. 

■ Appropriate tools for analysing employment effects 
An assessment of the impact of interventions on employment can be underta­
ken using the "step­by­step" approach presented in Part IV of this Volume. For 
an in­depth evaluation of employment effects, a wide variety of evaluation 
tools can be employed for the analysis of employment impacts. While these 
tools are extensively dealt with in Volume 3 of the MEANS Collection, a 
number of examples, drawn from across the spectrum of evaluation expe­
rience in this theme, are presented in this section. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Tools for  in-depth  analysis of  employment  impacts  can  be divided  into  four 
categories: 

•  descriptive tools that allow for the treatment of both qualitative and quanti­
tative  information  (e.g. full case-studies,  including  longitudinal  studies,  and 
surveys), 

•  comparative  tools  based  on  observation  (either  between  similar  cases  or 
between the subject case and a randomised control case), 

•  analytical tools that  focus  only  on the situation  being  analysed  (e.g. shift-
share analysis), 

•  models  which  are  built  on the  basis  of  economic  analysis  of  other  expe­
riences  and  simulations  of  varying  degrees  of  complexity  (e.g.  multipliers 
and  coefficients,  input-output  and  econometric  models  which  combine  a 
number of  coefficients). 

Tools for quantitative analysis, such as shift-share analysis, do not only need 
to  be  used  with  skill,  their  limitations  also  need  to  be  acknowledged.  The 
measurement  of  their  employment  effects  needs  to  be  interpreted  in  a 
manner which takes account of qualitative factors. 

Both the  level and  scale of the  intervention  being  evaluated  have a  number 
of implications for the type of tool  most  likely to be appropriate for the mea­
surement  of  employment  effects. The  investment  of  resources  needed  to 
produce dependable  simulations  means that full econometric  models, such 
as  the  HERMIN  macroeconomic  model, are  generally  available  for  only 
higher-level, larger-scale  interventions  which operate at the national  level or 
across  very  large  regions. Accordingly,  such  approaches  are more  likely  to 
be  of  most  direct  use  in  the  evaluation  of  the  effects  of  programmes 
designed  to  promote the development  and structural  adjustment  of  lagging 
regions,  or  in the  evaluation  of  the  overall  effect  of  the  variety  of  activities 
typically funded by the Structural Fund on a national level. 

On the  other  hand, simple  coefficients  are rarely  robust  enough  to  transfer 
between  situations  which  include  a  large  number  of  variables.  Ideally  their 
use  should  be  confined  to  very  similar  interventions  with  relatively  direct 
employment  effects  (e.g. those  which  are  targeted  on  similar  groups  in 
similar and smaller regions undergoing industrial reconversion or rural zones 
in the process of development and structural adjustment). 

Evaluating  impacts  on  employment 

The same  limitations  apply  to  comparative  analysis tools, since the  more 
complex  a situation  becomes, the more difficult  it is to find other  situations 
against  which  comparisons  can  be made. They  are, however,  particularly 
useful for establishing the policy-off  scenario of small-scale human resource 
interventions. 

Job creation or maintenance effects lend themselves to analysis by a variety 
of  top-down  approaches  (i.e. relatively sophisticated  statistical  techniques, 
modelling tools, etc.) and bottom-up  approaches  (i.e. surveys, case studies 
etc.). As for  interventions targeted at employability  effects, these are relati­
vely small and should therefore be analysed using the bottom-up approach. 

•  Surveys and case studies: Evaluating Training and Enterprise Council 
(TEC) services (United Kingdom) 

This  evaluation  sought  to  assess  the  overall  cost-effectiveness  of  a 
programme  of  business  counselling,  skills  training  and  information  and 
advice for  existing  small  and  medium  sized  enterprises  (SMEs). Additional 
employment was only one of a number of objectives of the programme and 
the evaluation also took  increased output, improved productivity,  enhanced 
capital  expenditure  and  profitability  into  account.  With  regard  to  employ­
ment  effects, the objectives  of the evaluation  were  primarily  to  assess  dis­
placement and to explore the possibility of any crowding out effects. 

The evaluation used: 
•  a  postal  questionnaire  sent  to  the  management  of  75  English  TECs 
(concerning their activities, targeting, etc.); 
•  personal  interviews with 20 TECs to provide qualitative information on the 
inputs  to  their  activities  (e.g. strategy  developed,  benefits  resulting  from 
different service approaches, etc.); 
•  six  case  studies  mainly  focused  on  customer  benefits  and  on  the 
companies' own views on the impact of the services on jobs and output. 

The  questionnaire  was  designed  to  distinguish  between  gross  and  net 
benefits.  Following  the  conventional  path  to  identify  the  net  impacts  on 
employment, the study attempted to assess (gross) effects, deadweight and 
displacement  effects.  All  of  the  results  of  the  case  studies  on  the  local 
economy were aggregated to the national level and net benefits on the local 
level were carefully distinguished from the net impact at national level. To do 
this, it was necessary to take into account  national displacement  as well as 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

macro-economic  crowding-out effects since gains in one TEC area may lead 
to  displacement  in another  area, and  if all areas  show  increased  economic 
activities, benefits are likely to be crowded out (e.g. by higher inflation rates). 

•  Comparative analysis: Measuring the effects of Swedish  employment 
training on income and employment  (Sweden)  1995 

Two  experimental  groups  consisting  of  4,000 trainees  who  completed  their 
training  in  1992  and  6,000  who  completed  it  in  1994, were  included  in  the 
survey.  A  very  high  proportion  of  these  were  individually  matched  with  a 
control  group  which  received  no training  at the  same time  as the  subjects. 
Some  policy  changes  had  taken  place  between  the  training  of  the  two 
cohorts, so a detailed  study  of these  particular  changes was  also  possible. 

The analysts used a fairly sophisticated control function (including values for 
background  variables  and  error)  to  trace  the  relationship  between  training 
and  income. The findings  obtained  showed  significantly  different  results for 
the  two  years, with  the  1994  trainees  having  significantly  higher  earnings 
than those who had not received training, but the 1992 trainees  experiencing 
no  significant  advantage.  For  those  who  ended  up working  in the  vocation 
for which they trained, earnings were higher than for the control group. 

In order to  measure employment  effects, regression analysis was used. The 
findings reported showed that while the control group had found it increasin­
gly  difficult  to  find  work  during  the  period  when  the  subject  group  was 
training, those who had subsequently  been trained found  it no more  difficult 
to  obtain  work.  Thus,  even  if their  position  had  not  improved  in  absolute 
terms,  it was  relatively  better  than  it would  otherwise  have otherwise  been. 

•  Statistical analysis: An ex post evaluation of the integrated action 
programme for Limburg using shift-share analysis (Belgium) 

The  goal  of  the  programme  being  analysed  was  to  support  job  creation  in 
new  activities  to  compensate  for  the  decline  of  employment  in the  mining 
industry. The evaluators tracked  employment  change during the  1980s  and 
early  1990s  on  the  basis  of  sectoral  and  other  characteristics  for  both 
Limburg  and  Belgium  as  a  whole.  Shift-share  analysis  was  then  used  to 
estimate the  number  of jobs created  by the  programme  in comparison  with 
what  would  have  happened  in its absence, on the assumption  that the  rate 
of decline in the traditional sectors was equal to the Belgian average. 

Evaluating  impacts  on  employment 

The  relatively  positive  experience  of  the  Limburg  region  (high  levels  of 
economic  growth,  new  firm  creation  and  employment)  was  attributed 
primarily to export  growth  in one or two  sectors and exceptional  growth  in 
the  number  of  small  enterprises.  Re-employment  patterns  amongst  ex-
miners  were  put  under  particular  scrutiny  in  light  of  the  objectives  of  the 
intervention, as were the  investment  instruments  chosen. On the  basis  of 
this analysis, specific  proposals  were made for a modification  of these ins­
truments to increase employment effects in subsequent  interventions. 

•  Multipliers and coefficients: Evaluating labour support schemes 
using standardised coefficients (France) 

This approach is used for the ex-post  and on-going  evaluation of effects  on 
employment  and unemployment  of  labour support  schemes financed  by the 
French Government. It takes into account both deadweight and attraction effects. 

For  each  single  measure  a coefficient  of  deadweight  and  a coefficient  of 
attraction  are calculated, based on the evidence of past evaluations and on 
intuitive adjustments. These coefficients are then applied to the variations of 
the stock  of  addressees  of  each  measure,  over  a given  period  of  time,  in 
order  to  obtain  the  effects  on  employment  and  unemployment  of  each 
measure. The effects on employment correspond to the difference  between 
the variation  in the stock  of addressees and the quota of these  addressees 
(obtained  by  applying  the  coefficient  of  substitution)  who  would  have 
obtained a job in any event, or who substitute for another worker. 

Calculations  of effects  on unemployment  also take  into account  the coeffi­
cient of attraction and the absolute value of the reduction  in  unemployment 
resulting from the  difference  between  employment  effect  and the quota of 
new addressees attracted to work by the measure. 

This  macroeconomic  approach  is especially  valuable  since  it  allows  the 
effects  of  cyclical  changes to  be expressed  as changes  in the  coefficients 
over time.  It is globally  useful as it gives a complete view  of existing  labour 
support  schemes  and of  the  net  effects  of  each  one on  employment  and 
unemployment.  However, it has a limited time horizon and does not  provide 
much information about job tenure and, in general, about the quality of jobs. 
The method  is easy to apply once the coefficients  have been obtained, but 
establishing the coefficients  in the first  place demands a high level of  infor­
mation input. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

•  Input-output analysis: Evaluating an objective 1 Operational 
Programme in Italy 

The tool  used  for  this  exercise  is a  Social  Accounting  Matrix  (SAM),  which 
integrates  Keynesian  income  multiplier  mechanisms  into  a standard  input-
output  model. In a SAM the various  interdependent  links of an economy  are 
quantified and simulated. The impact mechanism relies on global multipliers, 
obtained  from  a wider  matrix  system, which  add  additional  rounds  to  the 
input-output  mechanism. The  global  multipliers  are  able to  capture  effects 
arising  not  only  from  the  relations  linking  the  productive  sectors,  but  also 
from other relations linking economic actors. 

The  increase  in factor  revenues  (value-added)  generated  by  input-output 
mechanisms  is translated into an increase in income. The level of consump­
tion  then  rises  (according  to  the  average  consumption  rate),  creating  an 
increase  in  final  demand  which  corresponds  to  a  new  impulse  to  the 
economy and to a new activation of the process. Effects on employment  are 
calculated  on the  basis  of  a  hypothesis  on the  evolution  of  productivity  by 
sector,  which  is  related  to  the  changes  in  value  added  obtained  from  the 
input-output  application. 

The reference model for this exercise is a two-region (Campania and the rest 
of  Italy)  SAM. The  regional  SAM  tables  are  obtained  from  the  1988  input-
output  table  for  Italy, which  is split  using  the  results  of  two  surveys  on  the 
economy  of  Southern  Italy:  one  on  the  purchase  of  intermediate  goods  by 
firms  from  this  area  and  the  other  on  income  and  consumption  of  house­
holds. The SAM  used  has a low  level of disaggregation: two  factors  (labour 
and  capital)  and two  types  of  actor  (firms  and  households).  With  regard  to 
sector  disaggregation,  two  versions  are  provided:  one  with  four  sectors 
(agriculture,  industry,  building  industry  and  services)  and  another  with 
sixteen  (with  industry  and  services  split  into  sub-sectors).  The  model  is 
applied  in two  steps,  each  with  different  procedures,  in order to  separately 
capture demand-side and supply-side  effects. 

•  Macroeconomic  modelling: The HERMIN model in Ireland 

The  macro-economic  model  HERMIN  has  been  developed  by  national 
teams  in four  objective  1 countries:  Ireland, where it was initially developed, 
Portugal,  Greece  and  Spain. This  project  is aimed  at  creating  a  system  of 
harmonised  econometric  models  for  the  evaluation  of  the  medium-term 
economic  performance of the European economies. 

Evaluating impacts on employment 

As econometric models go, the Irish version of HERMIN is relatively small 
with 50 equations, about 150 identities and 60 exogenous variables. The 
standard version of HERMIN has four sectors: tradable (manufacturing 
industry), non­tradable (market services, building and utilities), agriculture 
and public. In the two main productive sectors (tradable and non tradable), 
simple production functions are used, while the agricultural sector has not 
been modelled because of the exogenous factors related to the CAP, which 
influence aggregate farming behaviour. The public sector is policy­driven, 
with expenditure and tax rates set as policy instruments, subject to budget 
constraints and borrowing sustainability. 

The consumption function of the model (based on Keynesian multiplier 
mechanisms) does not play an exclusive role in driving this model as it does 
in demand­side focused models. In this case the supply­side features of the 
industrial sector are also influential. From the production function, factor 
demands are derived in both productive sectors, on the basis of cost minimi­
sation subject to given levels of output. Levels of employment and invest­
ment are then dependent on the level of output, which is demand­driven in 
both sectors. In the tradable sector output is driven by world demand and 
competitiveness. In the non­tradable sector, on the other hand, actual 
output is driven by a weighted measure of final demand. 

The supply of labour is modelled in order to take into account migration 
flows, which are fairly significant in Ireland. The work­force participation rate 
is influenced by migration, unemployment (the discouraged worker effect) 
and the replacement ratio (i.e. the fraction of average earnings replaced by 
social welfare transfers). 

■ Summary 

The creation of employment effects is an objective in all areas of interven­
tion, and the capacity to create or maintain employment is an important 
factor in selecting projects for support. A second type of employment effect 
is sought by interventions in the area of human resources, primarily in 
training. In this case it is linked to the maintenance or improvement of the 
employability of individuals either in work or unemployed. 

In addition to the primary difference of job creation or maintenance effects 
and employability effects, employment effects vary widely. They occur at 
different points of the causal chain, they can be temporary or of a more long­

Transversal evaluation of impacts on intervention priorities 

term nature, they can be sustainable or transitory. For these reasons the 
aggregation of different types of employment effects is often not possible. 

When estimating net employment impacts, nature of deadweight, displace­
ment and substitution effects vary according to whether job creation or 
maintenance effects or employability effects are being evaluated. 

A great deal of experience has been accumulated in this area of evaluation 
and a wide range of bottom­up and top­down tools have been successfully 
used to evaluate employment effects. 

■ For m o re information see... 

Bradley J., Whelan K., Wright J. (1993), Stabilization and Growth in the EC 
Periphery, Aldershot: Avebury. 

Department of the Environment. (1988), An Evaluation of the Urban Develop­
ment Grant Programme, London: HMSO. 

Gautié J. et al. (1994), Les subventions à l'emploi ­ Analyse et expériences 
européennes, Cahier travail et emploi, La Documentation Française. 
Issaev B., Njikamp P., Rietveld P., Snickars F. (1982), Multiregional Economie 
Modelling: Practice and Prospects, North Holland. 

King J. (1990), Regional Selective Assistance 1985­88: An Evaluation by the 
Department of Trade and Industry, the Industry Department for Scotland and 
the Welsh Office Industry Department, London: HMSO. 
Manski CF., Garfinkel I. (eds.) (1992), Evaluating Welfare and Social Pro­
grammes, Harvard University Press. 

Njikamp P., Rietveld P., Snickars F (1986), Handbook of Regional and Urban 
Economics, North Holland. 

OECD. (1991 ), Evaluating Labour Market and Social Programmes ­ The state 
of a complex art, Paris: OECD. 
OECD. (1998), Employment Outlook, Paris: OECD. 

Schmid G., O'Reilly C, Schöman K. (eds) (1996), International Handbook of 
Labour Market Economics, Cheltenham: Edward Elgar. 
Storey D.J. (1990), "Evaluation of Policies and Measures to Create Local 
Employment", Urban Studies, vol.27, n°5. 

Evaluating the impact of structural interventions on equal 
opportunities 

■ Why do impacts on equal opportunities need to be evaluated? 

• The Community context 

The European Council has declared on several occasions (in Essen, Cannes 
and Madrid) that the promotion of the principle of equal opportunities 
between men and women is, along with the struggle against unemployment, 
a fundamentally important task facing the European Union and Member 
States. Glaring inequalities continue to exist between men and women in all 
areas. As far as the participation of women in the job market is concerned, 
their level of activity is lower than that of men in all the Member States of the 
European Union. In 1996, the employment of women over the age of 15 
varied between 34,6% and 58,7% (Eurostat). The wage gap remains signifi­
cant in all countries. Furthermore, women are generally more affected by 
unemployment than men in all age groups, particularly as regards long­term 
unemployment. Finally, most of the actions of the structural policies have dif­
ferentiated effects, reproducing discrimination between men and women. 

Efforts to promote equal opportunities at European level are not intended 
merely to introduce specific measures in favour of women, but to add a 
dimension to all actions and structural policies. This concept is known as 
"mainstreaming" (see Box 2). Mainstreaming is intended to Influence the 
strategies of Structural Funds policies so that they incorporate the 
dimension of equal opportunities" between men and women by systemati­
cally examining their respective situations. Policies must therefore be 
defined by taking into account the diversity of needs, including the needs of 
the various groups of men and women in society. 

The "Fourth Action Programme on Equal Opportunities for Women and Men 
(1996­2000)" is aimed at integrating equal opportunities into the preparation, 
implementation and monitoring of all policies and actions of the European 
Union and Member States, with due respect to their respective competencies. 

With regard to the Structural Funds, the most noteworthy event was the 
revision of the framework­regulation in 1993, in which equal opportunities 
are explicitly referred to, with particular emphasis on the struggle against 
unemployment among women. In 1996 the Council adopted a Resolution 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

reaffirming its political will in this respect.  In decision-making concerning  the 
approval  of  programmes,  a standard  clause  was  specifically  introduced  to 
guarantee compliance with this new obligation. 

Several  examples  of  good  practice  exist, taken  from the  successful  results 
of  the  NOW  1991-94  (New  Opportunities  for  Women)  experience,  and 
covering  a  broad  sample  of  measures.  NOW, the  first  Community  Initiative 
specifically  for women  (focused  on the  promotion  of  equal  opportunities  in 
the job  market),  has opened  new  avenues to  ensure that  women  are  taken 
into account  more fully by all the Structural Funds. 

From  the  beginning  of  the  new  period  of  Structural  Funds  programming 
(1994-1999), the principle of equal opportunities  has been considered as an 
objective  of  structural  policies  in their  entirety. There  has  thus  been  a  shift 
from a passive conception (respect for the principle) to an active  conception 
(objective of all structural policies) of equal opportunities. 

Today, equal opportunities  are no longer considered  only to  be the  problem 
of women. The promotion  of equal opportunities  is recognised for  its  ability 
to contribute  towards  socio-economic  development  in general and  regional 
readjustments in particular. 

In the context of Structural Fund interventions, the evaluation of the applica­
tion of the principle of equal opportunities  involves assessing the potential or 
actual impact on the respective situations of men and women and the extent 
to which equality is taken into account  in processes and mechanisms of pro­
gramming,  implementation,  monitoring  and  evaluation. This  latter  aspect 
involves,  for  example,  analysing  of  the  role  of  women  in  interventions, 
measuring positive developments and highlighting discrimination. 

•  The need to assess the application of the principle of equal 
opportunities 

Evaluation can play an essential role as a mainstreaming tool by ensuring that 
the principle of equal opportunities is taken into account  in structural policies. 
It may, in particular,  help to  enhance the effects  of  synergy for all the actions 
benefiting from Structural  Funds support.  In order to do so, it is necessary to 
apply techniques which take into account the specific nature of equal oppor­
tunities  between  men  and  women.  Through  the  definition  of  evaluation 
criteria  relating  to  this  theme,  and  of  corresponding  indicators,  evaluation 
can  be  used  to  systematically  examine  these  actions  and  policies  and  to 

Evaluating impacts on equal opportunities 

take into consideration their direct or indirect effects in terms of equal oppor­
tunities. 

Methodological difficulties are not as great in the assessment of the impact 
of specific actions in favour of women (who are, in this case, the direct bene­
ficiaries) as in the transversal evaluation of the effects of Structural Fund pro­
grammes in terms of equal opportunities. Interventions likely to produce 
direct effects on women are those focused on the development of human 
resources. Other types of intervention concerning infrastructure or the pro­
ductive environment do not necessarily include equal opportunities among 
their explicit objectives. Their effects are therefore more likely to be indirect. 

In the context of the evaluation of Structural Funds, a few isolated initiatives 
have been taken to evaluate the promotion of principle of equal opportuni­
ties. This experience has, however, been limited to a few cases of ex ante 
and intermediate evaluation. Ex post evaluation experience in the Structural 
Funds context is still embryonic. The NOW Community Initiative which was 
evaluated ex post is too specific to serve as a model in the framework of 
classical operational programmes. 

■ Defining what needs to be measured 

It is not possible to define ex nihilo what an evaluation has to assess, in so far 
as the strategy chosen to promote equal opportunities depends to a large 
extent on national and even regional contexts. Thus, the application of the 
principle of equal opportunities varies considerably from one Member State 
to the next. For certain countries (France, Portugal), it seems to be a general 
objective but without indications as to the strategy to adopt or actions to 
implement. In Spain and Germany equal opportunities are taken into consi­
deration in the context of specific actions, and are promoted by means of 
general measures. In Finland, the principle is visibly and thoroughly integra­
ted into all the components of programmes. 

• Analysing the process of programming interventions 

Because the programming of structural interventions is the result of 
decisions taken jointly (by the EU, state, regional and local authorities), it is 
necessary to examine whether, at all levels, there is a sufficiently strong 
political will to take into account the interests of women and to meet their 
specific needs. Keen awareness of the question is essential if suitable 
solutions are to be conceived and implemented (e.g. the setting up of 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

actions to  inform  and train  the  key  actors). The degree  of  awareness  is  not 
only  related  to  the  level  of  representation  of  women  in  decision-making 
bodies, even if the presence of spokespersons  is a contributing factor. 

Example: Training in occupations traditionally reserved for men. 
Training for women in traditionally  masculine occupations  has frequently been  unsuc­
cessful. Women are not  necessarily  recruited  by firms  in these sectors, or for jobs  for 
which they have been trained. Sometimes they are recruited but give up their jobs  due 
to  pressure  from  an  exclusively  masculine  environment.  An  ex  ante  evaluation  can 
help  to  reduce  this  failure  rate  by  programming,  in addition  to  the  training  measure, 
actions of the following  kind: awareness  in firms, promotion of recruitment  of women, 
advice and assistance to women after  recruitment. 

This example and others presented  in this chapter, are inspired by real situa­
tions. In some cases, however, they have been modified in order to provide a 
concise, but effective, illustration of an idea or a concept. 

that  the  resources  allocated  to  promoting  equal  opportunities  are 

The level of analysis  in the example  is upstream from the implementation  of 
the programme and thus of the production of effects on equal opportunities. 
The evaluator  must  examine the conditions  in which the  programmes  were 
designed and check: 
•  whether  the  announced  objectives  take  into  account  the  interests  and 
specific  needs of women; 
•  whether  groups of women and / or authorities responsible for equal  oppor­
tunities at national and regional level have been involved in the programming; 
•  whether  the  strategy  and  choice  of the  modes  of  intervention  have  relied 
upon a prior assessment of women's needs and potentialities; 
• 
coherent with stated objectives; 
•  the  proportion  of  interventions  programmed  that  are  able  to  facilitate  the 
achievement of priority objectives concerning equal opportunities; 
•  whether the  programming  documents  (CSF, SPD, etc.) provide  qualitative 
and quantitative criteria  and precise  indicators for assessing the considera­
tion given to equal opportunities; 
•  whether  some  measures  are likely to  run counter  to the  principle of  equal 
opportunities, risking the worsening  of the existing  imbalance between  men 
and women; 
•  whether the SPD is consistent  with  national and regional equal opportuni­
ties policies; 
•  the proportion of women in decision-making  bodies. 

Evaluating  impacts  on  equal  opportunities 

•  Analysing the conditions in which interventions are  implemented 

The perception of women's  interests is not always the same during the pro­
gramming of interventions and during their implementation. It is essential to 
analyse the conditions  of  implementation  in order  to  understand  how  they 
can account for the achievement or not of equal opportunity objectives. 

Example: Organisation of training for long-term  unemployed 
Presuming  the  monitoring  of  training  measures  revealed  that  the  drop-out  rate  of 
women was high, an intermediate evaluation would have made it possible to  examine 
the causes, e.g. family reasons (spouse's hostility), lack of infrastructure (lack of child-
care facilities,  means  of  transport,  care  facilities  for  an aged  or  ill  parents,  etc.)  or 
financial reasons (the allowance is too  low to cover child  minding costs, etc.). On the 
basis of such conclusions it becomes  possible for those responsible for the running of 
the scheme to take appropriate steps to remedy the situation. 

To ensure that the  principle  of  equal  opportunities  is taken  into  account  in 
the implementation of structural interventions, the evaluator must check: 
•  whether  the  selection  of  projects  has  included  the  application  of  the 
principle of equal opportunities as a condition in allocating budgets; 
•  whether groups of women and / or authorities responsible for equal oppor­
tunities are represented and active in the monitoring committees; 
•  whether the people responsible for programme management  are aware of 
all  issues  involved  in  promoting  equal  opportunities  (e.g. have  awareness 
campaigns on the subject taken place); 
•  whether  positive discrimination  (sometimes  referred to  by the  term  "affir­
mative  action",  i.e.  the  aiming  of  measures  at  a  particular  group  with  the 
objective of  eliminating  and  preventing  or compensating  for  disadvantages 
which  are a result  of  existing attitudes,  behaviour  and structures)  has  been 
applied to favour the participation of women in project initiation (e.g. whether 
there are women's NGOs among the promoters); 
• whether monitoring systems include the collection of gender differentiated data; 
•  whether  the  interventions  have  been  improved  during  their  implementa­
tion, to better meet women's needs; etc. 

•  Highlighting the effects of interventions on equal opportunities 

The main  methodological  difficulty  encountered  in this type  of  arises  from 
the fact that two concomitant  priorities  have to be evaluated: economic  and 
social cohesion  on the one hand and equal  opportunities  on the other. The 
latter is rarely formulated explicitly as an intermediate objective of  structural 
interventions, except in the case of specific measures or priorities. Moreover, 

Transversal evaluation of impacts on intervention priorities 

structural interventions are rarely, except for those of the ESF, aimed at 
people as direct beneficiaries. The effects of structural interventions are 
usually indirect and long term, which makes their evaluation complex. 

Application of the principle of equal opportunities to all interventions is a 
transversal objective that cannot be analysed immediately in its globality. It 
is therefore advisable to break down the effects into coherent sub­sets. The 
following typology is intended to provide an analytical grid, grouping 
together four types of possible effects in the specific context of Structural 
Funds, in terms of: 
• improving living conditions so as to better meet women's needs; 
• greater accessibility to the job market for women; 
• improving the situation of women at work; 
• promoting of the participation of women in the creation of socio­economic 
activities. 

Box 25 presents an objectives tree that draws on this typology to illustrate how 
this tool can be used to structure the mainly implicit objectives of actions 
promoting equal opportunities. Actions such as support for further or conti­
nuous education, professional training, counselling and orientation can all 
increase the employability of individuals. The introduction of more flexible 
work hours for women with children or aged relatives at home, leave organised 
in conjunction with school holidays, reduction in the physical aspect of jobs, 

Box 25 ­ Example of a thematic objectives tree for the priority 

of equal opportunities 

Global objectives 

Examples of specific objectives 

Balanced 
development 
for men 
and women 

Living conditions which 
meet women's needs 

ι ­ ­

• Improved access to neighbourhood 
services 
• Improved access to specialised medical 
treatment 
• Adaptation of transport systems 

Y 
Access for women to 
the job market 

• Restore employability 
• Adapt available jobs 
• Improving skills 

Women's situation 
at work 

« ­

• Reducing professional segregation 
• Improving working conditions 

Participation of women 

■*■ ­

so cio ­economic activities 

• Favour the creation of business 
or independent activities by women 

Evaluating  impacts  on  equal  opportunities 

can all contribute to  adapting jobs to women's  needs. As a consequence  of 
these actions, access for women to the job market can be improved. 

Furthermore the different intermediate objectives are not unrelated. If the living 
conditions  of women were improved  so that  needs could  be met  more ade­
quately,  the  other  three  types  of  impact  would  more  easily  be  achieved. 
Similarly,  improvements to the situation  of  women  in their jobs  may  improve 
conditions  of  access to  the job  market.  It  is for this  reason that, the  links 
between  intermediate  objectives  in the thematic  objectives  tree  have  been 
shown. 

In order to facilitate the work of evaluation teams, many examples of  effects 
are classified  in terms  of this typology  of  impacts and  presented  in Box 26. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Box 26  -  Example  of  effects  by category  of  impact 

Category 

Lifestyle 

Access for women 
to the job market 

Examples of effects 

•  Relaxing constraints related to responsibility for small children or for other 
persons, either temporarily (e.g. during job seeking) or over a longer period 
(e.g. investing In a middle-term social or professional project) 
•  Improving integration and reinforcing the capacity for initiative of women in rural 
areas, by providing more adequate public transport 
•  Reconciling family and professional constraints more satisfactorily 
•  More assistance in dealing with social problems, particularly through improved 
social services (quality family planning, support for victims of household violence, 
sexual harassment, etc.) 
•  Increased access to good quality care services for children and other depen­
dants 
•  Promotion of gender equality In parental leave 
•  More flexible work conditions 

•  Less sectoral and vertical segregation between men and 
women in education and training 
•  Improvement of job prospects for young women (guidance towards occupa­
tions with a future) 
•  Enhanced employability of women 
•  Provision of jobs which are better suited to meet women's requirements 
•  Adjustment of the content of training actions to meet the particular requirements 
of women (training in fields that improve their career prospects), and adjustment 
of the practical modalities of training to cater for women's specific constraints 
such as their daily schedule 
•  Personalised aid for the unemployed In finding a job. Including more appropriate 
support for women (e.g. for women trying to re-enter the job market) 
•  Setting up professional alternatives for women working in the underground 
economy, etc. 
•  Reconciling family and professional constraints more satisfactorily 

Women's situation 
at work 

•  Reduction of sectoral segregation by promoting a balanced representation of 
women and men in all economic sectors 
•  Jobs better suited to qualifications 
•  Increasing the proportion of women in senior positions 
•  better access to ongoing education, etc 
•  Reconciling family and professional constraints more satisfactorily 

Participation of 
women In the creation 
of new activities 

•  Better recognition and appreciation of the role of women in the creation 
of new businesses 
•  Improvement of women's access to services and infrastructures supporting the 
creation of enterprise, and to financial facilities for business development 
•  Greater participation of women in local development initiatives and in new 
sources of employment, etc. 
•  Reconciling family and professional constraints more satisfactorily 

Evaluating impacts on equal opportunities 

It is also possible that interventions, no matter their effectiveness and 
success, contribute to reproducing existing discriminatory patterns, thus 
having unfavourable or undesirable effects. It is very important that the 
evaluator analyses the over or under­representation of women in particular 
interventions and that the findings are contextualised. Women may for 
example be over­represented in measures not directly or immediately 
leading to employment, e.g. training measures, as opposed to apprentice­
ship schemes or unemployed aids. The importance of providing an explana­
tory socio­economic context to the findings is illustrated by the issue of part­
time work where the occurrence of such work can be considered as positive 
or negative depending, for example, on whether the part­time work was 
chosen or necessitated and taking in account the frequently less favourable 
working conditions, both in terms of work security and wages. When pro­
grammes are orientated towards traditionally male­dominated sectors, the 
evaluator could, in addition to determining to what extent women have parti­
cipated in measures, which types of measure and results, try to identify the 
conditions and necessary changes in these needed to promote increased 
gender balance in the sector in the future. 

■ Examples of impacts by area of intervention 

The approach used to integrate equal opportunities into programming 
documents should be twofold: an approach providing for measures expli­
citly in favour of women, and a transversal approach considering equal 
opportunities as a general guideline applicable to all the measures 
proposed. In the latter case, the promotion of equal opportunities is usually 
simply mentioned as a subsidiary of the general measures. In view of the 
absence of an operational translation of this priority, as well as its marginal 
nature, it is essential to examine how general measures incorporated into 
classical Structural Fund programmes can potentially contribute to equal 
opportunities. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

•  Basic  infrastructures 

Among  the  interventions  in the field  of  infrastructure, those  which  seem  to 
incorporate  an  equal  opportunities  dimension  concern  basic  infrastructure 
and facilities of a social nature. 

Example:  Child  c a re facilities  in  Portugal 

With  s u p p o rt  f r om  the  N OW  Initiative,  a  p l a y s c h o ol  w as  created  In  the  heart  of  the 

industrial  area  of  Evora  (Portugal),  c l o se  to  an  o c c u p a t i o n al  training  c e n t r e.  Child 

m i n d i ng  facilities  were  m a de  available  to  jobless  w o m en  wanting  to  a t t e nd  training 

c o u r s e s,  a nd  to  e m p l o y e es  of  f i r ms  in  t he  Industrial  area.  T he  p l a y s c h o ol  w as 

e q u i p p ed  with  a  playground  and  open-air  s w i m m i ng  pool  in  order  to  make  the  setting 

m o re  homely,  In  w h at  is  an  industrial  a nd  essentially  m a s c u l i ne  e n v i r o n m e n t.  T he 

c e n t re  t h at  h ad  h i t h e r to  o f f e r ed  t r a i n i ng  in  t r a d i t i o n a l ly  m a s c u l i ne  t r a d es  (e.g. 

carpentry),  had  been  frequented  mainly  by  m e n.  Today,  4 0%  of  the  participants  are 

w o m e n.  This  centre  has  enabled  s o me  of  t h em  to  find  j o bs  and  others  to  find  more 

s e c u re  e m p l o y m e nt  t h an  t h ey  f o r m e r ly  e n j o y e d.  T he  c h i ld  m i n d i ng  s e r v i c es  has 

helped  to  ease  the family  constraints  of  w o m en  w ho  consequently  have  the  possibility 

of  involving  themselves  in m e d i u m - t e rm  professional  projects. 

Outputs 

> 

Child  care 
facilities 

Use  of 
facilities 
for  lightening 
family 
constraints 

Increased 
availability 

Effects 

Better 
access 
to  training 

Qualifications 
improved 
and/or 
widened 

Successful 
projects 

O 

Better  access 
to job  market 

Improved 
jobs  situation 

Enhanced 
participation 
in  economic 
activities 

Priority 

I 

I 

Evaluating  impacts  on  equal  opportunities 

•  Human resources 

The  principle  of  equal  opportunities  is  more  visibly  incorporated  into 
measures concerning  human  resources. The  most  significant  effects  are in 
activities  involving  consultancy,  guidance  and monitoring; actions  in favour 
of training and occupational  integration; interventions  in education systems, 
training  and  employment;  and, lastly, on  work  organisation  and  modes  of 
production. 

Example:  Setting  up  measures  for  training, follow-up  and  rehabilitation  tailored 

to  suit  individual  requirements  (Portugal). 

Trainees  with  children  or  adults  under  their  responsibility  were  given  priority  in  training 

actions  and  in the allocation  of  additional  benefits. This enabled  both  w o m en  and  men 

to  reconcile  their family  and  working  lives. The training  also developed  methodologies 

and  specific  strategies  intended  for  w o m e n,  in the  framework  of  measures  providing 

assistance  and  rehabilitation  support  (individual  counselling, j ob  seeking  techniques, 

career  p l a n n i n g,  etc.).  C o n s e q u e n t ly  a  more  b a l a n c ed  p a r t i c i p a t i on  of  m en  a nd 

w o m en  in all sectors  of  activity  was  possible. 

Outputs 

> 

Financial  support 
for  people  with 
children or 
adults  In 
their  care 

System for 
providing support 

Individual 
rehabilitation 
course 

Effects 

Better reconci­
liation of family 
and working  life 

Easy access 
to  training 

Improved and 
diversified 
qualifications 

More  balanced 
participation 
in all  sectors 
of  activity 

Information 
Counselling 
Continuous support 

Quality of  training 
systems 

ΐ> 

Improved 
work 
situation 

Access 
to jobs 

Creation of 
economic 
activities 

Priority 

I 

I 

Transversal evaluation of impacts on intervention priorities 

• Productive environment 

Actions relating to the adaptation of productive activities and to local and 
regional development are likely to have an impact on the promotion of equal 
opportunities. 

Example: Creation of a European Business and Innovation Centre (Italy) 
A centre for the development of business innovation was created in the Pouilles 
region. The Business Information Centre in Puglia Sprind developed an international 
marketing methodology based on employing mainly women in the marketing division. 
Prospecting for business opportunities was based essentially on the ability of the 
women in the team to stimulate their interlocutors' curiosity and interest in their new 
products. Local products of SMEs were introduced into international distribution 
channels and the firms concerned enhanced their competitiveness on a global scale. 
This favoured the establishment of new SMEs in the region, specialising in agricultural 
products. The SMEs created jobs, most of which were occupied by women farmers 
living in and around the villages. 

Outputs 

> 

Creation for 
a development 
centre for 
innovative 
business in 
rural areas 

Establishment 
of other SMEs 

Creation of 
local activities 

Effects > 

Reduction of 
rural women's 
isolation 

Success of 
new enterprises 
created 
by women 

Access to 
employment 

4 

/ 
1 

Creation of 
economic 
activities 

Priority 

■ Defining and selecting indicators 

Volume 2 of the MEANS Collection is devoted to the subject of indicators for 
monitoring and evaluating Structural Fund interventions. The above Volume 
includes an information sheet devoted to the priority of equal opportunities. 

In order to remain as operational as possible, the use of indicators should be 
mainly limited to those which are quantifiable. Yet qualitative information 
must not be overlooked, given the complex and innovative nature of the 
objective of equal opportunities. 

The impacts of interventions on this priority are produced by an accumula­
tion of intermediate results derived from a series of heterogeneous outputs. 
Programme indicators should provide meaningful information on all the 

Evaluating  impacts  on  equal  opportunities 

steps in the process, ranging from the allocation of resources to the produc­
tion of impacts. 

A pragmatic  selection  should  be made of a small number  of ad hoc  indica­
tors relating to equal opportunities.  It should not be necessary to  undertake 
costly  collection  of  information,  nor  should  an  excess  of  information  be 
supplied to decision-makers: 
•  At  European  level, the  priority  need  is to  assess  and  clarify  the  conse­
quences of structural policies in relation to the extent to which equal  oppor­
tunities are being taken into account. 
•  At  programme  level, the  monitoring  committee  needs  to  know  to  what 
extent  resources  have  been  consumed  and  to  monitor  outputs. When  the 
programme  has  not  included  the  principle  of  equal  opportunities  in  its 
explicit  objectives, the  Monitoring  Committee  should  be  kept  informed  on 
the impact of the programme on men and women. 
•  At  project  level,  managers  must  monitor  in  detail  the  consumption  of 
resources and the outputs produced, as well as the reactions of the different 
of beneficiary groups, particularly women. 

Women are not a homogeneous group, but constitute half of the  population 
concerned  by  Structural  Fund  interventions.  Because  of  this, the  range  of 
indicators  should  be  systematically  broken  down  according  to  gender. A 
system  of  indicators  "sensitive  to  the  question  of  equal  opportunities" 
requires  the  close  integration  of  types  of  beneficiaries  and  types  of 
measures. This  makes  it  possible, for  example, to  assess  whether  women 
benefit in the same proportions as men from measures with a direct effect on 
employment  (such as recruitment  or employment  subsidies)  or whether,  on 
the contrary, they benefit only from training measures. 

Context  indicators,  by  providing  a baseline,  show  what  proportion  of  the 
targeted  population,  particularly  of  women, was  reached  or to  what  extent 
the needs of women in the region were met. A programme indicator (e.g. the 
number of women  beneficiaries of the programme who  kept their jobs  after 
having children) can be combined with a context  indicator  (e.g. the  regional 
percentage of women  between the ages of 25 and 35 who  interrupted their 
careers after  having children) to give an indication  of the contribution  of the 
programme  to  solving  one of  the  problems  faced  by  women  in the region. 

No universal rule exists for constructing a system of indicators for the appli­
cation of the  principle  of equal  opportunities. The quality of existing  indica-

Transversal  evaluation  of  impacts  on  intervention 

priorities 

tors has to be checked to determine whether they are acceptable. The most 
frequent shortcomings are: 
•  the absence of relations between the indicators and the targeted; 

•  the difficulty of using the indicators due to the lack of reference data for the 
constitution of the counterfactual situation; 
•  the absence  of  a sub-division  between  men  and women  in the  indicators 
when referring to individuals. 

A  system  of  pragmatic  indicators  taking  into account  the  principle  of  equal 
opportunities  can, however,  limit  the  amount  of  information  collected  and 
transmitted according to its relevance for each managerial level. 

In the  case  of  the  ESF, indicators  are  usually  presented  as  shown  in  the 
upper half of Box 27. When the principle of gender mainstreaming is applied, 
indicators are presented in the form shown in the table. 

Box 27 -  Example  of the  application  of the principle of gender  mainstreaming 

Measure XX : 
Number of beneficiaries : 
• Total 
•  by sex 

·  by age 
·  by level of qualification 

Gender mainstreaming implies the following breakdown of data: 

%Men 

% Women 

1. Total 
2.  by age 
3. by level of qualification 

•  Selecting and interpreting indicators for a given programme 

Programming  documents  reveal  an almost  total absence  of  precise  indica­
tors on how the principle of equal opportunities  is being taken into  account. 
As far  as the  participation  of  women  in the  CSF  and  SPDs  are  concerned, 
indicators  are very general and very  limited. At  best, women  are considered 
as a separate  category  of  beneficiaries,  like young  or  handicapped  people. 

In contrast,  more  specialised  monitoring  bodies  offer  increasingly  sophisti­
cated indicators. These are sometimes too general to be used in the evalua­
tion  of  a structural  programme  that  does  not  have  the  promotion  of  equal 
opportunities as its main objective. An example is the "index of  dissimilarity" 
which  serves  to  analyse  the  degree  of  segregation  between  the  sexes,  by 
sector of activity or by profession. 

Evaluating  impacts  on  equal  opportunities 

Thus, the evaluation team will most  often  have to  propose  ex  nihilo a list of 
indicators  which  incorporate  the  equal  opportunities  dimension  into  the 
global evaluation of a programme. Evaluators, however, cannot be expected 
to define specific  indicators  for the  promotion  of  equal opportunities  within 
this framework. It is therefore necessary to make a selection and to settle for 
relatively  rough  and  approximate  indicators to  assess the equal  opportuni­
ties dimension. Evaluation teams  should  use a solid analysis of the  context 
based  on  a diagnosis  of  the  problems  facing  women, the  identification  of 
prospects for development within the specific national or regional context or 
the  identification  of  the  most  important  aims  within  their  region  for  the 
promotion of women. 

The interpretation of information provided by indicators is not always easy. It 
is  essential  to  use  "control  variables"  on  the  local,  regional  or  national 
context,  particularly to check whether the effect  produced  is truly  beneficial 
for women. For example, an increase in part-time work for women could  be 
explained by the lack of other professional alternatives. It could also reveal a 
lack of infrastructure, child-minding facilities and a persistent lack of sharing 
of family  responsibilities. 

By way of an example, a system of indicators which could be used to track the 
extent to which the equal opportunities dimension has been taken into account 
within a programme of community  centre regeneration is presented in Box 28. 

Box 28 -  Example of a system  of indicators for the  establishment 

or regeneration  of community  centres 

Programme Indicators 
(Related to the intervention and its effects) 

Context Indicators 
(Related to the assisted area) 

Output indicators 

Facilities 

Capacity 

Result indicators 

Satisfaction 

Impact indicators 

Employment 

Purchased, constructed or 
refurbished buildings in m! 
(of which: area devoted to 
meeting women's needs, i.e. nursery, etc.) 

Maximum number of users that can be 
served (of which: maximum number 
of users of women-specific sources) 

% of user that are satisfied or very 
satisfied with the project 
(of which women/men) 

1 

Maximum number of users 
in nurseries in the region 

Number of users holding a permanent 
job (of which women/men) 

% of regional workers 
who use nursery services 

Source: Volume 2 of the MEANS Collection 

Transversal evaluation of impacts on intervention priorities 

■ Suitable sources of information 

Data available from the Statistics Office is usually used to quantify context 
indicators. For example, many indicators can be found at Eurostat in the 
NEW CRONOS database, a collection of about 70 million harmonised items 
of data for the Member States of the European Union. The Regio database, 
devoted to statistics on economic life in each Member State, presents data 
in the form of about 70 standard tables classified by subject domain. The 
data are all broken down by region, according to the nomenclature of the 
NUTS territorial units, and cover a wide range of information: demography, 
unemployment, labour surveys, etc. 

Statistics compiled from labour surveys primarily concern the distribution of 
working hours, the breakdown of jobs by age, gender and socio­professio­
nal category, and the level of unemployment with a breakdown of figures in 
terms of the reason, duration and modalities of job­seeking peculiar to each 
country and sector of activity. A statistical portrait of women living in the EU 
in the early 1990s is presented in the EUROSTAT publication "Women and 
men in the European Union". The document includes opinion polls that 
enrich a highly detailed comparative analysis of women's social and profes­
sional situation compared to that of men. 

Furthermore, the national institutes of certain Member States compile 
national statistics on the subject "women". For example, in Spain the 
Women's Institute (Instituto de la mujer) has created over 100 indicators to 
analyse disparities between men and women in the following fields: demo­
graphy, education, employment, health, policy and social benefits. 

■ S o me e x a m p l es of current practice in the European Union 

A detailed treatment of evaluation tools is provided in Volume 3 of the 
MEANS Collection. Some examples of the use of tools for assessing the 
extent to which the promotion of equal opportunities has been taken into 
account in Structural Fund interventions are presented in this section. 

The methodological singularity of this type of analysis lies in the complex 
and pervasive nature of the phenomena that has to be analysed. The 
construction of the evaluation must take into account the small number of 
former references and the lack of information available to date in this field. 

Evaluating  impacts  on  equal  opportunities 

An  assessment  of  the  impact  of  interventions  on the  priority  of  promoting 
equality  between  men  and women  can  be  undertaken  using the  "step-by-
step"  approach presented  in Part IV of this Volume. For an in-depth  evalua­
tion of impacts, however, other evaluation tools have to be employed. 

With the  exception  of the job  market, for  which  numerous  studies  exist  on 
discrimination  against  women  within  the OECD  area  as well  as at  national 
level, there are  not  sufficient  well-established  foundations  or  prior  informa­
tion available for a "hypothetical-deductive"  type approach, in other  words, 
a procedure which assumes the formulation of precise hypotheses for struc­
turing the tools for data gathering  and processing. The use of  closed ques­
tionnaires, for  example,  must  be  limited  to  certain  well-known  situations 
such as a survey  among trainees who  have attended the same course. It is 
thus appropriate to opt  for an  "inductive"  methodology  suited to  producing 
new  knowledge  about  unfamiliar  situations,  based  on  the  comparison  of 
qualitative observations. In addition to the use of focus groups, other corres­
ponding  techniques  are  unstructured  interviews,  full case studies  or,  more 
rarely, ethnographic observation. 

For  ex  ante  or  intermediate  evaluations, the  techniques  developed  in  the 
field of corporate  management  (e.g. Ishikawa diagram, "I had a dream" and 
the scenario technique) can serve to diagnose problems and find  innovative 
solutions. Descriptions of these tools can be found in most classic  corporate 
management  textbooks.  Other  suitable tools  are SWOT (Strengths /  Weak­
nesses, Opportunities / Threats) and colour vote. 

Ex  post  evaluation  requires  the  use  of  a  methodological  "package"  of 
different techniques  - e.g. questionnaire  surveys, full case studies and com­
parisons  of  several  population  samples  at  regional  and  national  level  -  in 
order to give a credible picture of the real impact. 

•  Ex ante analysis of projects financed by the ERDF in Spain 

In the framework  of the 1994-99 programmes, a case study  was carried out 
to  assess  the  extent  to  which  equal  opportunities  were  being  taken  into 
account.  On the  basis  of  answers  to  questionnaires  sent  to  all the  entities 
promoting  ERDF projects, the evaluator  identified the projects  most  directly 
related to the application of equal opportunities. Several equal opportunities 
criteria defined  by the evaluator  were  used to  perform  an in-depth  analysis 
identifying  projects  with  the  best  practices  in this  domain.  In total,  eight 
projects  with  the  following  potential  effects  were  chosen:  recognition  of 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

equality,  solidarity  between  women  and  self-esteem  building,  entry  by 
women  into  traditionally  male-dominated  sectors,  response  to  women's 
specific  needs,  promotion  of  dynamism  among  women, and  reconciliation 
of their working and family lives. 

The in-depth analysis of these eight cases helped to identify the characteris­
tics  of  projects  which  promote  equal  opportunities,  such  as the  adaptation 
of  the  form  and  content  of  training  actions  for  women, the  articulation  of 
measures to  ensure their  coherence  in terms  of  satisfying  equal  opportuni­
ties,  the  promotion  of  women's  participation  in  project  management,  the 
involvement  of  women's  associations  in  partnerships,  the  guarantee  of 
women's  participation  in project monitoring, and so on (see Box 29). 

Box  29  -  Standard form for  assessing the equal  opportunities 

dimension  in the framework  of ex ante  evaluation 

Name of the entity 
Title of the project 

Territory 
Objective (1,2,..) 

The project 

Objective 
Application of equal opportunities objectives (typology of effects) 
Typology of project promoters (% women) 
Beneficiary public (% women) 
Description of innovative actions and good practice 
Justification of equal opportunities strategies 
System of organisation and facilities developed 

Project programming 
framework 
in the ERDF 

Funds 
Programme 
Objective covered 
Axes, measures et actions 
Public and private co-financing 

Demonstrative value 
of actions 

Diffusion of results at local level 
Diffusion of results at regional level 
Transfer of results or methodology to other projects 
Multiplier effect 

Suggestions for the ap 

plication of "gender Mainstreaming": 

•  Analysis of implementation  of Finnish programmes. 

A thematic evaluation of equal opportunities was based on a matrix to assist 
commissioners and / or evaluators in their strategic  evaluation choice and in 
the  formulation  of  evaluative  questions.  This  approach  emphasises 
structure,  process,  results  and  impacts  on  women  and  men's  lives  and  on 
the  socio-economic  environment,  and  provides  evaluation  criteria  which 
take  into  account  the  principle  of  equal  opportunities. The  criteria  concern 

Evaluating  impacts  on  equal  opportunities 

the  different  roles  of  men  and  women,  women's  access  to  financial  aid 
provided  by the Structural  Funds, and the real needs of women in a specific 
context (see Box 30). 

Box 30 - Matrix for programme  assessment 

Objectives  of the  assessment 

Roles 

Access to  funds 

Needs 

Equal opportunities  criteria 

Structure 

Process 
1.  programming 
2.  Implementation 
3.  monitoring 
4.  evaluation 

Results 

Impact 

The matrix facilitated the selection of evaluative questions for which the eva­
luation had to provide a credible answer. For example, "what are the roles of 
women  and  men  in the  different  phases  of  design  and  implementation  of 
programmes?"  or  "to what  extent  does the  raison  d'être  of the  programme 
and  its  objectives  correspond  to  the  identified  needs  and  priorities  of  men 
and women  in the region?". The evaluation succeeded  in showing that allo­
wances granted  in the framework  of the ERDF programmes  did not  corres­
pond to Finnish women's real needs. 

•  An initiative for the evaluation of the application of equal opportuni­
ties in the framework of objective 2 programmes (Sweden) 

Within  the  framework  of  IQ-NET  (Networking  to  improve  the  quality  of 
objective  2  programmes),  a thematic  study  of  the  incorporation  of  equal 
opportunities  into these programmes  was  performed. The aim was to facili­
tate the exchange of information and experiences between the regions, with 
regard to the design, implementation, monitoring and evaluation of the equal 
opportunities dimension of objective 2 programmes. 

The thirteen objective 2 regions belonging to this network are all characteri­
sed by a particularly high unemployment rate among men. The priority of the 
intervention  is thus to  promote  the creation  of jobs. The  main  object  of  the 
study was, however, to examine the contribution  of objective 2  programmes 
to  the  promotion  of  equal  opportunities.  Within  the objective  2  framework, 
some  regions  incorporate  the  promotion  of  equal  opportunities  into  the 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

implementation  of  projects  and  then  into  their  evaluation.  Several  case 
studies were performed  to  answer  the following  evaluative question:  "What 
are the  employment  effects  of the  objective  2  programme,  bearing  in  mind 
the transversal  priority  given to equal  opportunities  by the Structural  Funds 
policy?". 

The  results of the  case  study  on the  Swedish  MASS  project  are worth  pre­
senting. The  project,  inaugurated  in January  1995  in the  region  of  Ånger­
manlandskusten,  was  aimed  at  providing  administrative  assistance  to 
several  SMEs  and  at  helping  them  improve  their  management.  Jobless 
women were the main beneficiaries of the project. 

A pilot  study was first conducted to define the problems  confronting  SMEs. 
It appeared  that they  encountered  problems  recruiting  skilled workers.  The 
second  step  related  the  needs  of  these  SMEs  to  job  applications.  14 
persons  were  subsequently  recruited,  of  whom  seven  were  women  pre­
viously  employed  in the  public  sector.  Five of  them  were  employed  on  a 
permanent  basis. 

The  project,  above  all,  helped  to  reduce  female  unemployment  and  to 
improve  the  competence  of  the  SMEs  involved.  It  also  contributed  to 
reducing  the  workload  of  managers  in these firms  and thus to  increase  the 
time they spent with their families. 

•  Evaluation of the participation of women in actions co-financed by the ESF 

This  evaluation, conducted  in  1991, was  the  first  thematic  study  on  equal 
opportunities and on the role of women in the Structural Funds. The origina­
lity of the study lay in the incorporation of the equal opportunities  dimension 
in all the  processes  and  methods  of  programming,  implementation,  mana­
gement and evaluation of ESF interventions. 

The construction of the methodological framework was based on  objectives 
defined by the Commission: 
•  qualitative and quantitative  analysis of the  role of women  in  programming 
documents; 
•  analysis of the extent to which of equal opportunities were being taken into 
account  in the  processes  and  methods  of  programming,  implementation, 
monitoring and evaluation; 
•  qualitative  and  quantitative  analysis  of  the  role  of  women  in the  actions 
implemented; 

Evaluating  impacts  on  equal  opportunities 

•  analysis of the coherence and complementarity  between actions financed 
by the  ESF and programmes  and actions set  up nationally and regionally  to 
promote equal opportunities, 
•  analysis  of  the  coherence,  synergy  and  complementarity  between  the 
NOW  programme  and  actions  financed  within the framework  of  objectives 
1,3 and 4. 

For  the  quantitative  analysis,  multiple  and  diverse  sources  of  information 
were used, in addition to the CSFs and OPs, e.g. annual reports  established 
either  by  the  national  and  /  or  regional  authorities,  or  by  the  training 
agencies; or even applications for support. 

For the qualitative analysis, interviews and / or questionnaires addressed  to 
the  actors  involved  in the  preparation  and  implementation  of the  CSFs  and 
OPs were  used, e.g.  ESF  Missions;  national, regional  and  local  authorities 
responsible  for  employment  and  occupational  training; training  agencies 
and private enterprise; and promoters and managers of actions  co-financed 
by the  ESF. National  and /  or  regional authorities  responsible for  policies  in 
favour  of  women, as well  as  correspondents  in each  Member  State  of  the 
Networks  dependent  on the  Unit  for  equal  opportunities  (DGV), were  also 
interviewed. National experts along with national and / or regional  managers 
responsible for the preparation and  implementation  of  NOW  in the  Member 
States were also contacted. 

The indicators used for the quantitative analysis concerned: 
•  resources  allocated  to  women,  through  mixed  and  specific  actions,  as 
compared to all available resources; 
•  number  of  women  beneficiaries of  mixed  and specific  actions,  compared 
to the total number of beneficiaries; 
•  percentage  of women  beneficiaries  by type  of  action, category  of  benefi­
ciary, level of qualification and sector of activity; 
•  proportion  of  actions  and  national  programmes  in favour  of  training  and 
employment of women, benefiting from ESF co-funding. 

The methodology  made provision for the following four typologies: types of 
measures / actions, categories of beneficiaries, sectors of activity, and quali­
fications. It included a series of key questions, grouped together by theme. It 
also provided for the constitution of a sample of mixed actions for each type 
of  measure,  spread  between  actions  with  a high  rate  and those  with  a  low 
rate of female participation, for the purpose of carrying out case studies. 

Transversal evaluation of impacts on intervention priorities 

■ S u m m a ry 

The promotion of equal opportunities between men and women is a priority 
for which very little evaluation experience exists, this is particularly the case 
for ex post evaluation. The incorporation of equal opportunities into Structu­
ral Fund evaluations is therefore an important challenge. 

Actions focusing on the development of human resources are those most 
likely to produce a direct impact on the situation of women. This type of 
impact is easier to evaluate than indirect impacts on the promotion of equal 
opportunities. 

Evaluation can play an essential role in promoting the strategy of gender 
mainstreaming by ensuring that the principle of equal opportunities is syste­
matically taken into account in structural policies. 

■ For m o re information see... 

Àlvaro­Page, M. (1996), Los usos del tiempo como indicadores de la discri­
minación entre géneros, Madrid: Ministerio de Asuntos Sociales; Instituto de 
la Mujer, N°46. 

Bamberger, M. (1996) "Understanding the Impact of Development Projects 
on Women: The Tunisia Institutional Development Fund Project" in Evalua­
tion for the 24st Century: New Topics for Evaluation. 

C.E. (1994), Les femmes et les hommes dans l'Union européenne: portrait 
statistique, Luxembourg: Office des publications officielles des Communau­
tés européennes. 

C.E. (D.G. V) (1998), 100 mots pour l'égalité, Glossaire de termes sur l'égalité 
entre les hommes et les femmes, Luxembourg: Office des publications offi­
cielles des Communautés européennes. 

Rubery, J.; Fagan, C. and Maier, F. "Occupational Segregation, Discrimina­
tion and Equal Opportunity", pp431­461 in Schmid, G.; O'Reilly, J.; and 
Schömann K. (1996) International Handbook of Labour Market Policy and 
Evaluation, Cambridge: EE, University Press. 

Thematic evaluations 
of domains of intervention 

In addition  to interventions  targeted at promoting  political  priorities,  interven 
tions  within  the  framework  of  the  Structural  Funds  take place  directly  in a 
wide range of domains.  These domains are largely situated  upstream  of  prio 
rities and may consequently  have an impact  on them.  For example,  interven 
tions  in  the  domains  of  transport  or  tourism  have  possible  environmental 
impact,  interventions  in  favour  of  RTD are  designed  to  have an  impact  on 
competitiveness  of industry, in particular  SMEs. 

The distinction  between  domains  of  intervention  and  political  priorities  has 
repercussions  on the organisation  of thematic  evaluations.  While the evalua 
tion  of priorities  can  take place  at any level above  that  of measure,  thematic 
evaluations  of  domains  will  be  transnational  in  nature,  taking  place  across 
programmes. 

The selection  of domains  has been made on the basis of recent  transnational 
evaluation experience acquired  within  the framework  of the Structural  Funds. 
The first  chapter  concerns  interventions  in  the  domain  of  RTD.  There is a 
considerable  amount  of evaluation  experience  in this domain,  but  evaluation 
practice  is  undergoing  a period  of  adaptation  to  match  the  evolving  policy 
context.  This structure  of this chapter  is similar  to that of the chapters  on the 
four political  priorities  in Part II. The second  chapter  is, on  the other  hand,  a 
rapid  survey  of  evaluation  practice  and  issues  relative  to  the  domains  of 
transport,  telecommunications  and  tourism. 

Evaluating impacts arising from interventions in the 
domain of Research and Technological Development RTD 

■ W hy evaluate the effects of interventions on RTD? 

For regions depending on agriculture and more traditional labour intensive 
industries, the opening up of national markets has meant increased compe­
tition from lower cost regions both within and outside the European Union. 
This has consequently led to a reduction of low skilled employment opportu­
nities and has exacerbated problems associated with economic develop­
ment and industrial restructuring. 

The decline of traditional sectors has been accompanied by an increasing 
reliance on new, rapidly expanding industries to generate economic growth. 
Because firms in these industries tend to be small or medium­sized, with 
generally limited resources for RTD investment, their research effort often 
has to be undertaken in co­operation with other firms or with research insti­
tutes to obtain optimal results. However, this type of collaborative arrange­
ment tends to occur most often in specific circumstances of which geogra­
phical proximity, the existence of non­pecuniary relations and training­
research­industry linkages are important factors. Particular examples of 
these phenomena that have inspired recent public policy thinking in the 
domain of RTD are the Italian "Districts" and the close links between training 
systems and industry found in Germany. 

As industrial restructuring has progressed, it has become increasingly evident 
that a process of technological polarisation is underway. The more technolo­
gically advantaged regions benefit from prior technological capacity, 
continued high levels of RTD expenditure, a well developed innovation culture 
in general, and the ability to further develop their innovatory capacity, whilst 
the more technologically disadvantaged regions fall further and further 
behind. This divergence of RTD capacity is reflected in the persistence of 
regional disparities in terms of competitiveness, trade, unemployment and 
growth, both within individual countries and across the EU as a whole. 

New insights into the process of innovation have meant that the linear 
"Science­Technology­Production" type model has given way to the concep­
tualisation of innovation as a dynamic, interactive and non­linear process. In 
addition, the understanding of the territorial dimension of RTD, through the 
framework of regional and national systems of innovation, underpins the 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

rationale of interventions aimed at economic and social cohesion. As a result 
of these advances, more emphasis  is being placed on the creation of endo­
genous technological  capacity  and on the diffusion  of  innovations  to  SMEs 
through  increased  co-operation  between  universities,  research  institutes 
and  enterprises,  as well  as  support  and  promotion  of  industrial  RTD. This 
change  of  orientation  reflects  the  fact  that  earlier  interventions  have  had 
generally disappointing  results in regions with low RTD potential. 

Central  to  this  change  of  emphasis  is that,  with  the  demise  of  the  linear 
model  of  innovation,  policies  focusing  on the  "technology  push"  approach 
are inadequate.  It  is now  realised that  technology  supply  and demand  have 
to be matched through an interactive  process  involving both producers  and 
users of new technologies.  Moreover, the development  of a region's innova­
tory  capacity  is intrinsically  linked to the tailoring  of  interventions  to  suit  the 
character  and to  anticipate  the  evolution  of  its  productive  structure.  Where 
SMEs  predominate,  one  of  the  major  requirements  in  RTD  intervention  is 
simply to help them appreciate the importance of innovation for competitive­
ness  in  an  open  economy.  The  present  generation  of  interventions  are 
therefore  largely  composed  of  "mission  oriented",  bottom-up  measures 
aimed at the wider  business and commercial  needs of  SMEs, their  demand 
for outside technology  and  integration of  it  into their  production  processes, 
their own capacity  in product  innovation and marketing, the financing, orga­
nisation  and  management  of  innovatory  activities,  and  the  creation  of  new 
high-tech firms. 

•  The Community  context 

The  importance  of  RTD for the competitiveness  of a region and the  need to 
develop  RTD capacity  is  recognised  within  the  Structural  Fund  framework. 
As a result, Structural  Fund support  for  RTD has risen dramatically  during  a 
period  in which  national  governments  have generally  been cutting  back  on 
science and technology  (S&T) expenditure. 

RTD interventions are financed under Structural Funds, with the ERDF being 
the major  source. RTD support  figures  prominently  in Community  Initiatives 
(INTERREG II, PME, ADAPT...) and is also the subject  of innovatory  actions: 
Regional  Innovation  Strategies  (RIS)  and  Regional  Technology  Transfer 
(RTT). In addition, leading  areas of scientific  and technological  research  are 
the object  of  interventions  associated  with Community  Support  Framework 
programmes. 

Evaluating  impacts  in  the  domain  of RTD 

Within  the  global  aim  of  the  EU's  RTD  policy,  which  is  to  strengthen 
European competitiveness  through the development  of scientific  and  tech­
nological  excellence,  Structural  Fund  interventions  are targeted  at  counte­
racting the concentration  of  RTD capacity  in certain countries, mainly in the 
North  of  the  EU (transnational  cohesion),  and  in certain  regions,  mainly  in 
and  around  countries'  capitals  and  a few  dynamic  regional  poles  (national 
cohesion). 

Structural  Fund  resources  have therefore  been  directed  to  more  peripheral 
regions with the object of developing their S&T base, particularly through the 
establishment  of  sustainable  links  between the  public  research  bodies  and 
regional industry. Over the long term, these interventions should increase the 
innovatory capacity of assisted regions either by reorienting their technologi­
cal  capacity  towards  the  faster  growing,  high  technology  sectors  or  by 
creating  indigenous  technological  capacity  and  therefore  reducing  the 
"technology  gap"  with  developed  regions. The  longer-term  impact  of  this 
process  will  be  increased  regional  specialisation  in  high  value-added 
products  and  services,  increased  growth,  and  enhanced  economic  and 
social cohesion. 

Recent experience of interventions  in the domain of RTD, for the  1989-1993 
programming  period,  has  shown  that  a number  of  problem  areas  in  RTD 
funding exist  in the  regions. Firstly, the relationship  between  increased  RTD 
funding,  increased  innovation  and  economic  development  is  neither 
automatic  nor  direct.  Attention  needs  to  be  paid  not  only  to  technology 
supply,  usually generated  upstream  from  industry  in  public  institutions,  but 
also to  the  demand  for  technology,  fashioned  by  the  needs  of  a  region's 
industrial actors  and their ability to  match  available technologies  with com­
mercial  opportunities.  However,  funding  during  this  period  tended  to  be 
concentrated  in  public  research  and  did  not  consequently  address  the 
practical RTD needs of local firms, particularly those of SMEs. 

In the  least  developed  regions  there  are  major  problems  of  absorption  of 
available  funds  due  to  the  fact  that  suitable  projects  are  often  difficult  to 
identify  or  authorities  have  little  experience  with  or  understanding  of  RTD 
issues. In certain  regions this  problem  has aggravated the  concentration  of 
funds  in  public  RTD  upstream,  with  the  consequent  mismatch  between 
public  technology  supply  and  industrial  technology  demand  (see  Box  31). 

Transversal evaluation of impacts on intervention priorities 

Box 31 ­ Evaluation of the impacts of CSFs for 

Research and Technology in Greece, Ireland and 
Portugal for the 1989 ­ 1993 programming period 

The objective of this evaluation was to assess the impact of interventions, 
make comparisons between the 3 countries studied and make recom­
mendations which would help improve both the efficiency and the effecti­
veness of CSF support for RTD in these countries. The study found that 
the national authorities in the three countries had largely used the oppor­
tunities provided by the funds to considerably enhance public infrastruc­
ture for RTD. It also reported that all three countries had had problems in 
the areas of industrial participation, technology transfer, overemphasis on 
technology supply, concentration of RTD around capital cities and human 
resource shortages. Problems of operational costs, achieving self­
financing targets, and management and evaluation also occurred. Conse­
quently, during the second programming period (1994­1999) more 
emphasis is being placed on better use of existing RTD capacity and less 
on building additional RTD capacity. 

The development of European RTD capacity in general, and in relation to 
cohesion in particular, represents an important strategic objective of the EU. 
This is reflected by the fact that interventions aiming to develop the techno­
logical potential of the regions amount to 7,600 million euros, approximately 
7% of all Community spending/ The evaluation of RTD interventions 
therefore has a central role to play in assessing the extent to which they are 
capable of achieving (or are achieving or have achieved) stated policy objec­
tives. The results and conclusions of these evaluations can then be used to 
inform the decision making process and consequently enhance the effecti­
veness of policies and programmes. A good example of this process is the 
evaluation of the impacts of CSF on research and technology in Greece, 
Ireland and Portugal for the 1989­1993 programming period (see Box 31). 

■ Defining w h at needs to be evaluated 

When evaluating Structural Fund interventions in the domain of RTD, it is 
important to bear in mind that the undertaking of RTD of sufficient quantity 
and quality is not an aim in itself. Nor is it a factor which when introduced into 
policy, leads to convergence towards a desirable goal. It is a prerequisite of 
sustained economic development and socio­economic cohesion. 

4. "Les Fonds structurels en 1996, huitième rapport annuel". This figure includes interven­
tions in the domains of R&D and innovation. 

Evaluating  impacts  in  the  domain  of RTD 

Whilst  it  is relatively straightforward  to  evaluate outputs  and  results of  RTD 
interventions,  it is more difficult  to  estimate the  impacts  produced.  Specific 
impacts  are often of an intangible nature (e.g. qualitative changes  in human 
capital)  and  impacts  at  the  socio-economic  level  only  occur  over  an 
extended time  period, making  impacts  difficult  to  attribute  to  interventions. 

The techno-industrial structure of a region is fashioned over time by a multi­
plicity  of factors. As a result, the  needs of  regions and therefore the  extent 
and  type  of  RTD  intervention  vary  greatly  from  one  region to  another.  For 
instance, in lagging regions and to a lesser degree in those with low popula­
tion densities, the supply  of facilities  and equipment, the number  of  resear­
chers, the  application  of  research  results  and the transfer  of technology  to 
local industry are problems that have to be addressed if endogenous scienti­
fic  and technical  capacity  is to  be  created.  It  is, however,  also  pertinent  to 
question to what extent interventions  in regions with little history of  research 
and  technology  should  be  undertaken  at  all. As  for  declining  industrial 
regions, interventions  are generally oriented towards  support for  investment 
in  specific  high-tech  sectors  (e.g. biotechnology,  microelectronics,  etc.), 
investment  in  key  locations  and  the  creation  of  investment  projects  to 
employ  highly trained personnel. 

•  Some key issues in RTD evaluations 

There are a number  of  key  issues related to  structural  support for  RTD that 
can be used to orient the evaluation of RTD interventions: 
•  Effectiveness  in  promoting  RTD capacity  and  capabilities:  What  consti­
tutes RTD capacity and capabilities?  Do interventions  promote the  intended 
results  and  impacts  in terms  of  RTD capacity  and  capability?  It should  be 
noted that  because the benefits  of  a single  research  project  are not  usually 
captured  by  or  do  not  generally  lead  to  specific  innovations,  particularly 
where collaborative  research is concerned, the results of  cost-effectiveness 
type analyses may be of limited worth unless RTD activities are appraised at 
a portfolio level. 
•  Strength of  linkages generated  by interventions  within the local  economy: 
Whilst  measures  of activity  (e.g. the number of SMEs involved in collabora­
tive RTD projects)  may be high, the  intensity  of  involvement  may differ  from 
firm to  firm.  Focusing  on the financial  participation  of  the  recipient  firms  or 
some  sort  of  qualitative  assessment  of  collaborative  ventures  can  help  to 
give an indication of the strength of these linkages. 
•  Skills  shortages  and  improved  qualification  levels:  In regions  undergoing 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

industrial restructuring, high unemployment  coexists with skills shortages in 
some of the more technologically  advanced  sectors. However, an  important 
question is the extent to which benefits of increased educational and training 
opportunities, especially at post-graduate level, are lost through the outward 
migration of addressees of supported training to other regions? 
•  When  improved  economic  performance  is  primarily  a  consequence  of 
process  innovations  leading  to  reduced  labour  costs,  initial  effects  on 
employment are likely to be a reduction in jobs. This may often be the case in 
regions  where  former  labour  intensive  industries  in  declining  industrial 
regions are attempting to remain competitive through capital investment and 
labour  substitution. The  relationship  between  innovation  and  employment 
creation or maintenance therefore warrants particular attention. 
•  Interactions  between  interventions  and  national  RTD  policy,  and  the 
regional distribution of RTD activities: The geographical distribution of public 
research capacity  is usually  strongly  influenced  by  national  strategic  consi­
derations.  Consequently,  in  more  centralised  states  this  leads  to  research 
facilities  tending  to  be concentrated  around  the  capital  city.  Conversely,  in 
more decentralised states research capacity  is usually found to be less cen­
tralised,  though  it  may  be  concentrated  around  regional  capitals.  Thus, 
national  RTD policies  and Structural  Fund interventions  can display  varying 
degrees of positive or negative synergies. 
•  Synergy  and  integration  with  other  EU  interventions:  Whilst  Support 
Framework  programmes  have,  until  recently,  been  concerned  with  more 
upstream  "pre-competitive"  research  projects,  and  Structural  Funds  with 
enhanced  research  capacity  and  technology  transfer,  the  reality  of  the 
situation  is that the  boundary  between these  approaches  is, in fact,  increa­
singly blurred. This overlap  suggests that there are possibilities  of  synergies 
and /  or  dangers  of  double  employment  between  these  two  policy  instru­
ments.  Furthermore,  research  projects  selected  under  the  Framework 
programme  are  chosen  on  the  basis  of  excellence  and  without  regard  to 
cohesion  and  structural  objectives.  This  generates  negative  synergies 
between these instruments when research capacity, as is generally the case, 
is concentrated  in the more dynamic and wealthier regions. 
•  Future  policy  orientations:  are structural  interventions  capable  of  overco­
ming  the  strong  tendency  towards  geographical  concentration  of  RTD, 
indeed,  is this  objective  relevant?  Or, on the  contrary,  should  interventions 
aim to concentrate RTD capacity in a relatively small number of regions so as 
to  achieve  critical  mass?  Should  Support  Framework  programmes  be 
tailored  to  take  account  of  the  concerns  of  structural  adjustment  and 

Evaluating  impacts  in  the  domain  of RTD 

cohesion?  How  can  overlaps  between  the  two  policy  instruments  be 
reduced  and  how  can  synergies  be  encouraged?  How  can  problems  of 
absorption and additionality  be overcome? 

The choice  of  level  of  study  of  RTD activities  is  largely  determined  by  the 
need  to  take  into  account  the  externalities  created  by  research  projects. 
Because  scientific  and technical  knowledge  are to  a certain  extent  public 
goods, the  estimation  of  effects  at  the  project  level  are  likely  to  seriously 
underestimate the benefits of the project accruing to society as a whole. It is 
for  this  reason  that  RTD  is  being  increasingly  studied  either  at  a  meso-
economic or macro-economic  level. Since Structural  Fund interventions  are 
geographically  delimited, a macro approach  seems to  be the  most  relevant 
level of analysis. Analysis at this level also allows interventions  and  projects 
to  be  examined  in the  larger  context  of  the  regional  or  national  innovation 
system of which they are an integral part. This does not exclude the possibi­
lity of  integrating  a meso-level  analysis  into an evaluation where a sector  or 
filière is of particular importance in a region. 

There  are four  main  channels  through  which  Structural  Funds  intervene  to 
reshape a region's RTD profile: 
•  Support  for  S&T infrastructures:  universities, associated  research  centres 
and public  laboratories, and structures  to  promote the diffusion  of results of 
research  projects. Activities  in this  area are aimed  at  enhancing  scientific 
and technological potential; 
•  Direct  support  for  industry:  subsidies  and  soft  loans to  support  R&D and 
technology  transfer  activities, and the creation  of  high-tech firms. Activities 
in  this  area  aim  at  enhancing  technological  potential,  increasing  links 
between  research and innovation and improving the financial framework  for 
performing RTD; 
•  Indirect  support  for  industry:  services  such  as technology  audits  and 
expert advice, etc., strengthening of innovation and technology centres, and 
efforts  to  increase  participation  in  other  EU  RTD  initiatives  (Support 
Framework,  Community  Initiatives,  etc.). These  activities  are  intended  to 
enhance  technological  potential  and  to  support  innovation  through 
improved links with research; 
•  Training: training for researchers and technology  managers with an overall 
aim of promoting an innovation culture. 

The suggested framework for evaluating RTD interventions  presented in Box 
32 is based on two  broad categories  of activities: the strengthening  of  S&T 

Transversal evaluation of impacts on intervention priorities 

potential and support for innovation. These categories group together a 
number of themes and sub­themes, some of which may warrant a more 
focused analysis depending on the RTD profile of the region under examina­
tion and the type of intervention. 

Box 32 ­ Framework for the evaluation of Structural Fund 

interventions in the domain of RTD 

Structural Fund interventions in the domain of RTD 

Strengthening ( 

1 

1 

■ 

I 

Enhancer 
of 
scientific 
potential 

Construction of 
university 
research 
centres and 

Grants for 
equipment 

Grants for 
projects 

Etc. 

1 

1 

1 

1 1 

1 1 

of an 111 links between I II of financial II 
"innovation II research | || framework for 11 
E ES | 

" ' ■ ­ " ■· * 

I 

ï 1 

Technology 
transfer 

1 1 

Seed and 
venture capital 

Construction of 
technology centres, 
science parks, 
support for SME 
research capacity 

Grants for equipping 
technology centres, 
measurement and 
testing equipment 

Training 

Researcher 
mobility 

Stimulation of 
activity 

Diffusion of 
results 

Grant funding of 
development projects 

Awareness 
raising activities 

Etc 

Etc. 

Etc. 

Promotion of 

Etc. 

Linkages 
between firms, 
universities. 
centres of 
research 

• Clarifying the objectives of RTD interventions 

In the case of RTD interventions, an objectives tree can be used to link up the 
outputs and results of RTD interventions to the specific objectives envisaged 
within the framework of the Structural Funds (see Box 33). 

Evaluating  impacts  in  the  domain  of RTD 

Box 33 -  Example  of a thematic  objectives tree  for 

RTD  interventions 

G l o b al  o b j e c t i v es 

E x a m p l es  of  s p e c i f ic  o b j e c t i v es 

Increased  technology 
transfer  between 
institutions  and 
enterprises 

• Increased number  of  collaborative 

projects  between SMEs and 
research  institutes 

• Increased number  of  post-graduate 

in-firm  placements 

Increased  Innovation 
by  enterprises, 
particularly  SMEs 

• Increased  number of  inter-firm 

RTD collaborations  Involving SMEs 

• Increased  number  of SMEs 

performing RTD 

Improved RTD 
Infrastructure 

•  Increased  number  of S&T parks 
•  Improved  university  research 

facilities 

•  Increased  number  of  new 

university  research  facilities 

Increased 
availability  of 
suitably 
qualified 
researchers  and 
technicians 

•  Increased  number of places on 

science and technology  oriented 
university  courses 

•  Increased  number  of  research 

grants available 

Links  between  intermediate  objectives 

Reduced  inter· 
regional  and 
infra-regional 
disparities  in 
RTD capacity 

Increased 
economic 
and social 
cohesion 

While specific  objectives  are strictly  subordinate to the intermediate  objec­
tives  in the tree, the  latter  are not  independent  of  each other.  For example, 
attaining the objective  of  improved  RTD infrastructure  is likely, by  providing 
facilities  for  firms  wishing  to  perform  RTD, to  contribute  to  innovation  by 
SMEs. 

While giving an overview of the structure of the objectives of an intervention, 
the objectives tree also provides an indication of what an impact typology for 
RTD-oriented  Structural  Fund interventions  might  look  like. Using the  inter­
mediate  objectives  of an intervention  as a starting  point, a typology  of  RTD 
impacts might look like this: 
•  Technology transfer; 
•  Industrial RTD; 
•  RTD infrastructure; 
•  Human resources. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

•  Net effects of RTD  interventions 

The net effects of structural interventions in the domain of RTD are difficult to 
evaluate. Impacts, especially at the socio-economic  level, often occur a long 
time after  RTD investments  have taken  place, which  makes their  attribution 
to  individual  interventions  difficult  to  ascertain.  RTD  impacts  are  heteroge­
neous and must therefore  be estimated using an array of indicators: sales of 
innovatory  products,  processes  and  services;  R&D  employment  created; 
technologies  acquired;  networks  created;  number  of  PhDs  obtained,  etc. 
They cannot therefore be summarised in the form of a single  indicator. 

Generally  speaking, the  estimation  of  net  effects  emphasises  the  assess­
ment  of  deadweight  (i.e. the  change  which  would  have  occurred  to  the 
activity  under examination  in a policy-off  situation).  However,  it is  important 
to  note  that  although  public  funding  dominates  upstream  research,  at  the 
industry  level  the  private  versus  public  funding  balance  varies  between 
sectors  and  the  importance  of  deadweight  will,  as  a consequence,  vary 
accordingly. 

It is for these reasons that the estimation of net effects  of  RTD  interventions 
is generally  undertaken  at the  results  level, by focusing  on changes  in RTD 
expenditure  resulting from an intervention in the domain. 

Example: From gross to net effects in the support for innovation in SMEs 
Assuming  that  support  for  a  venture  capital  fund  to  finance  research  projects 
performed  by  SMEs  has  led to  435  projects  being  launched  at  a cost  of  600  million 
euro.  It is however found that  120 projects  costing  150 million euro would  have  gone 
ahead without support (deadweight). The net effect of the intervention would therefore 
amount to 450 million euro. 

Another factor  which has to  be taken  into account  is the leverage effect (i.e. 
the increase  in privately financed  RTD which  results directly  from  Structural 
Fund expenditure  in RTD projects). Leverage along with substitution  is often 
explicitly  sought  by  policy  makers.  The  latter  results  from  interventions 
aiming  to  persuade  high-tech  enterprises  and  research  institutes  from 
outside the target  region to  relocate within the region in order to  increase its 
RTD capacity.  In the case where diversification of the regional economy  is a 
prime consideration,  substitution  of  innovative  activities for  activities  those 
in declining low-tech sectors will also be an explicit objective. 

Evaluating impacts in the domain of RTD 

■ Examples of i m p a c ts by area of intervention 

A number of different examples of RTD projects are presented below, cor­

responding with the three broad areas of intervention of the Structural 

Funds. 

* Basic infrastructure 

Interventions in the area of basic infrastructure usually have an indirect effect 

on RTD. For example, improvements in telecommunications networks may 

increase the scope for using telematics for long distance RTD collaboration. 

Infrastructure co­financed by Structural Fund interventions, having a more 

direct effect on RTD, includes facilities for university research, science and 

technology parks, etc. 

Example: The construction of a science and technology park in Northern Italy 
The Tecnoparco del Lago Maggiore in Piedmont was conceived to help entrepreneurs 
interested in technology and innovation to turn their ideas into successful businesses. 
The park, co­funded by the ERDF and the Piedmont region, provides rented facilities 
for research projects. Tenants of the Park also benefit from back­up services 
(business counselling, environmental safety and certification, computer­aided design, 
partner and finance search and the maintenance of meeting and training facilities) and 
support for innovative technology projects. The long­term objective of this project is to 
attract research investment and initiatives from outside the region and to stimulate the 
region's economy, allowing traditional industries to be replaced by innovative SMEs. 

Outputs  O 

Construction of 
buildings and 
provision of 
equipment 

Firms setting 
up in the park 

Effects 

New firms becoming 
established 

Jobs directly created by 
the firms located In the park 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

•  Productive  environment 

Interventions  in  the  area  of  the  productive  environment  cover  a  large 

spectrum  of  possible  measures  ranging  from  direct  support  for  RTD  invest­

ments to  support  for  services  aimed at assisting  the technological  activities 

of  firms,  particularly  SMEs  and  NTBFs,  and  at  creating  networks  involving 

both public  and private  institutions. 

Example: Promoting technology transfer in Aquitaine (France) 
The  economy  of Aquitaine  is dominated  by the presence  of large  enterprises  in  the 
armaments,  space  and aeronautics  sectors.  These  sectors  have, for a number of 
years, been  undergoing  large scale  restructuring  which has had dramatic  results not 
only  on their  own activities,  but also  on the activities  of  local  subcontractors  in a 
diverse range of sectors. One of the central priorities established in the  1994-96  SPD 
for  the region  was to adapt  the existing  potential  of the region's  SMEs  through a 
coherent policy for innovation and technology transfer. One of the interventions  within 
the  framework  of this  policy  has been  the setting  up of CREATI Aquitaine,  (Centre 
Régional d'Appui Technique et d'Innovation)  which in 1997 brought  together  10 large 
regional enterprises committed to developing a support  network of technical  services 
for  local  subcontractors  as well  as promoting  technology  transfer. The CREATI has 
also  been  involved  in a number  of projects  launched in partnership  between  partici­
pants and SMEs. These projects  have also benefited from co-operation  with  regional 
universities. 

Outputs 

> 

Effects 

O 

Advice  sessions 
financed  with 
experts in 
technology  Issues 

SMEs  entering  into 
collaborative  arrangements 

Production  of new 
products,  processes and 
services  incorporating 
acquired  technology 

Evaluating impacts in the domain of RTD 

• Human resources 

Interventions in favour of human resources cover such measures as financing 
university research grants in specific areas (e.g. technology transfer, etc.), 
vocational training in high­tech subjects, encouraging links between public 
research and industry (e.g. placements for researchers in industry, etc.). 

Example: Centros Tutelados (Spain) 

This project, set up by the Basque government in 1992, aims to develop a technologi­

cal network, support growth and create j o bs in local industries. The Centres support 

the development of local enterprises by undertaking their o wn research, by monitoring 

of scientific p r o g r e ss a c r o ss the w o r l d, a nd by their t r a i n i ng p r o g r a m m es w h i ch 

represent one of the major aspects of the centres' activities. Highly qualified technical 

and research staff undergo a t wo year training period in the Centres before going to 

work in regional SMEs to ensure that the knowledge they have gained is diffused into 

local industries. The Centres also organise training courses for local enterprises. Up to 

1997 over 1,400 trainees in nearly 900 enterprises had benefited from this service. 

Outputs 

> 

Organisation of 
external training 
courses 

Financing of places 
for technicians and 
researchers within 
the Centres 

Effects 

> 

Increased qualifications and 
skills of employees of 
client firms 

Increased overall skills level 
within participant firms 

Placing of researchers 
in regional companies 

Companies producing 
new or improved 
products, processes 
or services 

■ Defining and selecting indicators 

The evaluation of RTD interventions requires the construction of a system of 
indicators using both standard RTD indicators and ad hoc indicators that 
take into account the particularities of the specific evaluation. The monito­
ring of interventions should generally provide data for resource, output and 
result indicators, whilst the gathering of information on impacts requires the 
use of active data collection techniques. 

Resource indicators such as GERD, BERD and GOVERD5 give information 
on the structure of RTD financing and the consequent distribution of activity 
between public research and industrial RTD. When related to population or 
GDP, comparisons can be made between regions and countries. 

5. GERD ­ Gross Domestic Expenditure on Research and Development; BERD ­ Business 
Expenditure on Research and Development; GOVERD ­ Government Expenditure on 
Research and Development. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Output  indicators  focus  on the activities  of  operators  (contacts  with  addres­
sees, applications  processed, etc.), the number  of  hours of consultancy  and 
training sessions, number of supported research projects, networks and colla­
borative ventures financed, and the creation of infrastructure such as science 
parks, research facilities, etc. (measured in square metres or hectares). 

Result  indicators, in the case of  RDT infrastructure could  provide  information 
on the quality /  cost ratio of, for example, establishing a firm in a science and 
technology  park, or the  attractiveness  of the  park and  its environs to  resear­
chers. As for technology  services, results  indicators  largely focus on  measure 
addressees'  satisfaction.  Indicators  at  this  level  should  also  measure  the 
amount  of  private  money  invested  in the  supported  projects,  the  leverage 
effect  of an intervention. Another  useful  indicator  is the proportion of addres­
sees that undertaking RTD for the first time with the support of the programme. 

Results  indicators  should  also  provide  information  on  networking  between 
firms and institutes when this is the objective of an intervention. 

Impact  indicators  focus  on  wider  and  more  long-term  technological  and 
socio-economic  impacts. Because the developmental  effects  of  RTD occur 
over a long period and some specific impacts are difficult to measure (i.e. the 
knowledge  and  know-how  acquired  by  individuals),  impact  indicators  are 
generally limited to those informing on specific socio-economic  impacts of a 
more  tangible  nature,  for  example,  the  number  of  new  products  and 
processes  marketed,  their  sales  and  the  sales  of  related  patents  and 
licences, increased value added and direct employment effects  (researchers 
and technicians measured in FTEs). Because public and private research are 
increasingly  intertwined  -  indeed,  collaboration  between  public  research 
centres  and  SMEs  and  the  joint  development  of  innovations  is  one  of  the 
objectives  of  interventions  -  many  institutes  are  themselves  starting  to 
behave  like  private  firms:  patenting  innovations,  selling  licences  and 
marketing products, etc. Indicators for university  and research centre activi­
ties  within  the  framework  of  Structural  Fund  interventions  should  therefore 
focus on the more industrial and commercial manifestations  of public  sector 
research. 

By way of an example, a system of indicators that could be used to track the 
production of impacts arising from a project providing financial and informa­
tional  support  for  technology  acquisition  by  SMEs  is  presented  in  Box  34. 

Evaluating  impacts  in  the  domain  of RTD 

Box 34 -  Example  of a system of indicators for an  intervention 

providing financial and  informational  support for 
technology  acquisition  by  SMEs 

Programme  Indicators 
(Related to the intervention  and its effects) 

Context  Indicators 
(Related to the assisted  area) 

Output  indicators 

Activity of the 
operators 

Number  of contacts  between  operators 
andfimns looking for support  for 
technology  acquisition 
Number of advice sessions  organised 
with  experts  in technology  acquisition 

Number of addressees  Number  of SMEs receiving  support 

Volume of technical 
services  provided 

Hours of  consultancy  and 
training  services  dispensed 

Result  indicators 

Satisfaction 

%  of addressees  satisfied or  very 
satisfied with the  service 

Leverage effect 

Investment  in acquisition oriented  RTD 
by firms  benefiting from  financial  support 

Ratio of  BERD to  GERD 

Impact  indicators 

Internal technology 
transfer 

External technology 
transfer 

Technology 
acquisition 

Economic  impact 

Number  of  collaborative  arrangements 
between  research institutions  and 
assisted enterprises  after  one year 

Number  of assisted  firms  purchasing 
patents, licences  or entering  Into R&D 
collaboration  with foreign firms  after 
one year 

Number  of  new  products  or  processes 
incorporating  the acquired  technology 
marketed  by  assisted  firms  after 
one year 

% of  managers  of recipient firms who 
declare  that  the  Intervention  has 
relaxed constraints  in terms  of  access 
to new  technologies 

%  of  regional  managers  who  declare 
that  access to  new  technologies 
is a major  constraint 

Value added 

Employment 

Value added generated  after  one/three 
years in euro /  year 

Value added  generated 
in the  region  by  SMEs 

Net  employment  created after 
one/three years  (of which  women) 

Total number  of jobs  in SMEs 
(FTEs of  which  held by  women) 

Source: Volume 2 of  the MEANS  Collection 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Whilst these  indicators  concentrate  on the quantification  of  outputs,  results 
and impacts, it is also useful to have qualitative information about the effects 
of  interventions.  For example, data on the  number  of collaborative  projects 
between  research  institutes and firms, or the  number  of  innovations  arising 
from them can be enriched by qualitative information about the venture. But, 
because  this  sort  of  collaboration  is  intended  to  improve  the  competitive 
position  of  the  participating  enterprises,  for  reasons  of  commercial  confi­
dentiality  standard peer review or expert  panel procedures used for the eva­
luation  of  public  research  institutes  are  not  particularly  suited  to  this  task. 
However,  partners  could  be  asked  to  assess  the  quality  of  their  venture. A 
possible scoring  system for  measuring the  quality  of collaborative  ventures 
is presented in Box 35. 

Box 35  -  Example  of a score  card for  a joint  research  venture 

Assessment  of  the: 

•  Strategic  relevance  of the joint  research  activity  to  your  commercial 

activities 

0 

12 

3 

4 

•  Quality  of  the  research  performed  within the  project 

0 

12 

3 

4 

•  Quality  of  the  contribution  made  by your  partner(s) 
0 

12 

4 

3 

•  Extent  of  learning  engendered  in your  organisation  by 

the  partnership 

0 

12 

3 

4 

Legend: 
0  -  Very Poor;  1 - Poor; 2  - Low;  3  -  Good;  4  -  Excellent 

Evaluating impacts in the domain of RTD 

■ Suitable sources of information 

When evaluating RTD interventions within the framework of the Structural 
Funds, it is important to remember that the tools used must be suited to 
examining complex phenomena in detail. One must also bear in mind that 
the availability of monitoring data at the regional level is sometimes rather 
patchy, so that in some regions primary data collection will be an expensive 
and time­consuming part of an evaluation. 

Whilst the evaluation of RTD interventions is therefore likely to be heavily 
dependant upon primary data, sources do exist which provide relevant sta­
tistics at the firm level. 

Innovation surveys focus either directly on innovations or on firms and their 
innovatory activities. In the first instance, survey data on innovations 
deemed technologically or economically significant are collected. The 
second type has the advantage of taking into account both innovatory and 
non­innovatory firms as well as the users of innovations, allowing factors 
which restrict innovation and the existence and intensity of user­producer 
linkages to be examined. 

Data on patents is also available at the firm level, giving the technological 
classification of the innovation. Unlike data resulting from innovation 
surveys, patent data is available over a long period. Patents registered at the 
European Patents Office are available on the EPAT database, which has 
been in operation since 1978. 

Data are generally available at the national level (e.g. l'Observatoire des Sciences 
et les Techniques produces an annual report on S&T indicators for France) and 
statistics on national R&D expenditure are provided annually by the OECD. 

Relatively up­to­date European RTD statistics suitable for constructing 
context indicators are available from Eurostat in an annual publication which 
covers data on R&D personnel, R&D expenditure and patent applications 
broken down by region as well as government budget R&D appropriations. 
This source also provides "nearcasts" that provide estimates of current sta­
tistics that are being compiled. 

■ Appropriate tools for evaluating structural interventions 

in the domain of RTD 

While a detailed treatment of evaluation tools is provided in Volume 3 of the 
MEANS Collection, a number of tools particularly suited to evaluating 
impacts on RTD are presented below. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

While a great deal of experience  has been accumulated  in this domain, and 
standard  tools  for  this  type  of  intervention  have  a  long  track  record,  the 
increased  importance  of  interventions  towards  generating  "opportunities" 
for innovation through the creation of better conditions for learning, partner­
ships and networks, has led to new challenges in the area of RTD evaluation. 

First, these  effect  arising  from  RTD interventions  are difficult  to  realistically 
reduce to  simple  monetary variables, which  means that tools such  as cost-
benefit  analysis  and  return on  investment  type  analyses  are of  limited  use. 
This  latter  technique  is  restricted  to  analysing  private  returns  arising  from 
RTD expenditure, it therefore significantly underestimates total returns, since 
a large part of these are social returns generated through RTD spillovers  and 
externalities  (i.e. effects, either positive or negative, that to some extent take 
place  through  non-market  mechanisms  and  are  therefore  not  fully 
accounted for within the price system). 

Secondly,  because of the complexity  of the chain of  causality  between  RTD 
activities  and  their  socio-economic  effects,  increased  emphasis  has  to  be 
placed  on the  use of  tools  allowing  in-depth  qualitative  study. Thirdly,  the 
recognition  of  the  effects  of  RTD  spillovers  on  the  performance  of  firms 
requires  that  evaluations  simultaneously  take  a wider  approach  than  that 
afforded  by  project-by-project  examinations  of  interventions.  Finally,  the 
evolution in the nature of RTD interventions, the increased emphasis on eva­
luating  interventions  in the  area of  industrial  RTD and the  need to  take fully 
into account the spatial objectives of structural interventions, mean that eva­
luation techniques  in this domain are less able to rely on classic  RTD evalua­
tion tools such as expert  panels and bibliometric  indicators  in which a great 
deal of experience has been accumulated. 

In terms  of  the  alternative  approach  required, concrete  evaluation  expe­
rience  remains  rather  limited. The  tools  presented  below  are those  which 
have to a limited extent been used in evaluations in this domain or seem par­
ticularly suited to examining impacts of RTD oriented  interventions. 

By using questionnaire  surveys that  are carefully tailored to the characteris­
tics  of  the  population  concerned,  relatively  simple,  and  clearly  structured 
and designed to cover of the RTD themes depending on their importance  to 
the region or regions studied, evaluators can build up a picture of a region's 
innovation  system  in terms of the type and extent  of interventions, how they 
are implemented, the nature and extent of industrial  RTD, the effect of  inter­
ventions  on  regional  RTD  policies  and  the  organisation  of  RTD  activities. 

Evaluating  impacts  in  the  domain  of RTD 

Whilst informing on resource use, issues of organisational and  management 
quality, the  effectiveness  of  instruments  and  mechanisms,  surveys  should 
largely  focus  on  providing  quantitative  data  on  the  outputs  and  results  of 
interventions.  Questionnaires  may  also  have to  be  designed  to  reflect  the 
cover of programmes (i.e. regional or multi-regional). 

Experience from the EUREKA Initiative has shown that questionnaires  alone 
are  not  able  to  capture  the  richness  and  complexity  of  socio-economic 
effects  arising  from  RTD  interventions.  Furthermore,  experience  from 
Norway has shown that such impacts as extra sales, productivity  gains, net­
working  benefits  and  increased  skill-levels  may  take  over  ten  years  to 
become  evident.  Questionnaires  therefore  need  to  be  supplemented  by 
face-to-face interviews with key figures in the domain of RTD, focusing  both 
on  actual  and  expected  impacts.  The  interviewees  will  be  drawn  from 
programme  actors:  regional  and  local  programme  sponsors,  universities, 
etc.  More  importantly,  they  will  also  include  addressees  such  as  research 
centres,  SMEs  and  larger  companies  involved  in  programmes  along  with 
firms not directly involved (e.g. firms buying process innovations, rather than 
developing them themselves). 

Expert  panels  can  be  used  for  evaluating  interventions  targeted  at  public 
research. However,  because secrecy  is one of the ways firms  maintain their 
innovative  edge,  they  are  of  limited  use  when  private  or  collaborative 
research  activities  need to  be evaluated. On the other  hand, expert  panels 
are useful tools for undertaking  "quick and dirty" evaluations, when these are 
required. 

A number of carefully selected case studies can be used to provide  in-depth 
analysis  and comparison  of  individual  projects  and  measures. These  might 
focus  on projects of outstanding quality  in the area of the promotion of RTD 
capability.  This  technique  may  be  particularly  useful  in examining  internal 
(e.g. the setting  up of a RTD department)  or external (e.g. linkages  between 
firms or between firms and research centres) organisational changes engen­
dered by interventions. Case studies may also cover an entire region in order 
to better appreciate both the functioning of regional innovation systems and 
the  long-term  sustainability  of  a  region's  development.  In this  case  they 
might  look  at  competitiveness,  economic  development  and  employment, 
focusing on the demand side of RTD. Case studies should be designed with 
issues of extrapolation, aggregation and comparison in mind. 

Transversal evaluation of impacts on intervention priorities 

■ S u m m a ry 
The importance of RTD, especially at the industry level, is recognised as an 
essential prerequisite of sustained economic development. This recognition 
has led to a large increase in Structural Funds expenditure in this domain, 
accompanied by the reorientation of interventions towards downstream 
technological development. 

When estimating the net impacts of interventions, in addition to identifying 
deadweight additionality of Structural Fund expenditure is an important 
factor that should be taken into account. The assessment of leverage effects 
also represents an important element in appreciating the impact of interven­
tions on RTD. 

The evaluation of RTD interventions poses particular problems because of 
the intangible nature of many specific impacts and the long­term nature of 
socio­economic impacts. Moreover, the increased emphasis on the role of 
private sector RTD and the need to evaluate at a higher level than individual 
projects, mean that evaluations are less able to rely on assessments of 
expert panels, the established tool of RTD evaluation. 

■ For more information see... 

Archibugi D. and Pianta M. (1996), "Innovation surveys and patents as tech­
nology indicators: the state of the art" in: Innovation, Patents and Technolo­
gical Strategies. Paris: OECD. 
Commission européenne (1998), 2e Rapport européen sur les indicateurs 
scientifiques et technologiques 1998, Luxembourg: Office des publications 
officielles des Communautés européennes. 

European Commission (1999), Impact of Structural Funds (1994­1999) on 
Research Technology Development and Innovation (RTDI) in Objective 1 and 
6 Regions, European Commission, DG XVI. 
Eurostat (1997), R&D Annual Statistics 1997. Luxembourg: Eurostat. 
Georghiou L. (1998), "Issues in the Evaluation of Innovation and Technology 
Policy", Evaluation, Vol 4(1), 37­51. 
Lundvall B.­A. (éd.) (1992), National Systems of Innovation: Towards a 
Theory of Innovation and Interactive Learning. London: Pinter. 
OECD. (1994), The Measurement of Scientific and Technological Activities: 
Proposed Standard Practice for Surveys of Research and Experimental 
Development ­ Frascati Manual 1993. Paris: OECD. 
OECD. (1998), Policy Evaluation in Innovation and Technology: Towards 
Best Practices, OECD Proceedings, OECD: Paris. 
OST. (1996), Science & Technologie: Indicateurs 1996. Paris: Economica. 

Evaluating impacts arising from interventions in the 
domains of transport, telecommunications and tourism 

A small amount of experience in thematic transnational evaluations has been 
gained in the domains of transport, telecommunications and tourism. This 
experience can be characterised as both limited in terms of its transnational 
aspect and predominantly top­down in terms of evaluation design. The 
ambition of this chapter is to draw on this available experience and to com­
plement it with a number of points that should be considered when evalua­
ting interventions in these domains. 

■ Transport and telecommunications 

• Structural interventions in the domain of transport 

Structural Fund interventions in the domain of transport are part of a larger 
Community transport policy which also includes EIB funding and the Trans 
European Network (TEN) initiative. In addition to these instruments focused 
mainly on the funding of transport infrastructure projects, a substantial body 
of legislation, geared towards the harmonisation and liberalisation of market 
conditions within particular transport sectors, now exists. 

Transport infrastructure projects are funded by means of grants and loans 
that are largely allocated on the basis of project­specific appraisal and 
regional development strategy. For the cohesion countries, from both a 
physical and financial point of view, the planning and financing of transport 
infrastructure are among the most important areas of EU policy. 

Transport infrastructure investments are intended to affect socio­economic 
cohesion in two major ways: 
• by eliminating bottlenecks and reducing travelling times, costs and constraints 
on industry (e.g. of transporting raw materials, semi­finished and finished 
products, etc.) are reduced and economic productivity is improved (e.g. by 
allowing new organisational techniques such as just­in­time operations); 
• by increasing accessibility and mobility between regions, therefore 
reducing the effects of peripherality on business and individuals. 

This traditional argument is that improvements in transport and, in particular, 
transport infrastructure, foster economic development and reduce dispari­
ties between regions. This hypothesis is, however, questioned by some who 
argue that improved infrastructure may in some cases worsen disparities, at 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

least  in the  short-term,  by  opening  up  regions  to  imports  originating  from 
lower-cost  sources.  In addition to this  possible  undesirable  effect,  interven­
tions  within  the  framework  of  the  TEN  initiative,  which  account  for  the 
majority of investments  in this domain, are not aimed at achieving  cohesion 
but at increasing the overall competitiveness  of the  EU as a whole. Interven­
tions of this kind may therefore have greater positive effects in non-cohesion 
countries  in which  the  bulk  of  the  investment  in TEN  projects  occurs  and 
which  are  more  favoured  in  terms  of  accessibility  by  their  geographical 
position.  Nevertheless,  the  role  of  the  Community  in  identifying,  planning 
and  co-financing  the  TEN  is seen  as  an  important  factor  in countering  the 
national bias of most transport infrastructure projects. It also seems  possible 
that the tendency  of  improved  infrastructure to attract  individuals  and  busi­
nesses  to  the  more  dynamic  regional  poles  of  activity  at  the  expense  of 
cohesion  may be largely overestimated.  In the medium to  longer term it can 
be expected that there will be a degree of rebalancing of activity as the costs 
and  disadvantages  of  large  agglomerations  (e.g. office  rents,  pollution, 
congestion, etc.) start to outweigh the benefits. 

In addition  to  the  upgrading  and  construction  of  infrastructure,  there  is  a 
growing  emphasis  on  more  efficient  use of  existing  infrastructure  through 
high-technology  solutions  to  problems  of  road  congestion  and  transport 
related  environmental  problems  (e.g. Advanced Transport  Telematics  tech­
nologies). The effect  of this development  on cohesion  may  prove to  be  less 
than  was  bargained  for  since  this  sort  of  solution  seems  more  suited  to 
developed rather than peripheral parts of the Community. 

The  provision  of  transport  infrastructure  may  also  have  negative  effects 
arising  from  such  problems  as  road  congestion,  due  to  the  apparent 
capacity  for  demand  for  road space to  outstrip  supply.  Negative  effects  on 
the quality of life may result from air and noise pollution generated by the use 
of the infrastructure. 

The evaluation of the effects of interventions in this domain will depend  upon 
what  is considered  as the  contribution  of  transport  to  the  economy.  That 
contribution  largely  translates  into  reduced  costs  and  constraints  for  busi­
nesses  and  individuals.  It  also  has  a  strong  qualitative  aspect,  notably  in 
terms of safety and environment. 

•  Structural interventions in the domain of telecommunications 

The transition  from  an  industrial  to  an  information-based  service  economy 
holds  a  number  of  opportunities  for  the  more  peripheral  regions,  since  the 

Evaluating  impacts  of  interventions 

in  the  domains  of 

transport, 

effects  of  geographical  distance  on  business  activity  is  consequently 
reduced.  However,  the  possibility  of  benefiting  from  these  opportunities, 
depends  on these  regions  being  well  equipped  in télécoms  infrastructure 
and services. 

In light  of  advances  in technology,  which  have meant  that  telecommunica­
tions and  information technology  are increasingly  indistinguishable and that 
national telecommunications  monopolies  are  no  longer  tenable, the  role  of 
the EU has been twofold: 
•  First, it has put into place a common framework for a liberalised  European 
télécoms sector within  which the Member States  regulate their telecommu­
nications industries; 

•  Secondly,  it has promoted collaborative télécoms  RTD and supported  the 
modernisation of télécoms infrastructure. 

The effects of liberalisation policies in the domain of telecommunications  will 
lead to  an overall  improvement  in service  quality  and  efficiency.  However, 
their short-term  effects will  probably  accentuate  problems of cohesion. The 
immediate  effects  will  be  a  reduction  in  costs  for  heavy  consumers  of 
télécoms services in the more developed regions as the  cross-subsidisation 
of  services  stops  and  tariffs  are  rebalanced  to  reflect  real  costs.  For 
consumers  in  more  peripheral  regions,  the  same  process  may  lead  to 
increased  costs. Any  sizeable  positive  impact  occurring  in these  regions  is 
likely to be seen over the medium to  long-term as advantages  of  locating  in 
less  central  regions  outstrip  the  agglomeration  benefits  of  a  more  central 
locations. 

The  structural  change  brought  about  by  advances  in  telecommunications 
technology  not only  has an effect on institutions  and organisations, but also 
on the way  individuals  live their  lives. This  has led to  use of the term  "Infor­
mation  Society"  to  encompass  the  broader  aspects  of  this  process.  A 
dimension  of  increasing  importance  in the  "Information  Society"  is that  of 
telework  which  enables  employees  and  independent  workers  to  work  in 
remote  locations  using  the  telecommunication  network  to  receive  and 
transmit data, market their products, search for clients, etc. 

An  increased  focus  on these  broader  aspects  has  led to  the  EU playing  a 
highly  visible  role  in this  area.  However,  infrastructure  and  services  are 
mainly  financed  by the  private  sector  and  Structural  Fund  expenditure  has 
remained  modest,  amounting  to  less than  3%  of total  telecommunications 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

investments. Nevertheless, in lagging regions it has played an important role 
in the development of modern telecom  networks. 

In the case of telecommunications  Trans-European  Networks, the role of the 
Commission  is that of sharing financial  risk  by providing support  for  econo­
mically  beneficial  projects  which  are  of  common  interest  but  uncertain  or 
inadequate  in terms  of  financial  profitability.  Support  is provided  through  a 
number of  instruments: 
•  co-financing of feasibility studies and other technical  measures; 
•  subsidised interest rates; 
•  contributions to loan guarantee premiums. 

•  Some key issues for evaluations in the domains of transport and tele­

communications 

A  large proportion  of  Structural  Fund expenditure  in these two  domains  has 
tended  to  be concentrated  in support  measures  for  infrastructure  projects, 
especially  in the case of transport. Whilst this situation  is likely to continue in 
the foreseeable future, the increased interest in "software" transport  solutions 
and  Information  and  Communication  Technologies  (ICT) applications  deve­
lopment  means  that  less tangible  effects  will  need  to  be  increasingly  taken 
into account  when evaluations are performed. These two domains  have also 
been  the  subject  of  deregulation  policies  that  are  having,  or  will  have 
important  structural  repercussions. The  identification  of  synergies  (either 
negative  or  positive)  between  liberalisation  policies  and  regulatory  frame­
works  and  interventions  with  explicit  cohesion  objectives  represent  an 
important  element  of  evaluation. Another  important  aspect  of  infrastructure 
projects,  relating  to  cohesion,  is their  effect  on  accessibility  to  peripheral 
locations. 

Infrastructure  projects are generally  large and relatively few in number. They 
usually  concern  projects  that  either  take  the  form  of  the  construction  or 
modernisation  of  a  link  between  a  number  of  nodes  (e.g. roads,  railways, 
cables,  pipes,  etc.)  or  are  nodes  themselves  (e.g.  ports,  multimodal 
exchanges, and telephone exchanges). 

One  way  of  establishing  the  relative  importance  of  infrastructure  projects 
prior to  selecting a number  of them for  more detailed  analysis (i.e. for  inter­
mediate or ex-post  evaluation)  is, for example, to  rank them  in terms  of  the 
size of the intervention in budgetary terms. The importance of TEN projects, 

Evaluating  impacts  of  interventions 

in  the  domains  of 

transport, 

for example, is reflected  by a rating system which identifies  priority  projects 
and  other  projects  according  to  their  size. This  approach  is not,  however, 
necessarily  sufficient  to  comprehend  the full  importance  of  a project.  Other 
factors  that  could  be taken  into  account  when  ranking  projects  include 
social  and  financial  benefits  of  the  project,  and  the  participation  of  the 
private  sector.  Since  each  project  is  a component  of  a  system,  it  is  also 
important  to  examine  their  strategic  importance  within  the  system  itself.  It 
may  be  possible to  do  this  by examining the  importance  of the corridor  on 
which  the  project  is  situated,  in  relation  to  other  corridors  in the  region. 
Similarly,  it  may  be  possible  to  classify  nodes  according  to  their  strategic 
importance in the system. 

Interventions  in transport  infrastructure  occur  at the  start  of  a  long  causal 
chain.  Because  of  the  distance  between  an  intervention  and  its  eventual 
socio-economic  impacts (i.e. at the level of employment and GNP growth), is 
difficult  to  establish  clear  causal  links  between  this type  of  project  (i.e. the 
outputs  of  an  intervention)  and  these  global  impacts.  Evaluations  should 
therefore  concentrate  on  tracking  outputs,  results  and  specific  impacts  of 
interventions.  Furthermore,  because  medium  to  long-term  effects  of  infra­
structure  projects  on  cohesion  will  probably  to  some  extent  contradict  the 
short term  effects  (i.e. over time  economies  of agglomeration  may  become 
negative  and  relative  prices  will favour  locations  outside  central  poles  of 
activity), the timescale  of  evaluation  is of  the  utmost  importance  if the  full 
impact of an intervention is to be observed. 

In  addition  to  the  direct  effects  that  supported  infrastructure  projects 
produce,  they  also  give  rise  to  indirect  or  induced  effects  by, for  example, 
reducing  constraints  on the economic  activity  of indirect addressees. In this 
way  they  can  give  rise to  supply-side  effects  by  making  productive  struc­
tures more efficient  (e.g. by reducing delivery times) and reducing costs (e.g. 
by  reducing  distances,  travelling  times,  etc.). They  also  stimulate  demand 
(demand-side  effects)  by  mobilising  human  and  material  resources  during 
the construction period. 

While  Volume  2  of  the  MEANS  Collection  presents  an  information  sheet 
devoted to indicators for the domain of transport infrastructure, by way of an 
example, a system  of  indicators that  could  be used to track the  production 
of impacts arising from an upgrading project for a railway line is presented in 
Box 36. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Box 36  -  Example  of a  system  of indicators for  upgrading 
of a  railway  line for freight  operations 
Programme Indicators 
(Related to the Intervention and its effects) 

Context Indicators 
(Related to the assisted area) 

Output indicators 

Progress 

Quantity 

Quality 

Result indicators 

Speed 

Impact indicators 

Traffic flow 

Transport system 

Indirect economic 
effect 

Compliance with project duration 
Rate of completion 

Number of km upgraded 

% of line allowing speeds of at least 
a desired threshold value 
(i.e. 100 km/h) 
% of line length allowing trains of 
heaviest standard load 

Typical commercial speed of 
freight trains in that line 

Average speed on regional lines 

Freight traffic flows on the line 
(Tonnes / year) after one year 

Traffic flow of freight using the new 
infrastructure and withdrawn from road 
(Tonnes / year) 

Traffic flow to and from region 
(Tonnes / year) 

Modal distribution of traffic flows 
within the region (Tonnes / year) 

% of regional managers who declare 
that rail freight accessibility ¡s a major 
constraint 

Source: Volume 2 of the MEANS Collection 

Interventions  in telecommunications  infrastructure are also difficult to  link up 
to  socio-economic  impacts.  In this  case, evaluation  at the  impact  level  will 
normally  focus  on the  increased  use (e.g. penetration  rate, changes  in tele­
communications  traffic,  etc.)  of  teleservices  and  equipment  by  enterprises 
and individuals, dueto  improvements in the network. Where telecommunica­
tion  services  supported  by  Structural  Fund  interventions  are  concerned, 
these tend to be of an experimental nature (e.g. support for the setting up of a 
telework centre, support for on-line education services, etc.). This means that 
as well  as  looking  at  any  specific  socio-economic  impacts  that  they  may 
produce, the survival of projects is an important evaluation issue. 

While  Volume  2  of  the  MEANS  Collection  presents  an  information  sheet 
devoted to  indicators for the domain of telecommunications, an example of 
indicators that could  be used to track  impacts  produced  by the  deployment 
of a broadband  Hybrid Fibre Coax (HFC) network is presented in Box 37. 

Evaluating impacts of interventions in the domains of transport, 

Box 37 ­ Example of a system of indicators for the deployment 

of a broadband Hybrid Fibre Coax (HFC) network 

Programme Indicators 
(Related to the intervention and its effects) 

Context Indicators 
(Related to the assisted area) 

Output Indicators 

Progress 

Quantity 

Result Indicators 

Network performance 

Services supplied 

Impact indicators 

Connections 

% of HFC kms in the total network 
(In the considered region) 
% of buildings / apartments passed 

Compliance with project duration 
Rate of completion 

Number of HFC kms laid 
Number of buildings/apartments 
passed 

Bit rate Improvement 
Number of network failures 

Number of services 
Number of TV channels 
Hours of broadcasting supplied 
(through the network) 

Number of households that have 
connected their equipment (TV, PC) 
to the HFC network (after 6 months) 

Growth of PC and / or CATV 
penetration in households 
in the considered region 

ICT usage 

Number of subscriptions to the services 
Total hours of connections / month 

Growth in teleservices adoption rate 
In households in the considered region 

Source: Volume 2 of the MEANS Collection 

Given that the main actors in the domain of telecommunications are in the 
private sector and that some potentially profitable projects, which might 
have been financed exclusively by private actors, may be assisted by Struc­
tural Funds, the major preoccupation in estimating net effects is the need to 
identify deadweight (i.e. those investments that would have taken place in 
the absence of the intervention and their associated effects). 

■ Tourism 

• Structural interventions in favour of tourism 

The objective of interventions in this domain is to support the diversification 
of regional economies away from excessive dependence on agriculture and, 
to a lesser extent, on declining industries. However, while tourism has the 
capacity to play an important role in supporting socio­economic cohesion, it 
is generally characterised by a number of structural problems: 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

•  tourism demand  is highly seasonal, leading to infrastructures  being  under­
utilised for substantial  periods of the year, and to seasonal / casual  employ­
ment patterns; 

•  value  added  generated  by  tourist  activities  is often  weak,  leading  to  low 
profitability; 
•  product quality is highly variable; 
•  levels of training are relatively low; 
•  transport access is a limiting factor to tourist  development. 

Furthermore, the  development  of  tourism  needs  to  be  managed  in such  a 
way  as to ensure its sustainability  (i.e. by concentrating  on activities that  do 
not produce negative environmental  impacts and by mitigating any  negative 
impacts that do occur). 

Interventions  in the  domain  of  tourism  are therefore  essentially  aimed  at 
resolving these  problems  by, for  example, supporting  the  provision of  infra­
structure, widening  the  range of tourist  products, and  so  on.  Interventions, 
such as support  for the construction  of conference facilities, can  contribute 
to  reducing  seasonality  by  promoting  all  year  round  conference  tourism. 
Similarly, the development  of  new attractions  can  lead to  reduced  pressure 
on established tourist sites during the peak season. This type of  intervention 
also  has  an  impact  on  transport  access  (e.g. by  smoothing  out  seasonal 
travel  needs)  and  on  product  quality  and  profitability  (e.g. by  encouraging 
clients who demand higher quality products and services). 

Since tourism is a labour intensive activity relying heavily on locally  supplied 
products  and locally  based services, it is a domain  in which development  is 
largely endogenously  generated and which  has potentially  important  effects 
on employment and living standards, and thereby on the promotion of socio­
economic cohesion. 

•  Some key issues for evaluations in the domain of tourism 

Although certain interventions support relatively large and visible infrastructure 
projects, a large number of interventions in this domain are fragmented, hete­
rogeneous and  relatively small (e.g. support  for  improving  tourist  accommo­
dation, support for museums, heritage centres, training courses, etc.). 

The  development  of  tourism  is  heavily  dependent  on  qualitative  factors. 
Many  tourists  are  discretionary  visitors  (i.e. they  are  not  constrained  by 

Evaluating  impacts  of  interventions 

in  the  domains  of 

transport, 

factors such as business and family needs to visit a particular region), which 
means that they are highly sensitive to the quality and price of the product or 
service they  buy. The fact  that  tourism  has traditionally  had the status  of a 
commodity  is reflected in its lack of variety and its variable quality (as well as 
low  profitability  and  job  quality,  if  measured  in terms  of  low  training  levels 
and the casual nature of employment). Evaluation of interventions that aim to 
support  the  development  of  tourism  must  therefore  take  these  qualitative 
factors into account. 

In this  domain  it  is  highly  probable  that  a  number  of  projects  would  have 
been undertaken  In the absence of Structural  Fund interventions  (e.g. hotel 
modernisation, golfcourses, campsites, etc.). The calculation of deadweight 
is therefore  important  when  estimating  net  effects.  Since one  of  the  objec­
tives  of  interventions  in tourism  is to  develop  new  attractions  to  reduce 
pressure on established tourist sites during the peak season, substitution  is 
often an aim of interventions. On the other hand, visitors are only able to visit 
a limited number of sites during a stay. This implies that assisted sights  may 
benefit  at the  expense  of  unassisted  sites.  It  is therefore  necessary  to  be 
aware of possible displacement  effects. 

By  way  of  an example,  a system  of  indicators  for  tracking  the  creation  of 
impacts  by  interventions  supporting  the  creation  of  tourist  facilities  is 
presented in Box 38. 

Transversal evaluation of impacts on intervention priorities 

Box 38 ­ Example of support for the creation of tourist facilities 

(museums, theme parks, etc.) 

Programme Indicators Context Indicators 

(Related to the intervention and its effects) (Related to the assisted area) 

Output Indicators 

Activity of the 
operator 

Number of contacts with potential 
addressees 

Number of addressees 

Number of assisted economic units 

Total number of tourist facilities 

Capacity 

Maximum number of visitors per day 

Result Indicators 

Duration of visit 

Time normally required to visit 
the facility (hours) 

Cost of visit 

Average cost per visit of the facilities 
(euro per person) 

Impact indicators 

Fréquentation 

Attractiveness 
to Tourists 

Value added 

Jobs created 

Number of visits per year for supported 
facilities 

% of visits made by tourists from 
outside the region 

Value added generated in euro 
per year 

Value added generated In the 
tourist industry In euro / an 

Net jobs created (% of which are 
occupied women) 

Number of jobs in the tourist industry 

Source: Volume 2 of the MEANS Collection 

A more detailed treatment of indicators and other examples relating to 
tourism are provided in Volume 2 of the MEANS Collection devoted to indi­
cators for monitoring and evaluation. 

■ Examples of tools for evaluating structural interventions 
in t he d o m a i ns of transport, t e l e c o m m u n i c a t i o ns and 
tourism 

Transnational evaluations in the domains of transport and telecommunica­
tions have tended to be approached in a top­down fashion, relying heavily 
on documentary research and secondary data complemented by interviews 
with key actors, expert reports and, in some instances, case studies and the 
scores and weights technique. 

Evaluating 

impacts  of  interventions 

in  the  domains  of 

transport, 

The  objective  of  this  section  is to  present  a small  number  of  examples  of 
tools  used in these domains  within the framework  of transnational  thematic 
evaluations.  A  more  detailed  treatment  of  evaluation  tools  is  provided  in 
Volume 3 of the MEANS Collection. 

•  SWOT: Example of the strategic overview of transport issues for the 
Republic of Ireland 

SWOT analysis (Strengths, Weaknesses, Opportunities and Threats) is parti­
cularly  useful for  performing  a strategic  analysis  at the time  of the  concep­
tion or evaluation  of an infrastructrure  project.  Its implementation  involves a 
number of steps: 
1. Scanning the key issues and problems faced by a region; 
2. Ordering priorities; 
3. Identifying opportunities and threats (factors outside the project, region, etc.); 
4.  Identifying  strengths  and weaknesses  (factors  related to the  characteris­
tics of the project, region, etc.); 
5. Synthesising results and analysis of the SWOT matrix; 
6. Formulating strategies. 

In this  particular  example  a SWOT analysis was  undertaken to  examine the 
position  of  the  Republic  of  Ireland  with  regard  to  the  national  transport 
sector  and  in relation to  the overall  context  in which  it exists. The analysis, 
based on the factors summarised  in the matrix  presented  in Box 39, comes 
to  the  conclusion  that  the  overall  position  of  transport  for  the  Republic  of 
Ireland is one of weakness and inadequacy. This analysis allows the elabora-

Box 39  -  Example  of a SWOT analysis  matrix 

Strengths 

Weaknesses 

Industrial base relatively modern 
Tradition of open economy 
Educated workforce 
Unspoiled environment 
Proximity to sea 
Political stability 

Small size, peripheral location 
Poor indigenous resources 
Quality of domestic infrastructure 
Unsatisfactory access to transport 
Limited finance 
Unemployment, social problems 

Opportunities 

Threats 

SEM - access to mainland 
Role as entrepot for the USA? 
Shannon as European air hub? 
Suited to tourism and leisure 
Telecoms based services 
Mariculture, natural energy 

European centre moving eastwards 
Uncertainty of future EC funding 
Scotland and Wales, post Channel Tunnel 
Large industrial alliances 
Unstable Europe- social / economic 
Emigration of workforce 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

tion  of  strategy  options  aimed  at  eliminating  identified  weaknesses  or  at 
compensating for them through the development  of identified strengths. 

•  Scores and weights: Example of the  "Corridor Ranking  Framework" 

(Greece, Ireland and Portugal) 

This tool  is often  used  in the ex ante appraisal  of  projects, using a standard 
set of selection and ranking criteria. It usually involves two stages: 
1. Screening out projects that do not satisfy the relevant criteria; 
2. Ranking projects that have been retained according to the criteria. 

In  the  case  of  the  "Corridor  Ranking  Framework",  presented  here,  the 
relative  importance  of  corridors  depends  on  the  specific  objectives  that 
underpin  transport  infrastructure  interventions  and  that  contribute  to 
attaining the global objective of socio-economic  cohesion. 

The ranking  of corridors  in this three-country  evaluation was  undertaken  on 
the  basis  of  seven  criteria  that  were  themselves  ranked  according  to  their 
relative importance. The criteria used were the following: 
1. Does the corridor provide linkages to the rest of the European Community? 
2.  Does  the  corridor  connect  important  population  centres  (more  than 
50,000  inhabitants)? 
3. Does the corridor  provide links to external access  points (i.e. airports  and 
seaports)? 
4. Are traffic  volumes on the corridor  high and, as a result, does  congestion 
occur regularly, etc.? 
5. Does the corridor complete land and / or sea routes? 
6. Does the corridor facilitate international traffic? 
7.  Does the  corridor  integrate  remote  or  isolated  regions  with  development 
potential? 

In the case of evaluation of the transport sector in Greece, corridors  between 
economic  centres  were  ranked  according  to  these  criteria  reflecting  a 
number of transport-related  objectives  (e.g. improving  connections to  other 
European  countries,  linking  up  centres  of  high  population  concentration, 
improving links to ports, etc.) The method of scoring allows the relevance of 
the  project to  be taken  into account  by indicating whether the criteria apply 
to  the  project  fully,  partially  or  not  at  all.  The  criteria  are  also  ranked 
according  to  their  overall  importance. This  information  is  presented  in  the 
form of a matrix, an extract of which figures in Box 40. 

Evaluating impacts of interventions in the domains of transport, 

Box 40 

­ Extract from a 

'corridor ranking framework" applied 

to Greece 

1 
Trans­Euro­
pean Road 

• 

O 

• 

O 

o 

Potential 
candidate 
corridors 
for 
develop­
ment 

1) 
Athens ­
Patras 
2) 
Athens ­
Thessaloniki 
3) 
Thessaloniki­
Igoumenitsa 
4) 
Corinthois ­
Kalamata 
5) 
Igoumenitsa 
Patras 

2 
Connects 
population 
concent­
rations 
(cities > 
50k) 

• 

• 

loannina 

3 

O 

Decreaslr 

.apriority. 

3 
Links exter­
nal access 
points 

4 
Has traffic 
volumes 

5 
Completes 
land / sea 
routes 

6 
Facilitates 
International 
traffic 

• 

• 

O 

O 

• 

• 

• 

O 

O 

o 

O 

O 

• 

O 

o 

O 

O 

o 

o 

o 

■w. 

7> 
Inte­
grates 
remote / 
isolated 
areas 

O 

O 

• 

• 

• 

• Criteria fully applies 

φ Criteria applies partially 

O Criteria does not apply 

It is important to note that this example Is only one way of ranking projects. 
Another set of criteria would reflect another set of objectives or a different 
type of intervention (e.g. total project cost, financial return, socio­economic 
value, relationship to objectives, leverage effects, timing considerations, 
local geo­political factors, environmental safety and other related objectives, 
network effects, etc.). 

• Case studies: Example of transport sector projects (Ireland) 

Case studies were employed in Ireland to obtain both quantitative and quali­
tative data on the economic impacts of a number of specific transport 
projects. Surveys of users (local companies, road hauliers) and cost­benefit 
analysis were used to inform the case studies. 

Whilst the case studies selected (a by­pass, a motorway, regional airport 
development and a seaport) provided some information to complement 
macro­level data resulting mainly from secondary sources, their contribution 
to the evaluation could be considered as less than satisfactory for a number 
of reasons: the benefits arising from the projects were considered indivi­

Transversal  evaluation  of  impacts  on  intervention 

priorities 

dually  and  therefore  could  not  be  fully  appreciated  because  the  projects 
form  an integral  component  of  a system; and, secondly, competing  factors 
overwhelmed the specific effects arising from the individual  projects. 

Although case studies in this example provide useful illustrations of interven­
tions  and  their  effects, their  contribution  to the  evaluation  could  have  been 
increased  if a number of methodological  considerations had been taken  into 
account.  Firstly, the  selection  procedure  for  cases  should  be  transparent 
and the importance of the case in relation to overall  policy objectives  needs 
to  be clarified. Secondly, as part  of a transnational  evaluation, the  selection 
of cases should allow for the greatest possible degree of comparison  across 
countries and regions to be undertaken. Finally, the measuring of impacts at 
the  project  level  neglects  the  synergies  and  externalities  generated  by  an 
intervention. In order to take these factors into account, a case study  should 
focus on a number of key projects within a circumscribed geographical area. 

•  DELPHI: Example of forecasting the development of transport tele­
matics technologies 

The advantages of telematics  in the domain of transport  is that they  provide 
a way of optimising the use of existing networks through better traffic  mana­
gement  and  information. They  therefore  represent  a  partial  alternative  to 
road  infrastructure  investment.  In  this  particular  example,  the  DELPHI 
technique  was  used  to  forecast  the  likely  level  of  use  of  a  selection  of 
transport  telematics  technologies  in  the  year  2015  in  medium-sized 
European  cities.  The  DELPHI  technique  allows  experts  to  collectively 
perform quantitative estimations of potential impacts of  interventions. 

The study  itself  involved  a total  of 224 experts  in Advanced Transport Tele­
matics technology  (ATT) from the vast majority of Member  States. It focused 
on 18 different ATT technologies  and their possible adoption over the  period 
up  to  2015.  By  participating  in  a  consultation  process,  this  technique 
allowed the experts to provide information on the technologies they thought 
were  likely  to  be  adopted,  the factors  that  might  lie  behind  their  adoption, 
and those that might restrict or retard this process. 

A  technique  related  to  DELPHI, the  matrix  of  cross  impacts  (SMIC),  was 
used  in parallel with this study. This technique,  unlike  DELPHI, allows  inter­
actions  among events to  be considered  by refining the  probabilities  relating 
to the occurrence of individual future developments. The resulting probabili-

Evaluating  impacts  of  interventions 

in  the  domains  of 

transport, 

ties can be used directly  in the  planning  process or as the basis for  develo­
ping scenarios that can subsequently  be used for planning. 

In total,  126 experts were  involved from across the  EU. Six  possible  events 
were identified which could be broken down into two main groups: 
•  events involving the use of technologies to reduce transport  problems 
•  events involving traffic and transport problems or consequences). 

A first stage of analysis assessed the single probability of the occurrence of 
the six events  considered  separately. A second  stage examined  the condi­
tional  probability  of  the  events  forecast.  The  result  of  the  study  was  a 
selection  of  scenarios  chosen  on  the  basis  of  the  estimated  likelihood  of 
their occurrence. 

IV 
Framework for performing 
thematic evaluations 

Thematic  evaluations  can  be  performed  within  the  framework  of  a  single 
programme  or across a number  of programmes.  For regional or national level 
conclusions  to be aggregated  and compared  at European level, evaluations 
have to be similar in the way they are structured  so that they produce  results 
that are relatively homogenous.  The first chapter  of Part IV presents  a stan 
dardised procedure  which, although  flexible enough  to be adapted  to  indivi 
dual evaluation situations,  is sufficiently  robust  to ensure that this objective is 
attained. 

Firstly,  the  second  chapter  examines  the  basic  principles  of  transnational 
thematic  evaluation,  a number  of  organisational  issues associated  with  this 
type of evaluation and how  they are used to highlight  best practice.  There is, 
however, another approach  to undertaking  transnational evaluations which is 
to move away from  transnational  comparisons,  based  on synthesising  what 
are essentially a number  of individual  country  or regional  level  evaluations, 
towards a multi-country  or pan-European  impact  assessment approach.  The 
future  development  of  transnational  evaluations,  with  an  emphasis  on 
moving  towards  this new perspective,  is discussed.  The chapter  concludes 
with  a  brief  overview  of  three  evaluation 
tools  that  are  suited  to  this 
approach. 

A step­by­step approach for thematic analysis 

The methodological approach proposed below is intended to provide a 
practical model for assessing, in a way that is relatively standardised, the 
impacts produced in a particular priority or domain. The need for a standar­
dised approach is particularly important in the case of structural policies 
because programmes have to be monitored and evaluated throughout the 
European Union and, as far as possible, the results aggregated. This method 
can be used to visualise and specify the results of a programme and to 
produce sufficiently coherent information to enable the Commission 
services to synthesise data from a number of programmes. 

■ A "step­by­step" approach for a thematic evaluation 

On the basis of the different elements described in this volume, it is possible 
to establish the different steps required in a thematic evaluation at 
programme level. This six­phase procedure, comprised of eleven different 
steps, is set out in Box 41. 

In the case of an ex ante evaluation, the evaluation team is required to use 
the first six steps of the procedure. These correspond with the following 
three phases: preparation of the evaluation, strategic assessment and 
screening of the programme (screening involves the examination of projects 
and programmes to determine the degree of impact assessment which will 
be required). 

In cases of intermediate or ex post evaluation, all the steps are appropriate. 
In this instance, the thematic assessment matrix will be initially used to focus 
the evaluation and secondly to synthesise the results of the analysis. 

The greater part of the work recommended in this approach should in any 
case be carried out in the framework of the evaluation of a programme as a 
whole. Thus, it does not necessarily concern additional work, in so far as the 
steps described are not intended only for the assessment of the theme. They 
also serve as a reminder, while the evaluation is underway, to produce 
results on this theme. 

Transversal evaluation of impacts on intervention priorities 

Box 41  Transversal evaluation of a theme : 

"Step by step" approach for a programme 

Steps 

Outputs 

Preparation 

• Examine the programming documents. 

<1 > Identification of the peculiarities of the 

• Interview the programme managers and 
stakeholders. 

programme in relation to the theme: 
specific objectives, % of budget allocated 
to specific measures. 

• Collect available data (time series, 
Information drawn from the monitoring system, 
etc.) and record existing documentation (literature). 
• Analyse available statistics. 

<2> Assessment of the most Important Issues 

in the region in relation to the theme. 
Reference situation or context analysis. 
Information concerning programme 
outputs and results. 

Strategic assessment 

• Detail the transversal effects of the 
programme in relation to the theme according 
to a typology of impacts. 

<3> Detailed list of potential effects organised 
according to the generic 
criteria selected. 

• Complete the "thematic assessment matrix" 
(see Box 42). 

■:4> 

Classification of measures by category: 
favourable, unfavourable or neutral effect. 

Screening 

• Prepare a score card for important impacts 
assessing the effects of measures (see Box 43). 

<5> 10 to 20 score cards provisionally 

completed for the most interesting 
measures. 

• Consultation with programme managers and 
stakeholders. 

<6> 

Selection of measures for further 
analysis. 

• Collect and analyse information by means 
of the minimal case study technique, 
Interviews and / or focus groups. 

Scoping 

<7> 

Score cards completed on the basis 
of the analysis of selected measures. 

Programme level synthesis 

• Complete "summary table" (see Box 45) 
for each type of impact. 

<8> Summary tables providing a transversal 

view of impacts. 

• Detail the thematic assessment matrix. 

^9> 

Overall view of the value of programmes on 
the basis of types of impacts defined. 

Conclusions 

• Examine the progress made: success and <10> 
weaknesses of the programme. 

Useful conclusions for reviewing for the 
promotion of a priority or interventions in a 
domain. 

• Forward conclusions to the Commission <11 > 
services in a standardised format. 

Comparable view of the progress made 
in the EU in respect to the theme. 

A step­by­step approach for thematic analysis 

This above procedure is neither exhaustive nor exclusive. Nor is it intended 
to be a rigid framework. Evaluators have to determine just how far this 
approach is applicable to the programme in question and, if necessary, add 
descriptive comments and appropriate indicators. 

It should be noted that this procedure is not sufficient for the evaluation of 
the effects of specific measures in a domain: the screening (the identification 
and selection of measures for more detailed analysis) and scoping (the 
detailed analysis of impacts) of measures, in particular, needs to be further 
developed. 

This procedure is detailed below, using the priority of equal opportunities 
between men and women as an illustration. 

■ Initiating the procedure (steps 1 & 2) 

Step <1 > is intended to ensure that the evaluators have in­depth knowledge 
of the content of the programme concerned, including its objectives and 
information considered to be relevant. A thematic objectives tree can be 
drawn up, highlighting the objectives of the priority. This phase implies 
obtaining and consulting all the programming documents and interviewing 
programme managers. Based on the typology of potential effects described 
in the chapter on the promotion of equal opportunities, it consists of: 

• assessing the extent to which the chosen objectives include this priority 
and, where relevant, are consistent with EU policy, and checking whether the 
strategy is relevant from this point of view; 

• analysing the content of the programme in order to assess the extent to 
which the promotion of equal opportunities has been taken into account 
during the programming stage; comparing financial grants measure by 
measure, bearing in mind their specific (targeted) or general nature; making 
an inventory of the targeted publics of the different interventions in order to 
identify those interventions which explicitly aim at the priority. 

It is then essential to define the context indicators which measure the extent 
of the needs to be met or problems to be solved. Within the framework of ex 
ante evaluation, they help to show the raison d'être of interventions and to 
clarify the strategy to be chosen. They serve as a medium for the quantifica­
tion of clear and realistic objectives. Within the framework of intermediate or 
ex post evaluation, context indicators furnish information about the extent to 

Transversal evaluation of impacts on intervention priorities 

which the target has been reached, or the extent to which the needs in the 
region have been met. 

In step <2>, the problems encountered by the public (or a geographical 
zone) concerned are diagnosed, and the potentialities to be developed in the 
specific national or regional context are identified. A list can thus be drawn 
up of the main challenges to the effective promotion of equal opportunities. 
For this purpose it will be necessary to collect literature and have access to 
databases in order to analyse the national and regional socio­economic 
context and, more particularly, identify the expectations of potential benefi­
ciaries and political stakeholders. 

■ M a k i ng a strategic a s s e s s m e nt (steps 3 & 4) 

Step <3> consists of a thematic assessment of the programme measures. 
This implies the classification of each measure in terms of its potential 
impact on the promotion of equal opportunities. It is necessary first to draw 
up a typology of impacts. This can be done by drawing up a list of impact 
descriptors, based on the preceding contextual analysis, and grouping them 
according to the similarity of impacts (see Volume 4 of the MEANS Collec­
tion for a description of the use of the "impact mapping" technique in this 
respect). In the case of the promotion of equal opportunities between men 
and women, the types of impacts selected are: "Living conditions which 
meet women's needs", "Access for women to the job market", "Women's 
situation at work" and "Participation of women in the creation of economic 
activities". 

The thematic assessment matrix (see Box 42) of the contribution of structu­
ral programme interventions to objectives relating to this priority can be pro­
visionally completed in step <4>. In this way the most promising measures 
can be identified in terms of potential impacts. 

The use of this matrix allows the classification of interventions in terms of 
whether they have a favourable, unfavourable or neutral effect. Thus, it is 
possible to identify interventions that warrant further analysis during the 
phases of screening and scoping. Finally, the matrix enables the identifica­
tion of those objectives mentioned in the programme but which few 
measures help to achieve. 

A step­by­step approach for thematic analysis 

Box 42 ­ Thematic assessment matrix. Example relating to equal 

opportunities 

Type of impact expected / achieved on the promotion of equal opportunities 

Area of intervention 
(Measures) 

More appropriate 
living conditions 

Access to the job 
market 

Situation at work  Participation in the 

creation of socio­
economic activities 

Infrastructures 
Measure N°: 
Measure N°: 

Human resources 
Measure N°: 
Measure N°: 

Productive 
environment 
Measure N°: 
Measure N°: 

Added value of 
the programme 
In terms of equal 
opportunities 

Provisional 
assessment of 
impact 

/ /? ) X 

Beneficial effect Effect probably No significant Negative effect 

beneficial effect 

■ Screening and scoping (steps 5, 6 & 7) 

Step <5> concerns the identification and selection of measures for further 
analysis. It requires the creation of a score card for each measure identified 
as having a potential impact on the promotion of equal opportunities. An 
example is provided in Box 43. In the preparation of the score cards the 
following types of impact should be covered: 

• positive or negative; 

• direct or indirect; 

• temporary or permanent; 

• short or long term. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Box 43  -  Score  card. Example: assessing the  extent to  which 
a  measure takes  into account  of the  priority of  equal 
opportunities 

Sub-programme / priority: 
Measure (title): 
Objective: 
Budget (% of funding by the EU): 
Output: 

Potential positive impact of  Significant bene-  Hardly signifi- 
cant beneficial 
effect 

projects/actions on: 

ficial effect 

No significant 
effect 

Negative effect 

Living conditions better suited 
to women's needs 
Women's access to the job 
market 
Situation of women at work 
Participation by women in the 
creation of socio-economic 
activities 

1 

1 

1 

0.5 

0.5 

0.5 

0.5 

0 

X 

X 

X 

X 

Total number of points | 

| 

If the measure has a direct or explicit impact In terms of equal 

opportunities, multiply the score by 1.5 

Give details on the two most significant positive or negative effects: 

Secondly, the evaluators  are advised  to  consult  with  programme  managers 
and  stakeholders  about  the  results  obtained  during  the first  two  phases  of 
the  evaluation  (step  <6>). The objective  of this consultation  is to  select  key 
indicators  and  the  measures  that  are to  be the  subject  of  further  analysis. 

Until now there has been no attempt to quantitatively  measure the  changes 
that  have  occurred  in  relation  to  the  priority. At  this  stage, the  information 
summarised  in the  thematic  assessment  matrix  and  on  the  score  cards 
should  be  complemented  by  data  gathered  by  the  relevant  statistical 
agencies on the indicators selected. 

After the score  cards  have been the subject  of consultations, the  measures 
most  likely to have a significant  impact on the promotion of equal opportuni­
ties  are selected  for analysis  (step  <7>). This analysis  involves  an examina­
tion of impacts using minimal case studies, interviews and / or focus  groups 
(see  Box  44). This  technique,  based  on  the  limited  analysis  of  impacts  in 
order  to  consolidate  the  assessment  of  impacts  carried  out  on  a  strategic 
basis, is known as scoping. 

Cross programme 

thematic  evaluation 

Box 44 -  Interviews, focus groups and  case  studies 

Individual  interviews  can  be  used  for  all types  of  evaluation  where  the 
information needed is more complex than that which can be conveyed by 
questionnaire. Interviews range from being relatively informal and unstruc­
tured to  highly formalised,  based on  pre-determined  questions. They are 
particularly  useful  for  describing  procedures  and  processes  (e.g.  the 
implementation  of a programme)  and the  likely  impacts of an  intervention 
or  impacts  that  are difficult  to  quantify,  such  as  behavioural  changes  or 
intangibles.  Group  interviews  can  largely  be considered  as  a number  of 
simultaneous individual interviews. 

Focus  groups  actively  seek to  harness  the  dynamics  of the  interactions 
between the  individual  participants  and to  integrate  it into the analysis  of 
complex  issues. The  interaction  among  participants  is  particularly  useful 
when  opinions  about  a  certain  issue  are  not  fully  formulated.  The 
technique  can therefore  be used to  identify  the  needs  of a region, clarify 
the  objectives  of  interventions,  examine  the  impacts  of  innovative 
measures,  focus  or  refocus  the  implementation  of  a  programme,  or 
interpret the impacts of an intervention. 

Case studies  are used for the analysis of  specific  cases, ranging from  an 
individual  to  a geographical  region. They  can  also  be  used  to  analyse 
decision  making  processes.  Both  interviews  and  focus  groups  can  be 
used  as data  gathering  tools  within  the  framework  of  case  study.  Case 
studies are particularly useful for making detailed analyses of the complex 
process  of  programme  implementation  and  of  their  impacts. They  are 
often  undertaken  to  verify  causation  or to  qualitatively  test  and  validate 
measurements  of  effects  obtained  by  other tools. This tool  can  also  be 
used when there are very few units of analysis by matched  pairs of similar 
units. This  involves finding  two  units  of  observation  -  one of  which  is an 
addressee of a programme and one of which is not - that closely  resemble 
each  other  (e.g.  SMEs  in  the  same  sector,  with  the  same  number  of 
employees, etc.) and that are in a similar context (e.g. in a declining indus­
trial region where unemployment  is high, etc.). The relative evolution of the 
units can then be studied. 

Transversal evaluation of impacts on intervention priorities 

■ Synthesising the results (steps S & 9) 

The analysis by measure is essential for the construction of an evaluation of 
the programme as a whole. Evaluation results must also be produced at 
programme level, then at the higher levels. To globally assess the effects of 
measures at the level of each programme, a summary table (see Box 45) can 
be prepared for each of the four main types of impact, identified as being 
relevant In the case of structural interventions (step <8>). The summary table 
will be completed using: 
• the quantitative assessment provided by the score cards (far right-hand 
column of the matrix); 
• the analysis, providing both quantitative and qualitative data, performed 
during the scoping phase (cells of the matrix). 

Box 45 - Summary table of measures and their effects. Example 
of improving the suitability of living conditions to women's needs 

Summary table: synthesis of impacts in terms of living conditions better suited to women's needs 

Measures with a 
significant impact 

Budget 
(% of funding by 
the EU) 

Outputs 

Results which were 
expected and / or 
produced 

Impacts which 
were expected 
and / or produced 

N°: 
(title) 

N°: 
(title) 

N°: 
(title) 

N°: 
(titre) 

Total points or X ( in the case of a negative impact (source: score cards): 

General comments 
this type of impact: 

for 

Since the Commission will need to have an overall view of the impact of 
interventions on the priority of promoting equal opportunity between man 
and women, it may be useful to prepare a brief, very general summary on the 

Cross programme thematic evaluation 

(positive and negative) impact of each programme (<9>). The thematic 
assessment matrix, previously completed on a provisional basis can be used 
for this purpose (see Box 46). The matrix approach is sufficiently simple (without 
being simplistic) to meet practical needs. However, it remains necessary to 
analyse available indicators, in particular as far as context is concerned. 

The matrix approach makes it possible to achieve the standardisation 
needed for an overall view at European Union level. It serves to simplify an 
extremely complex reality and to present it in a form which is synthetic and 
comprehensible at policy level. 

The key steps and evaluation tools employed in this approach are summari­
sed in a highly simplified example in Box 46. Four measures initially enter the 
strategic assessment phase, of which three are considered sufficiently 
important to be retained for further study. At the screening phase Measure 4 
is eliminated which leaves Measures 1 and 3 to enter the scoping phase. The 
conclusions of the analysis made at the scoping phase are presented in a 
summary table and this is then used to complete the thematic assessment 
matrix. 

Box 46 ­ Summary of the key steps and evaluation tools 

Thematic assessment matrix Screening 

Step <4> and Step <9> Step <5> and <6> 

Scoping 
Step <7> 

Measure 

1 

2 

3 

4 

k 

f , 
) 

, I «* 

• 

1 

3 

> 4 

­ * ·! 1 ' 

­ * ¡ 3¡ 

_l 

S u m m a ry t a b le (one per t y pe of i m p a c t) 

S t ep < 8> 

Measure 

Resources 

Outputs 

Results 

Impacts 

1 

τ 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

I  Concluding  on  the  merits  of  t he  p r o g r a m me  as  regards 

the theme evaluated (steps 10 & 11) 

The  conclusion  of  the  evaluation  report  (step  <10>)  will  obviously  include 
recommendations  on  the  best  way  of  taking  the  evaluated  theme  into 
account when formulating objectives, programming and selecting  projects. 
So that a priority can be better taken into account  by decision-makers, the 
evaluation team must make significant efforts to: 
•  propose new strategies of intervention; 
•  strengthen the most promising  measures; 
•  formulate  recommendations  for  enhancing  the  potential  of  the  targeted 
social groups and areas. 

On  a  more  technical  level, recommendations  are  needed  so  that  in  the 
future  the  systems  of  monitoring  indicators  and  of  statistics  can  include 
information that will provide a better understanding of the situation. 

Finally, in step <11 > the conclusions  provided by the procedure will be suf­
ficiently  standardised  to  nourish  debates  and  comparisons  of  the 
promotion of equal opportunities between men and women (or other priori­
ties or domains) at the European level. 

Cross programme thematic evaluation 

■ Basic principles of transnational t h e m a t ic evaluations 

The preceding chapters have examined the evaluation of a programme from 
a thematic perspective. Evaluations at the programme level focus principally 
on the examination of programming undertaken by Member States. Whilst 
these evaluations allow the appreciation of interventions on a national / 
regional scale, they do not provide information that allows a Europe­wide 
analysis of impacts, the appreciation of Community value­added or a pan­
European strategic analysis of Structural Fund interventions. 

In order to achieve these more ambitious evaluation objectives, it is 
necessary to undertake thematic evaluations at transregional or transnatio­
nal level. It is important to note, however, that for this type of evaluation to be 
fully effective the methodology used should go beyond the simple supra­
national synthesis of a number of single evaluations (see Box 47, upper 
figure). The approach followed should be based on the collection, analysis 
and synthesis of data within a common and co­ordinated single evaluation 
framework across all the countries involved (see Box 47, lower figure). 

Transversal evaluation of impacts on intervention priorities 

Box 47 ­ Comparison of supra­national synthesis of programme 

evaluations and transnational evaluation 

Supranational synthesis of evaluations 
(a), (b) and (c) 

J_ 
Evaluation (a) 

Evaluation (b) 

Evaluation (c) 

Programme 
(a) 

Programme 
(b) 

I 

Programme 
(c) 

Single transnational evaluation 

Programme 
(a) 

J 

1 Programme ! 
; (b) : 
1 J 

1 Programme ! 
; (c) : 

ι I 

■ Organisational issues 

A major problem in commissioning thematic transnational evaluations is the 
question of how to take into account differing national or regional contexts, 
thematic specificities and interrelations between different themes and 
domains. The consequence of these complexities is that the drawing up of 
the terms of reference for an evaluation of this type is extremely problemati­
cal. It is for this reason that an essential first step is to set up a technical 
committee composed of experts and academics with relevant sectoral 
knowledge, and national experts who understand the national and regional 
socio­economic contexts in which the evaluation will take place. 

The role of the technical committee is to steer a feasibility study by evalua­
tors of two or three countries (carefully chosen for their representativeness 

Cross  programme  thematic  evaluation 

and relevance in terms of the theme of the evaluation). This study will involve 
the  elaboration  of  a  formal  pre-evaluation  document  in  which  priority 
questions, programmes to  be evaluated, timescale, budget  and the evalua­
bility  of  interventions  are  addressed  (e.g. Is the  evaluation  politically  and 
technically  opportune?  Is the  budget  and  timescale  adequate,  considering 
the  characteristics  of theme  and the  stock  of  existing  knowledge  that  may 
be mobilised?). 

When designing the evaluation the evaluation team should: 
•  draw  up evaluatory  questions that  disregard  country  and  regional  factors 
and concentrate solely on thematic aspects of the evaluation, and which are 
specifically centred around a European perspective; 
•  tightly define the criteria for the selection of case studies and sampling so 
as to  improve  comparability  and take  into  account  of  country-  and  region-
specific contextual factors; 
•  elaborate  an analytical  grid  which  takes  competing  factors  into  account. 

This work should be then validated by the technical committee. 

Case  studies  should  be selected  by the technical  committee  and  validated 
by the relevant national experts who would judge the pertinence of the cases 
chosen. The number  of case studies  performed  should  be relatively  limited 
and they should be likely to capture  strong structuring effects. These struc­
turing effects should  be analysed against a contextual  backdrop which may 
be regional, national, European or even international, depending on the par­
ticular cases selected. 

The output  of this process will be a document  which  is the result of  interac­
tions between actors with sectoral and national socio-economic  experience. 

When the final evaluation  report is produced  it is important that the  Member 
States  involved  in transnational  evaluations  should  have the opportunity  to 
verify the interpretation of the results of the evaluation. 

Transversal evaluation of impacts on intervention priorities 

■ Highlighting e x a m p l es of best practice 

Carefully designed and implemented transnational evaluations can comple­
ment existing programme level evaluation by highlighting examples of best 
practice and allowing experiences to be transferred between regions and 
countries. The objective of this section is to suggest a number of practical 
recommendations that are aimed at ensuring that evaluations are fully 
adapted to the goal of highlighting best practice. 

Since Community programmes are both complex and open ended, the way 
in which they are implemented is an important evaluation issue. The search 
for examples of best practice should therefore examine not only the impacts 
of programmes but also the process of implementation, and should make 
comparisons of delivery mechanisms in different countries. 

In order to find examples of best practice, it is important that populations of 
beneficiaries, geographical units (regions, sub­regions, towns, etc.), 
projects, interviewees, etc. selected for case studies should be similar in 
terms of the context in which they are situated. The contextual situation in 
which the selected case studies are embedded can be initially established 
by a SWOT analysis. Points of comparison also have to be established by 
which performances can be judged. This can be done using the benchmar­
king technique. 

The SWOT framework (Strengths­Weaknesses­Opportunity­Threats) can be 
used to undertake a strategic analysis of a programme and the region in 
which it is to be implemented (internal factors) as well as the overall policy 
environment (external factors). This tool can be used to outline an economic 
development strategy and is therefore particularly useful during the design 
and ex ante assessment stage of a programme. It can also be used for mid­
term and ex post evaluations to take into account contextual and identify 
problems of comparability. Box 48 illustrates this principle. 

Cross programme thematic evaluation 

Box 48 ­ Sampling with the objective of highlighting best practice 

Country 1 

Country 2 

Country 3 

SMEs 
medium­tech 
20­50 
employees 

  _ Λ_ 
e
.

t
x
e
t
n
o
C

d

.

c

.

b

.

a

SMEs 
medlum­tech 
20­50 
employees 

L 

Λ 1 

I 

1 

1 

1 
1 

1 

SMEs 
medium­tech 
20­50 
employees 

m ι 
• I 

I 

1 
1 

1 

■ 1 

1 

1 

1 

_. 

( · \ 
V · J 

Cross­country 
sample of various 
practices embedded 
in similar contexts 

If the theme which is the object of the evaluation is SMEs, the constitution of 
a cross­regional sample requires that the firms have similar characteristics 
(e.g. between 20 and 50 employees in a medium­tech sector of the 
economy). In order to assure comparability, the firms are drawn from similar 
contextual situations, minimising the difference in regional context. A direct 
comparison is then feasible across countries. 

Benchmarking consists of making a qualitative comparison between one 
intervention and another, comparable in terms of being embedded in the 
same contextual situation, which is reputed to be an example of best 
practice. By doing this, the strengths and weaknesses of the intervention 
being evaluated can be highlighted. This technique can also be used at a 
more strategic level to compare the organisation of programmes. 

■ Future directions for transnational evaluations 

Evaluation practice and experience in the field of transnational thematic eva­
luations have grown rapidly and a second generation of this sort of evalua­
tion, with an emphasis on highlighting best practice, is already developing 
(see Box 49). 

 
Transversal evaluation of impacts on intervention priorities 

While the type of transnational evaluation discussed above is capable of 
highlighting examples of best practice, it is not capable of providing a picture 
of pan­European achievements. This sort of study, which needs to take into 
account different national or regional contexts, could be used to inform 
European­wide strategic planning, for example, permitting the drawing up of 
a masterplan for Trans­European Networks or for the production of recom­
mendations and guidelines adapted to Community­wide conditions. 

Box 49 ­ The development of transnational evaluations 

1 ■ Experience of transnational 
evaluations 
• mainly based on documentary 
research, secondary data, and 
interviews with key actors 
• scores and weights 
• SWOT analysis 
• project case studies 

Examples: 
• Transportation 
• Telecommunications 
• Tourism 

$ 

2° Experience of transnational 
evaluations 
1° + additional tools including: 
• country­by­country reviews of 
existing literature 
• secondary data analysis 
• surveys 
• regional and project case 
studies 

Examples: 
• Business information 
Centres 
• Environment 
• RTD 
• SMEs 

^ 

3° Experience of transnational 
evaluations 
1° el 2° + additional tools such 
as: 
• research synthesis 
• expert panels 
• pan­European surveys 
• embedded case studies 
•others... 

In addition to the range of tools suited to thematic evaluations presented 
throughout this volume, there are a number of tools are suitable for underta­
king pan­European analysis. These tools are presented below. 

• Multi­country / multi­regional econometric analysis 

When surveys can be undertaken, the data collected can be used to 
construct ad hoc multi­country or pan­European econometric analysis of 
impacts. 

The results furnished by this type of analysis can transformed into coeffi­
cients which can subsequently be used to make a projection of the results 

Cross programme  thematic  evaluation 

for  either  particular  aspects  of  an  intervention  or  for  its  overall  projected 
effects.  The  use of  coefficients  also  requires  monitoring  data  to  provide 
information on the addressees of interventions and the distribution of expen­
diture,  etc.  Impacts  can  be analysed  both  in a cross-sectional  perspective 
(i.e. a variety of  interventions  at a given  moment  in time)  and over time (i.e. 
using time-series data). 

The  use of this tool  requires that  sampling  be correctly  undertaken.  Unlike 
the sampling technique used when benchmarking is the objective of an eva­
luation,  in this  case  samples  should  be  constituted  from  across  a  diverse 
range of country  and  regional  contexts.  In order  to  do this, a common, co­
ordinated  methodological  approach has to  be used which involves carefully 
controlled  data  collection  to  ensure  that  the  selection  of  units  of  analysis 
features the level of representativeness required. This approach is illustrated 
in  Box  50 where  careful  sampling  allows  a cross-regional  sample,  drawn 
from a variety of different contexts, to be constituted. 

Box 50 - Sampling with the objective  of  making  a pan-European  analysis 

Country  1 
SMEs 
medium-tech 
20  -  50  employees 

Country  2 
SMEs 
medium-tech 
20  -  50  employees 

Country  3 
SMEs 
medium-tech 
20  - 50  employees 

I 

1  1 

> 

| 

1 
1 

1 

t
x
e
t
n
o
C

e
.

d
.
c
.
b
.
a

1 

' 

1 
t 

1 

1 

1 

1 

i 
1 

f 

» 

•  \ 
· 
y 

f 
/ 
• 
V 

· 

· 

· 

 
 
 
Transversal  evaluation  of  impacts  on  intervention 

priorities 

Whenever  feasible,  comparative  analyses  should  be  performed  with  non-
European countries (e.g. the United States, Canada, Australia, etc.). 

•  Multi-site case studies with replication design 

Case  study  analysis  is  a  useful  tool  for  analysing  socio-economic 
phenomena  that  are  difficult  to  separate  from  their  contextual  conditions. 
The unit of analysis of the case study may be an individual, an organisation, a 
process,  an  industry,  a  region, etc. The  main  unit  of  analysis  (e.g. a  region) 
may  furthermore  contain  sub-units  of  analysis  (e.g. firms,  infrastructure 
projects, etc.). Case studies can either be single or multiple. 

The  use  of  multiple  case  studies  with  replication  design  is  a  particularly 
important  technique  for  strengthening  or  broadening  analytical  generalisa­
tions. This type of  case study  is based on a series of waves  (usually two  or 
three)  of  case  studies  carefully  chosen  to  support  a  given  proposition. 
Depending  on  the  design  of  the  evaluation  the  case  studies  will  produce 
either: 
•  similar results providing corroborative evidence; 
•  contrasting  results  but  with  similar  theoretical  underpinnings  (i.e.  the 
theory is corroborated); 
•  both replicated and contrasting results within the same study. 

If, however, the results produced in some of the cases are contradictory in some 
way, the initial proposition cannot be accepted. This may cause it to be retested, 
by means of a second and, when necessary, a third wave (see Box 51 ). 

Example: Providing advice to city administrations 
In a study designed to examine attempts by universities and research groups to colla­
borate  with  local  government  officials,  four  different  sets  of  replicated  case  studies 
were undertaken (see Box 51). The articulation of the propositions supported by each 
of  the  different  sets  allowed  an  overall  conclusion  to  be  reached  which  was  not 
evident at the outset  of the study. An initial group  of case studies  concluded that  uni­
versities and research centres  had failed to  help city administrations. A second  group 
concluded that this phenomenon  was not restricted to  universities  but included  other 
organisations. A third group showed that in some cases  universities  had been able to 
help  other  types  of  organisations.  Finally,  a fourth  group  showed  that  some  groups 
had  been  able  to  help  city  administrations  by  taking  implementation  issues  into 
account.  The  overall  conclusion  was  that  city  administrations  had  particular  needs 
when  receiving  advice  from  outside  bodies,  a conclusion  which  differs  significantly 
from the initial one. 

Cross  programme 

thematic  evaluation 

Box 51 - Example of multiple-case studies with replication design 

(Wave  1 ) 
Case studies  focusing 
on universities  and 
their  relations 
with  city 
acministrations 

(Wave 2) 
Case studies  on 
relations  between 
non-university  groups 
and  city 
administrations 

(Wave 3) 
Case studies  on 
relations  betwwen 
universities  and 
organisations  other 
than  city 
administrations 

(Wave 4) 
Case studies  on 
groups  concerned 
not  only  with  new 
ideas, 
but  also  their 
implementation 

Source: from Yin R.K. (1994), Case study Research: Design and Methods, London: Sage. 

•  Research synthesis 

Research or evaluation synthesis  is the analysis of existing research using a 
set  of  systematic  procedures  - or  meta-analysis  techniques  - for  summari­
sing quantitative results from  multiple studies. This technique  is based on a 
systematic  and  rigorous  approach  to  synthesising  past  research. Unlike  a 
literature review, research synthesis takes into account contradictory  results 
when analysing the existing stock of knowledge about a subject. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

This  technique  has  two  benefits:  first,  by  drawing  on  a  number  of  well 
designed  and  executed  studies,  the  resulting  analysis  is  highly  robust 
because  based  on  solid  and  reliable  knowledge;  secondly,  if  research 
synthesis shows that the existing base of knowledge  is not well-founded, it 
enables decision makers to avoid basing their policy making on  incomplete 
or  erroneous  information,  and  provides  guidance for  future  research. The 
procedure for performing research synthesis is summarised in Box 52. 

Box 52  - The seven steps  of  research  synthesis 

Siepi 
Selection of the questions  to be  addressed 

Step  2 
Collection  of the  all  relèvent documents  and identification  of  key  issues 

Step  3 
Development  of screening criteria and screening  out of irrelevent  material 

Step 4 
Development  and implementation  of critical  asessments  of  studies 

Step  5 
Redetermination of the appropriateness  of this technique  in the  light of the material  reviewed 

Step 6 
Undertaking  of the evaluation  synthesis 

Step  7 
Reporting the findings  of the  synthesis 

Conclusion 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

The development of thematic  evaluations within the framework of the Struc­
tural  Funds  has  been  shaped  by  the  evolving  policy  context  within  the 
European  Union. As  a consequence,  the  state  of  the  art  in this  area  was 
initially dominated  by experience in the evaluation of employment effects. As 
the decade  has  progressed, however,  environmental  concerns  and  equality 
between  men  and  women  have  climbed  the  political  agenda,  and  this  has 
been reflected in the accumulation of evaluation experience in these themes. 

At  the  same  time  as  this  process  of  accumulation  of  thematic  evaluation 
experience  has  been  occurring, the  need  for  comparisons  between  pro­
grammes  has  become  increasingly  important,  hence the  need  for  transna­
tional  thematic  evaluation.  Experience  in this  still  emerging  area of  evalua­
tion remains relatively limited for the moment. 

While  an  in-depth  evaluation  of  impacts  within  a  particular  theme  requires 
corresponding expert  knowledge which can only be provided  by specialists, 
the  central  premise  of  this  book  is that,  with  a  basic  understanding  of  a 
theme's characteristics, an impact assessment  can be designed or underta­
ken  by  evaluation  units  and  evaluators  without  in-depth  knowledge  of  the 
theme.  It  is on this  basis that  existing  thematic  evaluation  experience  and 
general  evaluation  expertise  have  been  combined  to  produce  this  book 
focusing on how to assess the impacts of Structural Fund interventions on the 
four political priorities of the EU and on number of domains of intervention. 

The  approach  taken  in  this  book  has  been  to  examine  thematic  impact 
assessment  in such  a way  that  methodological  considerations,  rather  than 
evaluation  themes,  are the  dominant  perspective. This  approach  has  been 
particularly  fruitful since  it has given  rise to a the construction  of an evalua­
tion  method  -  the  step-by-step  procedure  -  which  draws  upon  evaluation 
practices  and  tools  that  have  been  developed  within  the  various  thematic 
frameworks  An  example  of  this  is  the  screening  and  scoping  technique 
which  has  been  mainly  used for ex ante environmental  impact  assessment. 
Another  example  of  "cross-fertilisation"  between  themes  is that  of  mains­
treaming, which is finds its way into the step-by-step approach by way of the 
score card.  Moreover, this  is a strategy  which  is particular  to the  evaluation 
of  equality  between  men  and  women,  but  which  seems  to  be  a  promising 
approach for evaluating  impacts on SMEs within the framework  of  interven­
tions  aimed  at  improving  the  competitiveness  of  the  productive  sector.  At 
the conceptual level, the notion of sustainability which can be applied across 
a wide range of themes. Similarly, the concept  of leverage can be applied  to 

Conclusion 

all types  of  interventions  where the objective  is to  encourage  private  sector 
co-financing. Substitution, displacement and deadweight, concepts origina­
ting from the evaluation of employment  effects, are also applicable to  other 
themes. 

Finally,  although  a  large  part  of  this  Volume  has  focused  on  drawing  on 
existing  evaluation  experience,  the  future  of  the  evaluation  of  Structural 
Fund interventions  has not been forgotten. It is for this reason that a number 
of  tools  are discussed  that  seem  particularly  adapted  to  complementing 
transnational thematic  evaluation aimed at highlighting best practice with in-
depth evaluations of impacts at the pan-European level. 

Appendix 

Appendix  1 contains an index of tools classified by their thematic suitability in 
the form of a matrix. 

Appendix  2 presents  the  terms  of  reference  for  a programme  evaluation 
incorporating  explicit  instructions  relating  to  the  inclusion  of  Community 
political  priorities:  "Intermediate  Assessment  of  the  Highlands  and  Islands 
objective  1 Programme". 

Appendix  3 presents  the terms of reference  of a transnational  thematic  eva 
luation:  "Evaluation of  RTD actions  in  the case of Structural  Funds" (1998). 

Appendix  4 presents a list of boxes. 

The documents  presented  in Appendices  2 and 3 are not  intended  to serve 
as strict  models  upon  which  terms of reference  should  be based.  They can, 
nonetheless,  be  used  to guide  the  drawing  up  of  future  terms  of  reference 
which  the authors  should  tailor to  the specific  characteristics  of  the evalua 
tion to be performed  (e.g. in terms of particularities  of the theme,  evaluation 
tools to be chosen, organisational factors, etc.). 

Interviews 

Case 

Focus 

studies 

group 

Question­

naire 

SWOT 

surveys 

CBA 

and 

MCA 

Scores 

Screening 

and 

and 

DELPHI 

weights 

scoping 

Expert 

panels 

coefficients 

Input­

Macro­

and 

output 

ecowmethc 

statistical 

analysis 

models 

analysis 

Multipliers, 

Environ­

ment 

Page 54 

Page 55 

Page 52 

SMEs 

Page 73 

Page 75 

Page 73 

Page 73 

Page 73 

T3 
CD 

< 

ω 
5' 
3 

Employ­

ment 

Equal 

opportunities 

Page 101 

Page 101 

Page 103 

Page 104 

Page 104 

Page 126 

Page 125 

Page 123 

Page 126 

Page 123 

RTD 

Page 147 

Page 147 

Page 147 

Page 147 

·­*■ 

O 
o_ 
C/5 
03 
3 
O. 
O 

Π) 

O ^ -\ 
(Λ •a 
o 
=i o. 
Ξ" 
en 

Transport 

Page 163 

Page 161 

Page 162 

Page 164 

re 
3 
re 
C/5 

Appendix 2: Extracts from a Terms of Reference for a 
programme evaluation  incorporating thematic  elements 

Intermediate  Assessment  of the  Highlands  and  Islands  Objective 1 
Programme 

This document  explicitly presents the requirement to evaluate  Community 
priorities in the paragraph on "Specific Objective" of the evaluation. 

Examine  each  priority  and  measure  and  consider  their  appropriateness  in 
addressing the aim of the programme. Consideration should  be given to the 
number of measures, their suitability and scope and the continued  relevance 
of  targets,  indicators  and  outputs.  Recommendations  should  be  brought 
forward on amendments. Annex A sets out  in summary form the key  issues 
which  the  partnership  has  identified  in terms  of  implementing  the  SPD  by 
priority  and  measure. The final  report  should  address  each  of these  issues 
albeit in varying detail, and where appropriate, provide advice or recommen­
dations for the future implementation of the programme. 

Assess  performance  at  programme  priority  and  measure  level,  identifying 
cause of variance in terms of commitment and expenditure, having regard to 
external factors  and  match funding  (comment  on the  extent to  which  such 
factors can be accommodated within the existing implementation). 

Examine the synergy between the different Structural Funds, between Priori­
ties  and  between  the  SPD  and  Community  Initiatives.  To comment  and 
advise on linkages within the programme and the extent to which  objectives 
are  being  achieved  in  particular  job  creation,  environmental  sustainability 
and community  development. 

Details are given of key issues concerning various priorities 

PRIORITY 1 : SUPPORT FOR  BUSINESS 

•  Is there adequate balance across different  measures? 
•  Can different measures and different funds be better  integrated? 
• Are the targets appropriate in relation to the strategy? 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Measure  1: Financial support for the development of new ventures  and 
SMEs 

• Scheme coverage - does this require to be focused? - does it require better 
co-ordination. 
•  Number of schemes - Scope for more? Capacity for more? 
•  Leverage - Is there a need to establish  thresholds? 
•  Does there need to be greater of sectoral focus 

Measure  2:  Financial  support  for  research  and  development  appro­
priate to SMEs 

• Why are SMEs not more interested in R&D? 
•  Does product development require more emphasis? 
•  How can the measure be made more attractive? 

[etc.] 

PRIORITY 3: PRESERVATION AND  ENHANCEMENT  OF THE ENVIRON­
MENT 

• Are the measures too or not sufficiently  constrained? 
•  Is the scope of measures too  limited? 
•  How can a better integration between-funds  be achieved? 

Measure 1 : enhancement and protection of the environment and mana­
gement  schemes 

•  Is the scope of measure sufficient? 
• Are targets suitable? 
•  How can support  be adjusted in relation to the targeted  problem? 

Measure  2: pollution  control, waste  disposal  and  the  development  of 
recycling  systems 

•  How can statutory and non-statutory  responsibilities be better reflected in 
the system? 
•  How can re-cycling be encouraged? 
•  Can the innovatory character of projects be increased? 

[etc.] 

Appendix 3: An example of terms of Reference for a 
transnational thematic evaluation 

Evaluation of action in favour of RTD within the framework of the 
Structural Funds 

■ Background 

Presentation of the theme, related policy interventions, the context in 
which it is situated and the raison d'être of the evaluation. 

The importance of research and technological development (RTD) for 
improving the competitiveness of the productive sector through product and 
process innovation, especially by SMEs, is recognised in the context of 
Structural Funds. 

The Commission's First Action Plan for Innovation in Europe also stressed 
the need for all Community instruments to be mobilised to support RTD and 
innovation with an increased support from the Structural Funds through 
other forms of intervention such as Article 10 of the ERDF and some 
Community Initiatives (SME initiatives, ADAPT, LEADER II). 

The build up of a RTD capacity in assisted regions is a necessary but not a 
sufficient condition to enhance their competitiveness and economic perfor­
mance. The RTD investment effort should be related to the build up of the 
wider innovation system, in particular through the creation of durable links 
between the RTD system and the regional industrial structures. 

Within assisted regions, Community interventions for RTD and innovation 
under the Structural funds provide an opportunity to strengthen their capabi­
lities and to reduce the technology gaps with the most developed regions. 

Most objective 1 regions (and to a lesser extent objective 6 regions) face a 
number of problems relating in particular to the concentration of RTD related 
activities in the most developed areas, weak industrial participation as well 
as human resource shortages. In these regions, the provision of facilities and 
equipment and the availability of researchers are priorities as well as the 
application of research results and the transfer of innovation and technology 
to local business and industry. Most objective 2 regions, which often are 

Transversal evaluation of impacts on intervention priorities 

endowed with better RTD capacity, focus their actions towards the develop­
ment of specific segments in line with the needs of regional SMEs, and espe­
cially to technology and know­how transfer to local business and industry 
through regional and inter­regional co­operation networks. 

Given the complexities of developing RTD policies for both types of areas, it 
is imperative to have a detailed understanding of the current regional 
situation in order to determine what policies might be most appropriate. A 
region­specific analysis, according to characteristics of the objectives, 
enables current innovation and technology levels to be assessed and gaps 
to be identified in the policy framework. 

In view of the preparation of the post­1999 Structural interventions, the 
Commission is currently preparing a communication on RTD and innovation. 
Simultaneously, it is envisaged to launch a comprehensive thematic study to 
feed the reflection on these policy issues 

■ Aims and objectives of the study 

This section of the document should clearly list the central questions to be 
addressed by the evaluation. While these questions are obviously specific 
to the theme to be evaluated in terms of the programmes implemented 
and the impacts likely to be generated, they will be focused on key factors 
that are generally applicable to the evaluation of Structural Fund interven­
tions: 
• Impacts to be analysed 
• Effectiveness, efficiency, relevance and utility of interventions 
• Coherence and synergies between different levels of intervention (i.e. 
regional, national and European) 
• Coherence and synergies between different Community level interven­
tions (i.e. Community Framework programmes, Structural Fund pro­
grammes, Community initiatives, innovative or specific measures) 
• Future directions for policy 

The importance of research and technological development (RTD) for 
improving the competitiveness of the productive sector through product and 
process innovation, especially by SMEs, is recognised in the context of 
Structural Funds. 

The Commission's First Action Plan for Innovation in Europe also stressed 
the need for all Community instruments to be mobilised to support RTD and 

Appendix 

innovation  with  an  increased  support  from  the  Structural  Funds  through 
other  forms  of  intervention  such  as  Article  10  of  the  ERDF  and  some 
Community  Initiatives (SME initiatives, ADAPT, LEADER II). 

The build  up of a RTD capacity  in assisted  regions  is a necessary  but  not a 
sufficient  condition to enhance their competitiveness  and economic  perfor­
mance. The  RTD investment  effort  should  be  related to the  build  up of  the 
wider  innovation  system, in  particular  through the  creation  of  durable  links 
between the RTD system and the regional industrial structures. 

Within  assisted  regions,  Community  interventions  for  RTD and  innovation 
under the Structural funds provide an opportunity to strengthen their capabi­
lities and  to  reduce the technology  gaps with  the  most  developed  regions. 

Most  objective  1 regions  (and to  a lesser extent  objective  6 regions) face a 
number of problems relating in particular to the concentration of RTD related 
activities  in the  most  developed  areas, weak  industrial  participation  as well 
as human resource shortages. In these regions, the provision of facilities and 
equipment  and  the  availability  of  researchers  are  priorities  as well  as  the 
application of research results and the transfer of innovation and technology 
to  local  business  and  industry.  Most  objective  2 regions,  which  often  are 
endowed with better RTD capacity, focus their actions towards the develop­
ment of specific segments in line with the needs of regional SMEs, and espe­
cially to  technology  and  know-how  transfer  to  local  business  and  industry 
through regional and inter-regional co-operation  networks. 

Given the complexities  of developing  RTD policies for  both types of areas, it 
is  imperative  to  have  a  detailed  understanding  of  the  current  regional 
situation  in order  to  determine  what  policies  might  be  most  appropriate. A 
region-specific  analysis,  according  to  characteristics  of  the  objectives, 
enables current  innovation  and technology  levels to  be assessed  and  gaps 
to be identified in the policy framework. 

In view  of  the  preparation  of  the  post-1999  Structural  interventions,  the 
Commission is currently preparing a communication on RTD and innovation. 
Simultaneously, it is envisaged to launch a comprehensive thematic study to 
feed the reflection on these policy issues 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Scope 

The field  of  application  specified  should  delimit  the evaluation  according 
to  the  relevant  dimensions  such  as time,  geographic  zone, and  type  of 
intervention: 
•  On what  period should the evaluation focus? Will the evaluation  involve 
a comparison between programming  periods? 
•  Which  countries /  regions will  be  involved  in or excluded  from the  eva­
luation? 
•  Which policy  instruments would be included in the study? 

·-  Suggest  how  interventions  and  their  objectives  of  might  be categori­
sed. This will help give evaluators to better orient their proposals. 

Considering the content and policy relevance of RTD actions and the need to 
cover all relevant objectives  under Structural Funds, the study will be divided 
into two  lots. It will focus on the one hand on objective  1 and 6 regions (lot 1) 
and on the other hand on objective 2 regions (lot 2), but due consideration will 
be given where relevant to other forms of intervention such as Community  ini­
tiatives  and  innovative  actions  under  the  ERDF art.  10.  It  will  cover  both 
Structural Funds programming periods but particular attention will be given to 
the 1994-99 period, for which most information is available. 

The  study  will  also  consider  the  EU  Framework  Programmes  and  relevant 
national RTD policies including significant regional state aids for RTD. 

Regarding the actions or activities to be evaluated, there needs to be a clear 
classification of what makes up RTD activities. The coverage will be  broader 
than  RTD  in the  strict  sense  covering  largely  the  innovation  system.  The 
analysis  should  refer  to  a common  framework,  as  developed  under  the 
Action Plan for Innovation, in order to allow for a more coherent evaluation of 
RTD and innovation content of Structural Funds. For example, in objective 1, 
it should  be possible to  assess the catch-up  process  in terms  of  RTD infra­
structure,  and  to  distinguish  between  scientific  investments  (research 
centres,  research  projects...)  and  technological  investments  (technological 
parks, technology centres...). 

The suggested framework  encompasses the following  categories: 

(1) Enhancement of scientific and technological  potential 

•  Scientific potential (infrastructure, equipment, scientific  actions,) 
•  Technological  potential  (infrastructure,  equipment,  technological  actions,) 

Appendix 

(2) Support to Innovation 

• Promotion of an innovation culture, which includes human resources deve­
lopment through education and training measures, research mobility, stimula­
tion actions, dissemination of RTD results, promotion of public awareness, etc. 
• Regulatory and financial framework (legislative and regulatory environ­
ment, innovation financing through seed capital, venture capital or other) 
• Link between research and innovation , including technology transfer, 
promotion of networking, strengthening links between enterprises, universi­
ties and research centres 

In reviewing the different types of RTD innovation actions, it would also be 
interesting to consider relevant sectors of the Framework Programme such as: 
­ information and communication technologies, 
­ industrial technologies, 
­ life sciences, including bio­technologies and agriculture, 

in order to assess complementarity and synergy between Structural Funds 
interventions and the Framework Programme itself. 

As regards the development of RTD and innovation indicators, the Eurostat 
"Innovation manual" should constitute an essential source of information for 
this study. 

■ Content of the study 

This section outlines how the evaluation will be structured. It should 
present the methodological framework of the evaluation including: 
• evaluation tools to be used 
• which methods of data collection will be used 
• details of the types of analysis to be performed 

·■ The methodological framework should be drawn up in such a way as to 
ensure that the evaluators, while respecting its overall structure, feel able 
to use their experience and know­how to make practical suggestions on 
how the evaluation could be performed. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

It should comprise the following aspects: 

*  Methodological  framework 

This part will contain the development  of a methodology to allow for compa­
rative  analysis  and  measurement  of  performance  of  RTD  related  actions. 
This should include: 

(a) the adoption of a thematic approach requiring the comparative and simul­
taneous assessment of common themes and issues 

(b)  the  proposal  of  appropriate  indicators,  relating  to  inputs,  outputs, 
impacts, management and economic  context 

(c) the design of selected instruments to assess performance  of a sample of 
RTD related actions 

(d) the  design  of  questionnaire  forms  and  identification  of  key  informants 
(basis for field work) 

(e) the selection and implementation of case studies 

*  A  comparative  analysis  of  RTD  policies  and  programmes  and  the 
impact of Structural Funds  interventions 

In the  context  of  Structural  Funds, the  Commission  wishes  to  analyse  in  a 
comparative fashion: 

- t he  RTD  funding,  i.e.  the  amount  and  proportion  of  Structural  Funds 
budget  allocated  to  RTD,  and  a  comparison  with  national  or  other 
Community  funds  allocated  to  RTD for  each  relevant  objective,  member 
state and region; 

-  the importance assigned to different  RTD related actions and their diversity 
throughout the regions; 

- t he  quality  of  RTD  organisation  and  structures,  relating  in  particular  to 
management structures and evaluation procedures; 

-  the  effectiveness  of the  instruments  or  mechanisms  to  promote  RTD and 
innovation actions; 

-  the impact on human resources and training for RTD. 

*  Assessment  of  a  sample  of  RTD  related  actions  at  country/regional 

level (field work) 

This  part will consist  of  an in-depth  analysis  of  a significant  sample  of  RTD 
actions funded by the Structural Funds for each of the countries concerned. 

Appendix 

The selection of RTD actions will be based on criteria such as financial 
dimension and strategic importance. Fieldwork will be conducted in relevant 
countries or regions in accordance with a standardised methodology. 

• Assessment of Community value-added and best practices 

This part of the study will evaluate the RTD related actions funded by Structural 
Funds, and draw lessons for regional policy. Particular attention is to be paid to: 
- t he quality of RTD actions, and the changes observed between the 1989-
93 and 1994-99 programming periods; 
- the value added of Structural Funds actions in the field of RTD 

• the identification of best practices in terms of RTD industry linkages, of 
technology transfer mechanisms and institutional designs in the assisted 
regions and analysis of transferability conditions to other regions 
• the complementarity and synergy within Structural Fund programmes and 
with the EU RTD Framework Programme as well as with national RTD policies. 

• Synthesis of main findings and policy issues 

This part will draw conclusions and recommendations, with special regard to 
the future of structural policies. Some of the issues at stake have already 
been addressed in the first Cohesion Report and the current Communication 
on "RTD, Innovation and Cohesion". 

Nonetheless, further thought should be carried out on the following 
questions: 
• Should there be more RTD activities funded under the Structural Funds? 
• How should RTD actions be focused? 
• Have the Structural Funds been sufficiently considered in the Fourth 
Framework Programme and vice versa? 
• Should more be done in the future to achieve better synergy of both EU 
policies? 

■ Sources and methods 

The objective of this section is to outline the data collection stage of the 
evaluation. 

·■ Information should be provided on the balance between primary data 
generation and the use of secondary sources. This will allow evaluators to 
better plan the work programme and calculate its cost. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

The  development  of  the  evaluation  methodology  will  be  undertaken  in  co­
operation  with  the  Commission  services  and the  relevant  experts  selected 
for the study. The evaluation methodology  should comprise  both  qualitative 
and quantitative assessments. 

The study would involve two main elements: desk research and field work  in 
the regions concerned. 

•  Desk  research 

The consultant  should undertake a detailed  comparative  assessment  of the 
areas  for  which  he  will  be  appointed. The  review  will  concern  past  and 
current  programmes  (CSFs/SPDs,  OPs  and  other)  and their  main  compo­
nents  relating  to  RTD  activities  such  as  regional  analyses,  strategic 
approaches,  hierarchy of objectives, priorities and measures, indicators  and 
profiles, synergy between RTD measures and other components. 

The study  will draw  extensively  upon the  contents  of the  programmes,  and 
the  available  evaluation  exercises  for  those  programmes.  In  particular,  it 
should take into consideration the methodology and lessons drawn from the 
RTD  Thematic  evaluation  for  Ireland,  Portugal,  Greece  for  the  1989-93 
period. It should also take into account the results of recent evaluations such 
as the  mid-term  evaluations  (for  objective  1 regions)  and  the  evaluation  of 
RTPs (Regional Technology  Plans) under the ERDF art. 10. 

The consultant  is invited to collect data and statistics and to develop appro­
priate  indicators  which  may  be  relevant  for  evaluation  purposes  in order  to 
provide  key  elements  on  RTD  expenditure,  as  well  as  the  nature  and  the 
impacts of the actions funded. 

•  Field work 

In order to derive first hand information on the effects of RTD related actions, 
the consultant will interview key informants in the regions concerned. A short 
list  of  key  issues  (to  be  carefully  selected)  should  be  used to  structure  the 
questionnaire  or  checklists  for  use  in the  interviews.  A  number  of  case 
studies  will  be  selected  to  provide  in-depth  and  contextual  evidence  and 
information  on  best  practices  on  RTD  related  actions  under  the  Structural 
Funds. 

Appendix 

Deliverables 

The deliverables (inception report, interim and final reports, case study 
reports, etc.) that the evaluation will produce, both intermediate and final, 
should be listed and their contents detailed when necessary. 
• What format should be used for the production of documents? 
• What type of computer file should be used? 
• Which language(s) should be used? 

*■ The timetable for an evaluation is usually determined by the decision­
making calendar and not by methodological requirements. Rather than 
impose a timescale that may reduce the depth, scope or even quality of 
the evaluation, the production of an interim report can be an acceptable 
compromise. 

The proposed work programme and calendar should include: 
­ a detailed work programme covering essential questions and a methodo­
logical framework for the thematic study 
­ national reports covering the desk research and field work stages 
­ a final synthesis report, including a summary focusing on policies to be 
followed, which must be understandable and usable independently of the 
main report. 
­ All the texts and the corresponding database will be established or trans­
mitted (hard copy and diskette) in a format determined by DG XVI of the 
European Commission. 
­ Reports and documents will be presented in English, French or German. A 
supplementary budget is planned for translation into at least one of the two 
languages. 

Calendar 

The duration of the evaluation should be indicated along with any inter­
mediate deadlines. 

The study, divided into two separate lots will be completed in one year. 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Appendix 4: List of boxes 

Introduction 

Box 1 - Priorities, domains and themes 
Box 2 - Mainstreaming: a definition 

8 
11 

Technical considerations relating to the assessment of thematic 
impacts 

Box 3 - Using objectives trees for clarifying implicit and explicit objectives 
related to a given theme 
Box 4 - In-depth analysis of impacts: appropriate tools and constructing an 
evaluation method 
Box 5 - Typical strengths and weaknesses of sources of existing 
information 
Box 6 - Estimating impacts net of deadweight 
Box 7 - Additionality, deadweight  and leverage effects 

21 
23 
25 

16 

20 

Evaluating the environmental impacts of structural interventions 

Box 8 - An adaptation of the Driving Force-State-Response  (DF-S-R) 
model of sustainable development 
Box 9 - Detailed typology of environmental impacts of structural 
interventions 
Box 10 - Environmental objectives of Structural  Fund Interventions 
presented in the form of a thematic objectives tree. 
Box 11 - Example of a system of indicators for the construction of 
waste-water treatment plants 
Box 12 - Example of an environmental screening form 

33 

36 

37 

47 
53 

Evaluating the impact of structural interventions on the 
competitiveness of Small and Medium-Sized  Enterprises 

Box 13 - Definition of SMEs 
Box 14- Example of a thematic objectives tree for SME 
interventions 
Box 15 - Example of a system of indicators for an intervention  providing 
informational support for SMEs to promote exports 
Box 16 - Example of a factor analysis of the Catalan co-operative 
sector 

59 

65 

71 

74 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Box  17 - Example of the use of the case study technique in the 
evaluation of the NordTek programme, Denmark 

75 

Evaluating the impact of structural interventions on employment 

80 
80 

Box  18 - Different types of skill obsolescence 
Box  19 - Breakdown of different types of employment effects 
Box 20 - Production of employment  effects through structural 
interventions 
Box 21 - Structural  Fund interventions and employment  effect 
mechanisms 
Box 22 - Example of a thematic objectives tree for the promotion of employ­
ment 
Box 23 - Comparison of deadweight, displacement and substitution 
effects 
Box 24 - Example of a system of indicators for a return-to-work 
programme 

86 

89 

97 

82 

82 

Evaluating the impact of structural interventions on equal opportunities 

Box 25 - Example of a thematic objectives tree for the priority of equal 
opportunities 
Box 26 - Example of effects  by category of impact 
Box 27 - Example of the application of the principle of 
mainstreaming 
Box 28 - Example of a system of indicators for the establishment  and 
re-generation of community centres 
Box 29 - Standard form for assessing the equal opportunities dimension in 
the framework of ex ante evaluation 
Box 30 - Matrix for programme assessment 

121 

124 
125 

112 
114 

120 

Evaluating impacts arising from interventions in the domain of 
Research and Technological  Development 

Box 31 - Thematic evaluation of the impacts of CSFs for Research and 
Technology  in Greece, Ireland and Portugal for the 1989  -1993 
programming period 
Box 32 - Framework for the evaluation of Structural Fund interventions in 
the domain of RTD 
Box 33 - Example of a thematic  objectives tree for RTD 
interventions 

134 

138 

139 

Appendix 

Box 34 - Example of a system of indicators for an intervention  providing 
financial and informational support for technology acquisition by 
SMEs 
Box 35 - Example of a score card for a joint research venture 

145 
146 

Evaluating impacts arising from interventions in the domains of 
transport, telecommunications and tourism 

Box 36 - Example of a system of indicators for upgrading of a railway line for 
freight operations 
156 
Box 37 - Example of a system of indicators for the deployment of a 
broadband Hybrid Fibre Coax (HFC) network 
Box 38 - Example of support for the creation of tourist facilities (museums, 
theme parks, etc.) 
Box 39 - Example of a SWOT analysis matrix 
Box 40 - Extract from a "corridor ranking framework" applied to 
Greece 

160 
161 

157 

163 

A step-by-step approach for thematic analysis 

Box 41 - Transversal evaluation of a theme : "Step by step" approach for a 
programme 
Box 42 - Thematic assessment matrix. Example relating to equal 
opportunities 
Box 43 - Score card. Example: assessing the extent to which a measure 
takes into account of the priority of equal opportunities 
Box 44 - Interviews, focus groups and case studies 
Box 45 - Summary table of measures and their effects. Example of 
improving the suitability of living conditions to women's needs 
Box 46 - Summary of the key steps and evaluation tools 

170 

173 

174 
175 

176 
177 

Cross-programme thematic  evaluation 

Box 47 - Comparison of supra-national synthesis of programme evaluations 
180 
and transnational evaluation 
183 
Box 48 - Sampling with the objective of highlighting best practice 
Box 49 - The development of transnational evaluations 
184 
Box 50 - Sampling with the objective of making a pan-European 
analysis 
Box 51 - Example of multiple-case studies with replication design 
Box 52 - The seven steps of research synthesis 

185 
187 
188 

Transversal  evaluation  of  impacts  on  intervention 

priorities 

Index 

ethnographic observation, 123 
Europe-wide analysis of impact, 179 
evaluation research See research synthesis, 

187 

evaluation team, 16,120,169,178,181 
evaluatory question, 17 
expert panel, 148 
external coherence, 15 
externality, 137,148,164 

-F-

factor analysis, 74 
feasibility study, 180 
focus group, 73,123,175 
framework conditions, 62 
full-time employment, 94 
full-time equivalent (FTE), 85 

-G-

gross employment impact, 23, 87 
gross impact, 22 

-I-

impact descriptor, 172 
implicit objective, 15 
income multiplier effect, 24,25,82,88 
indicator, 18 
indicator (examples of), 47, 71,97,121,145, 

156,157 

indirect effect See induced effect, 155 
induced effects, 155 
information system, 20 
input-output model, 104 
internal coherence^ 5,38 
interview technique, 73,123,127,149,160, 
175 
Ishikawa diagram, 123 

job creation or maintenance effect, 77,80, 

88,101 

absorption, 86,133,137 
activity rate, 89 
ad hoc indicator, 44, 94,118 
additional effects, 25 
additionality, 25,137 
attraction effect, 89,103 

-B-

baseline, 23,43,119 
benchmarking, 182 
best practice, 63,123,182 
bibliometric indicator, 148 

-C-

case study technique, 54, 75101,123,125, 
149,160,163,175,181 
coefficients, 22,55,95,103,184 
colour vote, 123 
Community value-added, 179 
comparative analysis, 102,123,186 
confounding factor, 18,79,95 
context indicator,  17,72,119,122,147,171 
contingent valuation method, 55 
control group, 87,102 
cost-benefit analysis, 55,148,163 
cost-effectiveness analysis, 101,135 
counterfactual situation, 22,65,73, 87,99, 

119 

deadweight, 22, 38, 65, 87,101,103,140, 
159 
DELPHI technique, 164 
demand-side effect, 155 
displacement, 22,38,65,88,101,159 
domain of intervention, 129 
Driving Force-State-Response model, 33,43 

-E-

employability, 78, 79,95 
employability effect, 80,88,101 
employment effect, 25, 74,77,80 
environmental impact assessment (EIA),29, 

36,52 

leverage effect, 24,66, 70,140,144 

Transversal evaluation of impacts  on intervention  priorities 

-M-

strategic environmental assessment (SEA), 

32,50 

substitution, 22, 65, 88,103,140,159 
summary table, 176 
supplier effect, 25,84,88 
supply-side effect, 155 
sustainable development, 29 
sustainable employment, 84,94 
SWOT analysis, 123,161,182 
synergy effect, 65, 79,136,154,164 

-T-

technical committee, 180 
temporary employment, 84, 93 
terms of reference, 180 
thematic assessment matrix, 172,177 
transitory employment, 84 
typology of impact, 35,113,139,172 

-U-

unemployment, 77,98,125 
unemployment (effect on), 89 

macro-economic model, 104 
mainstreaming, 11,  70,107,120 
matched pairs, 175 
matrix of cross impacts (SMIC), 164 
mitigation, 42 
monitoring data, 73, 99,147,185 
multi-country econometric analysis of 
impact, 184 
multi-criteria analysis (MCA), 55 
multiple case studies with replication design, 

186 

multiple case study technique, 55 

-N-

net employment impact, 79, 87 
net impact, 22,38,66,101 

-O-

objectives tree,  16,37,65,86,112,139,171 

-P-

part-time employment, 94 
policy-off situation, 22 
political priority, 16,27 
positive discrimination (affirmative action), 

111 

primary data, 20, 71,147 

-Q-

questionnaire survey, 73,101,123,127,148 

reference situation, 17,96 
regression analysis, 102 

-s-

sampling, 181,185 
scenario technique, 123 
scoping, 50, 52,171,174 
score card, 146,173 
scores and weights, 160 
scoring See scores and weights 
screening, 50, 52,169 
shift-share analysis, 102 
spillover, 148 
statistical surveys, 21,98 
step-by-step approach, 169 

Acknowledgements 

The preparation of this Volume took place over several years from 1996 to 1998. The pre­
paratory work included the following steps: 

Several workshops and meetings which allowed the points of view of experts in different 
themes and the transversal (or multi-thematic)  perspective  of those responsible for the 
Structural Funds to come together. 

This process firstly gave rise to the production of two MEANS Handbooks on employment 
(Handbooks  3 and 6) as well as a methodological  note  on evaluating  human resource-
related interventions. These documents were used to provide the basis of the chapter on 
employment. 

Similarly the  chapters  on the environment  and equal opportunities  are both based on 
recent  methodological  notes  resulting from a confrontation  of the ideas of  experts and 
Commission officials. The consultation procedure employed during the preparation of the 
note on the environment  is particularly  noteworthy  for  its  use of the  Impact  Mapping 
technique (see Volume 3 of the MEANS Collection). 

The production of this Volume was coordinated by Anastassios Bougas and David Sweet 
(D.G. XVI.G.2  : Coordination of Evaluation) and involved several  Directorate-Generals  of 
the European Commission, notably D.G. II, V, VI, XI, XIV and XVI and in particular Philippe 
Goybet, Anne Bucher, Nelly  Bandarra, Miguel  Benito Alonso, R. Bradley, Ciaran Dearie, 
Sylvie Finne, Martin Hallet, Richard Harding, Mikael Kirosingh, Frédérique Lorenzi, Andrea 
Mairate, René Marty, Valéry Morard, Irmengard Nelles, Anastasios  Nychas, Julius Op de 
Beke, Sergio Piccolo, Brigitta Renner-Loquenz, Jacqueline Soulier Oliveira, Martin Tyrrell, 
Martine Voets and Dominique Van Der Wee. 

This Volume has been written by the Centre for  European  Evaluation Expertise, notably 
with the participation, Cristina Diaz Gomez, Andreas Esche, Connor  Kearney and Kevin 
Williams. It was translated into English by Liz Libbrecht. 

The authors wish to thank all those who contributed, either directly or indirectly, to the pro­
duction of  this  Volume, and  in particular  the following  experts:  Christine  Abrossimov 
(DARES), Mariano d'Antonio  (Universita da Roma), Peter Auer  (Wissenschaftszentrum, 
Berlin), Jean-Claude Barbier (Centre d'Etudes pour l'Emploi), Martin Bekker (B&A Groep), 
Peter  Bosch  fThe  European  Environment  Agency),  Marco  Causi (CLES  Sri), Antonio 
Estevan Estevan (Gabinete de Economia Aplicada), Marta Ferrés (Interés General), Chris­
topher  Hasluck  (University of Warwick), Andy  Hirst  (PA Cambridge  Economics Consul­
tants), Liisa Horelli (ITSU OY), Japp de Koning (Netherlands Economic Institute), Kierän A. 
Kennedy  (The Economic  and Social  Research  Institute), Ulla Koski  (Finnish  Ministry of 
Environment)Marie-Christine  Lefebvre  (Lconsult  SPRLHedwig  Lutz  (WIFO),  James 
Medhurst (Ecotec), Dr Heino Von Meyer.Chris Newton, (The Environmental Agency), John 
Rhodes (PA Cambridge Economics Consultants), Barry Sadler (Institute of Environmental 
Assessment), Claude  Seibel (DARES), Thanos  Skouras (Reseach Centre of the Athens 
University  of  Economics  & Business), Antoni  Soy  (Centro  d'Estudis  de  Planiflcacio), 
Joachim Spangenberg (Wuppertal Institute for Climate, Environment & Energy), Peter Tyler 
(PA Cambridge Economics Consultants), Pieter Van Run (CBEA), Claudia Villante (ISFOL). 

European  Commission 

Evaluating socio-economic  programmes 
Transversal evaluation  of  impacts on the  environment,  employment 
and other  intervention  priorities 

Luxembourg: Office for Official  Publications  of the  European Communities,  1999 

1999 —  215  p p .—  14.8x21  cm 

ISBN  92-828-6626-2 

Venta  ·  Salg  ·  Verkauf  ·  Πωλήσεις  ·  Sales  ·  Vente  ·  Vendita  ·  Verkoop  ·  Venda  ·  Myynti  · Försäljning 

BELGIQUE/BELGlE 

ÖSTERREICH 

Jean De Lannoy 
Avenue du Roi _02/Koninn__an 202 
B­1190 Bruxe«es_russo[ 
Tel. (32-2)538 43 08 
Fax (32-2) 538 06 41 
E­mail, jean de.LannoyOintoooard.be 
URL: http www jean­ae­iannoy De 

La librairie européenne/ 
De Europese Boekhandel 
Rue de la Loi 244/Wotsttaal 244 
B­1040 Pruid'es­Brussel 
Tel. (32-2)295 26 39 
Fax (32-2) 735 08 60 
E­mail: mailOlibeurop be 
URL: hrtpJ/www hbeurop.be 

Moniteur bel ge/Belg lac h S t u t s b l ad 
Rue de Louvam 40­42/leuvenseweg 40­42 
B­1000 Bruxelles/­russel 
Tôt. (32­2)552 22 11 
Fax (32­2) 511 01 84 

DANMARK 

J. H. Schultz Information A/S 
Heretedvang 10­12 
DK­2620 Albertslund 
Tit (45)43 63 23 00 
Fax (45) 43 63 19 69 
E­mail. schuttzOschuüz dk 
URL: htlp:/Avww Schultz dk 

DEUTSCHLAND 

Bundesanzeiger Vertag GmbH 
Veni iL'bsaDtei lung 
Amsterdamer Straße 192 
D­50735 Köln 
Tel (49-221)97 66 80 
Fax (49-221) 97 66 82 78 
E­Mail: vertnebObundesanieiger.de 
URL: rittp^/www.bundesanzeiger.de 

ΕΛΛΑΔΑ/GREECE 

G. C. Eleftheroudakl· SA 
International Bookslore 
Panepisiimrøu 17 
GR­10564 Athina 
Tel (30­1 ) 331 41 80/1/2/3/4/5 
Fax (30­1)323 9821 
E­mail: elebooksOnelof gr 

ESPAÑA 

Boletín Oficial del Estado 
Trafalgar. 27 
E 28071 Madrid 
Tel (34)915 38 21 11 (Libros). 
913 B4 17 15(Suscnp.) 
Fax (34) 915 38 21 21 (Libros). 
913 B4 17 14 (Suscflp.) 

E­mail: clientes O com boe es 
URL httpi/www boe es 
Mundi Prensa U b r o s, SA 
Castello. 37 
E­28001 Madrid 
Tel (34)914 36 37 00 
Fax ¡34)915 75 39 98 
E ­mail 11 br e na O mundiprensa. es 
URL: httpy/www.murtdipiensacom 

FRANCE 

Journal ottici«! 
Service des pubi­cali on s des CE 
26. rue Desaix 
F­75727 Paris Cedex 15 
Tel (33) 140 58 77 31 
Fax (33) 140 58 77 00 
URL: http //www journal'Officiel gouv Ir 

IRELAND 

Government Supplies Agency 
Publications Section 
4­5 Harcourt Road 
Dublin 2 
Tel (353­1)681 31 11 
Fax (353­1)47527 60 

ITALIA 

Licosa SpA 
Via Duca di Calabria. 1/1 
Casella postale 552 
1­50125 Firenze 
Tel (39) 055 64 83 1 
Fax (39) 055 64 12 57 
E­mail: hcosaOftbcc it 
URL: http //www.ftbcc.it/licosa 

LUXEMBOURG 

Messageries du livre SARL 
5. me Raiffeisen 
L ­2411 Luxembourg 
Toi (352)4010 20 
Fax (352) 49 06 61 
E mail mailOmdl lu 
URL­ http //www.mdl lu 

NEDERLAND 

SOU Servicecentrum Uitgever* 
CiriMorfei Planton straat 2 
Postbus 20014 
2500 EA Den Haag 
Tel (31­70) 378 90 80 
Fax (31­70) 378 97 83 
E­mail sduOsdu n 
URL: utip . www sdu nl 

Manz'sche Verlags­ und 
Universitátsbuchhandlung GmbH 
Kohlmarkt 16 
A­1014 Wien 
Tel. (43­1) 53 16 11 00 
Fax (43­1)53 1611 67 
E­Mail: bestellen β manz co at 
URL r, tip ://www. manz. at/index, htm 

PORTUGAL 

Distribuidora de Livros Bertrand L d .' 
Grupo Benrand. SA 
Rua das Tertas dos Vales. ■'■ ­ A 
Apañado 60037 
Ρ­2700 Amadora 
Tel. (351-1)495 90 50 
Fax (351­1) 496 02 55 
Imprensa NaclortaI­Casa da Moeda, EP 
Rua Marqués Sá da Bandeira, 16­A 
Ρ­1050 Lisboa Codex 
Tel.(351­1)353 03 99 
Fax (351­1)353 02 94 
E­mail del irtcmOmail lolepac pt 
URL: http^/www.incm.pl 

SUOMI/FINLAND 

Akateeminen Kirjakauppa/ 
Akademiska Bokhandeln 
Keskuskatu i /Cent raiga tan 1 
PL/PB 128 
FIN­00101 Helsinki/Helsingfors 
PJtfn (358­9) 121 44 18 
V : . i. (358­9)121 44 35 
Sähköposti. akatilausOakaloeminen.com 
URL. htlp^/inrww.akaieeminen com 

SVERIGE 
B T J AB 
Traktorvager 11 
S­221 82 Lund 
TIn (46­46) 18 00 00 
Fax (46­46) 30 79 47 
E­post bljfìu­pubQblj.se 
URL: hnp://www b1j.se 

UNITED KINGDOM 

The Stationery Office Lid 
International Sales Agency 
51 Nine Elms Lane 
London SW8 5DR 
Tel. (44­171)373 90 90 
Fax (44­171)873 64 63 
E­mail: ipa.enqumesOlheso.co uk 
URL http7/www the­stabonery­oftice co.uk 

ISLAND 

Boka bud La rusar Blondel 
Skólavordustig. 2 
15­101 Reykjavik 
Tel (354) 551 56 50 
Fax (354) 552 55 60 

NORGE 

S wet s Norge AS 
O s le n j ovo ie n 18 
Boks6512Et1erstad 
N - 0 6 0 6 O s_ 
Tel. (47­22) 97 45 00 
Fax (47­22) 97 45 45 

SCHWEIZ/SU ISSE/SVìZZERA 

Euro Info Center Schweiz 
_o OSEC 
Slamptenbachstrafle 85 
PF 492 
CH­8035Z_nch 
Tel. (41­1) 365 53 15 
Fax (41­1) 365 54 II 
E­mail eicsOosecch 
URL: htlp://www osee ch/eics 

BAL GARDA 

Europrees Euromedia Lid 
59. brvd Vilosha 
BG­1000 Solia 
Tel (359-2) 960 37 66 
Fax (359-2) 980 42 30 
E­mail MilenaCmboi cri bg 

CESKA REPUBLIKA 

ÚSIS 
NlS­prodejna 
H a velkova 22 
CZ­l30 00Praha3 
Tel (420­2)24 2314 86 
Fax (420­2) 24 23 11 14 
E­mail nkpospOdoc nis cz 
URL: http//ustscr cz 

CYPRUS 

Cyprus Chamber of Commerce and Industry 
PO Box 1455 
CY­1509 Nicosia 
Tel (357-2)66 95 00 
Fox (357­2) 66 10 44 
Email demelrapOccci org cy 

HHVATSKA 

Medialrade Lid 
Pavia Hatza 1 
HR­10000 Zagreb 
Tel. (385­1)481 94 1 
Fax (385­1) 481 94 1 

MAQYAflORSZAG 
Euro into Service 
Europa Hâz 
Margiiszigel 
PO Box 4 75 
H­1396 Budapest 62 
Tel- (36-1)350 80 25 
Fax(36­t)350 90 32 
E ­ m a 11 e υ r ο ι π ίο « î mail mata ν hu 
URL: http//www.eurotnlo.hiilnoex him 

MALTA 

Miller Distributors Ltd 
Malta International Airporl 
PO Box 25 
Luqa LG A 05 
Tel (356) 66 44 88 
F a x ( 3 5 6 ) 67 67 99 
E­mail: gwirthOusa.net 

POLSKA 

Ars Polona 
Krakowskie Przedmiescie 7 
Skr. pocztowa 1001 
PL­00­950 Warsiawa 
Tel (48­22)826 12 01 
Fax (4S­22) 826 62 40 
E­mail: ars pol ­ b e vy hsn com pi 

ROMANIA 

Euromedia 
Str. G­ral Berthelot Nr 41 
RO­70749 Bucuresti 
Tel (40­1)3154403 
Fax (40­11314 22 86 

R05SIYA 

CCEC 
60­letiya Oktyabrya Av 9 
117312 Moscow 
Tel. (7-095) 135 52 27 
Fax (7­095) 135 52 27 

SLOVAKIA 

Centrum VTI SR 
Nam Slobody, 19 
5K­61223 Bratislava 
Tel (421­7)54 41 83 64 
Fax (421­7) 54 41 83 64 
E­mail: europOlbb1.5tlk.stuba.sk 
URL: http .'.'www 5ltk.stuba sk 

5 LOVEHIJA 

Gospodarski Vestnlk 
Dunajska cesia 5 
SLO­10OO Ljublana 
Tel (386)613 09 16 40 
Fax (386)61309 1645 
E­mail: europe ovest m k si 
URL: httpJ/www.gveslnik.si 

TÙRKIYE 

Dünya Intotel AS 
100. Yil Mahallessi 34440 
TR­80O5O Bagolar­Istanbul 
Tel (90­212)629 46 89 
Fax (90­212) 629 46 27 
E­mail, intolel θ dunya­gazeie com.tr 

AUSTRALIA 

Hunter Publications 
PO Box 404 
3067 Abbolsford. Victoria 
Tel (61­3)94 17 53 61 
Fax (61­3) 94 1971 54 
E­mail: jpda «es Ooze mail com au 

CANADA 

Les «dillons La Liberté Inc. 
3020. chemin Samte­Foy 
G1X 3V Samle­Foy, Quebec 
Tel (1­418)658 37 63 
Fax (1­800) 567 54 49 
E­mail libertoOmodtom.oc ca 

Rr­nouf Publishing Co. Ltd 
5369 Chemin Canotek Road Unii 1 
K1J 9J3 Ottawa. Ontano 
Tel (1­613)745 26 65 
Fax(1 ­613)745 76 60 
E­mail: order deptOrenoutbooks.com 
URL: http //www renoufbooks com 

EGYPT 

The Middle East Observer 
41 Shenf Street 
Cairo 
Tel (20-2)392 69 19 
Fax (20­2) 393 97 32 
E­mail matoudaOmeobserver com eg 
URL: htlp7/www.meobserver com.eg 

EESTI 

INDIA 

Eesti Ka u b i nd us­Toost vi skods (Estonian 
Chamber of Commerce and Industry) 
Toom­Kooli 17 
E E­0001 Tallinn 
Tel (372)646 02 44 
Fax(372)646 02 45 
Email einloOkodaee 
URL: htlp .'www koda ee 

EBIC India 
3rd Floor. Υ Β Chavan Centre 
Gen J. Bhosale Marg 
400 021 Mumbai 
Tel (91­22)282 6064 
Fax ¡91­22)285 45 64 
E­mail: BbicOglasbmOl vsnl.net.in 
URL: htlpy/www ebondia com 

ROY luteiiintiun:il 

41. Mishmar Hayarden Street 
PO Box 13056 
61130 Tel Aviv 
Tel. (972-3)649 94 69 
Fax (972-3) 648 60 39 
E­mail: royilOnetvision.net il 
URL: htlp^/www foyinl.co il 

Sub­agent for the Palestinian Authority: 

Index Information Services 

PO Box 19502 
Jerusalem 
Tel. (972­2) 627 16 34 
Fax (972­2) 627 12 19 

Ρ Si­J apan 

Asahi Sanbancho Plaza «206 
7­1 Sanbancho. Chi yoda­ku 
Tokyo 102 
Tel (81­3)32 34 69 21 
F_x(81­3Í32 34 6915 
E­mail: books ü ps ι­japan co jp 
URL: httpiWwww psi ­japan com 

EBIC Malaysia 

Level 7. Wisma Hong Leong 
1B Jalan Perak 
50450 Kuala Lumpur 
Tel. (60­3) 262 62 98 
Fax (60­3) 262 61 98 
E­mail: ebic­klOmol.net.my 

Mundi Prensa Mexico. SA do CV 

Rio Panuco No 141 
Colonia Cuauhtemoc 
MX­06500 Mexico. DF 
Tel (52­5) 533 56 58 
Fax (52-5) 514 67 99 
E­mail: 1015452361 Ocompuserve com 

EBIC Philippines 

19th Floor, PS Bank Tower 
Sen Gil J. Puyal Avo. cor. Tindalo Si. 
Makati City 
Metro Manilla 
Tel (63-2)759 66 80 
Fax (63-2) 759 66 90 
E­mail: occpcomOglobe com ph 
URL: hltpJAwww eccp.com 

EBIC Sri Lanka 

Trans Asia Hotel 
115 Sir chittampalam 
A Gardiner Mawatha 
Colombo 2 
Tel. (94­1)074 71 50 78 
Fax (94­1)44 87 79 
E­mail: ebicsieitmin.com 

EBIC Thailand 

29 Vanisse Building. Btrt Floor 
Soi Chidlom 
Ploenchl 
10330 Bangkok 
Tel (66-2)655 06 27 
Fax (66-2) 655 06 28 
E­mail: ebicbkkOkscl5.thcom 
URL: httpJVvww obicbkk.org 

UNITED STATES OF AMERICA 

Berna n Associates 

4611­F Assembly Dnve 
Lanham MD207Ó6 
Tel (1­800) 274 44 47 (toll Iroe telephone) 
Fax (1­800) 865 34 50 (ton Iroe lax) 
E­mail: queryObeman.com 
URL http //www beman com 

Bille wenden Sla sich an ein Büro Ihrer 
Wahl/ Please contaci the sales office 
of your c h o i e ·/ Veuillez vous adresser 
BU bureau de venie de votre choix 

Office tor Official Publications 
ot lha European Communities 
2. rue Mercier 
L­2985 Luxembourg 
Tel (352)29 29­42455 
Fax (352) 29 29­42758 
E­mail: into info Oopoce cec be 
URL: httpJ/eur­op.eu int 

The  MEANS  collection: 
"Evaluating  socio-economic  programmes" 

This  six-volume  set  gives  wider  public  access  to  the  lessons  of  the  MEANS 
programme which the  European Commission  introduced with a view to  impro­
ving  and  promoting  evaluation  methods.  Its  publication  meets  a  growing 
need for  reliable  benchmarks  to  deal with  the  problems  of evaluation  encoun­
tered when implementing  public  measures in general and Structural  Fund ope­
rations  in  particular. 

In  this  context,  the  set  constitutes  a  unique  methodological  guide  that  can 
solve  the  technical  and  organisational  problems  arising  in  the  course  of 
evaluation  work  and for  which  the existing  literature  provides  little  help. 

The  collection  is  aimed  not  only  at  professional  evaluators,  for  whom  it  will 
provide a valuable  handbook,  but also at non-specialists,  in particular  manag­
ing  authorities  and  structural  programme  Monitoring  Committees. 

Evaluation  design  and  management 

Each  volume  is  designed  to  be  independent  of  the  others,  yet  they  are  also 
complementary. Together, they  deal with all aspects of the evaluation of  socio­
economic  programmes: 
I. 
II.  Selection  and  use of  indicators  for  monitoring  and  evaluation 
III.  Principal  evaluation  techniques  and  tools 
IV.  Technical  solutions  for  evaluation  within  a partnership  framework 
V.  Transversal  evaluation  of  impacts  on  the  environment,  employment  and 

other  intervention  priorities 

VI.  Glossary  of  300  concepts  and technical  terms 

OFFICE FOR OFFICIAL  PUBLICATIONS 

OF THE EUROPEAN  COMMUNITIES 

_-2985  Luxembourg 

EC Structural Funds 

iluating socio-economic programmes 

Glossary  of  300  concepts 
and technical  terms 

n Commission 

EC Structural Funds 

Evaluating socio-economic programmes 

Glossary  of 300  concepts 
and technical  terms 

Volume  6 

European Commission 

MEANS is a programme  of the European Commission. 
It is aimed at improving Methods  of Evaluating Structural Policies. 
The MEANS Programme and Collection  was supervised by 
Phillipe Goybet and Miguel Angel Benito Alonso, 
Unit for  the Coordination  of Evaluation in the Directorate for Coordination 
and Evaluation of  Operations of DG XVI 
Coordination-evaluation@fmb.dg16.cec.be 

The realisation of  the Programme was entrusted to the 
Centre for European Evaluation Expertise  (C3E) 
under  the direction  of Eric Monnier and Jacques Toulemonde. 
Responsibility for the opinions expressed in this 
book lies solely with  the authors of the text: 
C3E  -  13 bis Place Jules Ferry  -  F  -  69006 Lyon  -  France  -  c3e@c3e.fr 

A great deal of add tional information on the European Union is available 
on the Internet. 
It can be accessed through the Europa server (http://europa.eu.int) and 
on the Inforegio Website (http://inforegio.cec.eu.int). 

Cataloguing data can be found at the end of this  publication. 

Luxembourg: Office for Official Publications of the  European 
Communities, 1999 

ISBN  92-828-6626-2 

©  European Communities, 1999 
Reproduction is authorized, except for commercial purposes,  provided 
the source is acknowledged. 

PRINTED ON WHITE CHLORINE-FREE  PAPER 

Printed in Italy 

Foreword 

Developing  good  evaluation  practices 
has  become  a  priority  area  for  the 
Commission  and  is  directly  relevant 
to 
the  effective  use  of  Community 
resources. 
Indeed, the sound financial  management 
principles  which  the  Commission  seeks 
to  promote  via  its  SEM2000  (Sound  and 
Efficient  Management)  Programme  rely, 
specifically,  on  the  regular  evaluation  of 

the  Community  activities. 

fe^isi 
j| 

___. 
Jk  ^L 

^ ^ /
¿È 

^k 

^L 

__t 

I  _K__i 

The  Directorates  General  responsible  for 
structural  policy  have considerable expe-
rience in this area.  Since 1988 they have 
made  evaluation  an  integral  part  of  each  stage  in  the  process  of 
Community  Structural  intervention.  Thus,  the  programme  development 
stage  includes  a  prospective  evaluation;  the  implementation  stage  a 
strengthening of the monitoring systems and, at the end of the intervention, 
evaluation  establishes  what  has  been  achieved  in  terms  of  programme 
impacts. 
However, the capacity  of evaluation to  contribute  to the effectiveness  and 
transparency  of  structural  interventions  rests  ultimately  on  the  integrity  of 
the methods  adopted  to verify the credibility  of results. 
This  is why, four  years ago, on the  initiative for the  Directorate  General for 
Regional  Policy  and  Cohesion  (DG  XVI),  the  MEANS  programme  was 
initiated as a response to the demand for developing  evaluation  methodo­
logy.  This  also  include  practices  combining  both  general  scientific 
requirements  and  the  specific  requirements  of  Community  interventions 
whilst, at the same time, remaining  accessible to  a large audience. 
One of  the  more tangible  results  of  the  MEANS  programme  has  been  the 
publication of this six volume set by a group of independent  experts, focu­
sing  on  the  principal  elements  of  socio-economic  programme  evaluation. 
A short summary of this work can be found at the  beginning of the  present 
volume. This collection covers the various methodological  approaches  and 
innovations  that  have  taken  place  in the  context  of  structural  funds  eva­
luation.  It  will  enable  those  responsible  for  evaluation  to  develop  their 
approach  in  a  flexible  and  independent  manner  and  for  the  various 

programme  partners  to  exercise  greater  control  over  the  evaluation  pro­
cess.  Attending  the  conferences  of  evaluation  specialists  and  "clients" 
convened  in  Brussels  (1995),  Berlin  (1996)  and  Seville  (1998),  I  gained 
some impression  of the  interest that  new evaluation techniques  can gene­
rate.  I would  stress  that  the  development  of  evaluation  needs to  progress 
in  tandem  with  efforts  to  incorporate  evaluation  findings  in  the  decision 
making  process. This  implies  evaluation  as  a means  of  establishing  whe­
ther  policies  are equitable, effective  and relevant to the identified  needs. 
In satisfying  these  requirements,  evaluation  should  be  seen  as  a valuable 
instrument  affording  both  Member  State  and  Community  authorities  the 
necessary  guarantees  that  Community  funds  are  being  deployed  in  the 
best  interests of the  EU citizen. This is a cost worth  paying when the goals 
of  Economic  and  Social  Cohesion  and  the  European  Union  enlargement 
are at stake. 

Finally,  I wish  to  extend  my  thanks  to  the  members  of  the  Committee  of 
Independent  Experts  established  at  the  launch  of  the  programme.  Their 
expertise and standing  have contributed greatly to the quality and relevan­
ce  of  the  MEANS  programme.  I therefore  thank:  E.  Chelimsky  (USA),  ex-
President  of  the  "American  Evaluation  Association";  K.  Kennedy  (IRL), 
Director  of  the  "Economic  and  Social  Research  Institute"  (ESRI);  J.  R. 
Cuadrado  Roura  (SP),  President  of  the  "European  Regional  Science 
Association"; C. Seibel (F), Director of the "Statistiques  démographiques  et 
sociales de l'Institut  National de la Statistique et des Etudes  Economiques" 
(INSEE);  L.  Senn  (I),  Director  of  the  Department  of  Economie  Regional  of 
the  University  Bocconi;  E. Stern (U.K.),  Director  of the unit  of evaluation  of 
the  Tavistock  Institute;  A.  Sorber  (t)  (NL),  Head  of  the  "Policy  Analysis 
Department  of the Ministry of  Finance"; L. I. Strom (SV), Statens  Institut för 
Regionalforsking  (SIR);  L.  Tsoukalis  (GR.),  professor  at  the  University  of 
Athens;  H. Wollmann  (D), professor  at the  University  of  Humboldt  -  Berlin 
and founder  member of the  "European  Evaluation  Society". 

E. Landaburu 

The MEANS Collection: 

"Evaluating  socio-economic  programmes" 

The  evaluation  of  European  Union 
structural 
interventions  has  entered  into  a  phase  of  maturity, 
attested by the extension of work  carried out by both the 
Member  States and the Commission.  Regulations  based 
on  decisions  by the  Council  have  largely  contributed  to 
the  development  of  evaluation  practice  and  have 
stimulated  methodological  research. 

initiated  by  the  Commission 

Over  the  past ten  years, the  Commission  has  striven  to 
promote  and  improve  methods  for  evaluating  Structural 
Funds. The main  methodological  conclusions  have been 
debated  and  disseminated  as  part  of  the  MEANS 
in  1991. 
programme 
Achievements  to  date  provide  enough  material  for  a 
complete  and  accessible  publication  intended  for  a 
broad public: a six volume Collection covering all aspects 
of the evaluation  of socio-economic  programmes. These 
Volumes  are  written  with  a  perspective  that  is  broader 
than  the  current  administrative  framework,  for  this  is 
likely to evolve more rapidly than professional standards. 

information  on 

In general, the Collection  is aimed at a readership of non-
specialists  seeking  essential 
the 
evaluation of socio-economic  development  programmes. 
These  programmes  have  multiple  objectives,  a 
partnership  dimension  and  a  multi-sectoral  content  -  all 
characteristics which call for suitable evaluation  methods 
and  solutions  not  yet  provided  by  the  international 
technical literature. The complete Collection consists of 6 
volumes  designed  to  be  complementary  and  to  provide 
essential 
for 
professionals  wanting  to  independently  develop  their 
own evaluation practices. Each volume can, however, be 
read and  understood  without  having to  refer to the other 
volumes  in the Collection. 

recommendations 

information  and 

Glossary of 300 concepts and technical terms 

The  first  Volume  in  the  MEANS  Collection  entitled 
"Evaluation  design  and  management"  serves  as  an 
introduction  both to  the  Collection  and to the  evaluation 
of  socio-economic  programmes.  It  enables  readers  to 
understand  the  implications  of  evaluation  and  to  know 
what they can and cannot expect from an evaluation at a 
given  time  in  a  particular  context.  It  covers  all  the 
elements needed to steer an evaluation project from A to 
Z:  sharing  of  responsibilities,  defining  of  the  aims  and 
method  of  an  evaluation,  establishing  a  schedule  and 
budget, assessing the quality of the work performed and, 
finally, managing the use of  conclusions. 

The second Volume focuses on the "Selection and use of 
indicators  for  monitoring  and  evaluation".  It  deals  with 
the  subject  of  indicators  used  for  monitoring  and 
evaluation.  It  shows  how  indicators  can  be  used  to 
support  policy  decisions,  improve  management  and 
measure  the  overall  performance  of  programmes.  The 
volume  proposes  a  methodological  framework  and  a 
practical guide for the use of  indicators  in these  different 
situations. 

The  third  Volume  presents  a  selection  of  "Principal 
evaluation  techniques  and  tools"  applicable  to  the 
evaluation  of  socio-economic  programmes.  These  tools 
are taken from  relevant  scientific  disciplines: economics, 
sociology, management, geography, etc. Each one of the 
23 tools  is described  in the form  of  an information  sheet 
featuring  a  general  description,  the  principles  of  its 
implementation, a list of its strengths and weaknesses for 
the  evaluation  of  Structural  Funds,  an  example  of  its 
application  and a list of  references. 

A number of methods adapted to the evaluation of socio­
economic  programmes  are the object  of  a more detailed 
description 
in  the  fourth  Volume  entitled  "Technical 
solutions  for evaluation  within  a partnership  framework". 
Several  pilot  evaluations  were  implemented  within  the 
MEANS programme to design and test the following four 
innovations,  the  application  of  which  is  described  in 

The  MEANS Collection 

detail  in  this  Volume:  impact  mapping,  cross-impact 
matrix, score cards and multi-criteria  analysis. 

The  fifth  Volume  deals  with  "Transversal  evaluation  of 
impacts  on  the  environment,  employment  and  other 
intervention  priorities"  of  the  European  Union  and 
Member  States.  Among  the  questions  addressed  are 
impacts  in  terms  of  equal  opportunities  between  men 
and  women  and  the  competitiveness  of  SMEs.  The 
Volume provides the elements required for an analysis of 
those  impacts  which  are  not  necessarily  part  of  the 
explicit  objectives  of  all the actions evaluated. 

Finally,  the  last  Volume  is  a  "Glossary  of  concepts  and 
technical  terms".  The  compilation  of  the  Collection  has 
necessitated  the  rigorous  standardisation  and  definition 
of  the  terms  used. Each  of the first  five  volumes  has an 
index  of terms which, along with  hundreds of  definitions, 
are brought together  in the glossary. 

Table of contents 

I 

Introduction 

II  Evaluation Issues 
Utility  of  evaluation 
Types of evaluation 
Activities  related to evaluation 

III  Organisation  of evaluation 

What  is evaluated 
Stakeholders  in evaluation 
Organisation  of evaluation 
Preparation  of the evaluation 
Quality  of the evaluation 

IV  Criteria  and Indicators 

Internal  logic  of an intervention 
Evaluation criteria 
Measures and information  systems 
Indicators 

V 

Impacts 
Causality  analysis 
Socio-economic  effects 
Effects of synergy 
Employment  effects 
Economic  and social  development 

VI  Techniques 

Evaluation  methods 
Structuring techniques 
Information  gathering techniques 
Analytical techniques 
Judgement  techniques 

VII  Appendix 

Bibliography 

French Alphabetical  Index 

English Alphabetical  Index 

9 

15 
17 
25 
29 

31 

33 
37 
41 
45 
47 

51 
53 
61 
67 
73 

77 
79 
85 
89 
91 
93 

97 
99 
101 
103 
107 
111 

115 

117 

119 

127 

I 
Introduction 

Raison  d'être  of  this  Volume 

Evaluation  is an exercise  consisting  of  numerous  interactions  between the 
commissioner  and the evaluation team, on the one hand, and between the 
various  institutions  concerned,  on  the  other.  The  quality  and  utility  of  the 
evaluation depend to a large degree on the quality of these interactions. In­
teraction  can, however, easily lead to misunderstanding  if the vocabulary  is 
not precise and accurate. 

Within  the  context  of  the  European  Union's  socio-economic  programmes, 
problems  of  vocabulary  are frequently  encountered  because  evaluation  is 
performed  in  an  international  and  ¡ntersectoral  context,  between  people 
who  share  neither  the  same  mother  tongue  language  nor  the  same 
professional  references.  Moreover,  due  to  the  recent  development  of 
evaluation, many  actors  are unfamiliar  with the standard  vocabulary  which 
is not  always  stabilised. The  resulting communication  problems  can  waste 
time  and  energy,  and  even  compromise  the  quality  or  the  utility  of 
evaluation work. 

The  purpose  of  this  Volume  is to  contribute  towards  the  stabilisation  and 
clarification of the vocabulary. In the long run, it is also intended to bring the 
concepts  used  in  different  European  languages  closer  together.  The 
involved  in  commissioning, 
targeted  public  consists  of  all  persons 
performing  and  using  evaluations  situated  in  the  context  of  European 
economic and social cohesion policy. The book applies mainly to evaluation 
of a socio-economic  development  programme, but  may also be of  interest 
to  readers  who  conduct  or  perform  thematic  evaluations  or  in-depth 
evaluations  in  more  limited  domains.  That  is  why  the  term  "public 
intervention"  is systematically  used  in this  Volume,  in  a  generic  sense,  to 
denote any evaluation  object. 

The  choices  that  were  made 

This glossary  originated from the need to translate the work  of the  MEANS 
Programme  (Evaluation Methods  for  Actions  of  a  Structural  Nature)  into 
several languages. A first version in five languages was compiled  in 1994 in 
the  form  of  an  internal  Commission  document.  This  initial  effort  was 
welcomed  and  attracted  the  attention  of  evaluation  experts  in  various 
countries.  Many  of  them  spontaneously  contributed  to  a  second  version 
created  in  1996, with  improved  definitions  in four  additional  languages. By 
demanding  a  rigorous  effort  to  clarify  and  standardise  the  terms  used, 

Glossary of 300  concepts and technical  terms 

publication of the MEANS Collection, afforded an opportunity for  expanding 
the  glossary,  enhancing  its  coherence  and  for  publishing  this  Volume. 
Initially  the  Commission  chose  to  favour  the  coherence  of  the  glossary 
rather  than  the  number  of  languages  covered.  This  Volume  is  therefore  a 
first  step  which  covers  English  and  French,  and  which 
includes  a 
systematic  translation  from  one  language  to  the  other.  The  work  can 
subsequently  be extended to  other  European  languages. 

The  words  selected  for  this  volume  belong  essentially  to  the  evaluation 
vocabulary.  An  important  place  was  also  reserved  for  socio-economic 
impacts.  Each of the first five volumes  in the  MEANS Collection  is followed 
by an  index  which  refers to this  sixth  volume. 

the  European  Commission  and 

One  of  the  difficulties  encountered  in the  construction  of  this  work  derives 
from  the fact  that  several definitions  of  the same  word  often  co-exist,  both 
at 
in  scientific  and  professional 
communities.  In  these  instances,  a  single,  clear  choice  has  consistently 
been made so that a coherent whole can be achieved. However, to  facilitate 
the acceptance  of the definitions  chosen and to reduce  misunderstandings, 
the  main  competing  definitions  and  related  terms  currently  used  in  other 
sectoral  or  national  contexts  are also  mentioned. 

The  definitions  in  this  volume  are  presented  thematically  rather  than 
alphabetically,  along  the  same  lines  as  the  first  versions  of  this  glossary 
which  were  much  appreciated  by  users. The  themes  covered  are  listed  in 
the  contents. An  alphabetical  index  in  French  and  in  English  can  be  found 
in the annex. The glossary  can therefore  be used in three different  ways, as 
shown  in the following  box. 

Introduction 

How to use the glossary 

For readers who want to find the definition of a term 

>· Look for the term in the English index at the back of the book and go to the 

page on which that term is defined. 

For readers who want to translate a term from English to French or vice versa 

>■ Look for the term in the English or French index at the back of the book and 

go to the page on which that term is defined. The English terms are listed 
in italics. 

For readers who want to find their way around the vocabulary by starting with a 
theme 

> Look for the theme on the contents page and scan the pages concerning 

that theme. 

Λ",*-' 

Evaluation  issues 

This part  defines evaluation and its different  forms,  with an emphasis  on the 
utility  of  the  exercise  and  its  value-added  compared  to  similar  exercises 
such  as  monitoring  and  control.  For  a  complete  presentation  of  these 
themes, the reader is referred  to  Volume 1 of the Collection, particularly  the 
introduction  and  first  part  devoted  to  evaluation  issues  in  the  context  of 
European economic  and  social  cohesion  policy. 

Utility of evaluation 

Evaluation 

Evaluation 

Judgement on the value of a public intervention with reference to criteria 
and explicit standards (e.g. its relevance, its efficiency). 

The judgement primarily concerns the needs which have to be met by the 
intervention, and the effects produced by it. The evaluation is based on 
information which is specially collected and interpreted to produce the 
judgement. 

For example: evaluation of the effectiveness of a programme, cost-benefit 
evaluation of a project, evaluation of the validity of a policy, and evaluation 
of the quality of a service delivered to the public. 

Certain definitions exclude the judgement dimension and limit evaluation to 
the measurement of the intervention's effects. Other, more restrictive 
definitions, limit evaluation to the ex post estimation of effects. 

In certain contexts, evaluation focuses not on a public intervention but on 
a public organisation (e.g. evaluation of a university or a hospital). More 
generally, the term is used in human resource management for the 
evaluation of a person (e.g. annual evaluation interview) or in the financial 
domain to estimate the value of an enterprise. 

Journalistic use of the word is often in the sense of a vague estimation 
(" the number of victims is estimated at 1,000 people") or a monetary 
estimation (" this table is valued at 1 million euros"). 

Related term(s) : 

Appraisal 
Assessment 
Appréciation 
Examen 

■ Formative evaluation 

Evaluation endoformative 

Evaluation which is intended for managers and direct protagonists, in order 
to help them improve their action (feedback). 

Glossary of 300 concepts and technical terms 

Formative evaluation applies mainly to a public intervention during its 
implementation (on-going or intermediate evaluation). It focuses essentially 
on implementation procedures and their effectiveness and relevance. The 
distinction between formative and summative evaluation has wittily been 
summarised in the following way: "When the cook tastes the soup, it's 
formative; when the guests taste the soup, its summative". 

Related term(s) : 

Evaluation formative 

S u m m a t i ve evaluation 

Evaluation récapitulative S 

Evaluation performed for actors who are not directly involved in the 
management of the public intervention (political or professional institutions, 
foundations, the press, etc.). 

It produces a global and distant assessment. Evaluation is said to be 
summative or "recapitulative" when it aims at accounting for actions, and 
when it serves both for judging a public intervention and for helping the 
authorities responsible decide on its launching, maintenance, 
postponement, suppression and reproduction. In such instances, the users 
are external funding bodies and possibly other social actors who may 
decide for their own purposes on the appropriateness of adopting better 
practices identified during the programme evaluation. 

Related term(s) : 

Evaluation sommative 

Managerial evaluation 

Evaluation manageriale ■ 

An evaluative approach integrated into the management of public 
interventions, and aimed at recommending changes related either to 
decision-making (feedback, instrumental use) or to the behaviour of the 
actors responsible for the implementation of the intervention. 

The general approach of managerial evaluation is similar to that of new 
public management, and is aimed at addressing the problem of stagnating 

Evaluation  issues 

public  revenue. The underlying  question  can be formulated  as follows: how 
can  the  trade-off  between  the  different  sectoral  policies  be justified?  The 
dominant  approach  here  which  occurs  within  the  administration,  is that  of 
"  optimisation  " of  budgetary  resources. 

The  political  dimension  of  the  evaluation  act  is  overlooked.  The  quality  of 
managerial  evaluation  is  based  on two  principles: the  independence  of the 
evaluator  and the objectivity  of the method. 

For example, during the  1980s, most functions of the  British  administration 
were  privatised  or  delegated  to  semi-independent  agencies.  Demands  for 
effective  services  grew  at  the  same  time.  This  led  to  the  development  of 
Value For  Money  evaluations. 

Pluralistic  evaluation 

Evaluation 

pluraliste 

Evaluative  approach  designed  as  a  collective  problem-solving  process 
involving  all the  parties concerned. 

On the basis  of reliable information  acceptable to  all, value judgements  are 
formulated  by  seeking  agreement  within  an evaluation  authority  consisting 
of  political  and  administrative  officials,  as  well  as  spokespersons  for  the 
groups  concerned. 

In  France,  ad  hoc  evaluation  authorities  are  frequently  set  up  to  manage 
evaluation  work.  Thus,  in  the  framework  of  national  evaluation  of  the 
prevention  of  natural  hazards,  the  evaluation  authority  was  composed  of 
actors  from  diverse  origins:  ministries,  local  authorities,  public  agencies, 
associations, etc. Within the framework of policies co-financed  by the State 
and the  regions, the  steering  of certain  evaluations  is entrusted  to  political 
and  administrative  officials  at  both 
in  partnership  with 
representatives  of the groups  concerned. 

levels, 

Related  term(s)  : 

Partnership  evaluation 

Stakeholder  evaluation 

Evaluation partenariale 

Evaluation  participative 

Glossary of 300  concepts and technical  terms 

D e m o c r a t ic  evaluation 

Evaluation 

démocratique 

Evaluative  approach  aimed  at  enhancing  the  quality  and  transparency  of 
democratic  debate,  whether  through  the  promotion  of  an  awareness  and 
understanding  of  the  logic  and  effects  of  public  intervention,  or  through  a 
judgement  of  its  raison  d'être  and  effectiveness. 

Evaluation  is  defined  as  a  "judgement  on  the  value  of  the  public 
intervention"  by  democratically  elected  personalities.  Evaluation 
professionals  are  at  the  service  of  this  authority  and  have  an  exclusively 
technical  role. The  process  is  public. 

This  model  applies  in  the  case  of  an  experimental  intervention  which,  in 
view  of the  results  obtained  after  the first few  years,  must  be confirmed  by 
a vote. 

Cognitive  aim 

Finalité 

cognitive 

The  fact  that  evaluation  enables  its  users  to  become  aware  of  new 
information  on  an  intervention  and  its  effects. 

The  cognitive  contribution  of  an  evaluation  is  particularly  visible  when  the 
reader  of  the  evaluation  report  is  surprised  by  what  s/he  reads,  when  s/he 
discovers  something  new.  It  generally  corresponds  to  the  production  of 
new  information  (primary  data),  but  may  also  relate to  an  evaluation  which 
affords  its  users  access  to  secondary  data  about  which  they  were  not 
previously  aware. 

is  the 
A  dimension  of  the  cognitive  contribution  of  an  evaluation 
transformation  of  the  way  in  which  users  perceive  the  intervention,  its 
effects  and  relations  of  cause  and  effect  (impact  explaining  theory).  In that 
sense  it  is  a  conceptual  contribution.  Irrespective  of  its  aim,  a  successful 
evaluation  always  has a visible  and  recognised  cognitive  role. 

Related  term(s)  : 

Conceptual. 

Apport  conceptuel. 

Evaluation  issues 

Instrumental  aim 

Finalité 

instrumentale 

The fact  that  the  evaluation  results directly  in decisions  to  improve,  extend 
or  interrupt  a public  intervention. 

For  example,  an  intermediate  evaluation  report  recommended  a  series  of 
twenty-six  amendments  to  components  of  public  intervention.  A  few 
months  after  the  evaluation, the  authorities  responsible  for the  intervention 
agreed  on  a  series  of  adjustments.  Of  the  decisions  taken,  thirteen 
corresponded  to  recommendations  made  by the evaluation. 

realistic 

This contribution  is optimised when the evaluation team and steering group 
formulate 
recommendations  and  when  decision-makers 
appropriate them easily.  In this spirit, the status  of the recommendations  is 
clearly  explained  by  answering 
the  following  questions:  Does  the 
implementation  of this recommendation  have a financial  implication?  Can it 
intervene  in the short  term?  Is there  consensus  on the  recommendation? 

An  instrumental  use  is  difficult  to  prove  because  many  other  factors 
intervene at the time of  decision-making. 

Normative  aim 

Finalité 

normative 

The  fact  that  the  evaluation  proposes  arguments  to  form  or  modify 
judgements  on an intervention. 

An evaluation  may  have a normative  role when it  helps to clarify  criteria and 
define comparative norms, or when it identifies good  practice and exemplary 
projects. This contribution is essential when the evaluation serves to account 
to political authorities and the general public. Users compare the conclusions 
of the evaluation  with their own opinions. These conclusions  therefore  either 
maintain,  support  or  change  their  personal  judgement  on  the  intervention. 
However,  not  all  evaluations  serve  this  purpose  because  public  authorities 
are not always  prepared to call into question their  personal  convictions. 

Feedback 

Rétroaction 

Feedback  exists when the observation of  results and impacts  in the field is 
used  to  adjust  the  implementation  of  an  intervention,  or  to  make  more 

Glossary of 300  concepts and technical  terms 

radical  changes,  including  calling 
intervention. 

into  question  the  existence  of  the 

Feedback  is  the  main  purpose  of  evaluation  when  it  has  a  managerial 
and/or  formative  perspective.  It corresponds  to an  instrumental  use. 

Related  term(s)  : 

Feed-back 

Organisational  learning 

Apprentissage 

organisationnel 

The fact that  lessons  drawn from  experience are accepted  and  retained  by 
institutions  or organisations  responsible for  intervention. The  learning  goes 
beyond  feedback  in  so  far  as  the  lessons  are  capitalised  on  and  can  be 
applied to  other  interventions. 

Direct  learning  (called  "  single  loop  learning  ") exists  when the  users  learn 
that  the  intervention  has  been  a success  or failure. This  can  lead them, for 
example,  to  reduce  the  budgets  allocated 
least  successful 
interventions. 

to  the 

Indirect  learning  (called  "  double  loop  learning  ") exists when  users  realise 
that  they  must  call  into  question  their  basic  assumptions  (their  action 
theory). This may cause them, for example, to reorganise  implementation  of 
the  least  successful  interventions. 

Related  term(s)  : 

Single  loop  learning 

Double  loop  learning 

Apprentissage  en simple  boucle 

Apprentissage  en double  boucle 

Accountability 

Obligation, for the actors  participating  in the introduction  or  implementation 
of  a  public  intervention,  to  provide  political  authorities  and  the  general 

Rendre  des 

comptes 

Evaluation  issues 

public with information and explanations on the expected and actual  results 
of an  intervention, with  regard to  the  sound  use of  public  resources. 

From a democratic  perspective, accountability  is an important  dimension of 
evaluation.  Public  authorities  are  progressively 
their 
requirements  for  transparency  vis-à-vis  taxpayers,  as  to  the  sound  use  of 
funds  they  manage.  In  this  spirit,  evaluation  must  help  to  explain  simply 
where  public  money  was  spent,  what  effects  it  produced  and  how  the 
spending  was  justified.  The  addressees  of  this  type  of  evaluation  are 
obviously  political authorities  and, in  fine, citizens  via the  media. 

increasing 

For example, a training organisation  reports on the number of trainees  who 
benefited  from  its  services  and  the  qualifications  obtained.  A  managing 
authority  reports on the cost per net job created due to the intervention. The 
Commission  publishes  a  report  on  progress  made  in  terms  of  economic 
and social cohesion. 

Related  term(s)  : 

Imputabilité 

Redevabilité, 

Rendu-compte 

Responsabilité 

Types of  evaluation 

Overall  evaluation 

Evaluation 

d'ensemble 

Evaluation  of  an intervention  in its totality. 

Overall  evaluation  focuses  on  all  actions  financed  as  part  of  a  public 
intervention.  It  encompasses  all  the  tools  used,  all  groups  targeted,  all 
eligible  territories,  all  expected  impacts  and  all  relevant  themes.  Overall 
evaluation  often  constitutes  the  first  step  in the  screening  of  an  evaluated 
field,  before  a  second  stage  of  focalisation,  which  takes  the  form  of  in-
depth  or thematic  evaluations. 

Related  term(s)  : 

Global  evaluation 

Screening 

Radioscopie 

Thematic  evaluation 

Evaluation 

thématique 

Evaluation  which  transversally  analyses  a  particular  point  (a theme)  in  the 
context  of  several  interventions  within  a  single  programme  or  of  several 
programmes  implemented  in different  countries  or regions. 

The theme may correspond to an expected  impact  (e.g. competitiveness  of 
SMEs)  or  to  a  field  of  interventions  (e.g.  R&D).  The  notion  of  thematic 
evaluation  is similar  to that  of  an in-depth  study  (e.g. impact  of  support  for 
R&D on the competitiveness  of SMEs), but  it is a large scale exercise when 
conducted  on a European scale. 

In-depth  evaluation 

Evaluation 

approfondie 

Consists  of  focusing  evaluation  or  a  part  of  an  evaluation  precisely  on  a 
category of outputs, a group or on category of impacts. This permits a more 
detailed  analysis, unlike  overall  evaluation  which  aims at  being  exhaustive. 

Glossary of 300 concepts and technical terms 

Overall evaluation of a programme may be accompanied by an in-depth 
analysis of one or more questions. A decision may, however, be taken to 
launch an evaluation devoted entirely to the in-depth analysis of a single 
question. By focusing on a specific question it is possible to use more 
demanding and therefore more reliable evaluation techniques, particularly 
for observing the behaviour of addressees, for observing a comparison 
group or for analysing net effects. Since in-depth analysis focuses 
essentially on the observation of effects in the field, these studies concern 
intermediate and ex post evaluation. 

Related term(s) : 

Scoping 
Focalisation 

'& Ex ante evaluation 

Evaluation ex ante ■ 

Evaluation which is performed before implementation. 

For an intervention to be evaluated ex ante, it must be known with enough 
precision; in other words, a plan, at least, must exist. If the intervention still 
has to be planned from scratch, one would refer to a diagnosis of needs. 
This form of evaluation helps to ensure that an intervention is as relevant 
and coherent as possible. Its conclusions are meant to be integrated at the 
time decisions are made. Ex ante evaluation mainly concerns an analysis of 
context. It provides the relevant authorities with a prior assessment of 
whether development issues have been diagnosed correctly, whether the 
strategy and objectives proposed are relevant, whether there is 
incoherence between them or in relation to Community policies and 
guidelines, whether the expected impacts are realistic, etc. Moreover, it 
provides the necessary basis for monitoring and future evaluations by 
ensuring that there are explicit and, where possible, quantified objectives. 

Related term(s) : 

Prior appraisal 
Needs assessment 
Appréciation ex ante 
Evaluation a priori, 
Diagnostic 

Evaluation issues 

Mid-term evaluation 

Evaluation à mi-parcours ■ 

Evaluation which is performed towards the middle of the period of 
implementation of the intervention. 

This evaluation critically considers the first outputs and results, which 
enables it to assess the quality of the monitoring and implementation. It 
shows the translation into operational terms of initial intentions and, where 
relevant, points out de facto amendments to objectives. Through 
comparison with the initial situation, it shows the evolution of the general 
economic and social context, and judges whether the objectives remain 
relevant. It examines whether the evolution of policies and priorities of 
other public authorities raises problems of coherence. It also helps to 
prepare adjustments and reprogramming, and to argue them in a 
transparent manner. Mid-term evaluation not only relies strongly on 
information derived from the monitoring system, but also on Information 
relating to the context and its evolution. Mid-term evaluation has a 
formative character: it provides feedback on interventions of which it helps 
to improve the management. 

Mid-term evaluation is a form of intermediate evaluation. Other intermediate 
evaluations may be performed during the first or last years of 
implementation. 

Related term(s) : 

Evaluation intermédiaire 

On-going evaluation 

Evaluation chemin-faisant ■ 

Evaluation which extends throughout the period of implementation of an 
intervention. 

This form of evaluation accompanies the monitoring of outputs and results. 
It is too often confused with monitoring. The advantage of on-going 
evaluation is that it allows for good collaboration between the evaluation 
team and programme managers, which in turn favours a better 
appropriation of conclusions and recommendations. 

Glossary of 300  concepts and technical  terms 

On-going  evaluation  may  be  seen  as  a  series  of 
in-depth  studies, 
comprising  successive  analyses  of  evaluative  questions  which  have 
appeared  during  the  implementation.  For example,  an on-going  evaluation 
of  development  support  for  tourism  has  successively  considered  the 
following questions:  "how  has the public  reacted to the proposed support?" 
(first  year);  "is  the  evolution  of  the  tourist  market  making  the  assistance 
useless?"  (third  year);  "which  of  the  assisted  projects  can  be  considered 
cases  of  best  practice to  reproduce?"  (fourth year). 

Related  term(s)  : 

Rolling  evaluation 
Evaluation  in  itinere 

Ex  post  evaluation 

Evaluation  ex  post 

I 

Evaluation  which  recapitulates  and judges  an intervention  when  it  is over. 

It aims at accounting for the use of resources, the achievement  of  expected 
effects  (effectiveness)  and  of  unexpected  effects  (utility),  and  for  the 
efficiency  of  interventions.  It strives to  understand the factors of success  or 
failure,  as  well  as  the  sustainability  of  results  and  impacts.  It  also  tries  to 
draw  conclusions  which  can  be  generalised  to  other  interventions.  For 
impacts  to  have  the  time  to  materialise,  ex  post  evaluation  must  be 
performed  two  to  three  years  after  implementation  of  an  intervention.  For 
the  purpose  of  analysing  impacts,  ex  post  evaluations  are  likely  to  involve 
field  surveys  and  to  take  place over  long  periods  of  time. 

Related  term(s)  : 

Evaluation  a  posteriori 

M e t a - e v a l u a t i on 

Méta-évaluation 

Evaluation  of  another  evaluation  or of  a series  of  evaluations. 

The criteria  for judgement  are generally  reliability, credibility  and  utility. The 
term  is  sometimes  applied  to  the  auditing  of  an  evaluative  function  of  an 
organisation  (verifying  that  the  rules  concerning  evaluation  have  been 
applied  professionally).  Sometimes  meta-evaluation  is  used  to  refer  to  a 
synthesis  based  on a series  of  evaluations. 

Activities related to evaluation 

Audit 

Controle 

Verification of the legality and regularity of the implementation of resources. 

Auditing makes a judgement in terms of general criteria and standards, 
known and clarified beforehand, applicable to an entire political or 
professional field, and not specific to a public intervention. For example, in 
the case of assistance to a SME, an audit will check whether eligibility 
criteria have been met and whether the beneficiary firms have complied 
with the rules governing the use of assistance. The auditor is thoroughly 
familiar with the standards (e.g. legal or accounting knowledge) and his or 
her intervention is mandatory and inspires respect and authority. The main 
purpose of an audit is to find mistakes and evidence of dysfunctioning. The 
idea is to obtain a dissuasive effect. 

With time, the terms "control" and "audit" have extended further afield to 
encompass more activities than those corresponding to the original 
definition provided above. For example, certain audits or controls check 
whether the outputs have been produced with an adequate degree of 
efficiency and quality. Others examine whether the results and performance 
are sufficient compared to those obtained by other similar interventions. 

Control and audit also apply to the verification of the existence and good 
functioning of monitoring and evaluation procedures. 

Related term(s) : 

Control 

Audit 

Monitoring 

An exhaustive and regular examination of the resources, outputs and 
results of public interventions. 

Monitoring is based on a system of coherent information including reports, 
reviews, balance sheets, indicators, etc. Monitoring system information is 

Suivi ■ 

Glossary of 300  concepts and technical  terms 

obtained  primarily from operators  and is used essentially for steering  public 
interventions. When  monitoring  includes a judgement, this judgement  refers 
to the achievement  of operational objectives.  Monitoring  is also  intended to 
produce feedback  and direct  learning. It is generally the responsibility  of the 
actors  charged  with  implementation  of  an intervention. 

For example:  monitoring  of  the  consumption  of  budgets,  monitoring  of  the 
meeting  of  deadlines,  monitoring  of  the  percentage  of  SMEs  in  beneficiary 
firms,  monitoring  of  the  level of  qualifications  obtained  by trainees. 

Related  term(s)  : 

Management  audit 

Performance  audit 

Audit  de  performance 

Contrôle  de  gestion 

Pilotage 

Monitorage 

Additionality 

Additionnante 

The fact  that  Community  support  for  economic  and  social  development  is 
not substituted  for efforts  by national  governments;  in other  words, the fact 
that  the  beneficiary  State's  own  financing  remains,  globally,  at  least  equal 
to that  which  existed  before the  Structural  Funds'  contribution. 

Verification  of  the  implementation  of  this  principle  is  carried  out  at  the 
national  level  in  the  context  of  financial  control  and  not  of  evaluation  as 
such.  This  term  must  not  be  confused  with  additionality  which  applies  to 
the evaluation  of the  net  effects  of  an  intervention. 

Organisation  of evaluation 

The following  pages  cover  the main  terms  relating  to  the preparation  of an 
evaluation,  from  the  decision  to  launch  it  to  the  terms  of  reference.  The 
question  of  the  quality  of  evaluation  is  also  addressed.  For  a  complete 
presentation  of  these  themes,  the  reader  is referred  to  the second  part  of 
Volume 1 of the Collection,  devoted  to the preparation  of the different  forms 
of  evaluation,  and  to  the  fourth  part,  which  proposes  an  approach  to  the 
assessment of evaluation quality. 

What is evaluated 

Scope of evaluation 

Champ 

Precise definition of the evaluation object, of what is evaluated. 

The scope of the evaluation must be defined in at least four respects: 
operational (all or part of the domains of intervention, one or several related 
policies), institutional (all or part of the authorities), temporal (period taken 
into consideration) and geographical (one or more territories or parts of 
territories, a particular region, town, nature reserve, etc.). 

In the context of European cohesion policy, the evaluation object may be a 
programme, a measure or a project. To remain sufficiently general, this 
book uses the term public intervention and applies it, generically, to any 
evaluation object. 

Related term(s) : 

Evaluand 

Evaluanda 

■ Policy 

Politique 

A set of different activities (programmes, procedures, laws, rules) directed 
towards a single goal or general objective. 

These activities are often accumulated incrementally through the years. 
European economic and social cohesion policy is scheduled according to 
a precise time-frame, with a pluri-annual budget. This is not the case for the 
majority of policies, in the traditional sense of the term. 

Programme 

Programme 

Organised set of financial, organisational and human resources mobilised to 
achieve an objective or set of objectives in a given lapse of time. 

Glossary of 300 concepts and technical terms 

A programme is delimited in terms of a schedule and a budget. Programme 
objectives are defined beforehand; an effort is then made systematically to 
strive for coherence among these objectives. The three main steps in the 
life-cycle of a programme are design, implementation and ex post 
evaluation. A programme is always under the responsibility of an authority 
or several authorities who share the decision-making. Within the framework 
of European social and economic cohesion, programmes are generally 
broken down into axes, measures and projects 

Related term(s) : 

Axis 

Programme cycle 

Axe 

Cycle de programmation 

M e a s u re 

Mesure 

Within the framework of European economic and social cohesion policy, the 
basic unit of programme management, consisting of a set of similar 
projects and disposing of a precisely defined budget. 

Each measure has a particular management apparatus. Measures generally 
consist of projects. Many measures are implemented in the form of calls for 
projects. 

Related term(s) : 

Action 

Action 

■ Project 

Non divisible operation, delimited in terms of schedule and budget, and 
placed under the responsibility of an operator. 

For example: creation of a new training branch, extension of a purification 
network, carrying out of a series of missions by a consultancy firm. Within 

Projet m 

Organisation of evaluation 

the framework of European economic and social policy, the operator 
requests assistance which, after a selection procedure, is either attributed 
or not by the managers of the programme. Particularly careful ex ante 
evaluations are made of major infrastructure projects, using the cost-benefit 
analysis technique. 

Intervention 

Intervention ■ 

Any action or operation carried out by public authorities regardless of its 
nature (policy, programme, measure or project). 

Within the framework of European Economic and Social Cohesion policy, 
interventions take the following forms: Operational Programmes and 
Community Initiatives and Support Programmes for technical assistance 
measures and innovatory actions. Means of interventions employed are 
grants, loans, subsidised interest rates, guarantees, participation in equity 
and risk capital schemes or other forms of financing. Finally, the major 
domains of intervention are basic infrastructure, the productive environment 
and human resources. 

In this Volume, the term intervention is systematically used to designate the 
object of evaluation. 

Stakeholders in evaluation 

Stakeholder 

Protagoniste 

Individuals, groups or organisations with an interest in the evaluated 
intervention or in the evaluation itself, particularly: authorities who decided 
on and financed the intervention, managers, operators, and spokespersons 
of the publics concerned. 

The stakeholders may have legitimate interests which must be taken into 
account in an evaluation. They may also have purely private interests which 
are not legitimately part of the evaluation. 

For example, in the case of an intervention which subsidises the creation of 
new hotels, the stakeholders are the funding authorities/managers, the new 
hoteliers (direct addressees), other professionals in tourism, former 
hoteliers facing competition from the assisted hotels, tourists, nature 
conservation associations, and building contractors. 

Related term(s) : 

Partner 

Partenaire 

Partie prenante 

■ Funding authority 

Financeur 

Public institution which helps to finance an intervention. 

By extension, the term funding authority is also used for people who 
intervene on behalf of these institutions in the evaluation process: European 
Commission desk officers, officials from a national ministry; elected 
representatives from a regional or local authority. When private people or 
organisations are subsidised by the intervention, they also contribute part 
of the funds (leverage effect). These people or organisations are not, 
however, funding authorities but addressees. 

Glossary of 300  concepts and technical  terms 

Related  term(s)  : 

Decision-maker 
Décideur 
Maître  d'ouvrage 

M a n a g er 

Gestionnaire 

Public  (sometimes  private)  organisation  responsible  for  implementing  an 
intervention. 

Within  the  framework  of  European  economic  and  social  policy,  managers 
exist  at  the  programme  and  measure  levels.  By  extension,  the  term 
manager  is also used for people who intervene  in the evaluation  process on 
behalf  of  these  organisations. 

Related  term(s)  : 

Maitre  d'oeuvre 

Operator 

Opérateur 

Organisation which  implements, closest to the field, the resources  allocated 
to  an intervention. 

The  operators  are  the  last  links  in  the  chain  of  implementation  of  an 
intervention.  They  are  at  the  source  of  information  feeding  into  the 
include:  a  local  administration  which 
monitoring  system.  Examples 
distributes  direct  support  to  enterprises, a private company  which  has  built 
and  manages  a telephone  network,  a training  institute  which  trains  people 
funded  by  interventions. 

Within the framework  of  European  Union socio-economic  programmes,  the 
term  final  beneficiary  is  used  to  denote  public  or  private  organisations 
responsible  for  commissioning  work  or  for  distributing  public  assistance. 
Final  beneficiaries  are  required  to  report  on  their  actions  regularly  and 
systematically.  They  are  therefore  programme  operators  in  the  sense 
defined  in this  book. 

Related  term(s)  : 

Final  beneficiary 

Bénéficiaire  final 

Organisation of evaluation 

Project  promoter 

Porteur  de  projet 

Public  or  private  person  or  organisation  which  requests  and  possibly 
obtains  assistance  in  the  framework  of  an  intervention  for  a  given  project 
(e.g. rehabilitating  a run down  urban site; creating  a new training  branch). 

A  project  promoter  must  be  considered  to  be  an  operator  if  it  receives 
public  funds every year and if  it has to  report  regularly and  permanently  on 
the  project.  In  contrast,  it  must  be  considered  an  addressee  if  it  receives 
limited funding for a single  project. 

Direct  addressee 

Destinataire 

direct 

Person  or organisation  directly  affected  by the intervention. 

Addressees  receive  support,  services  and  information,  and  use  facilities 
created  with  the  support  of  the  intervention  (e.g.  a  family  which  uses  a 
telephone network that has been improved with public intervention support, 
or  a firm  which  has  received  assistance  or  advice).  Some  people  may  be 
addressees  without  necessarily  belonging  to  the  group  targeted  by  the 
intervention. Similarly, the entire eligible group does not  necessarily  consist 
of  addressees. 

The  term  "beneficiary"  is  often  used  in  the  sense  given  here  to  the  term 
"direct  addressee".  In this  glossary,  the  term  "addressee"  was  chosen  for 
two  reasons:  (1) the term  "final  beneficiary"  has  a different  meaning  in the 
evaluation  system  of  European  economic  and  social  cohesion  policy,  and 
(2)  the  term  "beneficiary"  is  associated  with  the  idea  of  a  positive  effect, 
which  can  be  misleading  because  the  evaluation  team  may  identify 
negative  impacts  when  investigating  effects  on direct  addressees. 

Related  term(s)  : 

End-user 

Recipient 

Bénéficiaire 

Usager 

Glossary of 300 concepts and technical terms 

Eligible public 

Public eligible ■ 

All the people and organisations for which an intervention is directly 
intended (e.g. people unemployed for over a year; entrepreneurs). 

An eligible public is composed of all potential direct addressees. Only some 
of these are affected by the intervention (see the notion of coverage rate). 
This concept is more limited than the term "concerned public" which also 
includes foreseen or unforeseen addressees. 

Related term(s) : 

Target group 

Public cible 

Indirect addressee 

Destinataire indirect i 

A person, group of persons or organisation which has no direct contact with 
an intervention, but which is affected by it via direct addressees (e.g. firms 
which have used technology transfer networks set up by a public 
intervention to innovate). 

Indirect effects are produced in the short or medium term, positively or 
negatively. The raison d'être of an intervention is to produce positive 
change for all its direct and indirect addressees. 

C o n c e r n ed group 

Public concerné 

All the persons or organisations which are affected or potentially affected, 
voluntarily or not, directly or indirectly, by the intervention. 

This concept is broader than the term "eligible public" (all the potential 
direct addressees) because it also includes indirect addressees. 

Related term(s) : 

Winners and losers 

Gagnants et perdants 

Organisation of evaluation 

Self-evaluation 

Auto-évaluation 

Evaluation of a public intervention by the organisation which participates 
directly in its implementation. 

This mode of organisation is rarely found in the context of programmes 
financed by the European Union. Due to cultural and geographic distance, 
it is more frequent in projects to assist developing countries. 

Internal evaluation 

Evaluation interne 

Evaluation of a public intervention by an evaluation team belonging to the 
administration responsible for the programme. 

Internal evaluation may be independent if the evaluation team has no 
hierarchical relation with the actors implementing the intervention. 
Otherwise, it would be self-evaluation. 

Related term(s) : 

In-house evaluation 

External evaluation 

Evaluation externe ■ 

Evaluation of a public intervention by people not belonging to the 
administration responsible for its implementation. 

For example, a team composed of private consultants, researchers or 
people belonging to public organisations unrelated to those responsible for 
the intervention. 

Commissionner 

Commanditaire 

Person or organisation which decides (or co-decides) to launch an 
evaluation. 

Glossary of 300 concepts and technical terms 

A commissioner has the advantage of making his or her decision visible by 
establishing a mandate. A commissioner may decide to steer the work of 
an evaluation team her/himself, or to constitute a steering group or 
evaluation authority for this purpose. 

Related term(s) : 

Mandant 

■ Evaluation team 

Equipe d'évaluation M 

The people who perform the evaluation. 

An evaluation team selects and interprets secondary data, collects primary 
data, carries out analyses and produces the evaluation report. An evaluation 
team may be internal or external. It may consist of a group of several 
organisations (consortium) or contract out the work. It may even consists of 
a single person. In this Collection, the term "evaluation team" has been used 
rather than "evaluator". The latter term, found in the management model, 
cannot easily be applied to the other two models (pluralistic and democratic 
evaluation) in so far as it is an evaluation authority which makes the final 
judgement on the basis of the work produced by an evaluation team. 

Related term(s) : 

Evaluator 

Evaluateur 

Chargé d'évaluation 

Steering group 

Groupe de pilotage 

Limited group composed of officials from those administrations most 
directly concerned by an intervention, sometimes accompanied by experts. 

The steering group meets frequently and its working procedures are fairly 
informal. The steering group acts as a mediator between the 
commissioners and the evaluation team, but also between the different 
partners concerned by an intervention. 

Organisation  of evaluation 

Evaluation  C o m m i t t ee 

Instance 

d'évaluation 

Steering  group  extended  to  include the  main stakeholders  in an  evaluated 
intervention. 

An  evaluation  may  involve  any  person  who  is  potentially  a  user  of  its 
recommendations,  any  person  who  has  an  interest  in  the  information 
produced,  and  any  person  who  is  likely  to  win  or  lose  in the  process. The 
main  categories  of  stakeholders  are  funding  authorities,  managers, 
operators and concerned groups. Stakeholders  invited to join an evaluation 
authority  express their  hopes and fears  ; they  improve the  relevance of  the 
questions  asked and their  presence  makes the evaluation  more credible. A 
wider  diversity  of  points  of  view  facilitates  their  confrontation  within  the 
steering  group,  and  helps  to  raise  the  degree  of  consensus  in  the 
interaction  and  the  robustness  of  interpretations.  Provided  it  can  take 
advantage of the different  points of view, pluralistic  evaluation enhances the 
richness and credibility of its conclusions. When the expectations of several 
partners are taken into account, conclusions are more readily accepted and 
viewed  with  greater  interest. 

Evaluation  setting 

Dispositif 

d'évaluation 

All  the  modes  of  organisation  of  a  given  evaluation  (relations  established 
between  the  commission,  the  steering  group  and  the  evaluation  team, 
involvement  of stakeholders, organisation of work, schedule and evaluation 
method). 

The term evaluation  protocol  applies more restrictively to only the  technical 
aspects  of the  system, without  referring to  its organisational  aspects. 

The setting  put  in  place for  a given  evaluation  is integrated  into  evaluation 
system  which  is  applied  to  the  institutional  context  of  the  intervention 
evaluated. 

Evaluation  system 

Système 

d'évaluation 

All  the  rules,  institutions,  procedures  and  resources  which  organise  the 
evaluative function  in a given  institutional  context. 

Glossary of 300  concepts and technical  terms 

The  evaluation  system  of  European  economic  and  social  cohesion  policy 
specifies the moment  at which  each programme  must  be evaluated, as well 
as  the  responsibility  at  the  different  evaluation  stages.  The  system  varies 
depending  on  the  country.  Each  evaluation  may  be  subject  to  a  particular 
setting  provided  that  the  basic  principles  of  the  system  as  a  whole  are 
respected. 

Preparation  of the  evaluation 

M a n d a te 

Mandat 

Document  signed  by the commissioners  to formalise their  wish to  evaluate 
an intervention. 

By  means  of  a mandate, a commissioner  delegates  the  performance  of  an 
evaluation to a steering group or evaluation authority. The mandate  specifies 
what  will  be  evaluated  (the scope  of  the  evaluation), the  reasons  for  which 
the evaluation has been launched, and the expected  use. The commissioner 
specifies  the  nature  of  the  expected  recommendations  resulting  from  an 
evaluation.  S/he  states  the  main  questions  that  the  evaluation  will  have  to 
answer,  which  generally  gives  an  initial  idea  of  the  evaluation  criteria.  S/he 
fixes a final date for  submitting  the  report  and may  also fix  a budget. 

Several  mandates  may  be  prepared  at  the  same  time  for  each  of  the 
exercises—ex ante, intermediate and ex post—planned  in the framework  of 
a pluri-annual  evaluation  plan. 

Related term(s)  : 

Evaluation  plan 

Plan  d'évaluation 

Pre-evaluation 

Préévaluation 

Rapid  prior  study,  aimed  at  identifying  stakeholders'  expectations,  at 
choosing and specifying evaluative questions, at checking whether it will be 
possible  to  answer  them,  at  establishing  the  evaluation  setting  and  at 
drawing  up terms of  reference. 

Where  relevant,  a  pre-evaluation  may  recommend  that  an  evaluation  be 
cancelled  or  postponed,  if  the  conditions  for  its  smooth  operation  and  its 
utilisation cannot  be met. The technical  part of a pre-evaluation  is called an 
evaluability  assessment. 

Glossary of 300  concepts and technical  terms 

Evaluability  a s s e s s m e nt 

Erude 

d'évaluabilité 

Technical  part  of  the  pre-evaluation,  which  takes  stock  of  available 
knowledge  and assesses whether technical and institutional conditions  can 
be  met  in  order  for  reliable  and  credible  answers  to  be  given  to  the 
questions  asked. 

Concretely,  it  consists  of  checking  whether  an  evaluation  team  using 
appropriate  evaluation  tools  will  be  capable,  in  the  time  allowed  and  at  a 
cost  compatible  with  existing  constraints,  to  answer  evaluative  questions 
with  a strong  probability  of  reaching  useful  conclusions. 

Terms  of  reference 

Cahier  des  charges 

The  terms  of  reference  define  the  work  and  the  schedule  that  must  be 
carried  out  by the evaluation  team. 

It recalls the regulatory framework  and specifies the scope of an evaluation. 
It  states  the  main  motives  for  an  evaluation  and  the  questions  asked.  It 
sums  up  available  knowledge  and  outlines  an  evaluation  method.  It 
describes  the  distribution  of  the  work  and  responsibilities  among  the 
people  participating  in  an  evaluation  process.  It  fixes  the  schedule  and,  if 
possible, the  budget.  It specifies the qualifications  required from  candidate 
teams  as  well  as the  criteria  to  be  used  to  select  an  evaluation  team.  It  is 
generally  completed  by  administrative  annexes. 

Related  term(s)  : 

Job  description 
Termes de  référence 

Evaluative  question 

Question  evaluative 

Question  asked  by  the  commissioner  in the  terms  of  reference  and  which 
the  evaluation  team  will  have to  answer. 

Theoretically,  questions  have 
(what 
happened?), causal (to what extent  is what  has happened really an effect  of 
the  intervention?)  and  normative  (is the  effect  satisfactory?).  An  evaluation 
generally  has several  questions. 

three  dimensions:  descriptive 

Quality of the  evaluation 

Credibility 

Crédibilité 

Quality of the results and conclusions of an evaluation when they are logically 
supported  by empirical facts and justified by an analysis of valid data. 

Credibility  depends  on  several  factors,  including:  reliability  of  data, 
soundness  of the  method, but  also the  reputation  of the evaluation team. 

Related term(s)  : 

Trustworthiness 

Impartiality 

Impartialité 

Quality  of  conclusions  and  recommendations  of  an  evaluation  when  they 
are justified  by explicit  judgement  criteria and  have not  been influenced  by 
personal or  partisan  considerations. 

An impartial evaluation takes  into account  the expectations,  interpretations 
and  judgement  criteria  of  all  legitimate  stakeholders,  Including  those  who 
have very  little  power  or  ability  to  express  themselves,  in  order  to  give  an 
opinion  on  the  "Res  publica".  Impartiality  is  an  essential  element  of  the 
quality  of an evaluation. 

Related  term(s)  : 

Neutrality 

Neutralité 

Reliability 

Quality  of  the  collection  of  evaluation  data  when  the  protocol  used  makes 
it  possible  to  produce  similar  information  during  repeated  observations  in 
identical  conditions. 

Fiabilité 

Glossary of 300  concepts and technical  terms 

Reliability  depends  on  compliance  with  the  rules  of  the  art  as  regards  to 
sampling and tools  used for the collection and recording of quantitative  and 
qualitative  information.  Moreover,  sound  reliability  implies  exhaustive  data 
collection  and the  appropriateness  of the  evaluative  questions  asked. This 
notion  is  important  not  only  for  primary  data  but  also  for  secondary  data, 
the  reliability  of  which  must  be carefully  checked. 

Related  term(s)  : 

Objectivity 

Soundness 

Objectivité 

Robustesse 

Construct  validity 

Validité  de 

construction 

Quality  of  an  evaluation  method  which  faithfully  reflects  the  changes  or 
needs that  are to  be  measured  or  identified. 

impacts  are  expressed 

Construct  validity  means  that  expected 
in 
sufficiently  precise  terms  and  that  observations  in  the  field  allow  for  a 
reliable  and  sensitive  analysis  of  the  object  of  observation.  For  example,  if 
the  impact  of  support  for  innovation  must  be  estimated  by  means  of  a 
survey  on  regional SMEs, the  notion  of  innovation  must  have been  defined 
precisely  and  the  questions  asked  must  correspond  precisely  to  these 
definitions  (e.g. number  of  new  products  or  production  procedures). 

Internal  validity 

Validité interne 

Quality  of  an  evaluation  method  which,  as  far  as  possible,  limits  biases 
imputable  to  data  collection  or  processing  techniques. 

For  example,  an  intervention  which, in directly  or  indirectly  assisted  SMEs, 
led to the  doubling  of the  innovation  rate per year for  1,000  jobs. This  type 
of  conclusion  is  sounder  if  the  collection  and  analysis  of  data  have  very 
precisely  taken  into  account  all the  specific  aspects  of the  intervention  and 
its context:  categories  of assisted firms, forms  of  innovation  specific  to  the 
region, etc. To obtain better  internal validity, it is necessary to strictly  control 

Organisation of evaluation 

a  series  of  parameters,  otherwise  an  artificial  situation  may  be  created 
which  would  limit  the  possibility  of  generalising  the  conclusions  (less 
external validity). 

External  validity 

Validité  externe 

Quality  of  an  evaluation  method  which  makes  it  possible  to  obtain 
conclusions  that  can  be  generalised  to  contexts  (groups,  areas,  periods, 
etc.) other  than that  of the  intervention  being evaluated. 

For example, evaluation  makes  it possible to conclude (1) that  in directly  or 
indirectly  assisted  SMEs  the  intervention  helped  to  double  the  innovation 
rate  and  (2)  that  similar  assistance,  attributed  to  SMEs  in  other  regions, 
would  probably  have exactly  the  same  effect.  Only  strong  external  validity 
allows  one to  extrapolate  from  lessons  learned  during  the  implementation 
of  the  evaluated  intervention.  It  is  particularly  sought  after  when  the 
evaluation  aims at  identifying  and validating  best  practice.  External  validity 
is  also  required  when  the  evaluation  uses  conclusions  of  evaluations 
performed  on  similar  interventions. 

Related  term(s)  : 

Representativeness 

Généralisabilité 

Représentativité 

IV 
Criteria and  Indicators 

The following  terms  concern  the logic  of the intervention,  from  the needs it 
meets to the inputs it mobilises,  the outputs  it accomplishes  and, finally, the 
impacts  it  produces.  This  is  the  logic  which  structures  evaluation  criteria 
and  makes it possible  to construct  judgement  criteria  and  indicators.  For a 
complete  presentation  of  these  themes,  the reader is referred  to  Volume 1 
of  the  Collection,  particularly 
the  beginning  of  the  third  part  entitled 
"evaluating  the  global  logic  of  a  programme",  as  well  as  Volume  2, 
especially the first part  devoted  to indicator-related  terminology. 

Internal  logic of an intervention 

Need 

Besoin 

Problem  or  difficulty  affecting  concerned  groups,  which  the  public 
intervention  aims to  solve  or overcome. 

Ex ante evaluation verifies whether the needs used to justify an  intervention 
are  genuine.  Mid-term  evaluation  may  involve  a survey  on  addressees,  to 
reveal  their  needs  and  opinions.  Needs  are  the  judgement  reference  of 
evaluations  which  use relevance and  usefulness  criteria. 

Strategy 

Stratégie 

Selection  of  priority  actions  according  to  the  urgency  of  needs to  be  met, 
the gravity  of problems to  be solved, and the chances of actions  envisaged 
being successful. 

In  the  formulation  of  a  strategy,  objectives  are  selected  and  graded,  and 
their  levels  of  ambition  determined.  Not  all  territories  and  groups  are 
concerned  by the same development  strategy. Ex ante evaluation  examines 
whether the strategy  is suited  to the context  and its  probable  evolution. 

Context 

Contexte 

The socio-economic  environment  in which  an intervention  is implemented. 

The  term  is  used  in  its  broadest  sense.  For  example,  in  the  case  of 
interventions  in favour  of  SMEs, the  context  includes  the  macro-economic 
situation  and the framework  conditions  (tax  laws, company  law, etc.). 

Related term(s): 

Framework  condition 
Socio-economic  environment 
Condition-cadre 
Environnement  socio-économique 

Glossary of 300  concepts and technical  terms 

Objective 

Objectif 

Clear, explicit  and initial statement  on the effects to  be achieved  by a public 
intervention. 

If the  objectives  are  not  stated  implicitly,  an evaluation  (and  particularly  ex 
ante evaluation)  may  help  to  clarify  them. A  quantitative  objective  is stated 
in  the  form  of  indicators  and  a  qualitative  objective 
in  the  form  of 
descriptors,  e.g.: 30%  of  all  outputs  must  be  accomplished  by  the  end  of 
the  third  year;  the  public  intervention  must  first  benefit  the  long-term 
unemployed.  Specific  objectives  concern  the  results  and  impacts  of  an 
intervention  on  direct  addressees.  A  global  objective  corresponds  to  the 
aim  of  the  intervention. The  aim  of  an  intervention  is to  produce  an  impact 
expressed  in global terms, e.g. reducing  regional disparities in development 
levels.  Objectives  may  also  be  intermediate.  Objectives  which  specify 
outputs  to  be produced  are called  operational  objectives. 

If  the  objectives  of  a  public  intervention  have  not  been  clearly  defined 
beforehand, the  evaluation  can try  to  clarify  them  afterwards.  In that  case, 
it  is preferable  to  refer to  implicit  objectives. 

Related  term(s): 

Aim 

Goal 

Global  objective 

Intermediate  objective 

Specific  objective 

Operational  objective 

Implicit  objective 

But 

Finalité 

Objectif  global 

Objectif  intermédiaire 

Objectif  spécifique 

Objectif  opérationnel 

Objectif  implicite 

Criteria and indicators 

Policy  priority 

Priorité 

politique 

The  funding  authorities'  wish  that  evaluation  should  examine  certain 
impacts  which  were  not  stated  as  objectives  when  the  intervention  was 
launched,  but which  represent  political  priorities  at that level. 

For example, in the framework  of  its economic  and social  cohesion  policy, 
the  European  Union  demands  that  evaluations  systematically  take  into 
account  impacts  on the  environment,  on the competitiveness  of  SMEs,  on 
the creation and maintenance  of jobs, and on equal opportunities  between 
men and women. 

Verifiable  objective 

Objectif  verifiable 

An  objective  stated  in  such  a  way  that  it  will  subsequently  be  possible  to 
check  whether  or not  it  has been achieved. 

A  way  of  making  an  objective  verifiable  is  to  quantify  it  by  means  of  an 
indicator  linked  to  two  values  (baseline  and  expected  situation).  An 
objective  may also be verifiable  if it is linked to a descriptor,  i.e. a clear and 
precise qualitative  statement  on the expected  effect. 

Related  term(s): 

Quantitative  objective 

Objectif  quantitatif 

Implementation 

Mise  en  oeuvre 

The operational  process  needed to  produce  expected  outputs. 

In 
the  context  of  European  Union  socio-economic  programmes, 
implementation  comprises  all or  part  of the following  tasks: mobilising  and 
distributing  allocated  inputs;  assigning  management  responsibilities  to 
operators;  selecting  calls  for  tenders  for  project  promoters;  and,  lastly, 
selecting and financing projects. To monitor and improve implementation, a 
monitoring  committee  is  set  up,  a  system  of  information  monitoring  is 
launched,  and audits and  evaluations  are performed. 

Glossary of 300 concepts and technical terms 

Related term(s): 

Implementation 

■ Input 

Ressource 

Financial, human, material, organisational and regulatory means mobilised 
for the implementation of an intervention. 

For example, sixty people worked on implementing the programme; 3% of 
the project costs were spent on reducing effects on the environment. 

Monitoring and evaluation focus primarily on the inputs allocated by public 
authorities and used by operators to obtain outputs. In this Collection, 
private inputs mobilised by assisted firms, for example, are considered to 
be results of public intervention. 

The above definition gives a relatively broad meaning to the word "input". 
Some prefer to limit its use to financial or budgetary resources only. In this 
case, the word "activity" can be applied to the implementation of human 
and organisational resources. The term "financial outputs" is sometimes 
used in the sense of consumption of budgetary inputs. 

Related term(s): 

Activity 

Financial realisation 

Means 

Resource 

Activité 

Intrant 

Moyen 

Réalisation financière 

Effect 

Socio-economic change resulting directly or indirectly from an implemented 
intervention. 

Effet U 

Criteria and indicators 

Effects include the results and impacts of an intervention, whether positive 
or negative, expected or not. In certain cases, the term "effect" is wrongly 
used to include outputs. 

Output 

Réalisation ■ 

That which is financed and accomplished (or concretised) with the money 
allocated to an intervention. 

A project promoter undertakes to produce an output in immediate 
exchange for the support granted. If this is not accomplished, the support 
is withheld or must be partly or entirely refunded. Operators are responsible 
for outputs and must regularly and systematically report on them to the 
monitoring committee. Outputs may take the form of facilities or works (e.g. 
building of a road, rehabilitation of an urban wasteland; purification plant; 
tourist accommodation). They may also take the form of immaterial services 
(e.g. training, consultancy, information). 

Related term(s): 

Product 

Realisation 

Extrant 

Produit 

Result 

Résultat 

Advantage (or disadvantage) which direct addressees obtain at the end of 
their participation in a public intervention or as soon as a public facility has 
been completed. 

Results can be observed when an operator completes an action and 
accounts for the way in which allocated funds were spent and managed. At 
this point s/he may show, for example, that accessibility has been improved 
due to the construction of a road, or that the firms which have received 
advice claim to be satisfied. The operators may regularly monitor results. 
They have to adapt the implementation of the intervention according to the 
results obtained. 

Glossary of 300  concepts and technical  terms 

Related  term(s): 

Immediate  outcome 

I m p a ct 

Impact 

A  consequence  affecting  direct  addressees  following  the  end  of  their 
participation  in an intervention  or after  the completion  of  public  facilities, or 
else  an  indirect  consequence  affecting  other  addressees  who  may  be 
winners  or  losers. 

impacts  (specific 

Certain 
impacts)  can  be  observed  among  direct 
addressees  after  a few  months  or  in the  longer term  (e.g. the  monitoring  of 
assisted  firms  after  two  years).  In  the  field  of  development  support,  these 
impacts  are usually  referred to  as sustainable  results. 

Some  impacts  appear  indirectly  (e.g. turnover  generated  for  the  suppliers 
of  assisted  firms).  Others  can  be  observed  at  the  macro-economic  or 
macro-social  level  (e.g. improvement  of  the  image  of  the  assisted  region); 
these  are  global  impacts.  Evaluation  is frequently  used  to  examine  one  or 
more  intermediate  impacts,  between  specific  and  global  impacts.  Impacts 
may  be  positive  or  negative, expected  or  unexpected. 

Related  term(s): 

Global  impact 
Intermediate  impact 
Outcome 
Specific  impact 
Sustainable  result 
Impact  global 
Impact  intermédiaire 
Impact  spécifique 
Résultat  durable 

U n e x p e c t ed  effect 

An  impact  which  is  revealed  by  evaluation  but  was  not  foreseen  when  an 
intervention  was  launched. 

Effet 

inattendu 

Criteria and indicators 

inductive  analysis 

Unexpected  effects  are  revealed  by 
techniques 
(particularly  case  studies).  They  are  not  part  of  the  objectives.  A  positive 
unexpected  effect  may  become  an implicit  objective.  Negative  unexpected 
effects  are  also  called  perverse  effects.  For  example,  if  assistance  is 
granted for the development  of a territory and, for that purpose, the territory 
is classified  "disadvantaged", it will  become  less attractive for  business  as 
a result. 

Related term(s)  : 

Perverse  effect 

Side  effect 

Effet pervers 

Retombée 

Evaluation  criteria 

Criterion 

Critère  M 

Character,  property  or consequence  of a public  intervention  on the basis of 
which a judgement  will  be formulated. 

For example, an employment  incentive programme may be judged  in terms 
of  "costs  per  job  created"  or  "percentage  of  support  benefiting  the  long-
term  unemployed"  (in  the  latter  case,  it  is  assumed  that  the  higher  the 
percentage, the  better  the  intervention). 

An evaluation criterion  must be explicit, that is, it must clearly show why the 
intervention  will  be  judged  better  or  worse.  Criteria  frequently  used  in 
evaluation  are  performance,  effectiveness,  equity  and  sustainability.  Thus, 
evaluation  criteria  may  refer to  different  social  values. 

To  be  used  in  an  evaluation,  a  criterion  must  be  accompanied  by  a  norm 
(level of  success  at which  an intervention  will  be considered  good  in terms 
of  this  criterion).  An  intervention  is  generally  judged  in  terms  of  several 
criteria. 

Related term(s): 

Social  value 

Valeur sociale 

Norm 

Level that the intervention  has to reach to  be judged successful, in terms of 
a given criterion. 

For  example,  the  cost  per  job  created  was  satisfactory  compared  to  a 
national  norm  based on a sample  of comparable  interventions. 

Norme 

Related  term(s)  : 

Standard 

Glossary of 300  concepts and technical  terms 

Weighting 

Pondération 

Used to  state that  one criterion  is of  more  or  less  importance  than  another 
one  in the formulation  of  a global judgement  on an  intervention. 

The weighting  of criteria can be formalised by expressing it as a percentage 
(the total  being  100%). Multicriteria  analysis  also  uses weighting. 

Rationale 

Raison 

d'être 

The fact  that  an  intervention  can  be justified  in  relation  to  needs  to  satisfy 
or socio-economic  problems to  solve. 

Ex ante evaluation  verifies the  real existence  of these  needs  and  problems, 
and ensures that they  cannot  be met  or solved  by  existing  private or  public 
initiatives.  Thus,  the  inadequacy  or  shortcomings  of  other 
initiatives 
(whether  private  or  public)  are  a  fundamental  element  in  the  programme 
rationale,  by virtue  of the  principle  of  subsidiarity. 

Related  term(s)  : 

Raison  d'etre 
Bien-fondé 

Relevance 

Pertinence 

Appropriateness  of the  explicit  objectives  of  an  intervention, with  regard  to 
the  socio-economic  problems  the  intervention  is meant  to  solve. 

Questions  of  relevance  are  particularly  important  in  ex  ante  evaluation 
because  the  focus  is on  the  strategy  chosen  or  its  justification. Within  the 
framework  of  mid-term  evaluation,  it  is  advisable  to  check  whether  the 
this 
socio-economic  context  has  evolved  as  expected  and  whether 
evolution  calls  into  question the  relevance  of  a particular  initial  objective. 

Related  terms: 

Appropriateness 
Adéquation 

Criteria and indicators 

Eligibility 

Eligibilité 

The  fact  that  a  region,  project  or  group  of  people  has  the  required 
characteristics  to  benefit from an intervention  or, more  precisely, to  receive 
assistance. 

For  example,  European  regions  eligible  for  support  attributed  to  lagging 
development  areas  must  have  a  GDP  lower  than  75%  of  the  European 
Union  average.  Eligibility  criteria  follow  directly  from  the  rationale  of  the 
intervention. 

Disparity 

Disparité 

The  fact  that  a  region  or  group  of  people  are  in  a  situation  which  differs 
significantly  from others. 

In general,  the  rationale  of  programmes  financed  by  the  Structural  Funds 
consists  of  reducing  socio-economic  disparities  between  regions  or  social 
groups. 

Related  term(s)  : 

Inequality 
Inégalité 

Effectiveness 

Efficacité 

The fact that expected effects  have been obtained and that objectives  have 
been achieved. 

Effectiveness  can  be  assessed  by  answering  the  following  questions,  for 
example:  "Could  more  effects  have  been  obtained  by  organising  the 
implementation  differently?"  or  "Which  are  the  most  successful  operators 
or  measures?".  An  effectiveness  indicator  is  calculated  by  relating  an 
output, result or impact  indicator to a quantified objective.  For example: the 
objective  in  terms  of  number  of  firms  created  was  as  high  as  85%;  the 
placement  rate  obtained  by  operator  A  is  better  than  that  obtained  by 
operator  B.  For the  sake of  clarity,  it  may  be useful to  specify  whether  one 
is referring to the effectiveness  of  outputs,  results or  impacts. 

Glossary of 300  concepts and technical  terms 

Related  term(s): 

Efficacy 
Success 
Effectivité 
Réussite 
Succès 

Efficiency 

Efficience 

The fact  that  the  effects  were  obtained  at  a reasonable  cost. 

Efficiency  may  be  assessed  by  answering  the  following  questions,  for 
example:  "Could  more  effects  have  been obtained  with  the same  budget?" 
or  "Have other  interventions  obtained the same effects  at a lower cost?". An 
indicator  of  efficiency 
inputs 
mobilised  by the quantity of effects obtained. For example: the average cost 
of training  a person  who  has been jobless  for  a long time  is 2,000  euro; the 
intervention should  achieve a cost  per job  created  of  less than  30,000  euro. 

is  calculated  by  dividing  the  budgetary 

For  the  sake  of  clarity,  it  could  be  useful  to  specify  whether  the  efficiency 
referred  to  relates  to  outputs,  results  or  impacts. The  efficiency  of  outputs 
is called the  unit  cost. 

Related  term(s): 

Cost  effectiveness 
Unit-cost. 
Value for  money 
Coût-efficacité 
Coût  unitaire 

Sustainability 

Durabilité 

The ability  of  effects  to  last  in the  middle  or  long term. 

Effects  are  sustainable  if  they  last  after  the  funding  granted  by  the 
intervention  has ceased. They  are not  sustainable  if an activity  is unable  to 
generate  its  own  resources,  or  if  it  is  accompanied  by  negative  effects, 
particularly  on the environment,  and  if that  leads to  blockages  or rejection. 

Criteria and indicators 

Related  term(s)  : 

Viabilité 

Pérennité 

Performance 

Performance 

The  fact  that  effects  were  obtained  at  a  reasonable  cost  and  that  the 
addressees  are satisfied  with them. 

For  example:  a water  purification  programme  has  a high  performance  rate 
if  the  per  capita  cost  is  limited  compared  to  similar  interventions,  if  the 
purification  plants  built  comply  with  quality  standards,  and  if  the  rivers 
concerned  are  less polluted. 

include  qualitative  dimensions. 

Efficiency  and  performance  are  two  similar  notions,  but  the  latter  extends, 
more  broadly,  to 
In  certain  contexts, 
performance  concerns  outputs  and  results  but  not  impacts.  In  other 
contexts, the term applies mainly to either outputs (World Bank -  Operations 
Evaluation  Department)  or  impacts.  The  meaning  of  the  word  performance 
is not yet  stable; it  is therefore  preferable  to define  it whenever  it  is used. 

Utility 

Utilité 

Í : 

The  fact  that  the  impacts  obtained  by  an  intervention  correspond  to 
society's  needs  and to  the socio-economic  problems  to  be solved. 

Utility  is  a  very  particular  evaluation  criterion  because  it  disregards  all 
reference  to  stated  objectives  of  an  intervention.  It  may  be  judicious  to 
apply  this  criterion  when  objectives  are  badly  defined  or  when  there  are 
many  unexpected  effects.  The  criterion  must,  however,  be  used  with 
caution  to  avoid  the  evaluation  team  being 
influenced  by  personal 
considerations  in  their  selection  of  important  socio-economic  needs  or 
problems.  Some  authors  have argued  for  a form  of  goal-free  evaluation. 

Related  term(s): 

Goal free  evaluation 

Evaluation affranchie  des  objectifs 

Glossary of 300  concepts and technical  terms 

Subsidiarity 

Subsidiante 

The principle which justifies that a public  authority  decides to  implement  an 
intervention  rather  than  to  leave  it  up  to  private  initiative  or  another  public 
authority. 

The  principle  of  subsidiarity  justifies  public  intervention  when  there  are 
shortcomings  in  the  private  sector  and  when  other  public  administration 
levels would  not  have  been as effective. Subsidiarity  is part  of the  rationale 
of  an intervention. 

In  the  European  context,  subsidiarity  means,  for  example,  that 
the 
Community  acts  in those  cases where an  objective  can  be achieved  better 
at the  European  level than  at the  level  of  Member  States taken  alone. This 
corresponds  to  the  notion  of  Community  value  added  which  is  frequently 
considered  a  criterion  to  take  into  account  during  the  evaluation  of 
programmes. 

Related  term(s): 

Community  value  added 

Valeur ajoutée  communautaire 

Measures and information systems 

Indicator 

Indicateur 

Measurement  of  an  objective  to  achieve,  a  resource  mobilised,  an  output 
accomplished, an effect obtained or a context variable (economic, social or 
environmental). 

The  information  provided  by  an  indicator  is of  a quantitative  nature  and  is 
used to  measure facts  or  opinions  (e.g. percentage  of  regional  enterprises 
which  have  been  assisted  by  public  intervention;  percentage  of  trainees 
who  claim  to  be  satisfied  or  highly  satisfied).  An  indicator  must,  among 
other things, produce simple information which is easy to communicate and 
easily  understood  by  both  the  provider  and  the  user  of  the  information.  It 
must  help  the  managers  of  public  intervention  to  communicate,  negotiate 
and decide. For that purpose, it should  preferably  be linked to a criterion on 
the success of the intervention. It must reflect precisely whatever  it is meant 
to  measure  (validity  of  construction).  The  indicator  and  its  measurement 
unit  must  be  sensitive,  that  is  to  say,  the  quantity  measured  must  vary 
significantly  when  a  change  occurs  in  the  variable  to  be  measured. 
Indicators  may  be  specially  constructed  by  the  evaluation  team  and 
quantified  by means of  surveys or statistical data. They are often  borrowed 
from  the  monitoring  system  or  statistical  series.  An  indicator  may  be 
elementary  or derived from  several  other  indicators  in the form  of  ratios  or 
indexes. 

Related term(s)  : 

Quantitative  data 

Sensitivity 

Donnée  quantitative 

Sensibilité 

Measurement  unit 

Used  to  observe  a  phenomenon,  change  or  variable,  and  to  place  it  on  a 
quantitative scale. 

Unité  de  mesure 

Glossary of 300  concepts and  technical  terms 

A  measurement  unit  allows  for  quantification.  An  elementary  indicator  is 
associated  with  a  measurement  unit  and  has  only  one  dimension  (e.g.  km 
of  motorway;  number  of  training  courses).  Some  measurement  units  are 
divisible  and others  not  (e.g. 20.3km were  built; 30 trainees  were  qualified). 
Measurement  units must  be harmonised  if indicators are to  be comparable. 

Related  term(s): 

Quantification 
Quantification 

Descriptor 

Descripteur 

A  description,  in the  form  of  a  concise,  clear  and  stable  statement,  of  an 
objective to  achieve  or an  impact  obtained. 

For  example,  one  can  examine  the  impact  of  a  measure  in terms  of  equal 
opportunities,  and conclude by choosing  a pre-established  descriptor  such 
as:  (1)  "the  measurement  enabled  most  women  addressees  to  qualify  for 
jobs  considered  to  be  reserved  for  men",  or  (2)  "the  measure  attracted  a 
large  proportion  of  women  addressees  who  wanted  to  qualify  for  jobs 
considered  to  be reserved  for  men". 

The  organisation  of  descriptors  in  the  form  of  a  structured  grid  may 
constitute  the  first  step  in  the  construction  of  an  indicator.  If  several 
descriptors  have  been  established  beforehand,  they  can  be  used  to 
construct  an  observation  grid.  By  means  of  this  grid  a  phenomenon  or 
change  can  be  observed  and  described  in  a  qualitative  and  structured 
way.  Evaluation  cannot  afford  to  exclude  from  its  scope  of  analysis  an 
important  objective  or  impact  simply  because  it  is  difficult  to  measure 
quantitatively  when  in fact  it  is considered  to  be  important.  In that  case,  it 
is preferable to  collect  qualitative  data  and to  structure them  by  means  of 
descriptors. 

Related  term(s) 

Observation  grid 
Qualitative  data 
Statement 
Donnée  qualitative 
Enoncé 
Grille  d'observation 

Criteria and indicators 

Scoring 

Notation ■ 

Choice of a level on a scale graduated in quantitative measurement units 
(e.g. a scale of 0 to 100 or -3 to +3) in order to represent the significance of 
an effect, need or element of quality. 

It is possible to construct an observation grid which is sufficiently structured 
to directly produce a score. The person who chooses the score is called the 
scorer or the assessor. Statistical analysis of results studies scoring 
systems. 

Related term(s): 

Rating 

Monitoring information system 

Système d'information de suivi 

Measures taken so that operators can collect and store information on 
inputs, outputs and results, and so that they can process and transmit this 
information regularly to the managers (e.g. progress reports). 

The monitoring information system also includes the monitors' syntheses 
and aggregations, periodically presented to the authorities responsible for 
the implementation (reviews, operating reports, indicators, etc.). In 
European Union socio-economic programmes, the key element in an 
information system is a system of indicators. 

Related term(s): 

System of indicators 
Système d'indicateurs 

Operating report 

Tableau de bord 

Presentation of essential information for the monitoring of an intervention, 
in the form of a small number of periodically quantified indicators. 

The operating report concerns the inputs mobilised, the outputs achieved 
and the results observed. It also consists of a few statistical indicators used 

Glossary of 300  concepts and technical  terms 

to  monitor  important  elements  In the context  (provided the  information  can 
be  obtained  without  too  much  delay).  The  operating  report  is  part  of  the 
monitoring  information  system. It  is often  produced  by  means of  data  base 
management  software.  The  advantage  of  an  operating  report  is  that  it 
allows  for  the  continuous  visualisation  of  the  main  parameters  of  public 
intervention  as well  as  rapid  feedback. 

Statistical  t e am 

Observatoire 

The  organisation  or  system  used  to  regularly  gather  or  disseminate 
statistical information on a given theme and over a long period (e.g. statistics 
on equal  opportunities;  environmental  statistics  in a nature reserve). 

Due to  its permanence, a statistical team can  produce chronological  series, 
something  which  an  evaluation  that  takes  place  in  a  limited  timeframe 
cannot  do.  When  they  exist,  statistical  teams  are  interesting  sources  of 
secondan/  data for  evaluations. 

Comparability 

Comparabilité 

Quality  of  an  indicator  which  uses  the  same  measurement  unit  to  quantify 
the  needs, objectives  or  effects  of  several  different  interventions. 

to  that  of  similar 

interventions).  Efforts  made  to 

Comparability  is  useful  for  establishing  norms  for  judgement  (e.g.  the 
average  cost  of  jobs  created  by  the  intervention  can  be  favourably 
improve 
compared 
comparability  involve  the  harmonisation  of  measurement  units  and  result, 
initially,  in  the  definition  of  standard  indicators,  i.e.  indicators  that  can  be 
used  in  several  regions  with  the  same  definition  for  the  same  sector  of 
intervention  (e.g.  number  of  SMEs  assisted,  defined  and  calculated  in  a 
comparable  way).  Secondly,  comparability  can  be  extended  to  key 
indicators,  that  is,  indicators  which  can  be  used  in  several  regions  and 
sectors  of  intervention. 

Related  term(s) 

Harmonisation 
Standard  indicator 
Harmonisation 
Indicateur  standard 

Criteria and  indicators 

Field  of  intervention 

Champ 

d'intervention 

A  set  of  interventions  which  are  similar  enough  for  their  indicators  to  be 
harmonised and for comparisons to be made between different  evaluations. 

For  example,  the  same  programme  can  predict  outputs  in  the  field  of 
research  and  development,  in  that  of  transport  infrastructure,  in  that  of 
training,  and  so  on.  Within  the  framework  of  European  cohesion  policy, 
fields of  intervention are grouped together  into three main categories:  basic 
infrastructure,  productive  environment  and  human  resources. 

Related  term(s): 

Secteur  d'intervention 

Indicators 

Context  indicator 

Indicateur  de 

contexte 

Measurement  of  an  economic,  social  or environmental  variable  concerning 
an  entire  region,  sector  or  group  in  which  public  intervention  takes  place 
(e.g. per capita  GDP, annual  number  of jobs  created  in the  region). 

Context  indicators  may  describe  a  basic  situation  before  an  intervention 
and  a desired  situation  after  intervention. They  are  generally  quantified  on 
the  basis  of  data  from  statistics  offices  or  statistical  teams.  They  apply  to 
an  entire  territory  or  group,  unlike  programme  indicators  which  apply  only 
to addressees  actually  affected  by  an intervention. 

Baseline 

Situation  de  base 

State  of  the  economic,  social  or  environmental  context,  at  a  given  time 
(generally at the beginning of the intervention), and from which changes  will 
be measured. 

The  basic  situation  is  described  by  context  indicators  which  describe  the 
economy,  socio-economic  environment,  concerned  groups,  etc. 

Related  term(s): 

Point zéro 

Situation  de  référence 

P r o g r a m me  indicator 

Indicateur de programme 

Indicator  which  concerns the  inputs  and outputs  of  the  intervention  as well 
as the  results and  impacts  on  its direct  and  indirect  addressees. 

Programme  indicators  may  include derived  indicators  measuring  efficiency, 
effectiveness  or  performance.  They  are  quantified  by  monitoring 
information  systems and also by evaluation when it produces  primary  data. 
When  they  concern  effects,  programme  indicators  measure  only  those 
which  affect  direct  and  indirect  addressees. 

Glossary of 300  concepts and  technical  terms 

Generic  indicator 

Indicateur 

générique 

Indicator  which  uses  the  same  measurement  unit  to  quantify  the  impacts 
resulting from  several outputs  of various  kinds (e.g. value-added  generated 
information 
in  the  domains  of  fishing,  tourism  and  new 
by  outputs 
technologies). 

Key  indicator 

Indicateur 

clé 

Indicator  likely  to  play  an  important  part  in comparisons  between  different 
interventions  and  in the  synthesis  of  conclusions  of  several  evaluations. 

Within  the  framework  of  European  cohesion  policy,  key  indicators  are, for 
example,  the  budgetary  absorption  rate  or  number  of  SMEs  assisted  (for 
the monitoring  of  implementation), or the cost  per job  created or per  capita 
GDP  (for  the  evaluation  of  impacts).  Key  indicators  must  be  both  standard 
and  generic  indicators. 

Absorption  rate 

Taux  de  consommation 

budgétaire 

Budgetan/  inputs  mobilised  in  proportion  to the  inputs  initially  allocated. 

Related  term(s): 

Taux  d'absorption 

Completion  rate 

Taux  de 

réalisation 

Percentage  of  initially  planned  outputs  and  which  have been  completed. 

The completion  rate of a major  infrastructure  project  is calculated  in terms of 
the stages of work  which were  initially planned and were actually completed. 
The completion  rate of an intervention  is the average of the completion  rates 
of  the  projects  comprising  that  intervention.  The  completion  rate  is  a  key 
indicator  for  monitoring the  implementation  of  an intervention. 

Related  term(s): 

Taux  d'avancement 

Criteria and  indicators 

C o v e r a ge  rate 

Taux  d'atteinte  du  public 

eligible 

Percentage  of  the  eligible  group  which  was  actually  affected  by  an 
intervention. 

The  coverage  rate  is  a  result  indicator  which  is  important  to  quantify  for 
monitoring  purposes.  The  exposure  rate  will  be  referred  to  indicate  the 
portion  of  the  population  targeted,  which  received  information  on  the 
programme. 

Related  term(s): 

Exposure  rate 

Taux de  couverture 

Taux  d'exposition 

ν 
Impacts 

The following  pages list terms relating  to the impacts  of public  interventions 
and,  more  particularly,  those  which  contribute 
towards  socio-economic 
development.  The vocabulary, partly  borrowed  from  economics,  generally 
uses  the  term  "effect"  rather  than  "impact".  Since  the  two  terms  are very 
similar, the original  economic  vocabulary  has been  retained. 

For a complete  presentation  of  these subjects,  the reader is referred  to  the 
third part of Volume 1 of the Collection,  devoted  to socio-economic  impacts 
at the micro  and macro  levels, the third part  of  Volume 4 devoted  to  effects 
of  synergy,  and  Volume  5  which  covers  impacts  on  employment, 
the 
environment  and  other  Community  priorities. 

Causality analysis 

Gross  effect 

Effet  brut 

Change  observed  following  a  public  intervention,  or  an  effect  reported  by 
the  direct  addressees. 

A gross effect  appears to be the consequence of an intervention  but  usually 
it  cannot  be  entirely  imputed  to  it.  The  following  example  shows  that  it  is 
not  sufficient  for  an  evaluation  merely  to  describe  gross  effects:  Assisted 
firms claimed  to  have created  500 jobs  owing to the support  (gross  effect). 
In  reality,  they  would  in  any  case  have  created  100 jobs  even  without  the 
support  (deadweight).  Thus,  only  400  jobs  are  really  imputable  to  the 
intervention  (net  effect). 

Net  effect 

Effet  net 

Effect  really imputable to the public  intervention and to  it alone, as opposed 
to  apparent  changes  or gross  effects. 

To evaluate  net  effects,  based  on  gross  effects,  it  is  necessary  to  subtract 
the  changes  which  would  have  occurred  in  the  absence  of  the  public 
intervention,  and  which  are  therefore  not  imputable  to  it  since  they  are 
produced  by  confounding  factors  (counterfactual  situation).  For  example, 
the  number  of employees  in assisted firms appears to be stable (change or 
gross  effect  equal  to  zero).  However,  it  is  estimated  that  without  support 
there  would  have  been  400  redundancies  (counterfactual  situation).  Thus, 
400 jobs  were  maintained  (net  effect). 

Related  term(s): 

Additional  Effect 
Effef  additionnel 
Effet  propre 

D e a d w e i g ht 

the  public 
Change  observed  among  direct  addressees 
intervention,  or  reported  by  direct  addressees  as  a  consequence  of  the 
public  intervention, that would  have occurred, even without  the intervention. 

following 

Effet 

d'aubaine 

Glossary of 300  concepts and technical  terms 

For example: a farmer  received assistance for the  building of a  self-catering 
cottage.  In the survey  he stated that the support  had enabled  him to  create 
better  quality  facilities,  but  that  he  would  have  built  the  cottage,  even 
without  support. Thus, there is deadweight  since the the construction  of the 
cottage  cannot  be  imputed  entirely  to  the  intervention.  Deadweight  can 
account  for  as  much  as  50%  of  the  gross  effect.  The  estimation  of 
deadweight  necessitates  a  survey  of  direct  addressees,  preferably  with  a 
comparative  analysis  of  non-participants. 

Related  term(s): 

Perte  sèche 

Poids  mort 

D i s p l a c e m e nt  effect 

Effet  de 

déplacement 

Effect  obtained  in an eligible  area at the  expense  of  another  area. 

Displacement  effects  may  be  intended  (e.g.  displacement  of  a  public 
administration  from  the  capital  to  a  region  undergoing  redeployment)  or 
unintended  (e.g.  10%  of  the  jobs  created  by  a  regional  development 
programme  resulted  in the  disappearance  of  jobs  in other  eligible  regions; 
a firm  used  programme  assistance to  move  its  premises from  the  centre  to 
the  outskirts  of  a town). When  they  are  not  intended,  displacement  effects 
must  be subtracted  from  gross  effects  to  obtain  net  effects. 

Substitution  effect 

Effet  de 

substitution 

Effect  obtained  in  favour  of  a  direct  addressee  but  at  the  expense  of  a 
person  or  organisation  that  does  not  qualify  for the  intervention. 

For  example, a  person  unemployed  for  a long  time found  a job  owing to  the 
intervention.  In  reality,  this  job  was  obtained  because  someone  else  was 
granted  early  retirement.  If  the  objective  was  the  redistribution  of  jobs  in 
favour  of  disadvantaged  groups,  the  effect  can  be  considered  positive.  An 
evaluation  determines,  with  regard  to  the  objectives  of  the  intervention, 
whether the substitution effect  can be considered  beneficial or not. When it is 
not  beneficial, the substitution  effect  must  be subtracted from gross  effects. 

Impacts 

Counterfactual  situation 

Situation 

contrefactuelle 

A  situation  which  would  have  occurred  in  the  absence  of  a  public 
intervention. 

For  example,  a firm  was  assisted  so  that  its  employees  could  be  retrained 
in  new  technologies.  No  redundancies  were  recorded  in the  following  two 
years.  It  is  estimated  that  without  the  assistance  (counterfactual  situation) 
50  jobs  would  have  been  lost.  By  comparing  the  counterfactual  and  real 
situations,  it  is  possible  to  determine  the  net  effects  of  the  public 
intervention.  Various  tools  can  be  used  for  the  construction  of 
the 
counterfactual  situation:  shift-share  analysis,  comparison  groups, 
simulation  using  macro-economic  models,  etc.  At  the  baseline,  the  real 
situation  and the counterfactual  situation  are identical.  If the  intervention  is 
effective  , they  diverge  while  it  is  underway. 

Related  term(s): 

Policy-off  situation 

Scénario  hors  intervention 

Confounding  factor 

Facteur 

concurrent 

Factors  independent  of  a public  intervention  which  are  partly or entirely  the 
cause of  changes  observed  among  addressees  (gross  effects). 

A  confounding  factor  can  hinder  or  amplify  an  expected  impact.  For 
example,  employment  increased  by  10%  in  a  group  of  assisted  firms. 
However,  this  increase  could  not  be  imputed  entirely  to  the  public 
intervention  because  the  assisted  firms  also  benefited  from  a  favourable 
macro-economic  context  (confounding  factor). In another example, close to 
80%  of  the  trainees  found  a  job  after  one  year.  However,  this  high 
placement  rate  was  partly  imputable  to  the  initial  qualifications  of  the 
trainees  which  were very  high at the time of their  recruitment  (skimming-off 
effect). 

When  a  causality  analysis  is  carried  out  as  part  of  an  evaluation,  it 
distinguishes  between  effects  imputable  to  the  public  intervention  (net 
effects)  and those  imputable  to  confounding  factors. 

Glossary of 300  concepts  and  technical  terms 

Related  term(s): 

Concurrent  factor 

Extraneous  factor 

Exogenous  factor 

Co-facteur 

Facteur  exogène 

Facteur  contextuel 

Causality  analysis 

Analyse  de 

causalité 

The study  of relations  of cause and effect  which  link  a public  intervention  to 
its  impacts. 

Causality  analysis  may  be  inductive.  In  this  case,  it  investigates  the 
mechanisms  likely to produce  impacts, as well as confounding  factors  likely 
to  have  an  influence.  Causality  analysis  may  also  be  deductive  (or 
hypothetico-deductive).  In  this  case,  it  examines  whether  assumptions 
about  impacts  are  not  contradicted  by  the  facts.  It  may  also  supply  a 
quantitative  estimation  of  impacts. 

Related  term(s): 

Deductive  analysis 

Inductive  analysis 

Analyse  deductive 

Analyse  inductive 

T h e o ry  of  action 

Théorie 

d'action 

All the hypotheses  used by funding authorities  and managers to explain  how 
a public  intervention  is going to  produce  its  impacts  and  achieve  its aim. 

The  theory  of  action  consists  of  relations  of  cause  and  effect  linking 
outputs,  results  and  impacts.  It  is  often  implicit,  or  at  least  partly  so. 
Evaluation  helps to  clarify  the theory  and for that  purpose  relies on a logical 
diagram  of  expected  impacts. 

Impacts 

Related  term(s): 

Programme  theory 

Referentiel 

Modèle  de  causalité 

Explanatory  theory 

All the assumptions  likely to  explain changes observed  following the  public 
intervention  (gross  effects). 

Théorie 

explicative 

likely  to  explain  gross  effects, 

The  scope  of  explanatory  theory  is  far  wider  than  that  of  the  theory  of 
action.  Like  the  theory  of  action,  it  encompasses  relations  of  cause  and 
effect  between  outputs,  results  and  impacts.  It  also  covers  any  other 
causes 
factors. 
Evaluation  relies  on  a list  of  explanatory  assumptions  established  with  the 
help of experts, based on research and evaluation in similar fields.  Inductive 
analysis 
the  construction  or 
improvement  of  explanatory  theory. 

techniques  also  contribute 

i.e.  all  confounding 

towards 

Extrapolation  coefficient 

Coefficient 

d'extrapolation 

Ratio  used to  estimate  an  impact  through  the  use of  monitoring  data. 

For  example,  if the  amount  of  investments  by assisted  SMEs  is  known, an 
extrapolation  coefficient  can  be  used  to  estimate  the  annual  value  added 
generated.  In  this  case,  the  extrapolation  coefficient  may  be  the  ratio  of 
annual value added  per euro invested. This ratio will be drawn from  relevant 
regional  or  sectoral  statistics. 

If the evaluation team uses existing extrapolation coefficients, it is important 
for  it to  explain  and justify  its choices. A coefficient  can  be deduced from  a 
questionnaire  survey  (provided  the  sample  is representative)  conducted  as 
part  of  the  same  evaluation  or  a preceding  one. A  coefficient  may  also  be 
taken from  an evaluation  or  research  on  similar  interventions,  provided  the 
conclusions  can  be  generalised  (external  validity).  A  coefficient  can  be 
established  for  the  needs  of  an  evaluation  by  an  expert  panel  or  by  a 
network  of  experts,  using the  Delphi  panel  technique. 

Socio-economic effects 

Direct  effect 

Effet  direct  U 

Effects  of  a  public  intervention  on  its  direct  addressees,  excluding  all 
repercussions  on other  groups. 

For example: investment  support  has direct effects on assisted  businesses 
(production  costs  and  capacity;  creation  or  maintenance  of  jobs).  New 
infrastructure  has  a  direct  effect  on the  people  and  enterprises  which  use 
it. 

Indirect  effect 

Effet  indirect 

I 

Effect  which  spreads  throughout  the  economy,  society  or  environment, 
beyond  the direct  addressees  of  the  public  intervention. 

Indirect  "internal" effects, which  are spread through  market-based  relations 
(e.g.  effect  on  suppliers  or  on  the  employees  of  an  assisted  firm),  are 
distinguished  from  external  effects  or  "externalities"  which  are  spread 
through  non-market  mechanisms  (e.g.  noise  pollution;  cross-fertilisation 
within  an  innovation  network). 

Related  term(s): 

Effet  induit 

First  round  effect 

Effet 

primaire 

Effect  of  a  public  intervention  on  direct  addressees  and  on  the  inner  circle 
of  indirect  addressees. 

For  example,  the  impact  on  assisted  firms  and  their  customers,  suppliers 
and  employees.  First  round  effects  may  be  analysed  by  means  of 
evaluation  surveys. They are, by  nature, micro-economic,  unlike  secondary 
effects  which  concern  circles  of  increasingly  indirect  addressees. 

Glossary of 300  concepts  and  technical  terms 

Related  term(s): 

Knock-on  effect 
Impact  de premier  ordre 

S e c o n d a ry  effect 

Effet 

secondaire 

All  the  impacts  produced  by  an  intervention, over  and  above  its first  round 
effects,  that  is, apart  from  direct  addressees  and  the  first  circle  of  indirect 
addressees. 

Potentially,  the  mechanisms  of  secondary  effects  stretch  in  concentric 
circles  throughout  the  entire  economy.  They  are  generated  by  supplier 
effects,  income  multiplier  effects,  impulsion  effects,  etc.  The  evaluation  of 
secondary  effects  is based  on  macro-economic  reasoning  and  techniques. 

Externality 

Externante 

Effect  of  a  private  action  or  public  intervention  which  is spread  outside  the 
market. 

For example: a firm  pollutes  a  river  and causes  an economic  loss for  a fish 
farm  downstream;  an  engineer  leaves  the  firm  in  which  he  or  she  was 
trained  and  applies  his  or  her  know-how  in  a  new  firm  which  he  or  she 
creates.  By  their  very  nature,  externalities  trigger  private  choices  which 
cannot  be  optimised  through  the  mechanisms  of  market  competition.  Only 
collective  and  often  public  decisions  are  able  to  promote  positive  external 
effects  and  prevent  negative  ones.  A  large  proportion  of  financial  support 
allocated  within  the  framework  of  European  cohesion  policy  is  aimed  at 
promoting  positive external  effects  which  businesses  do  not  seek  to  create 
themselves  spontaneously. 

L e v e r a ge  effect 

Effet  de 

levier 

Propensity  for  public  intervention  to  induce  private  spending  among  direct 
addressees. 

In cases where  public  intervention  subsidises  private investments,  leverage 
effects  are  proportional  to  the  amount  of  private  spending  induced  by  the 

Impacts 

subsidy.  Leverage effects  must  not  be confused with additional  effects  (see 
net  effect).  Nor  do  they  refer to  the  principle  of  additionality  which  applies 
to  European  social  and economic  cohesion  policy  (see additionality). 

D e m a n d - s i de  effect 

Effet  de 

demande 

Effect  which  spreads  through  growing 
intermediate  consumption  of 
enterprises  (supplier  effect)  and  through  the  income  distributed  within  the 
assisted  region,  and  which,  in  turn,  generates  spending  by  households 
(multiplier  effect).  When  public  financing  ceases,  demand  side  effects 
disappear. 

Supplier  effect 

Effet 

fournisseur 

Secondary  effect  on  companies  supplying  goods  and  services 
businesses  which  are the direct  addressees  of  a public  intervention. 

to 

There  may  be several  cycles  of  supplier  effects,  if the  analysis  is  extended 
to the  suppliers  of  suppliers,  etc.  Input-output  analysis  is  used to  estimate 
all  supplier  effects  in a given  territory. 

Income  multiplier  effect 

Effet  multiplicateur 

de 

revenu 

Secondary  effect  resulting  from 
generated  by the  public  intervention. 

increased 

income  and  consumption 

Multiplier  effects  are cumulative  and  take  into  account  the fact  that  part  of 
the  income  generated  is spent  again  and  generates  other  income,  and  so 
on  in  several  successive  cycles.  In  each  cycle,  the  multiplier  effect 
diminishes  due  to  purchases  outside  the  territory.  The  effect  decreases 
much faster  when  the territory  is small and when  its  economy  is open. 

Supply-side  effect 

Efref  d'offre 

Secondary  effect  which  spreads  through  the  increased  competitiveness  of 
businesses  and thus  of  their  production. 

Glossary of 300 concepts and technical terms 

The main mechanisms at play are increased productive capacity, increased 
productivity, reduced costs, and the diversification and reinforcement of 
other factors of competitiveness such as human capital, public facilities, the 
quality of public services, innovation networks, etc. 

Structuring effect 

Effet structurant 

Sustainable changes in socio­economic structures following a public 
intervention. 

Structuring effects are changes which last after the public spending has 
ceased. They include sustainable effects at the micro­economic level and 
supply­side effects, but not demand­side effects. Structuring effects must 
not be confused with structural adjustment, which strives for the 
convergence of the macro­economic variables of a country towards 
international standards, particularly in terms of public finances and inflation. 

Related term(s): 

Lasting effect 
Effet structurel 

■ Impulsion effect 

Effet d'entraînement ■ 

Secondary effect which spreads through investments induced upstream 
and downstream from the sector affected by the intervention. 

For example, the construction of a large infrastructure project generates the 
creation of new businesses in the region. These continue to expand after 
the work has ceased. 

■ N e t w o rk effect 

Effet de réseau 

Secondary effect which spreads through special and lasting relations 
between businesses in the same territory or branch, and between those 
same businesses and public or para­public organisations such as research 
centres, universities, etc. 

Network effects are partially based on non­market interpersonal relations. 
Thus, they are in part "externalities". 

Effects of synergy 

Internal  coherence 

Cohérence 

interne 

Correspondence  between the different  objectives  of the same  intervention. 

Internal coherence  implies that there is a hierarchy  of objectives, with those 
at the  bottom  logically  contributing  towards those  above. 

Related  term(s): 

Consistency 

External  coherence 

Cohérence 

externe 

Correspondence  between  the  objectives  of  an  intervention  and  those  of 
other  public  interventions  which  interact  with  it. 

If  a  national  policy  and  a  European  Union  socio-economic  programme  are 
implemented  in  a  complementary  manner  in  the  same  territory  for  the 
purpose of developing SMEs, it can be said that there is external coherence. 

Complementarity 

Complémentarité 

The  fact  that  several  public  interventions  (or  several  components  of  an 
intervention)  contribute  towards the  achievement  of the same  objective. 

Complementarity  may  be  functional,  if  the  objective  is  functional  (e.g. 
developing  a  sector  of  activity,  creating  a  network).  It  may  be  territorial,  if 
the objective concerns a territory  (e.g. integrated  local development  policy). 

Synergy 

Synergie 

The  fact  that  several  public  interventions  (or  several  components  of  an 
intervention)  together  produce  an  impact  which  is greater  than  the  sum  of 
the  impacts they  would  produce  alone  (e.g. an  intervention  which  finances 

Glossary of 300  concepts and technical  terms 

the  extension  of  an  airport  which,  in turn,  helps  to  fill  tourist  facilities,  also 
financed  by the  intervention). 

Synergy  generally  refers  to  positive  impacts.  However,  phenomena  which 
reinforce  negative  effects,  negative  synergy  or  anti-synergy  may  also  be 
referred to  (e.g. an  intervention  subsidises  the diversification  of  enterprises 
while a regional  policy  helps to  strengthen  the dominant  activity). 

Employment effects 

Employment  effect 

Improvement 
structural  modification  of the  labour  market,  following  an intervention. 

in  employability,  creation  and  maintenance  of  jobs,  or 

Effet 

d'emploi 

Job  created  or  maintained 

Emploi  créé  ou  maintenu 

New jobs created owing to the public  intervention, or threatened jobs  which 
the  public  intervention  helped to  safeguard. 

Employability 

Employabilité 

An  individual's  ability  to  find  or  keep  a  job  in  a  given  socio-economic 
environment. 

Employability  concerns  the  appropriateness  of  skills  in  relation  to  the 
requirements  of  the  labour  market,  so  that  the  individual  concerned  can 
keep  his  or  her  job  or  find  a  (new)  job  in  reasonable  conditions  in  a 
reasonable  length  of  time.  A  public  intervention  in  favour  of  employability 
concerns  human  resource  development  and  particularly  training.  The 
employability  of  an  individual  can  be  examined  indirectly  on  the  basis  of 
pre-established  factors  (e.g. qualifications,  experience,  mobility,  existence 
of job  offers). 

Transition  rate 

Taux  de 

transition 

Percentage of addressees whose social integration  has improved, following 
the  intervention. 

The  improvement  may  correspond  to  the  people's  recruitment  (placement 
rate)  or  to the  continuation  of  training  on  a  higher  level. The transition  rate 
is evaluated  after a period of  months  (for example,  12 months  after the  end 
of  a training  course). 

Glossary of 300  concepts and technical  terms 

Related  term(s): 

Placement  rate 

Taux de  placement 

Skimming-off  effect 

Consequence  of the recruitment  of  participants for a training course,  which 
favours those  most  likely to  obtain the  best  results. 

Effet 

d'écrémage 

is  an  unintended  effect  which 

This 
the  measurement  of 
performance  rather  than  the  satisfaction  of  the  needs  of  those  who  are 
most  disadvantaged. 

favours 

P e r m a n e nt  job 

Emploi 

permanent 

A  job  generated  by  an  intervention,  which  lasts  after  public  support  has 
ceased. 

By contrast, a temporary  job  lasts  only  until the end of  an  intervention. The 
creation  of  permanent  jobs  is a sustainable  effect  of  an intervention. 

Full-Time  Equivalent  (FTE) 

Equivalent 

temps  plein 

(ETP) 

Conversion of  part-time jobs into quantities corresponding to full-time jobs. 

For  example,  an  intervention  to  support  exports  enables  firms  to  find  new 
markets and to create 8 full-time and 22 part-time jobs. The impact  in terms 
of  full-time  equivalents  is 19. 

Economie and social development 

Economic a nd social cohesion 

Cohésion économique et sociale 

The fact that there are unacceptable gaps between regions, territorial 
categories (e.g. urban / rural) or social groups, from the point of view of their 
level of economic development and social integration. 

Economic and social cohesion relates primarily to disparities in regional 
income and quality of life, as well as to all differences between social 
groups' access to employment. It is the main objective of the European 
policy of the same name (see Article 130 A of the Treaty). 

Sustainable d e v e l o p m e nt 

Développement durable ■ 

Increase in economic activity which respects the environment and uses 
natural resources harmoniously so that future generations' capacity to meet 
their own needs is not compromised. 

By contrast, unsustainable development is characterised by the destruction 
of natural resources. This has negative repercussions on long-term 
development potential. 

Related term(s): 

Développement viable 

■ Mitigation 

Action aimed at reducing the potential negative effects of a public 
intervention on the environment. 

Atténuation 

Endogenous d e v e l o p m e nt 

Développement endogène 

Increase in economic activity based on internal competitive advantages 
within a region or territory. 

Glossary of 300  concepts and technical  terms 

The  main  factors  of  endogenous  development  are  human  capital, 
enterpreneurial  spirit,  local  savings  and  local  innovation  networks.  By 
contrast,  exogenous  development  favours  the  inward  transfer  of  capital, 
technology,  know-how  and  skills. 

Competitiveness 

Compétitivité 

The  capacity  of  a  firm,  in  a  competitive  socio-economic  environment,  to 
defend  or  increase  its  market  share over  the  medium  and  longer  term,  and 
to  generate wealth. 

The  notion  of  competitiveness  can  apply  to  a  single  firm  or  to  all  the 
businesses  in  a  sector  or  region.  It  is  directly  related  to  the  notion  of 
competitive  advantage, an advantage which an enterprise, region or  sector 
must  possess  or  develop  if  it  is  to  be  competitive  in  a  segment  of  a 
particular  market. 

SWOT analysis is used to  identify  existing advantages (called  "strengths"  or 
success  factors)  and  disadvantages  that  need  to  be  reduced  (called 
"weaknesses"). 

Diversification 

Diversification 

Deployment  of  the  production  of  a  firm  or  a  region  beyond  its  traditional 
specialisation,  with  a  view  to  creating  new  activities  and/or  setting  up  in 
new  market  segments. 

Policies which create incentives and support for diversification  are intended 
for  all  enterprises  established  in market  segments  which  are threatened  or 
in crisis. 

Equal  opportunities 

Egalité des chances 

Equal  access  for  women  and  men  to  employment,  at  the  same  level  of 
remuneration  and  social advantages,  in a given  socio-economic  context. 

This  impact  relates  to  the  principle  of  equal  rights  and  equal  treatment  of 
women  and  men.  It  means,  first,  that  everybody  is  free  to  develop  their 

Impacts 

personal  aptitudes  and  to  make  choices  without  being 
limited  by 
stereotyped  gender  roles  and,  secondly,  that  particular  differences  in 
behaviour,  aspirations  and  needs,  between  women  and  men, are not  to  be 
valued  too  highly  nor  considered  too  critically.  The  principle  of  equal 
opportunities  may 
for 
discrimination.  The  evaluation  of  impacts  on  equal  opportunities  includes 
the  mainstreaming  of  gender  at  all stages. 

require  unequal 

to  compensate 

treatment 

M a i n s t r e a m i ng 

Intégration 

Systematically  taking  into  account  the  specific  priorities  and  needs  of 
women  and  men  in  all  dimensions  of  an  intervention, from  the  design  and 
implementation  stage to  monitoring  and  evaluation. 

For example, equal  opportunities  is mainstreamed  in evaluation  in so far as 
evaluation  systematically  distinguishes  men  and  women  among  the  direct 
and  indirect  addressees  of  a public  intervention, and among  operators  and 
decision-makers,  when  this  distinction  concerns  needs,  results  and 
impacts.  By  extension,  the  principle  of  integration  may  also  apply  to  other 
impacts  considered  to  be  priorities. 

Related  term(s): 

Intégration  stratégique 

Environmental  I m p a ct  A s s e s s m e nt  (EIA) 

Evaluation  d'Impact 

sur  l'Environnement 

(EIE) 

Study  of  all  the  repercussions  of  an  individual  project  on  the  natural 
environment. 

EIA  is  a  compulsory  step  in  certain  countries  in  the  selection  of  major 
infrastructure projects. By contrast, strategic  environmental assessment (SEA) 
refers  to  the  evaluation  of  programmes  and  policy  priorities.  EIA consists  of 
two steps: screening, which refers to an initial overall analysis to determine the 
degree  of  environmental  evaluation  required  before  the  implementation  is 
approved; and scoping which determines which impacts  must be evaluated in 
depth.  The  evaluation  of  environmental  impacts  examines  expected  and 
unexpected  effects. The latter are often  more  numerous. 

VI 
Techniques 

lists 

the  main 

This  part 
the  design  and 
implementation  of  an  evaluation.  For  a  complete  presentation  of  the  main 
evaluation  techniques,  the  reader  is referred  to  Volume  3 of  the  Collection, 
which  includes  23  detailed  information  sheets. 

terms  used 

technical 

in 

Evaluation methods 

M e t h od 

Méthode 

Complete  plan of  evaluation  team's  work. 

A method  is an ad hoc  procedure, specially  constructed  in a given  context 
to  answer  one  or  more  evaluative  questions.  Some  evaluation  methods 
are  of  low  technical  complexity,  while  others  include  the  use  of  several 
tools. 

Evaluation  design 

Protocole 

d'évaluation 

Technical  part  of  the  evaluation  plan,  designed  for  data  collection  and 
analysis. 

M e t h o d o l o gy 

Méthodologie 

Strictly  speaking,  this  is  the  science  of  the  construction  of  evaluation 
methods. 

The  technical  design  of  an  evaluation  is,  however,  a  matter  of  know-how 
and  expertise  rather than a purely  scientific  procedure. In practice, the term 
methodology  is often  used  in the  sense  of  a  method, as defined  above. 

Tool 

Outil 

Standardised  procedure  used  to 
regression  analysis  or questionnaire  survey). 

fulfil  a  function  of  evaluation  (e.g. 

Evaluation tools  serve to  collect  quantitative  or  qualitative  data,  synthesise 
judgement  criteria,  explain  objectives,  estimate  impacts,  and  so  on. A  tool 
has  a specific  name,  is  described  in  a  manual,  and  can  be  taught.  Its  use 
may  be  free  or  may  involve  payment  of  a  fee  if  the  tool  is  protected  by  a 
registered  trademark.  An evaluation  may  use one or  more  tools. 

Glossary of 300  concepts and technical  terms 

Related  term(s): 

Technique 

Technique 

Structuring techniques 

Strengths,  W e a k n e s s e s,  Opportunities,  Threats  (SWOT) 

Forces, 

faiblesses,  opportunités, 

menaces 

Analysis  of  internal factors which can  be relied on (strengths) or which  need 
to  be compensated  for  (weaknesses),  as well  as external  factors  which  are 
favourable  (opportunities)  or  unfavourable  (threats). 

This is an evaluation tool which is used to check whether a public  intervention 
is suited  to  its context. The tool structures  debate  on strategic  orientations. 

Logical  f r a m e w o rk 

Cadre 

logique 

Tool  used to  structure the  logic  of  a  public  intervention. 

It  is based  on  a matrix  presentation  of the  intervention, which  highlights  its 
outputs,  results, and  specific  and global  impacts.  Each  level of  objective  is 
associated with one or more verifiable indicators  of success, as well as with 
influencing  success  or  failure  (confounding 
the  conditions  and  risks 
factors).  This  tool  is  frequently  used  for  the  evaluation  of  development 
support  projects.  By  contrast,  it  is  ill-suited  to  the  evaluation  of  socio­
economic  programmes  because  its  level of  simplification  is too  high. 

Objective  tree 

Arbre  des 

objectifs 

Hierarchical  classification  of  the  objectives  of  a  public  intervention,  linking 
each  specific  objective  to  its global  objective. 

The  objectives  tree  of  a  programme  makes  the  overall  logic  explicit.  It  is 
used for  clarification. 

Concept  mapping  of  impacts 
Cartographie 

conceptuelle 

des 

impacts 

Tool  used  for  the  clarification  of  explicit  and  implicit  objectives,  based  on 
the  identification, grouping  together  and grading  of  expected  impacts. 

Glossary of 300  concepts and technical  terms 

The  concept  mapping  of  impacts  is  implemented  in  a  pluralistic  and 
participatory  way, so that  a large  number  of  participants  can  be involved. It 
may  result  in  the  selection  of  indicators  that  are  associated  with  the  main 
expected  impacts. 

Colour  Vote 

Vote  coloré 

Technique  used to run meetings, based on a visual presentation  of  opinions 
in a group. 

In an  evaluation  situation  in which  several  stakeholders  are involved, it  can 
be  used  to  structure  discussion  and  collective  reflection  by  facilitating  the 
expression  of  all  points  of  view.  It  is  therefore  a tool  likely  to  enhance  the 
impartiality  of  the  evaluation  in the context  of  a pluralistic  approach.  It  may 
be  used  to  clarify  and  grade  evaluative  questions,  to  choose  evaluation 
criteria, to  validate  conclusions,  and  to  formulate  recommendations.  Some 
variations  of this tool  require  computer  support. 

Related  term(s) 

Abaque  de  Régnier 

Matrix  of  cross  impacts 

Matrice  des  impacts 

croisés 

Tool  used  to  highlight  potential  complementarity,  duplication  and  conflict 
between  the  impacts  of  the  different  components  of  an  intervention,  or 
between  the  impacts  of  different  interventions. 

For  example,  the  tool  presents  the  different  measures  of  a  programme  in 
lines  and  columns.  In  each  cell  a  score  between  -3  and  +3  is  given  to 
describe the  potential  synergy  between  a measure  situated  on a  line and a 
measure  situated  in a column. 

Information gathering techniques 

Primary data 

Données primaires 

Data collected directly in the field, by means of a survey carried out by the 
evaluation team on the groups concerned by the intervention. 

Primary data play an important role in the cognitive contribution of the 
evaluation. They are added to data already available at the start of the 
evaluation (e.g. former research and evaluations, monitoring data, statistics). 
Primary data are particularly useful for analysing impacts and needs in depth. 

S e c o n d a ry data 

Données secondaires 

Existing information, gathered and interpreted by the evaluation team. 

Secondary data consists of information drawn from the monitoring system, 
produced by statistics institutes and provided by former research and 
evaluations. 

Cross sectional data 

Données par coupe transversale 

Data collected in a comparative way, at a given time, on several categories 
of individuals or facts. 

For example: comparative unemployment rates in European regions; 
comparative placement rates for women and men. 

■ T i me series 

Série temporelle 

Data collected on the same population, in a comparative way, at regular 
intervals during a given period. 

Overall variations in the characteristics of a given population are observed 
over time. Statistics institutes and statistical teams are the main sources of 
time series. 

Glossary of 300  concepts and technical  terms 

Related  term(s): 

Série  chronologique 

Longitudinal  data 

Données 

longitudinales 

Time  series  relating  to  repeated  comparative  observations  of  the  same 
individuals, at  regular  intervals, during  a given  period. 

In  the  evaluation  context,  these  data  are  obtained  by  observing  the 
evolution  of  a sample  of  addressees. 

Related  term(s): 

Données  de  cohorte 

S a m p le 

Echantillon 

Part of  a group  selected for  a survey. 

In  its  statistical  sense,  the  word  "population"  applies  to  all  the  people, 
projects  or  objects  likely  to  be  observed,  e.g.:  all  the  assisted  firms  or 
individuals;  all  the  firms  in  a  region;  or  an  entire  targeted  group.  If  the 
sample  is representative, the survey  results can be generalised to the entire 
surveyed  population  (internal  validity)  or  to  similar  populations  (external 
validity).  A  sample  is  representative  if  it  has  the  same  distribution  of 
characteristics  as  the  population  from  which  it  is  taken,  and  if  it  is  large 
enough  (generally  at  least  a thousand  individuals). 

Questionnaire  survey 

Enquête  par 

questionnaire 

A  set  of  structured  questions  addressed  to  a  sample  of  people  or, 
exhaustively,  to  a group  of  people. 

A  questionnaire  consists  of  a  list  of  questions  of  which  the  sequence  is 
determined  in  advance.  Depending  on  whether  the  questions  are  open  or 
closed,  the  interviewee  is  either  free to  formulate  his  or  her answers  as  he 
or  she  wishes,  or  is  given  predefined  statements  (descriptors)  from  which 

Techniques 

to choose. A questionnaire  can be undertaken  by post, telephone, e-mail  or 
face-to-face  interview.  If  the  questionnaire  survey  is  exhaustive  or  if  it 
concerns  a representative  sample, and provided that  most  of the  questions 
are closed, it  may  be used for gathering  quantitative  data and for  informing 
indicators. 

Individual  interview 

Entretien 

individuel 

Technique  used  to  collect  qualitative  data  and the  opinions  of  people  who 
are concerned  or  potentially  concerned  by the  intervention,  its  context,  its 
implementation  and  its  effects. 

Several types of  individual  interview  exist, including  informal  conversations, 
semi-structured  interviews  and structured  interviews. The latter  is the  most 
rigid  approach  and  resembles  a  questionnaire  survey.  A  semi-structured 
interview  consists  of  eliciting  a  person's  reactions  to  predetermined 
elements, without  hindering  his or  her freedom to  interpret  and  reformulate 
these  elements. 

Focus  group 

Entretien  de  groupe 

Survey technique  based  on a small group  discussion. Often  used to  enable 
participants to form an opinion on a subject with which they  are not familiar. 

The  technique  makes  use  of  the  participants'  interaction  and  creativity  to 
enhance and consolidate the information collected. It is especially  useful for 
analysing  themes  or  domains  which  give  rise to  differences  of  opinion  that 
have to be reconciled, or which concern  complex  questions  that  have to  be 
explored  in depth. 

C a se  study 

Etude  de  cas 

In-depth  study  of  data  on a specific  case (e.g. a project,  addressee, town). 

The  case  study  is an  appropriate  tool  for  the  inductive  analysis  of  impacts 
and  particularly for innovative interventions for which there is no explanatory 
theory  of  impacts. A  case  study  is concluded  with  a  monograph  presented 

Glossary of 300  concepts  and technical  terms 

in a narrative form. A series of  case studies  can  be carried out  concurrently, 
in  a  cumulative  or  iterative  way.  The  latter  consists  of  conducting  several 
series of  case studies, carefully  selected  to  verify  an  assumption. 

Ethnographic  observation 

Observation 

ethnographique 

In  situ,  non-disruptive  observation  of  the  daily  activity  of  actors  and/or 
addressees  of  the  evaluated  intervention. 

tries  to  understand  the  situation  "from  the 

The  researcher 
inside". 
Ethnographic  observation  is useful in little known  situations or when  access 
to  the  field  is  difficult.  It  is  used  to  collect  very  detailed  information,  with 
audio-visual  recordings  where  relevant,  on  a  few  typical  situations  of 
implementation  of the intervention. It also serves to identify  all the effects  of 
the  intervention  and the  influence  of the  context. 

Analytical techniques 

C o m p a r i s on group 

Groupe de comparaison 

Group of people or organisations which has not participated in a public 
intervention and which is compared with a group of participants for the 
purpose of analysing the net effects of an intervention. 

If the two groups are randomly selected samples, the comparison group is 
called the control group because it is comparable, in every way, to the so­
called "experimental" group. Irrespective of the mode of selection of the 
comparison group, its characteristics must be as similar as possible to 
those of the direct addressees. For example, the evaluation must make a 
"before­after" comparison between a group of direct addressees and 
another group constituted by means of the matching pair technique. 

Related term(s): 

Analyse comparative 

■ Control group 

Groupe témoin ti 

Comparison group consisting of eligible people or organisations which have 
been excluded from all participation in the intervention by a process of 
random selection. 

Apart from its non­participation in the intervention, the control group is, 
from every point of view, comparable to the group of participants. It has 
also been exposed to the same variations in the socio­economic context 
(confounding factors). When a group of participants and a control group are 
compared, the influence of confounding factors is the same on both sides 
(provided the two groups are large enough). 

Related term(s): 

Affectation aléatoire 

Groupe de contrôle 

Glossary of 300 concepts and technical terms 

M a t c h i ng pair 

Binome apparié 

Technique for constructing a comparison group. 

For each individual in the sample of direct addressees being surveyed, an 
individual is chosen who resembles it as closely as possible and who has 
not participated in the intervention. 

Regression analysis 

Analyse de régression 

Statistical tool used to make a quantitative estimation of the influence of 
several explanatory variables (public intervention and confounding factors) 
on an explained variable (an impact). 

Regression analysis is a tool for analysing deductive causality. It is based 
on an explanatory logical model and on a series of preliminary 
observations. The tool can be used in varying ways, depending on whether 
the variables of the model are continuous or discrete and on whether their 
relations are linear or not. 

Covariation 

Covariance 

The fact that two variables systematically evolve in the same direction or in 
opposite directions. 

If there is little covariance, there is no statistically reliable relationship 
between them. A large degree of covariance between A and Β indicates an 
assumption of causality but does not prove it. (Is A the cause of B? or is Β 
the cause of A? or are A and Β the consequence of something else?). 

Related term(s): 

Corrélation 

Factor analysis 

Statistical analysis tool used to identify all correlation within a large quantity 
of data concerning many individuals (e.g. survey results), and to represent 
it in a simplified way in the form of a two- or three-dimensional space. 

Analyse factorielle 

Techniques 

The tool  reveals  groupings  and  suggests  relations  of  cause  and  effect.  It is 
an inductive  causality  analysis  tool. 

Shift-share  analysis 

Analyse 

structurelle-résiduelle 

Tool  for  evaluating  regional  policy,  which  estimates  the  counterfactual 
situation  by  projecting  national  economic  trends  onto  the  economy  of  a 
given region. 

The  basic  assumption  of  this  technique  is that,  in the  absence  of  regional 
policy,  the  evolution  of  economic  variables  in the  region  would  have  been 
similar  to  that  of  the  country  as  a whole.  Comparison  between  the  policy-
off  and  policy-on  situations  is concluded  with  an  estimation  of  the  macro-
economic  impact  of  regional  policy. The optimum  conditions  for  using  this 
tool  rarely  exist. 

Input-output  analysis 

Analyse Entrées-Sorties 

Tool  which  represents  the  interaction  between  sectors  of  a  national  or 
regional  economy  in the  form  of  intermediate  or final  consumption. 

Input-output  analysis serves to estimate the  repercussions  of a direct  effect 
in  the  form  of  first  round  and  then  secondary  effects  throughout  the 
economy.  The  tool  can  be  used  when  a  table  of  inputs  and  outputs  is 
available. This  is usually  the case at the  national  level but  more  rarely  so at 
the regional level. The tool is capable of estimating demand-side effects  but 
not  supply-side  effects. 

M a c ro  e c o n o m ic  model 

Modèle 

macro-économique 

Tool  used  to  simulate  the  main  mechanisms  of  a  regional,  national  or 
international  economic  system. 

A  large number  of  models  exist,  based  on  widely  diverse  macro-economic 
theories. This type of tool  is often  used to simulate future trends, but  it may 
also  serve  as  an  evaluation  tool.  In  this  case,  it  is  used  to  simulate  a 
counterfactual  situation,  and  thus  to  quantitatively  evaluate  net  effects  on 

Glossary of 300  concepts and technical  terms 

most  of  the  macro-economic  variables  influenced  by  public  actions,  i.e.: 
growth,  employment,  investment,  savings,  etc.  The  models  are  generally 
capable of estimating  demand-side effects far more easily than  supply-side 
effects. 

Delphi  panel 

Enquête  Delphi 

Procedure  for  iterative  and  anonymous  consultation  of  several  experts, 
aimed at  directing their  opinions  towards  a common  conclusion. 

The  Delphi  panel  technique  may  be  used  in  ex  ante  evaluation,  for 
estimating  the  potential  impacts  of  an intervention, for  example. 

Judgement techniques 

Expert  panel 

Panel 

d'experts 

Work  group  which  is  specially  formed  for  the  purposes  of  the  evaluation 
and  which  meets  several  times. 

The experts  are recognised  independent  specialists  in the evaluated field of 
intervention. They  collectively  pronounce  a judgement  on  the  value  of  the 
public  intervention  and  its  effects.  An  expert  panel  serves  to  rapidly  and 
inexpensively  formulate  a  synthetic  judgement  which  integrates  the  main 
information  available  on  the  programme,  as  well  as  information  from  other 
experiences. 

Related  term(s): 

Peer  review 

Evaluation  par  les pairs 

B e n c h m a r k i ng 

Etalonnage 

Qualitative  and  quantitative  comparison  of  the  performance  of  an 
intervention, with  that  which  is  reputed  to  be  the  best  in the  same  domain 
of  intervention  or  in a related  domain. 

Benchmarking  is facilitated  when, at the  national  or  regional  level, there are 
league tables  of  good  and  not  so  good  practice. 

Related  term(s): 

Benchmark 

Best  practice 

League  table 

Classement 

Etalon 

Meilleure  pratique 

Glossary of 300  concepts and technical  terms 

Multicriteria  analysis 

Analyse 

multicritère 

Tool  used to  compare  several  interventions  in relation  to  several criteria. 

Multicriteria  analysis  is  used  above  all  in  the  ex  ante  evaluation  of  major 
projects,  for  comparing  variants.  It  can  also  be  used  in  the  ex  post 
evaluation  of  an  intervention,  to  compare  the  relative  success  of  the 
different  components  of the  intervention. Finally, it can be used to  compare 
separate  but  similar  interventions,  for  classification  purposes.  Multicriteria 
analysis  may  involve weighting, reflecting the relative importance  attributed 
to  each of  the criteria.  It may  result  in the formulation  of a single  judgement 
or  synthetic  classification,  or  in  different  classifications  reflecting  the 
stakeholders'  different  points  of  view.  In  the  latter  case,  it  is  called 
multicriteria-multijudge  analysis. 

Related  term(s): 

Multicriteria-multijudge  analysis 

Analyse  multicritère-multijuge 

Cost-effectiveness  analysis 

Analyse coût-efficacité 

Evaluation  tool  for  making  a judgement  in terms  of  efficiency. 

This tool  consists  of  relating the  net effects  of  the  intervention  (which  must 
be  determined  separately)  to the  financial  inputs  needed  to  produce  those 
effects. The judgement  criterion  might,  for  example,  be the  cost  per  unit  of 
impact  produced (e.g. cost  per job created). This unit cost  is then compared 
to  that  of  other  interventions  chosen  as  benchmarks. 

Cost-benefit  analysis 

Analyse 

coût-avantages 

Evaluation tool for judging the advantages of the intervention from the  point 
of  view  of  all the  groups  concerned,  and  on the  basis  of  a monetary  value 
attributed  to  all the  positive  and  negative  consequences  of the  intervention 
(which  must  be estimated  separately). 

Techniques 

When  it  is  neither  relevant  nor  possible  to  use  market  prices  to  estimate  a 
gain  or  a  loss, a fictive  price  can  be  set  in various  ways. The first  consists 
of  estimating  the  willingness  of  addressees  to  pay  to  obtain  positive 
impacts  or  avoid  negative  impacts.  The  fictive  price  of  goods  or  services 
can  also  be  estimated  by  the  loss  of  earnings  in  the  absence  of  those 
goods  or services  (e.g. in cases  of  massive  unemployment, the fictive  price 
of  a  day's  unskilled  work  is  very  low).  Finally,  the  fictive  price  can  be 
decided on directly  by the administrative  officials  concerned  or the  steering 
group.  Cost-benefit  analysis  is  used  mainly  for  the  ex  ante  evaluation  of 
large  projects. 

VII 
Appendix 

Bibliography 

GAO  (1991),  Designing  Evaluations,  Washington  :  United  States  General 

Accounting  Office,  Program  Evaluation  and  Methodology  Division. 

Guba  E. et  Lincoln  Y. (1989), Fourth  Generation  Evaluation',  Newbury  Park: 

Sage  Publications. 

Hadji  C.  (1990), L'évaluation,  règles  du jeu,  Paris  :  ESF  Editeur  (collection 

Pédagogies). 

Monnier  E.  (1992),  Evaluation  de  l'action  des  pouvoirs  publics,  Paris  : 

Economica. 

OECD  (1986)  'Glossaire  sur  l'évaluation  des  politiques  internationales  de 
développement  de  l'OCDE';  in  :  Methods  and  Procedures  in  Aid 
Evaluation  :  A  Compendium  of  Donor  Practice  and  Experience, 
Paris: OECD. 

Patton  M.Q.  (1987),  Utilization  Focused  Evaluation',  Newbury  Park  : Sage 

Publications. 

Rossi  P.  H.  et  Freeman  H.E.  (1993),  Evaluation  -  A  Systematic  Approach, 

Newbury  Park  : Sage  Publications. 

Scriven  M.  (1991),  Evaluations  Thesaurus,  Newbury  Park  :  Sage 

Publications. 

Sohm  E.D.  (1978), Glossary  of  Evaluation  Terms, Genève  : UNO. 

Vedung  E. (1997), Public  Policy  and  Program  Evaluation,  New  Brunswick  : 

Transaction  Publishers.Index  Alphabétique  Français 

French Alphabetical Index 

French alphabetical  index 

Abaque  de  Régnier  Voir  Vote  coloré 
Action  Voir  Mesure 
Activité  Voir  Ressource 
Additionnante  30 
Adéquation  Voir  Pertinence 
Affectation  aléatoire 

Voir  Groupe 
Analyse  comparative 

témoin 

Voir  Groupe  de  comparaison 

Analyse  coût-avantages  112 
Analyse  coût-efficacité  112 
Analyse  de  causalité  82 
Analyse  de  régression  108 
Analyse  deductive 

Voir Analyse  de  causalité 
Analyse  Entrées-Sorties  109 
Analyse  factorielle  108 
Analyse  inductive 

Voir Analyse  de  causalité 

Analyse  multicritère  112 
Analyse  multicritère-multijuge 

Voir Analyse  multicritère 

Analyse  structurelle-résiduelle  109 
Apport  conceptuel 

Voir Finalité  cognitive 
Appréciation  Voir  Evaluation 
Appréciation  ex  ante 

Voir Evaluation  ex  ante 

Apprentissage  en  double  boucle 

Voir Apprentissage 

organisationnel 

Apprentissage  en  simple  boucle 

Voir Apprentissage 

organisationnel 

Apprentissage  organisationnel  22 
Arbre  des  objectifs  101 
Atténuation  93 
Audit  Voir  Contrôle 
Audit  de  performance  Voir  Suivi 
Auto-évaluation  41 
Axe  Voir  Programme 

B 

Bénéficiaire  Voir  Destinataire  direct 

Bénéficiaire  final  Voir  Opérateur 

Besoin  53 

Bien-fondé  Voir Raison  d'être 

Binôme  apparié  108 

But  Voir  Objectif 

Cadre  logique  101 
Cahier  des  charges  46 
Cartographie  conceptuelle 

des  impacts  101 
Champ  d'évaluation  33 
Champ  d'intervention  71 
Chargé  d'évaluation 

Voir Equipe  d'évaluation 
Classement  Voir  Etalonnage 
Coefficient  d'extrapolation  83 
Co-facteur  Voir Facteur  concurrent 
Cohérence  externe  89 
Cohérence  interne  89 
Cohésion  économique  et  sociale  93 
Commanditaire  41 
Comparabillté  70 
Compétitivité  94 
Complémentarité  89 
Condition-cadre  Voir  Contexte 
Contexte  53 
Contrôle  29 

Contrôle  de  gestion  Voir  Suivi 
Corrélation  Voir  Covariance 
Coût  unitaire  Voir  Efficience 
Coût-efficacité  Voir  Efficience 
Covariance  108 
Crédibilité  47 
Critère  61 
Cycle  de  programmation 

Voir  Programme 

Glossary of 300  concepts and technical  terms 

Décideur  Voir  Financeur 
Descripteur  68 
Destinataire  direct  39 
Destinataire  indirect  40 
Développement  durable  93 
Développement  endogène  93 
Développement  viable 

Voir Développement 

durable 

Diagnostic  Voir Evaluation  ex  ante 
Disparité  63 
Dispositif  d'évaluation  43 
Diversification  94 
Donnée  qualitative  Voir  Descripteur 
Donnée  quantitative  Voir  Indicateur 
Données  de  cohorte 

Voir Données 

longitudinales 

Données  longitudinales  104 
Données  par  coupe  transversale  103 
Données  primaires  103 
Données  secondaires  103 
Durabilità  64 

indirect 

inattendu 

Effet  fournisseur  87 
Effet  inattendu  58 
Effet  indirect  85 
Effet  induit  Voir Effet 
Effet  multiplicateur  de  revenu  87 
Effet  net  79 
Effet  pervers  Voir Effet 
Effet  primaire  85 
Effet  propre  Voir Effet  net 
Effet  secondaire  86 
Effet  structurant  88 
Effet  structurel  Voir Effet  structurant 
Efficacité  63 
Efficience  64 
Egalité  des  chances  94 
Eligibilité  63 
Emploi  créé  ou  maintenu  91 
Emploi  permanent  92 
Employabilité  91 
Enoncé  Voir  Descripteur 
Enquête  Delphi  110 
Enquête  par  questionnaire  104 
Entretien  de  groupe  105 
Entretien  individuel  105 
Environnement  socio-économique 

Echantillon  104 
Effectivité  Voir  Efficacité 
Effet  56 
Effet  additionnel  Voir Effet  net 
Effet  brut  79 
Effet  d'aubaine  79 
Effet  de  demande  87 
Effet  de  déplacement  80 
Effet  de  levier  86 
Effet  de  réseau  88 
Effet  de  substitution  80 
Effet  d'écrémage  92 
Effet  d'emploi  91 
Effet  d'entraînement  88 
Effet  direct  85 
Effet  d'offre  87 

Voir  Contexte 

Equipe  d'évaluation  42 
Equivalent  temps  plein  92 
Etalon  Voir  Etalonnage 
Etalonnage  111 
Etude  d'évaluabllité  46 
Etude  de  cas  105 
Evaluanda  Voir  Champ 
Evaluateur  Voir Equipe  d'évaluation 
Evaluation  17 

Evaluation  à  mi-parcours  27 
Evaluation  a  posteriori 

Voir Evaluation  ex  post 

Evaluation  a  priori 

Voir Evaluation  ex  ante 

Evaluation  affranchie  des  objectifs 

Voir  Utilité 

Evaluation  approfondie  25 

French alphabetical  index 

Evaluation  chemin-faisant  27 
Evaluation  démocratique  20 
Evaluation  d'ensemble  25 
Evaluation  d'Impact  sur 
l'Environnement  95 
Evaluation  endoformative  17 
Evaluation  ex  ante  26 
Evaluation  ex  post  28 
Evaluation  externe  41 
Evaluation  formative 

Voir Evaluation 

endoformative 

Evaluation  in  itinere 

Voir Evaluation 

chemin-faisant 

Evaluation  intermédiaire 

Voir Evaluation  à  mi-parcours 

Evaluation  interne  41 
Evaluation  manageriale  18 
Evaluation  par  les  pairs 
Voir Panel  d'experts 

Evaluation  partenariale 

Voir  Evaluation  pluraliste 

Evaluation  participative 

Voir Evaluation  pluraliste 

Evaluation  pluraliste  19 
Evaluation  récapitulative  18 
Evaluation  sommative 

Voir Evaluation 

récapitulative 

Evaluation  thématique  25 
Examen  Voir  Evaluation 
Externante  86 
Extrant  Voir  Réalisation 

Facteur  concurrent  81 
Facteur  contextuel 

Voir Facteur  concurrent 

Facteur  exogène 

Voir Facteur  concurrent 
Feed-back  Voir  Rétroaction 
Fiabilité  47 
Finalité  Voir  Objectif 

Finalité  instrumentale  21 
Finalité  normative  21 
Financeur  37 
Focalisation 

Voir Evaluation 

approfondie 

Forces, faiblesses,  opportunités, 

menaces  101 

Gagnants  et  perdants 

Voir  Public  concerné 

Généralisabilité  Voir  Validité  externe 
Gestionnaire  38 
Grille  d'observation  Voir  Descripteur 
Groupe  de  comparaison  107 
Groupe  de  contrôle 

Voir  Groupe 

témoin 
Groupe  de  pilotage  42 
Groupe  témoin  107 

H 

Harmonisation  Voir  Comparabilité 

I 

Impact  58 
Impact  de  premier  ordre 

Voir Effet  primaire 
Impact  global  Voir  Impact 
Impact  intermédiaire  Voir  Impact 
Impact  spécifique  Voir  Impact 
Impartialité  47 
Implementation  Voir Mise  en  oeuvre 
Imputabilité  Voir Rendre  des  comptes 
Indicateur  67 
Indicateur  clé  74 
Indicateur  de  contexte  73 
Indicateur  de  programme  73 
Indicateur  générique  74 

Glossary of 300  concepts and technical  terms 

Indicateur  standard 

Objectif  quantitatif 

Voir  Comparabilité 
Inégalité  Voir  Disparité 
Instance  d'évaluation  43 
Intégration  95 
Intégration  stratégique  Voir  Intégration 
Intervention  35 
Intrant  Voir  Ressource 

Voir  Objectif  verifiable 
Objectif  spécifique  Voir  Objectif 
Objectif  verifiable  55 
Objectivité  Voir  Fiabilité 
Observation  ethnographique  106 
Observatoire  70 
Opérateur  38 
Outil  99 

M 

Maître  d'oeuvre  Voir  Gestionnaire 
Maître  d'ouvrage  Voir  Financeur 
Mandant  Voir  Commanditaire 
Mandat  45 
Matrice  dimpacts  croisés  102 
Meilleure  pratique  Voir  Etalonnage 
Mesure  34,  67 
Méta-évaluation  28 
Méthodes  99 
Méthodologie  99 
Mise  en  oeuvre  55 
Modèle  de  causalité 

Voir  Théorie  d'action 

Modèle-macroéconomique  109 
Monitorage  Voir  Suivi 
Moyen  Voir  Ressource 

N 

Neutralité  Voir 
Norme  61 
Notation  69 

Impartialité 

Objectif  54 
Objectif  global  Voir  Objectif 
Objectif  implicite  Voir  Objectif 
Objectif  intermédiaire  Voir  Objectif 
Objectif  opérationnel  Voir  Objectif 

Panel  d'experts  111 
Partenaire  Voir  Protagoniste 
Partie  prenante  Voir  Protagoniste 
Pérennité  Voir  Durabilité 
Performance  65 
Perte  sèche  Voir Effet  d'aubaine 
Pertinence  62 
Pilotage  Voir  Suivi 
Poids  mort  Voir  Effet  d'aubaine 
Point  zéro  Voir Situation  de  base 
Politique  33 
Pondération  62 
Préévaluation  45 
Priorité  politique  55 
Produit  Voir  Réalisation 
Programme  33 
Projet  34 
Protagoniste  37 
Protocole  d'évaluation  99 
Public  cible  Voir Public  eligible 
Public  concerné  40 
Public  eligible  40 

Quantification  Voir  Unité  de  mesure 

Radioscopie 

Voir Evaluation  d'ensemble 

French alphabetical index 

Raison d'être  62 
Réalisation  57 
Réalisation financière  Voir Ressource 
Redevabillté Voir Rendre des comptes 
Referentiel  Voir Théorie d'action 
Relevance  62 
Rendre des comptes  22 
Rendu-compte 

Voir Rendre des comptes 

Représentativité  Voir Validité externe 
Ressource  56 
Résultat  57 
Résultat durable  Voir Impact 
Retombée  Voir Effet inattendu 
Rétroaction  21 
Réussite  Voir Efficacité 
Robustesse  Voir Fiabilité 

Scénario  hors  intervention 

Voir Situation contrefactuelle 

Secteur  d'intervention 

Voir Champ d'intervention 

Sensibilité  Voir Indicateur 
Série chronologique 

Voir Série temporelle 

Série temporelle  103 
Situation contrefactuelle  81 
Situation de base  73 
Situation de référence 

Voir Situation de base 

Stratégie  53 
Subsidiante  66 
Succès  Voir Efficacité 
Suivi  29 
SWOT  101 
Synergie  89 
Système d'évaluation  43 
Système  d'indicateurs 

Voir Système d'information de suivi 

Système dinformation  de suivi  69 

Tableau de bord  69 
Taux de réalisation  74 
Taux  d'absorption 

Voir Taux de consommation 
budgétaire 
Taux  d'avancement 

Voir Taux de réalisation 

Taux de couverture  75 
Taux  d'exposition 

Voir Taux d'atteinte du public 
eligible 

Taux de consommation budgétaire  74 
Taux de couverture 

Voir Taux d'atteinte du public 
eligible 

Taux de placement  91 
Taux de  placement 

Voir Taux de transition 

Technique  Voir Outil 
Théorie d'action  82 
Théorie explicative  83 

U 

Usager  Voir Destinataire direct 
Utilisation cognitive  20 
Utilité  65 

Valeur ajoutée  communautaire 

Voir Subsidiarité 
Valeur sociale  Voir Critère 
Validité de construction  48 
Validité externe  49 
Validité interne  48 
Viabilité  Voir Durabilité 
Vote coloré  102 

English Alphabetical  Index 

English alphabetical  index 

Absorption  rate  74 
Accountability  22 
Action  See  Measure 
Activity  See  Input 
Additional  Effect  See  Net  effect 
Additionality  30 
Aim  See  Objective 
Appraisal  See  Evaluation 
Appropriateness  See  Relevance 
Assessment  See  Evaluation 
Audit  29 
Axis  See  Programme 

Β 

Baseline  73 
Benchmarking  111 

Case  study  105 
Causality  analysis  82 
Cognitive  aim  20 
Colour  Vote  102 
Commissionner  41 
Community  value  added 
See  Subsidiarity 

Comparability  70 
Comparison  group  107 
Competitiveness  94 
Complementarity  89 
Completion  rate  74 
Concept  mapping  of  impacts  101 
Conceptual  See  Cognitive  aim 
Concerned  group  40 
Concurrent  factor 

factor 

See  Confounding 
Confounding  factor  81 
Consistency  See  Internal  coherence 
Construct  validity  48 

Context  53 
Context  indicator  73 
Control  See  Audit 
Control  group  107 
Cost  effectiveness  See  Efficiency 
Cost-effectiveness  analysis  112 
Counterfactual  situation  81 
Covariation  108 
Coverage  rate  75 
Credibility  47 
Criterion  61 
Cross  sectional  data  103 

D 

Deadweight  79 
Decision-maker 

See  Funding  authority 

Deductive  analysis 

See  Causality  analysis 

Delphi  panel  110 
Demand-side  effect  87 
Democratic  evaluation  20 
Descriptor  68 
Direct  addressee  39 
Direct  effect  85 
Disparity  63 
Displacement  effect  80 
Diversification  94 
Double  loop  learning 

See  Organisational 

learning 

93 

Economic  and  social  cohesion 
Effect  56 
Effectiveness  63 
Efficacy  See  Effectiveness 
Efficiency  64 
Eligibility  63 
Eligible  public  40 
Employability  91 

Glossary of 300  concepts and technical  terms 

Employment  effect  91 
Endogenous  development  93 
End-user  See  Direct  addressee 
Environmental  Impact  Assessment  95 
Equal  opportunities  94 
Ethnographic  observation  106 
Evaluability  assessment  46 
Evaluand  See  Scope 
Evaluation  17 
Evaluation  Committee  43 
Evaluation  design  99 
Evaluation  setting  43 
Evaluation  system  43 
Evaluation  team  42 
Evaluative  question  46 
Evaluator  See  Evaluation 
Ex  ante  evaluation  26 
Ex  post  evaluation  28 
Exogenous  factor 

team 

See  Confounding 

factor 

Expert  panel  111 
Explanatory  theory  83 
Exposure  rate See  Coverage  rate 
External  coherence  89 
External  evaluation  41 
External  validity  49 
Externality  86 
Extraneous  factor 

See  Confounding 

factor 
Extrapolation  coefficient  83 

Factor  analysis  108 
Feedback  21 
Field  of  intervention  71 
Final  beneficiary  See  Operator 
Financial  realisation  See  Input 
First  round  effect  85 
Focus  group  105 
Formative  evaluation  17 
Framework  condition  See  Context 
Full-Time  Equivalent  92 

Funding  authority  37 

Generic  indicator  74 
Global  evaluation 

See  Overall  evaluation 
Global  Impact  See  Impact 
Global  objective  See  Objective 
Goal  See  Objective 
Goal  free  evaluation  See  Utility 
Gross  effect  79 

H 

Harmonisation  See  Comparability 

I 

Immediate  outcome  See  Result 
Impact  58 
Impartiality  47 
Implementation  55 
Implicit  objective  See  Objective 
Impulsion  effect  88 
Income  multiplier  effect  87 
In-depth  evaluation  25 
Indicator  67 
Indirect  addressee  40 
Indirect  effect  85 
Individual  interview  105 
Inductive  analysis 

See  Causality  analysis 

Inequity  See  Disparity 
In-house  evaluation 

See  Internal  evaluation 

Input  56 

Input-output  analysis  109 
Instrumental  aim  21 
Intermediate  impact  See 
Intermediate  objective  See  Objective 
Internal  coherence  89 

Impact 

English alphabetical  index 

Internal  evaluation  41 
Internal  validity  48 
intervention  35 

Monitoring  information  system  69 
Multicriteria  analysis  112 
Multicriteria-multijudge  analysis  See 

Multicriteria  analysis 

Job  created  or  maintained  91 

N 

Key  indicator  74 
Knock-on  effect  See First  round  effect 

Lasting  effect  See  Structuring  effect 
League  table  See  Benchmarking 
Learning  22 
Leverage  effect  86 
Logical  framework  101 
Longitudinal  data  104 

M 

Macro-economic  model  109 
Mainstreaming  95 
Management  audit  See  Monitoring 

Manager  38 
Managerial  evaluation  18 
Mandate  45 
Matching  pair  108 
Matrix  of  cross  impacts  102 
Means  See  Input 

Measure  34 
Measurement  unit  67 
Meta-evaluation  28 
Method  99 
Methodology  99 
Mid-term  evaluation  27 
Mitigation  93 
Monitoring  29 

Need  53 
Needs  assessment 

See  Ex-ante  Evaluation 

Net  effect  79 
Network  effect  88 
Neutrality  See 
Norm  61 
Normative  aim  21 

Impartiality 

0 

Objective  54 
Objective  tree  101 
Objectivity  See  Reliability 
Observation  grid  See  Descriptor 
On-going  evaluation  27 
Operating  report  69 
Operational  objective  See  Objective 
Operator  38 
Outcome  See  Impact 
Output  57 
Overall  evaluation  25 

Partner  See  Stakeholder 
Partnership  evaluation 

See  Pluralistic  evaluation 
Peer  review  See  Expert  panel 
Performance  65 
Performance  audit  See  Monitoring 
Permanent  job  92 
Pertinence  62 
Placement  rate  See  Transition  rate 
Pluralistic  evaluation  19 

Glossary of 300  concepts and technical  terms 

Policy  33 
Policy  priority  55 
Policy-off  situation 

See  Counterfactual 

situation 

Pre-evaluation  45 
Primary  data  103 
Prior  appraisal  See Ex-ante  evaluation 
Product  See  Output 
Programme  33 
Programme  cycle  See  Programme 
Programme  indicator  73 
Programme  theory 

See  Theory  of  action 

Project  34 
Project  promoter  39 

Qualitative  data  See  Descriptor 
Quantification  See  Measurement  unit 
Quantitative  data  See 
Quantitative  objective 

Indicator 

See  Verifiable  objective 

Questionnaire  survey  104 

Raison  d'etre  See  Rationale 
Rating  See  Scoring 
Rationale  62 
Realisation  See  Output 
Recipient  See  Direct  addressee 
Regression  analysis  108 
Reliability  47 
Representativeness  See  Reliability 
Resource  See  Input 
Result  57 
Rolling  evaluation 

See  On-going  evaluation 

Scope  of  evaluation  33 
Scoping  See  In-depth  evaluation 
Scoring  69 
Screening  See  Overall  evaluation 
Secondary  data  103 
Secondary  effect  86 
Self-evaluation  41 
Sensitivity  See 
Indicator 
Shift-share  analysis  109 
Side  effect  See  Unexpected  effect 
Single  loop  learning 

learning 

See  Organisational 
Skimming-off  effect  92 
Social  cost  benefit  analysis  112 
Social  value  See  Criterion 
Soundness  See  Reliability 
Specific  impact  See  Impact 
Specific  objective  See  Objective 
Stakeholder  37 
Stakeholder  evaluation 

See  Pluralistic  evaluation 

Standard  See  Norm 
Standard  indicator  See  Comparability 
Stastitical  team  70 
Statement  See  Descriptor 
Steering  group  42 
Strategy  53 
Structuring  effect  88 
Subsidiarity  66 
Substitution  effect  80 
Success  See  Effectiveness 
Summative  evaluation  18 
Supplier  effect  87 
Supply-side  effect  87 
Sustainability  64 
Sustainable  development  93 
Sustainable  result  See  Impact 
SWOT  101 
Synergy  89 
System  of  indicators 

Sample  104 

See Monitoring  information  system 

English alphabetical index 

Target group See Eligible public 
Technique See  Tool 
Terms of  reference  46 
Thematic  evaluation  25 
Theory of action  82 
Time series  103 
Tool  99 
Transition rate  91 
Trustworthiness See Credibility 

u 
Unexpected  effect  58 
Unit-cost  See Efficiency 
Utility  65 

Value for money See Efficiency 
Verifiable objective  55 

w 
Weighting  62 
Winners  and losers 

See Reached group 

Acknowledgements 

The  preparation  of  this  Volume  took  place  over  several  years  from  1993  to  1998. 
Preparatory  works  included  the following  steps  : 

An  initial  version  of  the  glossary  in five  languages  was  produced  in  1993  in the  form 
an internal  document  for  the  European  Commission. 

A second version was  produced in 1995 which included nine languages and took  into 
account  the comments  of  an international  panel  of  experts. 

This  Volume  is  the  direct  result  of  the  work  undertaken  to  ensure  that  the  whole  of 
the  MEANS  Collection  is based  on a coherent  vocabulary. 

The production  of this Volume was coordinated  by  Miguel  Benito Alonso and Andrea 
Mairate (D.G.  XVI.G.2  : Coordination of  Evaluation) and involved several  Directorate-
Generals  of  the  European  Commission,  notably  D.G.  II, V, VI,  XIV  and  XVI  and  in 
particular  Philippe Goybet,  Anastassios  Bougas  and  David  Sweet. 

This  Volume  has  been  written  by  the  Centre  for  European  Evaluation  Expertise, 
notably  with  the  participation  of  Kristian  Colletis-Wahl.  It was  translated  into  English 
by  Liz  Libbrecht. 

The  authors  wish  to  thank  all  those  who  contributed,  either  directly  or  indirectly,  to 
the  production  of  this  Volume,  and  in  particular  the  following  experts:  Dr  Ahonen 
(University  of  Tampere),  X.  Ballart  (Universitad  Autonoma  de  Barcelona),  G.Becher 
(Prognos  A.G),  M.L.  Bemelmans-Videc  (University  of  Nijmegen),  M.  Meneguzzo 
(Universita  Bocconi),  Ch.  Pollitt  (Brunei  University),  Thanos  Skouras  (Athens 
University  of  Economics  and  Business),  Markus  Spinatsch  (Switzerland)  ), E. Vedung 
(Uppsala  University). 

European  Commission 

Evaluating  socio-economic  programmes 
Glossary  of  300  concepts  and technical  terms 

Luxembourg: Office for  Official  Publications  of the  European  Communities,  1999 

1999 —  135  pp. —  14.8  χ 21  cm 

ISBN  92-828-6626-2 

Venta · Salg ■ Verkauf ·  Πωλήσεις  ·  Sales  ·  Vente  ·  Vendita  ·  Verkoop  ·  Venda  ·  Myynti  · Försäljning 

8ELG1QUE/BELGIE 

ÖSTERREICH 

Jean De Lannoy 
Avenue du Ro· 202/Komngsiaan 202 
B­1190 Bruielles. Brussel 
Tel (32­2) 538 43 OS 
Fax (32­2) 538 08 41 
E­mail |oan de lannoyOintoboard be 
URL: http //www« je an­de­lan no y be 

LB librairie européenne; 
De E u r o p e ·· Boekhandel 
Rue do la Lo· 244/Wetstraat 244 
B­1040 Bmxe«es,_russei 
Tel (32­2)295 26 39 
Fax (32­2) 735 08 60 
E­mail mailOhbeurop be 
URL hrtpy/www iibourop be 

Moniteur belge/Belgisc h Staatsblad 
Roe de Louvam 40­4_leuvenseweg 40­42 
B­1000 BruxeUes/Bmssei 
Tel (32­2)552 22 11 
Fax (32­2) S1101 84 

DANMARK 

J. H. Schultz Information A/S 
Herstedvang 10­12 
DK­2620 Albertslund 
TH (45,43 63 23 00 
Fax(45)43 63 19 69 
E­mail Schultz O Schultz dk 
URL http //www schuiu.dk 

DEUTSCHLAND 

Β unde aanzei ger Verlag GmbH 
V ert »eOsabt _ lung 
Amsterdamer Strafle 192 
D­5073S Koir­
Tci (49­221)97 6 6 80 
Fax (49­221) 97 66 B2 78 
E­Mail vertrieb O b undesanzeiger de 
URL hrp .'.'www bundesanzeiger de 

EAAA_A/GREECE 

Q. C. Eleftheroudakis SA 
Intemationat Bookstore 
Paneptsnmiou 17 
GR­10564 Athma 
Tel (30­1)331 41  8 0 Ί / _ _ 4 /5 
Fax  (30-1)323 98  21 
Email  etebooksOnetor  gr 

ESPANA 

Bolelin  Oficial  del  Estado 
Trafalgar.  27 
E  28071  Madrid 
Tel  (34)915  36 21  11 (Libros). 
913 B4  17  15(Suscnp) 
Fax (34) 915  38 21 21  (Libros). 
913 64  17  14<SuSCnp) 

Ε-max  ctientesOcom  boe  es 
URL. http //www  boe  es 
Mundi  Prensa  Libros.  SA 
Caste*d.  37 
E-26001  Madnd 
Tel  (34)914  36 37  00 
Fax (34) 915  75 39 98 
E-mail  librerlaOmundiprensa  os 
URL  hnpy/wwwrnundprensacorn 

FRANCE 

Journal  officiel 
Service des publications  dos  CE 
26.  rue  Dosai« 
F-757_7 Pans Cedex  15 
Τ«  (33)  140 58  77  31 
Fax  (33)  140 56  77 00 
URL  http/.'www ¡ournal­otriciel gouv tr 

IRELAND 

Govern m· ni Supplies Agency 
PuOkcaiions Section 
4­5 Harcourt Road 
Dublin 2 
Tel (353­1)661 31 11 
Fax (353­1) 475 27 60 

ITALIA 

U c o s a S pA 
Via Duca di Calabria, i/l 
Casana postale 552 
1­50125 Firenze 
Tel (39) 055 64 83 1 
Fax (39) 055 64 12 57 
Ε­mai kcosaOtibcc ri 
URL hnpj/www ftbcc it/licosa 

LUXEMBOURG 

Messageries du livra SARL 
5, rue Raiffeisen 
L 2411 Luxembourg 
Tel (352) 40 10 20 
Fax (352) 49 06 61 
E mail mailOmd) lu 
URL http//www m_ lu 

NEDERLAND 

SOU Servicecentrum Uitgevers 
C n π stoffel Planlijnstraal 2 
Postbus 20014 
2500 EA Den Haag 
Tel (31­70) 376 96 60 
Fax (31­70) 376 97 63 
E­mail sduOsdu nl 
URL: http //www sdu nl 

Manz sch« Verlags­ und 
Universitet sbuchfi i nd lung GmbH 
Kohlmarkl 16 
A­1014 Wien 
Tet. (43­1)53 1611 00 
Fax (43­1) 53 16 11 67 
E­Mail bestellen O manz co ai 
URL' http //www manz at/Index htm 

PORTUGAL 

Distribuidora de Livros Bertrand L d .' 
Grupo Bertrand, SA 
Rua das Tenas dos Vales. 4­A 
Apartado 60037 
Ρ­2700 Amadora 
Tel (351­1)495 90 50 
Fax (351­1)496 02 55 

Imprensa Nacional­Casa da Moeda, EP 
Rua Marquas Sá da Bandeira. 16­A 
Ρ­1050 Lisboa Codex 
Tet. (351­1)353 03 99 
Fax (351­1)353 02 94 
E­mail, del incmOmaJ telepac pt 
URL: htlp7/www inem pt 

SUOMIFINLAND 

Akateeminen Kirjakauppa/ 
Akademiska Bokhandeln 
Keskuskatu i Ce ni raígala n ι 
PtVPB 128 
FlN­00101 Helsinki Helsingfors 
Ρ Λη (358­9) 121 44 1Θ 
F ./fax (358­9) 121 44 35 
Sähköposti akatilausOakaleeminencom 
URL: http ■.www akateeminen corn 

SVERIGE 

B T J AB 
Traktorvägen 11 
S­221 82 Lund 
Ttn (46­46) 16 00 00 
Fax (46­46) 30 79 47 
E­posi btfeu­pubObt] se 
URL: httpJ/wwwbij se 

UNITED KINGDOM 

The Stationery Office Ltd 
International Sales Agency 
51 Nine Elms Lane 
London SW8 SDR 
Tel (44­171)873 90 90 
Fax (44­171)873 94 63 
E­mail: tpa.enquinesetttesQ.couk 
URL: http //www. the­S lal ione ry­off ice co uk 

ISLAND 

Bokabud Lar u ur Blondel 
S* Ol av or Ou s! ig 2 
IS­101 Reykjavik 
Tel (354)551 56 50 
Fax (354) 552 55 60 

NORGE 

S wets Norge AS 
Ostenioveien 1Θ 
Boks 6512 Etterstad 
N­O606OS1O 
Tel (47­22)97 45 00 
Fax (47­22) 97 45 45 

: ; o i W L : _ ! ; t j : : , Si SVIZZERA 

Euro Info Center Schwell 
co OSEC 
Stampfenbach Straße 85 
PF 492 
CH­8035 Zürich 
Tel (41­1)365 53 15 
Fax (41­1) 365 54 11 
E­mail eicsOoseccn 
URL http .'.Www osoc ctvoics 

BÁLGAHUA 

Europress Euromedia Ltd 
59. btvd Vilosha 
BG­1000 Sotia 
Tel (359­2) 980 37 66 
Fax (359­2) 980 42 30 
E­mail MilenaOmbox eil bg 

C E SKA RE PUBLIKA 

ÚSIS 
NlS­prodejna 
Havelkova 22 
C Z ­ l 3 O 0 0 P r a h a3 
Tel (420­2)24 23 14 86 
Fax (420­2) 24 23 11 14 
Email nkpospOdec ms « 
URL: riMp usiscr cz 

CYPRUS 

HRVATSKA 

Medialraoe Ltd 
Pavia Haua ι 
HR­10000 Zagreb 
Tel. (385­1)491 94 11 
Fax (385­1)481 94 11 

MAGYARORSZAG 

Euro Info Service 
Europa Haz 
Margitszigel 
PO Box 475 
H­1396 Budapest 62 
Tel (36­1)350 80 25 
Fax (36-1)350 90 32 
E­mail euiomloOmail mala v hu 
URL hrtp//wwweuroinfo.hu/inaex.htm 

MALTA 

Miller Distributors Ltd 
Malta International Airport 
PO Box 25 
Luqa LQA 05 
Tel (356) 66 44 68 
F a x ( 3 5 6 ) 67 67 99 
E­mail, gwirth O usa net 

POLSKA 

Ars Polona 
Krakowskie Przedmiescie 7 
Skr pocztowa 1001 
PL­00­950 Warszawa 
Tel. (48­22)826 12 01 
Fat (48­22) 826 62 40 
E­mail: ars po' O bevy hsn com pi 

ROMANIA 

E u r o m e d ia 
Sir G­ralBerthek)lNr4l 
RO­70749 Bucuresti 
Tel (40­1)31544 03 
Fax (40­1) 314 22 86 

ROSSIYA 

CCEC 
60­tetiya Ofctyabrya Av 9 
117312 Moscow 
Tel ( 7 - 0 9 5 ) 1 35 52 27 
Fax (7095) 135 52 27 

SLOVAKIA 

Centrum VTI SR 
Nam Slcbody 19 
SK­81223 Bratislava 
Tel (421­7)54 41 B3 64 
Fax (421­7) 54 41 B3 64 
E­mail europOtbbl sltk stuba sk 
URL http://wwwsrtk._uba sk 

SLOVENUA 

GospodarsM Veslnlk 
Dunaiska cesta 5 
SLO­1000 Ljublana 
Tet ( 3 8 6 ) 6 1 3 0 9 1 6 40 
Fax ( 3 8 6 ) 6 1 3 0 9 1 6 45 
E­mail europOgveslrwksi 
U RL http://www gvestmk si 

TURKIYE 

Du nya Infoisi AS 
100. Yil Mahallessi 34440 
TR­80050 Bagolar ­Istanbul 
Tel (90-212)629 46 89 
Fax (90-212) 629 46 27 
E­mail: inlotetOdunya­gazetecom Ir 

AUSTRALIA 

Hunter Publications 
PO Box 404 
3067 Abboisford. Victoria 
Tel (61­3)94 1 7 5 3 61 
Fax (61­3) 94 19 71 54 
E­mail (pdaviesOozemail com au 

CANADA 

Les éditions La Libert* Inc. 
3020. chemin Sainte­Foy 
G1X 3V Sainle­Foy. Ouébec 
Tel (1­418)658 37 63 
Fax (1­800) 567 54 49 
E­mail liberteOmedKVTi qcca 
Renouf Publishing Co. Ltd 
5369 Chemin Canotek Road Unit 1 
KU 9J3 Ottawa. Ontano 
Tel (1­613)745 26 65 
Fax(1­613) 745 76 60 
Email Order deptOrenoufbOOkS com 
URL. httpJ/www renoufbookj.com 

EGYPT 

Cyprus Chamber of Commerce end Industry 
PO Bo« 1455 
CY­1509 Nicosia 
Tel ( 3 5 7 2 ) 66 95 00 
Fax (357­2) 66 10 44 
E­mail demetrapOccci orgcy 

Tel (20­2)392 69 19 
Fax (20­2) 393 97 32 
E­mail matoudaOmeobserver.com ι 
URL httpy/www meobserver.com eg 

EESTI 

INDIA 

Eesti Ksubandus­Toosluskoda (Estonian 
Chamber ol Commerce and Industry) 
Toom­Kook 17 
E E­0001 TaH.nn 
Tel. (372)646 02 44 
Fax (372) 646 02 45 
E­mail emtoOkoda.ee 
URL: http //www koda ee 

EBIC India 
3rd Floor. Y B Chavan Centre 
Gen J. Bhosale Marg 
400 021 Mumbai 
Tet (91­22)282 60 64 
Fax (91­22) 2B5 45 64 
E­mail: ebicOgiasbm01.vsnl.net.in 
URL: http.//www ebcmdiacom 

ROY International 

41. Misturar Ha>arden Street 
PO Box 13056 
61130 Tet Aviv 
Tel (972-3)649 94 69 
Fax (972­3) 648 60 39 
E­mail royilOnetvision.net.il 
URL http '/www royint co il 

Sub­agoni lor the Palestinian Authority: 

Index Information Services 

PO Box 19502 
Jerusalem 
Tel (972­2)627 16 34 
Fax (972­2) 627 12 19 

JAPAN 

PSI­Japan 

Asahi Sanbancho Plaza «206 
7­1 Sanbancho. Chiyoda­ku 
Tckyo 102 
Tel (81­3)32 34 69 21 
Fax (81­3) 32 34 69 15 
E­mail: booksOpsi­japancojp 
URL http_'n*ww psi­japan com 

EBIC Malaysia 

Level 7. Wisma Hong Leong 
18 Jalan Perak 
50450 Kuala Lumpur 
Tel (60­3)262 62 98 
Fax [60­31 262 61 98 
E­mail: e_c­klOmol.net my 

Mundi Prensa Mexico, SA de CV 

Rio Panuca No 141 
Colonia Cuauhtemoc 
MX-06500 Mexico. DF 
Tel (52-5) 533 56 58 
Fax (52­5) 514 67 99 
E­mail 101545 2361 Ocompuservö com 

EBIC Philippines 

19tti Floor. PS Bank Tower 
Sen. Gii J Puyat Ave cor Tindalo SI 
Makali City 
Metro Manilla 
Tel. (63-2) 759 66 30 
Fax (63­2) 759 66 90 
E­mail. eccpcomOglobe.com ph 
URL http^/www eccp com 

EBIC Sri Lanka 

Tr ans Asia Holel 
115 Sir chittompalam 
A. Gardtner Mawalha 
C_ombo2 
Tel (94­1)074 71 50 78 
Fax (94­1) 44 87 79 
E­mail: ebtcsIOitmincom 

EBIC Thailand 

29 Varassa Building. 8lh Floor 
Soi Ch idiom 
Ploenchit 
10330 Bangkok 
Tol (66-2)655 06 27 
Fax (66­2) 655 06 26 
E­mail e_cbVkOksc15.lhcorrt 
URL: http/www.ebtcbkk org 

UNITED STATES OF AMERICA 

Beman Associates 

4611 F Assembly Dnvo 
U n h am MD20706 
Tel (1­600) 274 44 47 (toll (ree telephone) 
Fax ( 1 ­6001 865 34 50 (loll free fax) 
E­mail: queryObeman.com 
URL: MtpVwww beman com 

Bitte wenden Sie sich an ein Büro Ihrer 
Wahl/ Please contaci the sales off Ico 
ol your choice/ Veuillez vous adresser 
au bureau de vente de votre choix 

O f l ke for Official Publications 
of lhe Europeen Communities 
2. 'm­· Mercier 
L­2985 Luxembourg 
T_ (352)29 29­42455 
Fax (352) 29 29­42758 
E­mail: info infoOopoce cec be 
URL. http://eur­op.eu int 

The  MEANS  collection: 
"Evaluating  socio-economic  programmes" 

This  six-volume  set  gives  wider  public  access  to  the  lessons  of  the  MEANS 
programme  which the  European Commission  introduced with a view to  impro­
ving  and  promoting  evaluation  methods.  Its  publication  meets  a  growing 
need for  reliable  benchmarks  to  deal with the  problems  of  evaluation  encoun­
tered when implementing  public  measures in general and Structural Fund ope­
rations  in  particular. 

In  this  context,  the  set  constitutes  a  unique  methodological  guide  that  can 
solve  the  technical  and  organisational  problems  arising  in  the  course  of 
evaluation  work  and  for  which  the  existing  literature  provides  little  help. 

The  collection  is  aimed  not  only  at  professional  evaluators,  for  whom  it  will 
provide a valuable  handbook,  but also at non-specialists,  in particular  manag­
ing authorities  and structural  programme  Monitoring  Committees. 

Evaluation  design  and  management 

Each  volume  is  des gned  to  be  independent  of  the  others,  yet  they  are  also 
complementary. Together, they deal with all aspects of the evaluation of socio­
economic  programmes: 
I. 
II.  Selection  and  use of  indicators for  monitoring  and  evaluation 
III.  Principal  evaluation  techniques  and  tools 
IV.  Technical  solutions  for  evaluation  within  a partnership  framework 
V.  Transversal  evaluation  of  impacts  on  the  environment,  employment  and 

other  intervention  priorities 

VI.  Glossary  of  300 concepts  and  technical  terms 

OFFICE FOR OFFICIAL PUBLICATIONS 
OF THE EUROPEAN COMMUNITIES 

L-2935 Luxembourg