Review of evaluations of Cohesion policy 2007-2013 and a structured approach to assessing evaluation plans for 2014-2020 : final report.
Date: 2016-01-14
Institutions: Directorate-General for Regional and Urban Policy
Subjects: Cohesion Fund, EU financing, European Regional Development Fund, economic and social cohesion, project evaluation, regional policy, report

Review of evaluations of 
Cohesion policy 2007-2013 
and a structured approach to 
assessing evaluation plans for 
2014-2020  
(Contract number 2014CE16BAT045)

Final Report 

September 2015 

Regional and 
Urban Policy

ISM
ERI
EUR
OPA

ISMERI EUROPA 

Evaluation Helpdesk – Final Report 

EUROPEAN COMMISSION 

DG Regional and Urban Policy 
Evaluation and European Semester 

Contact: Daniele VIDONI 

E-mail: Daniele.VIDONI@ec.europa.eu 

European Commission 
B-1049 Brussels 

EUROPEAN COMMISSION 

Review of evaluations of 
Cohesion policy 2007-2013 
and a structured approach to 
assessing evaluation plans for 
2014-2020 

Final Report 

Directorate-General for Regional and Urban Policy 

2015 

EN 

Europe Direct is a service to help you find answers 
to your questions about the European Union. 

Freephone number (*): 

00 800 6 7 8 9 10 11

(*) 

The information given is free, as are most calls (though some operators, phone 

boxes or hotels may charge you).

LEGAL NOTICE 

This document has been prepared for the European Commission however it reflects the views only of the 
authors, and the Commission cannot be held responsible for any use which may be made of the information 
contained therein. 

More information on the European Union is available on the Internet (http://www.europa.eu). 

Luxembourg: Publications Office of the European Union, 2015 

ISBN 978-92-79-52296-3 
doi: 10.2776/197632 

© European Union, 2015 
Reproduction is authorised provided the source is acknowledged. 

CONTENTS 

ABSTRACT ................................................................................................................... 5 

INTRODUCTION ............................................................................................................ 6 

1  ASSESSMENT OF EVALUATIONS ................................................................................. 7 

1.1  Features of evaluations of Cohesion policy in 2007-2013 ......................................... 7 
1.2  The 8 evaluations subject to peer review ............................................................. 14 

2  ASSESSING THE QUALITY OF EVALUATION PLANS: A STRUCTURED TEMPLATE ............... 20 

3  CONCLUDING COMMENTS - LESSONS LEARNED FROM CARRYING OUT THE PROJECT ...... 28 

4 

 
 
 
 
 
 
Abstract 

This  is  report  synthesises  the  results  of  a  project  undertaken  as  a  follow-up  to  the 
activities  of  the  network  of  independent  evaluation  experts  which  monitored  the 
performance  of  Cohesion  policy  over  the  2007-2013  programming  period.  It  updates 
the  findings of that project by summarising the features of the evaluations carried out 
on  EU  co-financed  programmes  after  the  previous  project  came  to  an  end..  It  also 
outlines  the  results  of  an  in-depth  peer  review  of  8  selected  impact  evaluations 
undertaken  as  a  continuation  of  the  exercise  .started  during  the  previous  project.  In 
addition,  it  sets  out  a  suggested  structured  means  of  assessing  the  evaluation  plans 
which Managing Authorities across the EU are required to prepare for each Operational 
Programme in the 2014-2020 period. An underlying purpose of the project was to help 
prepare the ground for  an Evaluation Helpdesk to assist the Commission to carry out 
its  new  responsibilities  as  regards  evaluations  in  the  current  period.  The  report 
concludes by listing  the main lessons learned  from the project  which are  relevant for 
the operation of the proposed ‘Evaluation Helpdesk for Cohesion Policy’. 

5 

 
 
 
 
 
Introduction1 

In  the  2007-2013  programming  period  Member  States  were  given  the  freedom  to 
carry  out  evaluations  of  Cohesion  programmes  according  to  their  needs,  instead  of 
being  required  under  the  regulations  to  undertake  specific  ones.  One  of  the  tasks 
given  to  the  network  of  independent  evaluation  experts  set  up  by  DG  Regional  and 
Urban Policy to monitor the performance of programmes co-financed by the ERDF and 
Cohesion Fund across the EU over the period was to record the evaluations carried out 
in their countries and their features. The main findings were that: 

 

the  number  of  evaluations  varied  considerably  across  countries,  with  some, 
especially  in  the  EU12,  undertaking  a  significant  number  while  others 
undertook very few; 

  most  of  the  evaluations  carried  out  focused  on  the  processes  involved  in 
managing  the  funds  and  implementing  programmes  rather  than  on  their 
effects; there were consequently only a comparatively small number of impact 
evaluations aimed at assessing the results of interventions; 

  evaluations  also  varied  markedly  across  policy  areas  without  reflecting  the 
relative  scale  of  expenditure  or  necessarily  the  need  for  policy-makers  to 
understand more about the effects. 

A further task of the same project was to subject a selection of impact evaluations of 
EU co-financed interventions to peer  review by a panel of evaluation of experts, with 
the  aim  of  encouraging  Member  States  to  undertake  more  of  this  type  of  evaluation 
and to demonstrate through good examples how best to do it.. From this, it emerged 
that  relatively  few  of  the  evaluations  carried  out  were  of  good  quality.  Common 
deficiencies  were  evaluation  questions  being  defined  too  vaguely,  the  adoption  of  an 
inappropriate methodology or the ineffective application of appropriate ones, a failure 
to use relevant or suitable data and a disconnect between the analysis carried out and 
the policy conclusions drawn from it.  

The project came to an end at the beginning of 2014 and a new project was initiated 
later in the year to record evaluations of 2007-2013 programmes completed after the 
expert  network  had  finished  its  work  and  to  continue  the  peer  review  process.  An 
additional  task  was  to  prepare  a  structured  means  of  assessing  the  evaluation  plans 
which  Managing  Authorities  (MAs)  were  required  to  produce  for  each  Operational 
Programme (OP) in the 2014-2020 period. The underlying purpose was to prepare the 
ground  for  setting  up  an  Evaluation  Helpdesk  to  assist  the  Commission  to  carry  out 
the  tasks  required  under  the  regulations  for  the  new  period  –  to  review  and  agree 
evaluation  plans  and  to  monitor  and  report  on  the  evaluations  of  programmes 
undertaken by MAs. 

The  following  report  summaries  the  results  of  the  project  and  highlights  the  main 
points to come out of it. It is divided into three parts. Part 1 summarises, in the first 
section, the features of evaluations carried out on ERDF and Cohesion Fund supported 
programmes over the 2007-2013 period. In particular, it compares those completed in 
the latter part of 2013 and during the course of 2014 with those completed in earlier 
years,  which  were  summarised  in  the  Synthesis  Report  of  the  network  of  evaluation 
experts referred to above.  

In the second section, it summarises the peer reviews of 8 selected impact evaluations 
undertaken  by  a  panel  of  evaluation  experts  at  two  meetings,  one  in  June  this  year 
and the other in July. The aim, as before, was to assess the strengths and weaknesses 
of  the  evaluations  of  EU  co-financed   interventions in  order  to  encourage  MAs  across 
the EU both to undertake  similar evaluations and to do so in a way which is likely to 
produce meaningful results.  

1 Andrea Naldini of ISMERI Europa and Terry Ward of Applica are responsible for Part 1 of this report. Elliot 
Stern and Terry Ward for Part 2. Other members of ISMERI Europa and Applica assisted in the preparation, 
notably Enrico Wolleb of ISMERI and Lydia Greunz, Ruggero Fornoni and Francesca Liberati of Applica. 

6 

 
 
 
                                                 
Part  2  sets  out  a  structured  template  for  assessing  the  evaluation  plans  prepared  by 
MAs for  the  OPs they  are responsible for.  The template covers the various functions 
which  plans  need  to  fulfil  and  the  arrangements  for  doing  so  in  order  to  help 
Commission  officials  who  need  to  approve  the  plans  to    judge  their  quality  and  the 
extent to which they achieve their purpose. 

Part 3 summarises the lessons learned from the project which can help in the setting 
up of an Evaluation Helpdesk on Cohesion policy and its operation.  

1  Assessment of evaluations 

1.1  Features of evaluations of Cohesion policy in 2007-2013 

The  evaluations  undertaken  in  the  latter  part  of  2013  and  in  2014  on  ERDF  and 
Cohesion  Fund  programmes  should  be  considered  along  with  those  carried  out  in 
earlier years. The latter were summarised in the final synthesis report produced by the 
network  of  independent  evaluation  experts  charged  with  monitoring  the  performance 
of Cohesion policy over the 2007-2013 period2. By the end of 2014, the indications are 
that  most  of  the  evaluations  planned  by  Member  States  for  the  period  had  been 
completed,  the  main  ones  still  remaining  being  a  few  ex-post  evaluations  which  are 
due to be finished by the end of 2015. 

Although  evaluation  of  programmes  was  not  compulsory  for  the  2007-2013  period, 
and  while  a  number  of  evaluations  were  carried  out  in  some  countries  and  not 
published,  a  relatively  large  number  were  undertaken  and  made  publicly  available. 
While  not  all  of  these  have  necessarily  been  identified,  information  seems  to  have 
been  compiled  on  most  of  them.  This  gives  a  reasonable  indication  of  both  the 
variation in evaluation activity across countries and  the types of evaluation that were 
carried  out,  in  terms  of  the  themes  or  policy  areas  covered  as  well  as  the  methods 
used. 

1.1.1  Number of evaluations by Member State and Objective 

At  the  end  of  2014,  information  had  been  compiled  on  87  additional  evaluations  of 
ERDF and Cohesion Fund programmes completed after those reported by the network 
of experts up to the third quarter of 2013 or so. These increased the total number on 
which details are available from 830 to 917 (Table 1).  

Almost  half  of  the  additional  87  evaluations  were  carried  out  in  just  two  countries, 
Italy (24) and Poland (15), though this may reflect the way that the information was 
collected  –  by  web  search  and  by  asking  the  experts  who  had  been  members  of  the 
evaluation  network,  not  all  of  whom  replied  –  rather  than  the  true  situation.  In  the 
case  of  Italy,  the  relatively  large  number  is  a  result  of  the  delayed  start  of  the 
evaluation activity, while in  Poland, it is in line with the  great many undertaken over 
the  whole  programming  period  and,  indeed,  represents  a  reduction  in  the  rate  of 
activity. Other countries with a significant number of evaluations are Germany (7), the 
UK  (9)  and  Hungary  (9).  In  Germany  and  the  UK,  most  of  the  evaluations  were 
produced  by  only  one  or  two  OPs  (Brandenburg  in  the  first  and  Wales  in  the  latter) 
where there seems to  have a particular commitment to evaluations. According to the 
information compiled, 13 Member States  did not produce any evaluation in  2013 and 
2014, which was also the case for the Cross Border Cooperation (CBC) programmes. 

Over the period as a whole, by far the largest number of evaluations were carried out 
in Poland (265), with the second largest number being undertaken in Italy. These two 
countries together, therefore, accounted for over 40% of the total number carried out 
in the EU on the 2007-2013 programmes, though it should be emphasised that these 
are  only  the  ones  which  it  has  been  possible  to  identify.  At  the  other  extreme,  no 

2 Include reference to the 2013 Evalnet Synthesis Report 

7 

 
 
 
                                                 
evaluations at all are recorded for Luxembourg, though the scale of the ERDF received 
over the period was small. 

If  the  number  of  evaluations  carried  out  is  related  to  the  number  of  Operational 
Programmes (OPs), a somewhat different picture emerges. The largest number is then 
recorded by Lithuania, where 42 evaluations were undertaken on the two OPs, and on 
the 10 priorities included within these programmes. Poland ranks only second in these 
terms  with  an  average  of  just  over  13  evaluations  per  OP  and  1.6  per  priority.  It  is 
one  of  only  4  countries,  however,  where  at  least  one  evaluation  was  carried  out  per 
priority  over  the  period  (the  others  are  Denmark  and  the  Czech  Republic  as  well  as 
Lithuania). This, it should be recalled, is the requirement in the current programming 
period, that each priority should be subject to at least one evaluation at least one over 
the period.  

Table 1 Evaluation reports in the last two years (2013-2014) and in the 
whole period by country and indicators of the evaluation activity in EU27 
countries 

6.9 

0.0 
0.0 
0.0 
1.1 

2013-2014 

2007-2014 

%  No. 
42 
17.2  265 
6 
82 
10 
9 
27.6  109 
18 
41 
31 
3 
14 
25 
24 
5 
73 
9 
4 
2 
16 
29 
9 
29 
16 
2 
11 
0 
100.0  884 
33 
100.0  917 
41.4  519 
58.6  398 

Lithuania 
Poland 
Cyprus 
Czech Republic 
Latvia 
Estonia 
Italy 
Romania 
Hungary 
Portugal 
Denmark 
Bulgaria 
Slovakia 
Austria 
Slovenia 
France 
Belgium 
Ireland 
Malta 
Sweden 
UK 
Finland 
Germany 
Greece 
Netherlands 
Spain 
Luxemburg 
Total MS 
Cross-border 
EU27 
EU12 
EU15 
Note: Countries ordered by number of evaluations per OP 
* ERDF and Cohesion Fund plus national funding 
Source: Information collected from network of national experts during period 201-2013 and 
under the Evaluation Helpdesk project 

Evaluations 
per OP (No) 
21.0 
13.3 
6.0 
5.9 
5.0 
4.5 
3.9 
3.6 
3.2 
3.1 
3.0 
2.8 
2.8 
2.7 
2.5 
2.4 
2.3 
2.0 
2.0 
2.0 
1.8 
1.8 
1.6 
1.6 
0.5 
0.5 
0 
3.6 
0.5 
2.9 
6.8 
1.7 

Evaluations 
per priority 
(No.) 
4.2 
1.6 
0.9 
1.0 
0.8 
0.6 
0.8 
0.8 
0.6 
0.4 
1.5 
0.6 
0.6 
0.9 
0.4 
0.4 
0.6 
0.5 
0.3 
0.6 
0.4 
0.4 
0.3 
0.2 
0.1 
0.1 
0 
0.7 
0.1 
0.6 
1.1 
0.3 

Public funds* 
per OP (EUR 
mn) 
153 
262 
97 
326 
436 
378 
322 
1,015 
611 
538 
137 
473 
473 
53 
787 
242 
222 
206 
428 
124 
336 
234 
813 
1,178 
897 
3,189 
0 
390 
331 
388 
342 
447 

% 
4.6 
28.9 
0.7 
8.9 
1.1 
1.0 
11.9 
2.0 
4.5 
3.4 
0.3 
1.5 
2.7 
2.6 
0.5 
8.0 
1.0 
0.4 
0.2 
1.7 
3.2 
1.0 
3.2 
1.7 
0.2 
1.2 
0.0 
96.4 
3.6 
100.0 
56.6 
43.4 

No. 
6 
15 
0 
0 
0 
1 
24 
2 
9 
4 
0 
0 
2 
0 
1 
1 
2 
0 
0 
0 
9 
0 
7 
0 
0 
4 
0 
87 
0 
87 
36 
51 

2.3 
10.3 
4.6 
0.0 
0.0 
2.3 
0.0 
1.1 
1.1 
2.3 
0.0 
0.0 
0.0 
10.3 
0.0 
8.0 
0.0 
0.0 
4.6 
0.0 

0.0 

Overall,  fewer  evaluations  were  undertaken  in  EU15  countries  than  EU12  Member 
States and even fewer relative to OPs (only a quarter of the number) or priorities. The 

8 

 
 
 
  
  
 
 
 
 
number  was  particularly  small  in  Spain  given  the  amount  of  funding  going  into 
programmes,  so  that in  total  only  one  evaluation  was  carried  out  over  the  period  for 
every EUR 3.2 billion of public funding. This figure was much smaller in Greece, but it 
was  still  the  case  that  the  amount  was  around  EUR  1.2  billion  on  average  for  each 
evaluation  undertaken.  The  average  amount  was  also  over  EUR  1  billion  in  Romania 
where funding was divided between  a relatively small number of  OP  and priorities as 
compared  with  other  countries,  so  that  the  scale  in  these  terms  was  correspondingly 
larger than elsewhere. 

As  implied  by  the  distribution  of  evaluation  across  countries,  most  of  the  evaluations 
carried  out  were  in  Convergence  regions,  where  most  of  the  funding  went.  Just  over 
60%  of  the  evaluations  over  the  period  were  in  these  regions,  over  80%  of  them  in 
regions  in  the  EU12  (Table  2).  Only  18%  of  the  evaluations  on  programmes  in 
Convergence  regions  were,  therefore,  undertaken  in  the  EU15  (in  Greece,  Spain, 
southern  Italy,  Portugal  and  so  on)  which  is  only  just  over  half  of  what  would  be 
expected given the share  of funding going to such regions in these countries (almost 
35% of the total going to Convergence regions).  

Table 2 Evaluations carried out by Objective and country group, 2007-2014 

EU12 

EU15 

EU27 

EU12 

EU15 

EU27 

EU12 

EU15  EU27 

Number  

% of total by MS group 

% of total by 
Objective 

Convergence 

439 

96 

535 

84.6 

26.3 

60.5 

82.1 

17.9 

100 

Competitiveness 

Conv. + Comp. 

10 

70 

226 

43 

236 

113 

1.9 

13.5 

61.9 

11.8 

26.7 

12.8 

4.2 

61.9 

95.8 

38.1 

Total 
41.3 
Note: Evaluations of Cross-border programmes are excluded. 'Conv.+Comp.' are mainly 
national programmes 

58.7 

100 

884 

100 

519 

365 

100 

100 

100 

100 

Source: Information collected from network of national experts during period 201-2013 and 
under the Evaluation Helpdesk project 

1.1.2  Time profile of evaluations completed 

As would be expected, the number of evaluation carried out on the 2007-2013 period 
was small in the first three years of the period and then increased markedly from then 
on.  Less  expectedly  perhaps,  the  number  reached  a  peak  in  2011  and  then  declined 
progressively  from  then  until  2014  (Figure  1).  In  2011,  therefore,  almost  300 
evaluations were completed across the  EU on  ERDF and Cohesion Fund programmes, 
around a third of all the evaluations carried out over the period. The implication seems 
to be that for many OPs, the evaluations undertaken were aimed at providing a mid-
term  review  of  programmes  so  as  to  examine  the  progress  being  made  up  to  that 
point  and  to  identify  areas  or  aspects  which  needed  to  be  modified.  This  includes 
consideration  of  the  desirability  of  shifting  funding  between  policy  areas  in  order  to 
increase the rate of expenditure of funding as well as to respond to the effects of the 
crisis.  

Figure 1 Number of evaluations of 2007-2013 programmes by year of 
completion , EU27 

9 

 
 
 
  
300

250

200

150

100

50

0
2007

2008

2009

2010

2011

2012

2013

2014

Source: See Table 1 

Mid-term  evaluations  of  programmes,  following  the  pattern  of  the  previous 
programming  period  when  they  were  obligatory,  were  particularly  evident  in  France, 
Germany, the Netherlands and the UK. They were also evident in countries which were 
hit  especially  hard  by  the  global  recession  and/or  were  experiencing  tightening 
constraints  on  public  finances,  such  as  the  Baltic  countries  or  Portugal,  where  there 
was a pressing need to increase the rate of expenditure both to avoid the risk of de-
commitments  and  to  counter  the  effects  of  recession.  In  other  countries,  such  as 
Poland,  Romania,  Slovakia,  Finland  and  Sweden,  evaluations  were  focused  more  on 
procedures  for  managing  funding  and  on  monitoring  the  outcome  of  expenditure  in 
particular policy areas. 

1.1.3  Evaluations by type 

The types of evaluation carried out over the period also varied over time in much the 
way that would be expected (Figure 2):  

  Procedure  and  implementation-oriented  evaluations,  i.e.  those  aimed  at 
examining  the  way  funding  is  managed  or  projects  are  selected  and  carried 
out,  were  by  far  the  most  numerous  in  the  early  years,  particularly  in  2010, 
reaching a peak in 2011, the number then declining quickly.  

  Evaluations designed to monitor progress and achievements became significant 
in number in 2010 and then increased markedly in 2011 to reach a peak only 
slightly below that of process evaluations before declining but at a slower rate 
than the latter and so being the most numerous type of evaluation in 2012, 

  Result-oriented  evaluations,  concerned  to  assess  the  effect  of  interventions, 
increased gradually from 2009 on, though at a slower rate than the other two 
types of evaluation reflecting the time taken for programmes to start producing 
results.  Unlike  the  other  two  types  of  evaluation,  the  number  continued  to 
increase up to 2013 when it reached a peak of 66 across the EU as a whole and 
was the most common type of evaluation. This was also the case in 2014. 

Figure 2 Evaluations by year of completion and type, EU27 (numbers)  

10 

 
 
 
  
140

120

100

80

60

40

20

0
2007

2008

2009

2010

2011

2012

2013

2014

1. Procedure/implementation

2. Monitoring/achievement

3. Result

Source: See Table 1 

In  2013  and  2014  taken  together,  therefore,  result-oriented,  or  impact,  evaluations 
accounted  for  over  half  (50%)  of  the  evaluations  completed  on  ERDF  and  Cohesion 
Fund co-financed interventions, whereas in 2012, they accounted for only 22% of the 
total and in the 5 years before that for just 17%.  

It can be expected that the number of such evaluations will be more numerous in the 
current programming period, given the emphasis in the new Regulations on the results 
of  the  funding  provided.  In  the  same  way  as  in  the  last  period,  however,  it  is  likely 
that  other  types  of  evaluation,  concerned  with  examining  procedures  and  assessing 
progress in the implementation of programmes will predominate up to 2018 or so. 

1.1.4  Evaluations by method 

In  line  with  the  increasing  number  of  result-oriented  evaluations  carried  out  in  the 
latter years of the programming period,  there  was also a growing use of quantitative 
methods (Table 3). 

Table 3 Evaluation reports by principal method used, 2007-2014 

Quantitative 

Quantitative and qualitative 

Qualitative 

TOTAL 
Source: See Table 1 

2007-2013 

2013-2014 

2007-2014 

No. 

76 

225 

529 

830 

% 

9.2 

27.1 

63.7 

100.0 

No. 

27 

17 

43 

87 

% 

31.0 

19.5 

49.4 

100.0 

No. 

103 

242 

572 

917 

% 

11.2 

26.4 

62.4 

100.0 

Over  the  2007-2014  period  as  a  whole,  qualitative  methods  were  the  most  common 
ones used, in part reflecting the focus on procedures and assessing progress. Overall, 
evaluations  using  only  qualitative  methods  accounted  for  almost  63%  of  the  total 
number  carried  out,  while  another  26%  used  such  methods  in  combination  with 
quantitative  ones.  In  Malta,  Denmark  and  the  Netherlands,  all  of  the  evaluations 
undertaken over the period used qualitative methods, while in Finland, France, Poland, 
Romania Slovakia and the UK, 80% did so.  

Evaluations  using  quantitative  methods  alone  or  primarily,  therefore,  made  up  just 
11%  of  the  total,  while  counterfactual  evaluations  together  with  cost-benefit  analysis 
(CBA),  accounted  for  only  over  4%.  Counterfactual  evaluations  were  confined  to  13 
countries,  with  Hungary  and  Italy  having  the  largest  numbers  (respectively  11  and 
10).  The  number  of  such  evaluations  carried  out  across  the  EU,  however,  was 
considerably more than in the previous programming period (2000-2006), when only 

11 

 
 
 
 
 
 
 
one  evaluation  was  based  on  CBA,  though  some  used  this  method  combined  with 
others. 

Moreover,  the  use  of  counterfactual  methods  increased  over  the  period,  with  some 
13%  of  evaluation  carried  out  between  the  latter  part  of  2013  and  the  end  of  2014 
using  this  method  as  compared  with  only  3%  in  the  earlier  years.  The  use  of  other 
quantitative methods also increased, another 20%  of evaluations using some form of 
quantitative method other than counterfactual or CBA. This was particularly so in EU12 
countries,  where  a  significant  number  of  evaluations  were  based  on  econometric 
models, an approach made easier by the fact that 5 of the countries are single NUTS 2 
regions  and  most  received  support  under  the  Convergence  Objective  only. 
Accordingly,  the  model  can  be  applied  to  the  whole  country  without  the  need  for 
regional differentiation and the data that this entails. 

The  use  of  counterfactual  methods  was,  as  might  be  expected,  concentrated  almost 
entirely  in  results-oriented  evaluations,  while  other  quantitative  methods  were  also 
used in evaluations aimed at assessing progress. Qualitative methods were most often 
used in procedure/implementation-oriented evaluations and, indeed, 86% of the latter 
used such methods alone. Qualitative methods, however, were in many cases used in 
result-oriented,  or  impact,  evaluations,  frequently  in  the  form  of  a  theory-based 
approach.  Indeed,  almost  70%  of  result-oriented  evaluations  used  either  qualitative 
methods alone or a mix of these with quantitative methods. 

Among  the  229  result-oriented  evaluations  168  were  concerned  with  assessing  the 
impact.  Evaluations  using  qualitative  approaches,  including  theory  based  ones, 
represented  56%  of  these  and  quantitative  methods  44%.  Of  the  latter,  22%  used 
counterfactual  methods  to  assess  the  impact  and  13%  applied  other  econometric 
techniques.  

Figure 3 Impact evaluations by method used (%)  

22% 

Counterfactual Impact
Econometric approaches, CIE +
qualitative methods

Econometric and I/O models

56% 

13% 

Other quantitative methods

9% 

Theory based approaches and
qualitative methods

Source: See Table 1 

It is difficult to prescribe the method, or mix of methods, to be used for different types 
of  evaluation,  which  depends  on  a  range  of  factors  –  the  evaluation  questions,  the 
budget,  the  existing  knowledge  and  the  data  available,  in  particular.  It  is  evident, 
however, that the use of quantitative methods, which has so far featured in a minority 
of evaluations, becomes much more important with the emphasis on results as it is in 
the present period. But this requires the relevant data being available to carry out the 
analysis.  Over  the  past  period  the lack  of  such  data  on  the  output  and  results  of  the 
intervention  (which  should  generally  come  from  the  monitoring  system)  and  of 
comparable  data  to  use  in  the  counterfactual  exercise  to  represent  the  context  in 
which  the  intervention  took  place  has  severely  constrained  the  possibility  of  applying 
quantitative methods. Many evaluations, therefore, have had little option but to  have 
recourse to surveys of funding recipients and providers to produce the data to use in 
order to try to capture the achievements of interventions. 

12 

 
 
 
 
1.1.5  Evaluations by policy area 

Over the programming period as a whole – or at least up to 2014 - over a third of the 
evaluations  carried  out  were  ‘multi-area’,  in  the  sense  of  covering  more  than  one 
policy  area,  in  many  cases  all  those  covered  by  an  OP  –  i.e.  the  evaluations  were 
designed to assess the performance of the programme as a whole or the way in which 
it was being implemented (Table 4). This was particularly the case in respect of cross-
border  programmes,  almost  all  of  the  relatively  few  evaluations  carried  out  on  these 
being  multi-area  ones.  A  further  10%  were  concerned  with  the  capacity  to  manage 
funding and to implement programmes or with the institutional arrangements in place, 
while  4%  were  targeted  at  assessing  the  extent  to  which  interventions  addressed 
transversal  issues,  such  as  gender  equality  or  sustainability,  and/or  their  effect  on 
them.  Accordingly,  only  just  under  half  of  evaluations  were  focused  on  a  particular 
policy area, such as enterprise support, which accounted for 16% of the total number 
carried out.  

Over  the  latter  part  of  the  period,  from  part  way  through  the  second  half  of  2013  to 
the  end  of  2014,  there  was  an  increase  in  the  proportion  of  evaluations  on  an 
individual  policy  area,  these  accounting  for  just  over  two-thirds  of  the  total  carried 
out.  The  increase  was  particularly  marked  as  regards  evaluations  of  support  for 
investment  in  environmental  infrastructure  and  other  measures  to  protect  or  to 
improve  the  environment,  which  increased  by  more  than  three-fold  in  proportionate 
terms. This might reflect, at least in part, the substantial sums put into this area and 
the  apparent  delays  in many  countries  of  carrying  out  projects.  Other  areas  in  which 
evaluation  activity increased  were  enterprise  support,  RTDI  and  the  various  activities 
covered  by  ‘territorial  development’,  many  of  the  evaluations  in  the  first  two  areas 
being  aimed  at  assessing  the  results  of  interventions.  By  contrast,  the  proportion  of 
evaluations  targeted  on  investment  in  transport  declined  even  further  with  only  one 
being carried out over the period. 

Table 4 Evaluations by policy area, 2007-2014 

RTDI 
Enterprise support and ICT 
Human resources 
Transport 
Environment 
Energy 
Territorial development* 
Capacity 
building 
Multi-area 
Transversal aspects 
Total 

+ 

Institution 

2007-2013 

2013-2014 

2007-2014 

No. 
59 
128 
26 
38 
33 
16 
105 

87 
305 
33 
830 

% 
7.1 
15.4 
3.1 
4.6 
4.0 
1.9 
12.7 

10.5 
36.7 
4.0 
100 

No. 
8 
19 
1 
1 
12 
4 
14 

7 
19 
2 
87 

% 
9.2 
21.8 
1.1 
1.1 
13.8 
4.6 
16.1 

8.0 
21.8 
2.3 
100 

No. 
67 
147 
27 
39 
45 
20 
119 

94 
324 
35 
917 

% 
7.3 
16.0 
2.9 
4.3 
4.9 
2.2 
13.0 

10.3 
35.3 
3.8 
100 

* This covers a range of areas, such as tourism, cultural activities, social infrastructure and 
urban and rural regeneration 

Source: See Table 1 

Over  the  period  as  a  whole,  therefore,  a  disproportionately  small  number  of 
evaluations  were  carried  out  on  investment  in  transport,  which  was  also  the  case  in 
respect  of  environmental infrastructure  and  measures  to  support  the  development  of 
renewables  and  improvements  in  energy  efficiency,  despite  the  increase  in  the  later 
years.  While  the  projects  undertaken  in  these  areas  typically  take  many  years  to 
complete, often extending over more than one programming period, and even longer 
to generate observable  effects, it remains the case that they are ‘under-evaluated’ in 
relation to the substantial amounts of funding they absorb.  

13 

 
 
 
  
1.1.6  Summary of main features of evaluations  

The  main  features  of  the  evaluations  carried  out  on  2007-2013  programmes  which 
come  out  of  the  above  review  are  very  much  in  line  with  those  summarised  in  the 
Synthesis report  on the performance  of Cohesion policy over this period published at 
the beginning of 2014: 

 

there  were  many  more  evaluations  of  programmes  undertaken  in  the  2007-
2013  period  than  previously,  a  large  proportion  of  them  focused  on  particular 
policy areas in line with DG Regional Policy intentions when the regulations for 
the  period  were  introduced  which  gave  MAs  more  freedom  to  choose  which 
particular interventions or aspects of policy to evaluate; 

  many  more  evaluations  were  carried  out  in  the  EU12  countries  than  in  the 
EU15,  reflecting  the  need  to  assess  the  way  the  management  process  was 
working,  given  their  limited  experience  of  Cohesion  policy  regulations,  as  well 
as the effects of interventions.  

  at  the  same  time,  there  was  more  limited  evaluation  activity  in  a  number  of 
EU15  Member  States,  especially  in  the  south  of  Europe  where  the  scale  of 
funding  remained  relatively  large.  Indeed,  the  difference  between  countries  in 
the  number  of  evaluations  carried  out  was  substantial,  raising  questions  over 
the  adequacy  of  the  evidence  base  for  determining  how  EU  funding  should  be 
spent  and  on  what.  If  the  regulations  are  respected,  however,  this  will  not  be 
the case in the present period; 

 

there  was  a  marked  increase  in  impact  evaluations  aimed  at  assessing  the 
results of interventions as the period went on; 

  a  significant  and  increasing  number  of  impact  evaluations  used  counterfactual 
methods  to  assess  the  effects  of  interventions  over  the  period;  this  was 
accompanied by the growing use of quantitative data, though a lack of suitable 
data has limited the application of such methods. 

1.2  The 8 evaluations subject to peer review 

The  second  task  of  the  project,  as  noted  above,  was  to  continue  the  process  of 
subjecting  a  selected  number  of  impact  evaluations  to  peer  review.  It  was  also  to 
prepare summaries of the more interesting ones for the evaluation library maintained 
by  DG  Regional  Policy.  Of  the  30  or  so  evaluations  of  this  type  identified  as  having 
been carried out in 2013 and 2014, detailed summaries were produced for 15 of them 
(see  Interim  Report,  June  2015)  which  seemed  to  be  the  most  interesting  from  a 
demonstration  perspective.  From  these,  7  were  chosen  for  peer  review,  the  choice 
being  made  on  the  basis  not  so  much  of  their  perceived  quality  but  of  their 
representative nature, the policy areas covered – i.e. giving priority to areas in which 
relatively few evaluations have been  carried out  – and the Member States concerned 
so  as  to  achieve  a  reasonable  balance  between  them.  In  addition,  an  ex-ante 
evaluation  of  an  OP  for  the  2014-2020  period,  which  also  seemed  to  be  reasonably 
representative of those carried out, was selected for review.as well. As previously, the 
focus  of  the  peer  review  was  on  the  method  used,  its  appropriateness  and  the  way 
that  it  was  applied,  the  quantitative  data  and  the  qualitative  evidence  compiled  and 
analysed and their reliability, and the justification for the  policy conclusions drawn..  

1.2.1  Summaries of the 8 evaluations reviewed 

Summaries  of  the  8  evaluations  reviewed,  four  at  the  first  meeting  and  four  at  the 
second, and the main points to come out of the review, are set out below.  

  R.  Mamede,  T.  Fernands,  A.  A.  Silva,  ‘Counterfactual  impact  evaluation  of 

EU funded enterprise support in Portugal’, published in 2014.  

This  is  a  counterfactual  impact  evaluation  of  support  to  enterprises  under  the 
POE/PRIME programme co-financed by the ERDF in the period 2000-2006. The 
objective  was  to  assess  the  impact  of  the  ERDF  co-financed  support  on  the 

14 

 
 
 
survival  rate  of  enterprises  and  the  net  change  in  both  total  and  skilled 
employment  as  well  as  the  average  cost  per  job  directly  created  by  the 
intervention.  The  study  is  essentially  an  academic  one  using  a  relatively 
advanced  technique  (Coarsened  Exact  Matching)  to  identify  the  control  group 
with  which  the  performance  of  companies  receiving  support  could  be 
compared. The aim was to reduce the problem of ‘causal attribution’  – i.e. the 
difficulty of identifying the factors responsible for differences in performance or, 
in this case, of isolating the effect of the support provided from other potential 
factors. 

The  study  found  that  the  intervention  tended  to  increase  the  survival  rate  of 
the companies supported and to increase both the number of people employed 
and, more especially, the number of skilled people employed. It concluded that 
the  overall  cost  per  job,  in  terms  of  the  public  funding  provided,  was  around 
EUR 55,000 over a three-year period. It stopped short, however, of considering 
whether this represents good value for money and whether the main aim of the 
intervention,  which  was  to  strengthen  the  competitiveness  of  the  companies 
supported, was achieved or not.  

  Europos  socialiniai,  teisiniai  ir  ekonominiai  projektai  (a  Lithuanian  public 
institution),  ESTEP  Vilnius  UAB,  ‘Evaluation  of  the  efficiency  of  the  EU 
support  to  waste  management  in  Lithuania  and  identification  of 
financing priorities for 2014-2020’, published in 2014. 

This evaluation is intended to assess the effects of EU support since 2000, first 
under  ISPA  (Instrument  for  Structural  Policies  for  Pre-Accession)  and  then 
under  the  ERDF  and  Cohesion  Fund,  on  improving  waste  management  in 
Lithuania  and  to  draw  implications  for  policy  in  the  current  programming 
period,  especially  as  regards  funding  priorities.  A  theory-based  approach  is 
used together with a range of qualitative and quantitative methods.  

The conclusion reached is that EU funding was appropriately targeted and was 
effective  in  achieving  its  objectives,  including  in  terms  of  the  costs  of  the 
projects  concerned.  In  practice,  however,  the  evidence  suggests  that  over  15 
years, there has been no significant improvement in the management of waste 
despite  the  funding  allocated  to  the  issue.  Moreover,  over  the  2007-2013 
period,  only 10 of 29 projects were  either completed or near to  completion at 
the time of the evaluation. Nevertheless, the target set at the beginning of the 
period had almost been achieved, which suggests either that the target was set 
much  too  low  or  that  other  factors  apart  from  the  funding  provided  were 
responsible for the small improvement. 

  N.  Bryer,  G.  Williams,  T.  Fanning,  M.  Munday,  (Old  Bell  3  Ltd,  Regeneris  and 
‘Evaluation  of  Local  Investment  Funds  (LIF)’, 

Cardiff  University), 
published in 2013. 

This assesses the net impact of the Local investment funds (LIFs), launched in 
2009  and  co-financed  by  the  ERDF  as  part  of  the  Convergence  OP  for  West 
Wales and the Valleys, on job creation and business performance as well as to 
examine the perceptions of recipients on the way the programme has operated. 
To  do  so,  the  evaluation  uses  a  mix  of  qualitative  and  quantitative  methods, 
including  surveys  of  the  companies  receiving  support  and  interviews  with 
relevant personnel and a comparison of the performance of the companies with 
those not supported. The study found differences in  the number of businesses 
supported and the number of start-ups assisted and jobs created between the 
three areas in which the scheme was implemented. This, it is argued  might be 
due  in  part  to  differences  in  the  way  targets  were  defined  rather  than  to  any 
differences in the performance of the schemes. 

The  assessment  of  the  study  was  that  the  methods  of  analysis  were  clearly 
explained but the links between them might have been drawn more  clearly to 
make  the  theory  on  which  the  evaluation  was  intended  to  be  based  more 

15 

 
 
 
evident.  There  was  equally  a  need  to  explain  the  data  used  more  fully  and  to 
assess  their  reliability  since  they  mostly  took  the  form  of  judgements 
expressed by recipients of funding, which might lack objectivity. 

  Laboratory  Research  &  Consultancy,  ‘Impact  evaluation  of  revitalization 
for  Kujawsko-Pomorskie 

projects 
Voivodeship for 2007-2013’, published in 2013. 

implemented  under  the  ROP 

The evaluation is aimed at assessing the impact of urban regeneration schemes 
under  the  Regional  Operational  Programme  for  2007-2013  in  the  Kujawsko-
Pomorskie voivodeship in Poland, or, more precisely, their effect on the quality 
of  life  of  people  of  people  living  in  the  areas  covered  by  ‘Local  Revitalisation 
Programmes’  (LRPs).  It  uses  a  number  of  different  qualitative  methods  to  do 
so,  including  in-depth  interviews  with  people,  telephone  interviews,  focus 
groups  and  case  studies.  The  study  concluded  that  the  LRPs  had  helped  to 
increase  the  attractiveness  of  town  centres  and  the  development  of  tourism, 
even  though  it  noted  that  a  lack  of  data  and  the  fact  that  only  26%  of  the 
initial targets had been met made it difficult to assess the achievements of the 
programme. 

The  view  was  that  it  is  not  clear  on  what  evidence  this  finding  is  based.  The 
theory of how the many and varied measures included in the programme were 
intended  to  achieve  its  objective  needs  to  be  more  clearly  set  out  and  the 
choice  of  indicators  used  to  monitor  the  effects  of  the  programme  better 
explained and justified. 

The evaluations reviewed at second meeting in July 2015 were: 

  G.  Pellegrini,  M.T.  Raggi,  A.  Spegni,  D.  Cirillo,  and  M.  De  Castris,  ‘Report  on 
the  Guarantee  Fund  -  NOP  Research  and  Competitiveness  2007-2013’, 
PromuovItalia, published in 2014. 

This assesses the effectiveness of the National Guarantee Fund (NGF) which is 
an established financial instrument in Italy. The fund provides both ‘first level’ 
guarantees  with  regard  to  company  bank  loans  and  ‘second-level’  guarantees 
to secure the first-level ones. The objectives of the evaluation are to: a) assess 
the volume of guarantees provided in the period 2000-2013 and the breakdown 
by  region  and  sector  as  well  as  by  firm  size;  b)  assess  the  impact  of  NGF  on 
growth  in  the  region;  c)  analyse  the  effect  of  the  programme  on  interest  rate 
and  loan  size.  To  do  so,  the  evaluation  adopts  a  quantitative  approach.  Three 
different  methods  are  applied:  1)  a  standard  Ordinary  Least  Squares  (OLS) 
regression  to  estimate  the  gross  effect  of  the  intervention;  2)  a  shift-share 
analysis  to  separate  the  national  effects  from  the  territorial  one;  3)  an 
Instrumental  Variable  (IV)  regression  to  isolate  the  exogenous  effects.  The 
findings are that the NGF had positive effects on SMEs access to credit. It also 
had  a  moderately  positive  effect  on  employment  (a  10%  increase  in  the 
number  of  guarantees  tended  to  lead  to  a  0.1%  increase  in  employment). 
There  was,  however,  a  North-South  divide,  with  the  fund  reducing  interest 
rates in the South and increasing the amount of credit extended in the North. 

The study was considered to be particularly relevant given the increasing use of 
FIs  in  the  application  of  Cohesion  policy.  The  findings  indicate  that  such  a 
scheme  can  have  very  different  results  depending  on  the  regional  context.  In 
this  specific  case,  the  difference  appears  to  be  determined  mainly  by  the 
characteristics  of  the  companies  and  the  collateral  they  provide  rather  than 
different assessments of risks by banks in the South of Italy as opposed to the 
North. However, there is question over whether the focus of analysis should not 
have been the lending institutions rather than firms and on whether the level of 
bank  investments  had  changed  as  a  result  of  the  NSF  rather  than  on  the 
change in employment generated by the companies. 

16 

 
 
 
  M. Iordan, A. Gramillano and F. Salvagnini, (SC ACZ Consulting SRL, t33 SRL, 
IRIS  SRL),  ‘Impact  evaluation  of  Key  Area  of  Intervention  4.3.  Support 
for  the  development  of  micro-enterprises  –  ERDF  ROP  2007-2013’, 
published in 2013. 

This is aimed at assessing the impact on employment of intervention  ’KAI 4.3’ 
which provides direct financial support to micro-enterprises in Romania. It uses 
a  range  of  quantitative  techniques  to  do  so  -  Difference-in-Difference, 
Regression  Discontinuity  Design  and  Propensity  Score  Matching  -  combined 
with  qualitative  analysis  (interviews  with  managing  authorities,  regional 
agencies, beneficiaries, and other stakeholders). The evaluation finds evidence 
of  a  large,  statistically  significant,  effect  on  job  creation,  namely  that 
beneficiary  companies,  on  average,  created  three  more  full  time  equivalent 
jobs (around 60% more) than non-beneficiary ones.  

No  effect,  however,  was  found  on  enterprise  turnover,  though  the  effect  on 
employment  remained  positive  and  statistically  significant  4  years  after  the 
intervention.  This,  accordingly,  suggests  that  productivity  declined  in  the 
companies supported relative to others, an implication which is not picked up in 
the study and which raises a question-mark over the results. Equally, there is 
question  over  why  the  analysis  producing  the  results  was  restricted  to  only  a 
proportion of the enterprises receiving funding. 

  Ecorys,  ‘Ex-ante  Evaluation  of  Operational  Programme  Regions  in 

Growth for the 2014-2020 programming period’, published in 2014. 

The  evaluation  assesses  the  programme  for  the  present  period  in  Bulgaria.  It 
focuses,  in  particular,  on  the  alignment  of  the  planned  interventions  with 
identified  needs,  the  internal  and  external  coherence,  the  application  of 
horizontal  principles  (e.g.  sustainable  development  and  equal  opportunities,) 
and  the  integration  of  environmental  considerations.  It  adopts  a  qualitative 
approach,  including  interviews  with  stakeholders,  analysis  of  the  intervention 
logic, a SWOT analysis and a review of best practice in other EU Member States 
and of the indicators and workload. It finds that the objectives are in line with 
both  national  and  EU  ones  as  well  as  with  identified  need,  that  horizontal 
principles  are  respected,  output  indicators  are  adequate  but  modifications  to 
result indicators are needed, the budget is in line with investment priorities, the 
plan  for  strengthening  administrative  capacity  seems  appropriate  and  there 
should be only minor adverse effects on the environment. 

The main question raised at the meeting was whether an ex ante evaluation of 
this  kind,  which  is  required  under  Cohesion  policy  regulations,  should  go 
beyond  procedural  issues,  of  assessing  conformity  with  EU  and  national 
objectives and so on, to examining critically the likelihood of the  interventions 
planned  achieving  the  objectives  set  and  whether  other  measures  might  be 
more effective. 

  Dirección  General  de  Fondos  Comunitarios  -  Secretaría  de  Estado  de 
Presupuestos y Gastos Europos social, ‘Impact evaluation of the Structural 
Funds  co-financed  investment  in  Water  Sanitation  and  Wastewater 
Treatment in Spain’, published in 2014. 

The  aim  is  to  assess  the  outcome  of  investment  in  wastewater  treatment 
carried  out  over  the  2007-2012  period  in  Spain  and  in  particular  whether 
Structural  Funds  support  helped  to  improve  water  quality  challenges  and 
whether  achievements  were  in  line  with  targets.  The  evaluation  applies  a  mix 
of  qualitative  and  quantitative  methods,  the  former  mainly  comprising 
document  research  and  the  latter  using  two  different  methods:  propensity 
score  matching  (PSM)  and  difference-in-differences  (DID).  The  evaluation 
found that implementation lagged behind the original plans, with less than half 
of  the  budget  allocated  being  spent  and  that  there  was  no  statistically 
significant  effect  on  the  amount  of  water  treated.  However,  investment  in 

17 

 
 
 
water  sanitation  and  treatment  has  had  a  significant  effect  on  the  amount  of 
water  reused, 
in  municipalities  supported 
compared to the control group. 

increasing  water  availability 

Although  the  analysis  is  difficult  to  follow,  the  method  used  seems  to  have 
been applied satisfactorily. The main question raised was why a  counterfactual 
analysis  had  been  applied  at  all,  given  that  assessment  of  changes  in  water 
treated and in its quality ought to be a simple matter of comparing the data for 
the municipalities where investment had taken place with others.  

1.2.2  Lessons learned from the review of the 8 evaluations 

The  main  points  to  come  out  of  the  review  of  the  8  evaluations  are  summarised 
below: 

  Counterfactual evaluation are sometimes very ‘method-focussed’ and driven by 
technical  considerations.  They  often  neglect  the  policy  implications  of  the 
findings.  While  a  number  of  the  evaluations  reviewed  are  technically  strong, 
the  results  do  not  always  have  clear  policy  implications  or  are  not  interpreted 
from a policy perspective.  In particular, the issue  of value for money tends to 
be ignored, as in many cases do the wider effects on, for example, enterprises 
which have not received support and the employees working in them. 

  The data used are often those which are available rather than those which  are 
best suited to answering the evaluation question or which are most meaningful 
given  the  intervention  being  examined  and  its  objective.  Schemes  which  are 
aimed  at  strengthening  competitiveness,  therefore,  are  in  many  cases, 
assessed  in  terms  of  their  effect  on  employment,  even  though,  in  the  short-
term at least, there is no necessary relationship between the two.  

 

 

 

In  practice,  therefore,  if  the  data  ideally  needed  for  the  analysis  are  not 
available, there is often a failure to consider using alternative data which  may 
be  less  than  ideal  but  which  are  more  relevant  as  regards  the  intervention 
being analysed but which can be compiled even though this might require some 
effort.  

It  is  important  that  evaluators  are  ‘honest’  about  the  data  and  method  used, 
that  they  are  clearly  explained,  that  any  shortcomings  in  both  regards  are 
indicated  and  that  sensitivity  and  robustness  checks  are  carried  out. 
Accordingly,  in  the  case  of  data  collected  through  interviews  or  surveys,  the 
questions asked, the sample selection process, any ‘cleaning’ or adjustment of 
the data and so on  need to be clearly set out. A good evaluation  should draw 
attention to its limitations and provide all the information required to interpret 
the results and their implications correctly. 

In general, data information collected in this way needs to be verified so far as 
possible  against  other  information,  ideally  quantitative  data,  before  any 
conclusions  on  the  effect  of  policy  are  drawn  from  it.  Such  ‘triangulation’  is in 
many  cases  a  necessary  step  to  producing  meaningful  evaluation  findings, 
especially  where  qualitative  information  is  concerned,  particularly  when  it 
comes  from  surveys  of  beneficiaries  or  the  bodies  providing  the  funding  who 
may well not be objective in their responses.  

  Counterfactual  methods  are  not  always  the  best  or  most  suitable  approach  to 
use.  This  is  particularly  so  if  the  aim  is  to  identify  causal  mechanisms  or  the 
processes  by  which  an  intervention  produces  its  effects.  In  these  cases,  case 
studies or other qualitative methods are more likely to be more appropriate. As 
a general rule, the evaluation method to be used should be determined by the 
evaluation questions to be answered, with the simplest method being used and 
more complex methods only if this is insufficient. 

  Qualitative  methods  are  often  considered  to  be  ‘easier’  or  ‘less  costly’  than 
quantitative  ones,  whereas  in  practice,  the  reverse  is  as likely  to  be  the  case. 
In practice, if carried out seriously, qualitative impact evaluations can in many 

18 

 
 
 
cases  be  much  more  time-consuming  and  demanding  than  quantitative  ones, 
which if the data are  available can simply involve the application of  a suitable 
estimation technique.  

  Many qualitative evaluations claim to be theory-based but do not spell out the 
underlying  theory  which  they  are  supposed  to  be  based  on.  Theory-based 
evaluations need, therefore, to explain clearly the theory which it is intended to 
test,  the  various  processes  by  which  the  intervention  being  examined  is 
assumed  to  achieve  its  effects  and  the  steps  by  which  the  analysis  will  be 
carried out.  

  A literature review targeted on the issues which are relevant for the evaluation 
to be undertaken is usually important for identifying the approach and methods 
which  are  best  designed  to  answer  the  evaluation  questions.  Many  theory 
based evaluations, in particular, however, give the impression of ‘starting from 
scratch’ and fail to take account of what is already known from previous studies 
carried out. 

  There is a need for meta-evaluations to be undertaken to assemble the existing 
evidence from past evaluations, or research studies, on the effects of particular 
interventions in different policy areas in order not only to inform policy-makers 
but also to provide the starting-point for evaluations in the areas concerned. 

  Ex ante evaluations have the potential to make a significant contribution to the 
preparation  of  Cohesion  policy  programmes.  They  need,  however,  to  be 
incorporated closely into the decision-making process, which is not always the 
case at present. They can then be used to assess the likely effectiveness of the 
measures that it is planned to take and to compare them with alternative ways 
of  achieving  the  objectives  set  so  as  to  help  to  improve  the  way  that  EU 
funding is spent. 

19 

 
 
 
 
 
2  Assessing the quality of evaluation plans: a structured 

template 

For  the  2014-2020  programming  period,  Managing  Authorities  for  the  first  time  are 
obliged to prepare  Evaluation Plans in respect  of the programmes co-financed by the 
Structural  Funds.  Guidance  on  how  to  prepare  these  plans  is  contained  in  the 
Commission’s Guidance Document on Evaluation Plans (April 2014). An important shift 
which  is  reflected  in  this  guidance  is  a  focus  on  identifying  and  analysing  results  of 
programmes  rather  than  on  management  procedures  and  implementation  processes. 
This reflects in turn the policy emphasis in the present programming period on results 
and  on  the  formulation  of  programmes  which  are  best  designed  to  achieve  Cohesion 
policy objectives. The guidance document states: ‘The evaluation planned should also 
allow the Managing Authority to conclude on the impact of the programme in relation 
to the targets of the Union strategy for smart, sustainable and inclusive growth’. 

The new emphasis is itself a major reason for the stress on effective planning ahead of 
evaluations, since the orientation of policy towards results includes a demonstration of 
‘impact’  so  requiring  identification  and  careful  preparation  and  design  of  the 
evaluations which need to be carried out in support of such a policy. Good evaluation 
planning, as past experience shows, is, therefore, an important pre-condition for high-
quality evaluations being carried out which, in turn, are an essential complement to a 
results-oriented policy.  

This  template  is  intended  to  complement  the  Commission’s  Guidance  Document  by 
providing  a  means  of  assessing  the  evaluation  plans  produced  by  Managing 
Authorities.  At  the  same  time,  it  gives  a  way  for  Managing  Authorities  themselves  to 
check  that  the  Plans  conform  to  what  is  expected  of  them  by  setting  out  in  a 
structured  way  the  various  objectives  that  they  need  to  achieve  and  the  kinds  of 
action they need to take in order to do so. 

There are six main focus areas in the Commission’s Guidance document: 

  Management and planning 

  Responsibility and coordination 

  Design and methods 

  Data availability and data systems 

  Skills and expertise 

  Evaluation use and communication 

The  content  of  each  of  these  focus  areas  is  outlined  in  the  table  below  under  three 
headings: 

1)  Topic which simply labels what needs to be covered 

2)  Objective  which  indicates  what  needs  to  be  achieved  in  respect  of  each  topic 
or, from the perspective of the reviewer, the aspects which are expected to be 
shown to be in place in each case 

3)  Examples  of  action,  which  lists  examples  of  the  steps  to  take,  or  the  steps 
which  have  been  taken,  for  the  plan  to  achieve  the  objective  concerned  and 
which,  accordingly,  enable  an  assessment  to  be  made  of  whether  or  not  the 
objective is likely to be met and whether or not the various aspects of the plan 
have been put in place. 

The  template  enables  the  quality  of  Evaluation  Plans  to  be  judged  at  three  possible 
levels: 

  At  a  first,  or  basic,  level  (Level  1),  it  provides  a  checklist  against  which  the 
various aspects, or topics, which need to be covered by the plan - such as the 
use  of  evaluation  evidence  or  data  requirements  –  can  be  cross-referenced  to 
ensure that they are present.  

20 

 
 
 
  At  a  second  level  (Level  2),  it  identifies  the  various  objectives  which  the  plan 
needs to meet in respect of each topic, such as a detailed timetable listing all 
the evaluations planned and an indication of how the findings will feed into the 
policy-making  process  or  specifying  the  sources  of  the  data  which  the 
evaluation will use.  

  At  a  third level  (Level 3),  the  template  lists  the  kinds  of  action  which  need  to 
be taken, or the arrangements which need to be in place, in each broad focus 
area  for  the  objectives  in  respect  of  the  different  aspects  or  topics  to  be 
achieved. This provides a basis for assessing what has been done in practice in 
each case and to judge whether the steps taken have been, or are likely to be, 
effective. Such an assessment can be more  or less detailed. At a minimum, it 
can  involve  a  simple  check  of  whether  or  not  each  kind  of  action  has  been 
undertaken. At the other extreme, the details of each of the actions which has 
been undertaken and of each of the aspects which have been put in place can 
be examined in some depth to verify that the plans are fit for purpose.  

It is the degree of detail which is given in the plan about the arrangements which have 
been made to carry out the evaluations and to make effective use of the findings and 
the extent to which they are coherent and have been carefully designed to achieve the 
different objectives which determine its quality.  

It  should  be  emphasised  that  the  examples  of  action  given  in  the  template  are 
precisely  that  and  are  not  intended  to  be  either  a  comprehensive  list  of  the  means 
which are taken to achieve the objectives that the plan needs to attain or prescriptive 
in  the  sense  of  defining  what  Managing  Authorities  should  do  in  this  regard.  New 
issues  which  evaluations  need  to  take  account  of  are  likely  to  arise  over  time,  or 
existing ones to become more important. Plans will, accordingly, need to be designed 
to accommodate these or to give added weight to them in the evaluations carried out. 

Focus area 

Topic 

Objective 

Examples of action  

Evaluation 
function 

An operating and 
effective evaluation 
function  

Use of available 
evidence 

Systematic use of 
available information, 
especially findings of 
previous evaluations, 
to fully analyse need, 
resources, priorities 
and potential 
interventions as well 
as the identification of 
where to focus future 
evaluations  

i

g
n
n
n
a
l
p
d
n
a

t
n
e
m
e
g
a
n
a
M

Staff with specialist and 
administrative expertise are 
in place to plan, procure, 
coordinate and manage 
evaluations over the 
programming cycle 

Reviews of evidence from 
previous evaluations, a 
synthesis of findings, 
reviews of existing research 
literature and consultations 
with experts, including the 
academic community, have 
been carried out in order 
both to determine what is 
known about the effects of 
interventions and how best 
to implement them and to 
identify the gaps in 
knowledge where future 
evaluation effort needs to 
be focused 

21 

 
 
 
 
 
 
Focus area 

Topic 

Objective 

Examples of action  

Time planning  

A timetable which lists 
all the evaluations 
which are planned and 
which indicates when 
each of them will be 
carried out and how 
the findings will feed 
into decision making 
and reporting 

Quality 
management 

A quality management 
plan covering the 
whole evaluation cycle 

Partnership 
Involvement 

Involvement of 
relevant partners and 
stakeholders in 
making the evaluation 
plan and in 
implementing it 

An outline ‘chart’ or time 
schedule has been prepared 
showing how the findings of 
evaluations will relate to: 

  the various stages of 

implementation of the 
programme  

  management reviews of 
various aspects of the 
programme and decision 
points 

  annual implementation 
and progress reporting  

  reports required at the 

end of the programming 
period, particularly in 
2022 the report 
summarising the 
evaluations carried out 

A quality management plan 
is in place which includes: 

  the application of quality 

criteria to past 
evaluations where they 
are to be used as 
evidence in support of 
plans 

  the establishment of a 
quality review process 
for each deliverable of 
the evaluations 

  a peer review panel or 
similar to ensure that 
independent 
assessments and 
feedback can be 
provided 

Partners have been 
explicitly involved, or have 
been consulted, in the 
formulation of the 
evaluation plan 

Arrangements have been 
made for partners to be 
involved in reviewing and 
summarising evaluation 
findings 

MA 
Coordination 

A clear set of 
responsibilities for the 
design, coordination, 
delivery and 
monitoring of 

A responsible coordination 
unit is in place with stated 
powers and obligations at 
each stage of the evaluation 
cycle 

22 

n
o

i

i
t
a
n
d
r
o
o
c
d
n
a

y
t
i
l
i

b
i
s
n
o
p
s
e
R

 
 
 
 
 
 
Focus area 

Topic 

Objective 

Examples of action  

evaluations within an 
OP 

Arrangements are in 
place to exchange 
information over 
cross-cutting aspects 
of evaluations, such as 
the gender dimension, 
and to facilitate 
mutual learning 

An adequate and 
flexible budget for 
evaluation and related 
activities has been 
fixed 

Cross MA 
Coordination 

Budget 

Evaluation 
design 

The rationale for all 
evaluations is set out 
and key evaluation 
questions are specified 

The division of labour 
between coordination unit, 
MAs and Monitoring 
Committees and other 
authorities, agencies or 
units has been specified 

Areas of interdependence 
have been identified (e.g. 
integrated evaluations that 
affect a particular target 
group). 

Procedures are in place to 
exchange information and to 
ensure that responsibilities 
are clear 

The cost of carrying out all 
the evaluation activities 
(including data collection 
and training where 
necessary) has been 
estimated and a budget set 
aside to cover them 

The budget is clear about 
who pays for what and has 
been agreed by the relevant 
budget holders 

Each planned evaluation (or 
groups of evaluations) is 
linked to programme 
objectives, priorities, the 
pursuit of common EU goals 
or horizontal issues (such as 
gender equality or 
environmental sustainability 

The purpose of all 
evaluations is stated 

The types of evaluation 
questions to be posed are 
set out for each evaluation 
(or groups of evaluation)  

The design of evaluations 
makes explicit reference to 
the evidence already 
available 

Selection of 
designs and 
methods 

Appropriate 
approaches, or 
methods, are indicated 
for each category of 
evaluations 

Evaluation approaches are 
identified for each 
evaluation and are matched 
to the evaluation questions 
asked 

For evaluations which are 
intended to assess 
efficiency, the results of 
interventions will be 

23 

s
d
o
h
t
e
m
d
n
a
n
g
i
s
e
D

 
 
 
 
 
 
Focus area 

Topic 

Objective 

Examples of action  

Results 
orientation 

Expected results are 
indicated and 
indicators identified 

Contribution to 
results  

Data 
requirements 

The evaluation 
designs, or the 
methods which it is 
planned to adopt, are 
capable of linking 
cause and effect  

Data requirements for 
each evaluation are 
identified and the 
particular data needed 
are specified  

Data availability  Data sources have 
been reviewed and 
any gaps or 
deficiencies in the data 
identified 

s
m
e
t
s
y
s

a
t
a
d
d
n
a

y
t
i
l
i

b
a
l
i
a
v
a

a
t
a
D

compared with the costs 
involved 

The use of counterfactual, 
statistical, theory-based 
approaches are clearly 
justified 

Innovative methods if 
anticipated are also justified 

A logic model or theory of 
change links the 
intervention rationale of the 
interventions to be 
evaluated to the results 
expected 

A set of indicators based on 
available data is identified 
and are matched to the 
specific objective to be 
achieved 

The rationale for their 
adoption of specific 
evaluation approaches is 
made clear.  

Data needs are linked to the 
subjects of the planned 
evaluations, the methods to 
be used and the time 
schedule for carrying them 
out 

A review of data sources has 
been made including, where 
relevant, on entities or 
activities which are not the 
subject of intervention in 
order to provide a suitable 
basis for a counterfactual 
analysis, as well as on unit 
costs where the efficiency of 
interventions is being 
assessed. 

Measures are set out to fill 
any gaps or to correct any 
deficiencies which have 
been identified and 
responsibilities for 
implementing these 
measures have been clearly 
assigned 

Comprehensive 
data sets 

Statistical systems 
appropriate to results 

Consideration has been 
given to: 

24 

 
 
 
 
 
 
 
 
 
Focus area 

Topic 

Objective 

Examples of action  

indicators are available 

  monitoring data 

  existing national 
statistics and 
administrative data (e.g. 
on employment and the 
number of new 
businesses set up) 

  Social media, GIS data 

and other relevant 
sources of ‘big data’  

  data to be collected as 
part of the evaluations 

Independence of evaluators 
is demonstrated by: 

  accountability lines, or 
the demarcation of 
responsibilities, being 
specified that ensure a 
clear separation from 
programme 
implementation and 
those responsible for 
this 

  authority being given to 
those responsible for 
evaluations to 
commission and manage 
them within agreed 
policy lines and 
operating procedures 

  an obligation being 

expressed to support 
the independence of 
evaluators to carry out 
evaluations and to 
report findings and 
conclusions without 
interference 

Evaluation 
independence 

The independence of 
evaluators and the 
units responsible for 
evaluations is assured 
by the Managing 
Authority  

e
s
i
t
r
e
p
x
e
d
n
a

s
l
l
i
k
S

Internal 
expertise 

Evaluation 
networks and 
providers 

Those employed in the 
evaluation unit or body 
responsible for 
evaluations have a 
basic level of 
understanding of 
evaluations and of the 
different methods in 
use 

The evaluation unit or 
body is linked to 
sources of evaluation 
expertise and to 
potential providers of 

Those recruited or already 
in position have basic 
training in evaluation 
approaches and in the 
management of evaluations 

Those in senior positions 
have had previous 
experience of OP 
evaluations 

The evaluation unit or body 
has established links with 
external expertise such as 
professional networks, 
university departments and 
specialist consultancies or a 

25 

 
 
 
 
 
 
Focus area 

Topic 

Objective 

evaluations 

Examples of action  

plan is in place to identify 
such expertise and to form 
links with the evaluators 
concerned 

Concrete plans exist to 
communicate information on 
the evaluations planned, to 
hold meetings for this 
purpose and to carry out 
other activities to raise 
awareness of what is 
planned 

Training and 
development 

There is capacity for 
evaluation training and 
development and 
plans are in place to 
provide training and to 
facilitate development 

Training and development 
needs for internal 
(Managing Authority) staff, 
particularly the staff in the 
evaluation unit, have been 
identified 

Sources of training and the 
development of expertise 
for both internal staff and 
potential external evaluators 
have been located 

The various users of 
evaluations and the 
particular use they will 
make of the findings are 
identified, including: 

  implementing agents 

  Managing Authorities 

  National Authorities  

  European Commission 

The guidelines which have 
been put in place or are 
intended include: 

  what information will be 

shared with which 
agencies 

  what evaluation findings 
will be communicated to 
the public and how this 
will be done 

  plans for the regular 
communication of 
details of the 
evaluations and their 
findings to the European 
Commission 

Evaluation 
users 

The users of 
evaluation findings are 
identified 

Evaluation 
communication 

Guidelines are in place 
or are envisaged 
regarding information 
sharing and 
dissemination 

n
o

i
t
a
c
i
n
u
m
m
o
C
d
n
a

e
s
U

26 

 
 
 
 
 
 
Focus area 

Topic 

Objective 

Examples of action  

Analysis and 
comparison at 
European level 

Planning to enable the 
results and findings of 
evaluations to be 
compared at European 
level 

The plan makes clear that 
each evaluation report will 
include: 

  the Terms of Reference 
for the evaluation (and 
preferably the cost) 

  an outline of the 

objectives and expected 
results 

  details of the method(s) 

and data used 

  the findings and 

conclusions 

  the lessons learned for 

the programme or 
particular intervention 
concerned, for future 
policy and for evaluation 
procedures and methods 
themselves 

27 

 
 
 
 
 
 
3  Concluding comments - lessons learned from carrying out the 

project 

Since  the  project  was  a  pilot  one,  in  the  sense  of  preparing  the  ground  for  an 
Evaluation  Helpdesk  to  support  the  Commission  in  fulfilling  its  responsibilities  with 
regard to evaluations in the 2014-2020 programming period,  it is important to draw 
lessons  from  the  experience  of  carrying  it  out.  These  lessons  are  briefly  summarised 
below.  

The  role  of  national  experts:  Although  the  pilot  project  did  not  require  an 
exhaustive list of evaluations to be compiled, it was still challenging to identify the ‘up 
to  100’  carried  out  from  the  latter  part  of  2013  to  the  end  of  2014  which  the  ToR 
required.  (In  the  event,  92  were  found.)  The  difficulty  stems  from  the  fact  that 
evaluations  are  not  systematically  published  by  the  Managing  Authorities  responsible 
or are made readily available online. Those that are available tend to be published in 
the  national  language  which  is  an  additional  complication.  National  experts  were, 
therefore,  needed  to  identify  the  evaluations,  which  highlights  the  importance  of 
having  in  place  an  effective  network  of  people  with  good  relations  with  Managing 
Authorities  who,  accordingly,  can  keep  themselves  informed  about  the  evaluations 
which are carried out.  

Language and translation: It is important to keep in mind that most evaluations are 
published  in  the  language  of  the  country  concerned  and  although  summaries  in 
English are sometimes included, they are generally not sufficient to assess the quality 
of  the  evaluation.  The  pilot  project  showed  that  Google  translations  can  be  of  some 
use  for  evaluations  in  a  language  not  understandable  by  any  of  the  team  members, 
but  in  many  cases,  especially  for  those  written  in  one  of  the  smaller  country 
languages, they were good enough only to give an indication of the contents. This is a 
further  reason  for  involving  national  experts  who  can  help  both  in  producing  English 
summaries  and  in  assessing  the  quality  of  the  evaluation.  The  pilot  project  also 
showed  that  it  is  essential  to  take  account  of  the  time  it  takes  to  make  satisfactory 
translations when planning the peer review process. (It would be useful in this regard 
if all evaluation were published with an English executive summaries which would help 
not only the task of the Helpdesk but also to the diffusion of the evaluation findings.) 

Selection  of  evaluations  for  peer  review:  The  choice  of  the  evaluations  for  peer 
review was based only to a limited extent on their quality,  which is difficult to assess 
from  a  cursory  reading,  especially  of  an  English  summary.  A  few  of  the  evaluations 
reviewed, therefore, were not of high quality. This, however, was not a major issue in 
that  there  were  as  many  lessons,  if  not  more  to  be  learned  from  them  as  from  the 
better  evaluations,  especially  those  in  the  less  commonly  covered  policy  areas. 
Indeed,  there  is  a  risk  that  such  lessons  would  not  come  out  if  the  choice  of 
evaluations  for  peer  review  were  restricted  to  what  appear  to  be  the  best  ones  only, 
which could mean that those relating to particular policy areas were not covered at all.  

Participation at peer review meetings: Discussions at peer review meetings often 
cover issues which are of general relevance for those both commissioning evaluations 
and undertaking them. Although the main points made can be summarised in reports 
which could be published on the Commission website, consideration might be given to 
inviting  Managing  Authorities  to  participate  at  the  meetings.  One  possibility  in  this 
regard might, for example,  be to hold meetings in a succession of different countries 
and to invite a few of the people responsible for evaluations in the country concerned 
to  attend.  Consideration  might  also  be  given  to  inviting  authors  of  evaluations  to 
sessions  where  the  particular  points  arising  from  the  review  in  question  could  be 
discussed.  

Structured  approach  for  assessing  the  quality  of  evaluation  plans:  The 
template produced in this regard, which is set out above, was prepared in the light of 
the evaluation plans submitted by Managing Authorities but without directly discussing 
it  with  them.  Since  it is  yet  to  be  used  for  the  purpose  it  is intended  to  serve,  there 
are  no  lessons  as  such  that  can  be  drawn  from  it.  Accordingly,  it  would  be  useful  at 
the next stage to get the reaction of those  responsible for drawing up the plans to it 

28 

 
 
 
and  to  discuss  possible  modifications  with  a  selected  number  of  them  so  as  to  make 
the template as relevant as possible as an aid to preparing plans as well as assessing 
them.  

29 

 
 
 
HOW TO OBTAIN EU PUBLICATIONS 

Free publications: 

•  one copy: 

via EU Bookshop (http://bookshop.europa.eu); 

•  more than one copy or posters/maps: 

from the European Union’s representations (http://ec.europa.eu/represent_en.htm);  
from the delegations in non-EU countries 
(http://eeas.europa.eu/delegations/index_en.htm);  
by contacting the Europe Direct service (http://europa.eu/europedirect/index_en.htm) 
or calling 00 800 6 7 8 9 10 11 (freephone number from anywhere in the EU) (*). 

(*)  The information given is free, as are most calls (though some operators, phone boxes or hotels may 
charge you). 

Priced publications: 

•  via EU Bookshop (http://bookshop.europa.eu). 

Priced subscriptions: 

•  via one of the sales agents of the Publications Office of the European Union 

(http://publications.europa.eu/others/agents/index_en.htm). 

 
 
 
 
 
 
 
 
K
N
-
0
4
-
1
5
-
7
6
2
-
E
N
-
N

[

C

a

t

a

l

o

g

u

e

n

u

m

b

e

r

]

doi: 10.2776/197632