Assessing the performance of integrated territorial and urban strategies : challenges, emerging approaches and options for the future.
Date: 2018-03-05
Institutions: Directorate-General for Regional and Urban Policy
Subjects: European Investment Fund, European Regional Development Fund, Structural Funds, added value, economic and social cohesion, regional and local authorities, report, urban economy

Assessing the performance of 
integrated territorial and 
urban strategies: 

Challenges, emerging 
approaches and options for 
the future 

Martin Ferry, Irene McMaster Arno van der Zwet – European Policies Research Centre 
January 2018 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
EUROPEAN COMMISSION 

Directorate-General for Regional and Urban Policy 
Unit 03 – Inclusive Growth, Urban and Territorial Development 

E-mail:  REGIO-URBAN-TERRITORIAL@ec.europa.eu  

European Commission 
B-1049 Brussels 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
EUROPEAN COMMISSION 

Assessing the performance of integrated 
territorial and urban strategies: 

Challenges, emerging approaches and 
options for the future 

Directorate-General for Regional and Urban Policy 

2018 

EN 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Europe Direct is a service to help you find answers  
to your questions about the European Union. 

Freephone number (*): 

00 800 6 7 8 9 10 11 

(*) 

The information given is free, as are most calls (though some operators, phone 

boxes or hotels may charge you). 

LEGAL NOTICE 

This document has been prepared for the European Commission however it reflects the views only of the 
authors, and the Commission cannot be held responsible for any use which may be made of the information 
contained therein. 

More information on the European Union is available on the Internet (http://www.europa.eu). 

Luxembourg: Publications Office of the European Union, 2018 

ISBN: 978-92-79-80392-5 
doi: 10.2776/96617 

© European Union, 2018 
Reproduction is authorised provided the source is acknowledged. 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Assessing the Performance of Integrated Territorial Development Strategies 

Contents 

1. 

INTRODUCTION ..................................................................................... 3 

2.  MEASURING  THE  EFFECTIVENESS  OF  URBAN  AND  TERRITORIAL 

STRATEGIES: KEY CONSIDERATIONS ..................................................... 5 

2.1  Generic challenges ............................................................................ 5 
2.2  Specific challenges ............................................................................ 5 
2.3    Strong variation in what has to be measured ....................................... 6 
2.4  Capturing added value ...................................................................... 9 
2.5  Addressing different audiences .......................................................... 10 

3.  EXISTING METHODOLOGIES ................................................................ 13 

3.1  Evolution of Cohesion Policy evaluation .............................................. 13 
3.2  Evaluation of other urban development initiatives ................................ 17 

4.  EMERGING  APPROACHES  TO  ASSESSMENT  IN  TERRITORIAL 

STRATEGIES ......................................................................................... 20 

4.1  Monitoring systems .......................................................................... 20 
4.2    Indicator sets………………………………………………………………………………………………21 
4.3    Emerging evaluation approaches……………………………………………………………….22 

5.  OPTIONS FOR FUTURE ASSESSMENTS OF EFFECTIVENESS .................. 25 

5.1    Indicators…..……………………………………………………………………………………………….28   
5.2    Evaluation......................................................................................31 

6.  CONCLUSIONS ..................................................................................... 31 

Annex 1:  Framework for generating different types of knowledge.............39 
Annex 2:  Assessment methodologies.........................................................41 

1 

 
 
 
 
 
 
 
 
 
 
Assessing the Performance of Integrated Territorial Development Strategies 

2 

 
Assessing the Performance of Integrated Territorial Development Strategies 

1.  INTRODUCTION 

This report examines the development of methodologies to measure the effectiveness 
of  provisions  in  Article  7  ERDF  and  Article  36  of  the  CPR  (ITI).  There  is  currently 
relatively  little  information  in  relation  to  measuring  the  effectiveness  of  the  new 
territorial  provisions.  Guidance  on  Integrated  Sustainable  Urban  Development1  ITI 
scenarios published by the Commission makes limited reference to the development of 
indicators for territorial provisions and the evaluation of these approaches.2 

The report is based on a range of information sources accessed at different stages in 
the research (see Figure 1) : 

  a review of approaches developed for measuring the effectiveness of integrated 
place-based  strategies  (e.g.  results  from  the  ex-post  evaluation  of  urban 
development in 2007-13, practices identified as part of the Urban Development 
Network and URBACT); 

  a stocktake of approaches to measurement, based on those identified in Task 3 

of the project (case studies); and 

  a  focus  group  held  in  Brussels  in  November  2016,  with  the  participation  of 
stakeholders  from  Commission  services,  programme  authorities,  organisations 
representing  local  and  regional  authorities  and  experts.  It  provided  an 
opportunity to present and discuss key methodological challenges and develop 
insights to be integrated into the methodological report. 

Figure  1 :  Key  steps  in  developing  a  methodology  for  evaluating  the 
effectiveness of territorial provisions 

Review:  existing 
methodologies for 
measuring 
effectiveness 

Stocktake:  
approaches 
suggested in   
strategy case studies 

Idea generation:  
involving experts in a 
focus group 

Develop 
methodological 
report 

Following  this  introduction,  Section  2  highlights  the  key  considerations  involved  in 
assessing  the  achievements  of  the  integrated  place-based  strategies.  Section  3 
reviews  existing  methodological  approaches  to  assessing  territorial  provisions  under 
Cohesion  policy  and  under  urban  development.  Section  4  explores  emerging 
approaches  to  the  assessment  of  territorial  and  urban  strategies  launched  under  the 
territotial  provisions  set  out  for  2014-2020.  Looking  forward,  Section  5  outlines 
options  for  developing  frameworks  for  future  assessments.  Section  6  presents 

1 European Commission (2014) Guidance for Member States on Integrated Sustainable Urban 
Development (Article 7 ERDF Regulation), 
http://ec.europa.eu/regional_policy/sources/docgener/informat/2014/guidance_sustainable_urb
an_development_en.pdf (downloaded 10/01/18) 
2  European  Commission 
Investments, 
(2015)  Scenarios 
http://ec.europa.eu/regional_policy/sources/docoffic/official/reports/pdf/iti_en.pdf  (downloaded 
10/01/18). 

Integrated  Territorial 

for 

3 

 
 
                                           
Assessing the Performance of Integrated Territorial Development Strategies 

conclusions  and  recommendations  on  a  framework  and  approach  for  measuring 
effectiveness.  

4 

 
 
Assessing the Performance of Integrated Territorial Development Strategies 

2.  MEASURING 

THE  EFFECTIVENESS  OF  URBAN  AND  TERRITORIAL 

STRATEGIES: KEY CONSIDERATIONS 

2.1  Generic challenges 

The  importance  of  assessing  the  impact  and  results  of  public  policy  interventions  is 
clear.  It  is  expected  that  integrated  place-based  strategies  and  their  interventions 
should  be  developed  with  results  in  mind.  Policy-makers,  stakeholders  and 
beneficiaries  require  knowledge  and  understanding  of  the  progress  and  effectiveness 
of the strategies; transparency and accountability have to be ensured. It is important 
to gather information with which to better understand the role of the interventions and 
an  evidence  base  comparable  to,  and  where  possible  compatible  with,  other 
interventions. 

The  most  appropriate  methodology  for  measuring  the  effectiveness  of  the  new 
territorial provisions depends on a wide range of factors, including the scale and scope 
of  the  approach  and  the  quality  and  quantity  of  the  data  available.  Moreover, 
territorial  approaches  are  by  their  very  nature  highly  shaped  by  context.  They  are 
diverse  in  terms  of  their  thematic  content  and  geographic  scales,  e.g.  covering 
selected parts of an urban area, major agglomerations, small-to-medium-sized towns, 
wider areas including urban areas and their surroundings. As such, important issues to 
consider in developing an assessment approach include the following.  

  The impact of diversity on the availability and comparability of statistical data, 
e.g.  it  can  be  challenging  to  set  common  ‘benchmarks’  against  which  the 
achievements  of  strategies  can  be  compared.  In  some  contexts  the 
development and adoption of an integrated place-based  strategy is a  valuable 
result  in  itself  and  represents  added  value;  in  others,  the  approach  is  simply 
building on established practices and should be viewed as a means to  achieve 
additional results.  
Identifying the results of the strategies is recognised as complex in the context 
of  multiple,  more  dominant  and  complexly  interwoven  determinants  of 
economic  growth.3  Integrated  place-based  strategies  cover  potentially  broad 
themes with comparatively small amounts of money.  

 

  Approaches  to  strategy  management  and  implementation  vary:  in  some 
Member  States,  strategies  are  embedded  in  existing  urban  development 
arrangements  or  wider  interventions,  which  may  make  it  difficult  to  directly 
attribute achievements to the integrated place-based strategies alone. 

  The  information  required  to  assess  the  effectiveness  of  a  strategy  in  a  given 
territory is likely to be different from the information required by policy-makers 
at the regional, national and/or European levels.  

  How can the softer ‘added value’ objective of these initiatives be assessed? For 
instance,  how  can  potential  contributions  to  strengthening  human  and  social 
capital  among  the  actors  and  networks  involved  in  integrated  place-based 
strategies be incorporated in evaluating effectiveness?  

2.2 

Specific challenges 

There are specific challenges and complexities that are inevitable given the particular 
characteristics of these territorial and urban strategies.  

  First,  there  are  implications  of  measuring  initiatives  that  can  cover  multiple  ESI 
funds.  Strategies  may  be  well  integrated  internally,  but  for  the  purpose  of 
measuring  the  contribution  to  particular  OPs,  strategy  activities  must  also  be  put 

3 OECD (2007) 2007 Economic Review - European Union. Paris: Organisation for Economic 
Cooperation and Development, Paris, p.129. 

5 

 
 
                                           
Assessing the Performance of Integrated Territorial Development Strategies 

into  the  architecture  of  monitoring  systems  that  are  often  designed  around  the 
requirements of separate funds (ERDF and ESF). 

  Second,  there  is  the  challenge  of  combining  measurement  of  impacts  under  a 
specific, limited number of policy themes with assessments of integrated territorial 
results. Consequently, the strategy's integrated effects might be virtually lost due 
to the obligation to display particular (thematic) contributions separately. 

  Third,  it  should  be  noted  that  these  strategies  have  only  been  in  operation  for  a 
short  period  of  time.  Operations  are  still  being  launched  and  in  some  Member 
States  this  approach  to  territorial  development  is  being  undertaken  for  the  first 
time. 

2.3 Strong variation in what has to be measured 

Significant  variation  across  territorial  strategies  must  also  be  taken  into  account  in 
developing  methodologies  to  assess  the  acheievements  of  territorial  and  urban 
strategies.  The  Final  Report4  accompanying  this  paper  has  noted  this  variation  and it 
has implications for approaches to measuring effectiveness. 

  Size  of  Funding.  The  approach  taken  to  strategy  assessment  obviously 
depends on the level of funding of the investment: the focus on assessment is 
strong where larger investments are involved and there is likely to be a larger 
budget  for  more  sophisticated  or  rigorous  methodologies.  At  Member  State 
level, there is variation in the allocation of resources to these investments, but 
case  study  research  illustrates  the  wide  range  of  individual  strategy  budgets 
that have been set.  In  a selection of 30 case  studies from across the  Member 
States,  the  average  ITI  budget  was  €164  million.  However,  this  includes  a 
range  of  funding  levels  from  less  than  €1  million  (in  Vejle,  Denmark),  up  to 
€100  million  in  several  cases  (e.g.  Tartu,  Limburg,  Zagreb,  Lille),  and  very 
large  investment  budgets  (e.g.  €793  million  in  Katowice,  €902  million  for  ITI 
Azul in Spain, €1.1 billion for the Danube Delta ITI in Romania).   

  Thematic  Content.  The  thematic  content  of  strategies  also  has  clear 
implications  for  evaluation,  relating  to  the  demands  of  evaluating  particular 
policy fields and also to the number of fields included in integrated place-based 
strategies.  For  instance,  thematic  evaluation  such  as  innovation  or  equal 
opportunities  could  be  feasible  where  the  strategy  has  a  limited  number  of 
thematic  objectives  and  investment  priorities.  Strategies  are  linked  to  the 
thematic  objectives  (TO)  laid  down  in  the  CPR  and  the  associated  investment 
priorities  (IP).  The  most  commonly  included  themes  in  the  strategies  concern 
the low-carbon economy, environmental protection and resource efficiency, and 
promoting  social  inclusion.  However,  there  is  significant  variation  in  the 
thematic  scope  of  the  strategies.  Although  there  is  strong  variation  within 
Member  States,  strategies  in  Poland,  Cyprus,  Italy,  Lithuania,  Latvia,  Poland 
and Hungary have a  relatively high number of  IPs. The Azul strategy in Spain 
has  over  50  IPs  whereas  in  Denmark,  Estonia,  Ireland,  Slovenia  and  Sweden, 
the strategies include only two or three IPs. 

  Territorial  Coverage.  Whatever  approach  to  evaluation  is  taken,  a  key 
performance  measurement  concern  is  the  spatial  area  within  which  impacts 
should  be  assessed.  This  is  crucial  for  strategy  operations  which  by  definition 
have a territorial focus, and therefore the spatial dimension takes on a central 

4 van der Zwet A, Bachtler J, Ferry M, McMaster I and Miller S (2017) Strategies for integrated 
development: how are ESIF adding value in 2014-20? Final Report to the European Commission 
(DG Regio), European Policies Research Centre, University of Strathclyde, Glasgow. 

6 

 
 
 
                                           
Assessing the Performance of Integrated Territorial Development Strategies 

for  analysis  has 

relevance  in  judging  their  performance.5  A  key  issue  is  reconciling  different 
geographies  and  associated  datasets  for  the  measurement  of  effectiveness 
across district, municipality and city boundaries, and the geographical coverage 
of domestic and EU interventions.  The selection of the most appropriate spatial 
impact,  particularly  the 
implications  measuring 
area 
assessment  of  additionality,  potential  displacement  effects  (i.e.  how  much  of 
the  outcome  displaced  other  outcomes  outside  of  the  target  area),  leakage  of 
outputs  or  outcomes  that  benefit  those  outside  the  target  spatial  area  and 
multiplier  effects.  For  instance,  in  Nicosia’s  strategy,  results  indicators  have 
been specialised based on the need to measure the wider results of the actions 
implemented  beyond  the  direct  results  recorded  in  the  intervention  area  of 
each municipality.  

The  areas  covered  by  the  case  studies  also  vary  considerably,  again  with 
implications  for  the  methodological  approach  to  assessing  achievements.    As 
noted  in  the  Second  Interim  Report  of  this  study,  the  case  studies  include 
various  territorial  configurations,  based  on  statistical  analysis  or  predefined 
criteria,  administrative  status,  economic  functions,  societal  challenges,  etc. 
These  include:  networks  of  cities  (e.g.  examples  in  CZ  and  FI);  functional 
urban  areas  mainly  found  in  Eastern  Europe  (e.g.  PL  and  CZ,  SK);  umbrella 
municipal organisations (e.g. PL and FR) ; crossing administrative boundaries ; 
and specific areas in the cities (e.g. The Hague). In some cases, certain priority 
themes  (often  social  inclusion)  are  targeted  at  specific  areas  whereas  others 
cover the whole city.  

  Role in Cohesion policy architecture. Again, there is significant variation in 
the way in which the case study strategies fit into the broader Cohesion policy 
programme architecture (see Table 1).  

o  There  are  strategies  that  are  covered  by  a  small  number  of  projects  in 
an  Operational  Programme  (e.g.  the  Vejle  strategy  is  seen  as  a 
framework strategy for the two projects it supports) or are contained in 
a  single  measure  within  a  programme  Priority  (e.g.  Tartu  strategy  is 
funded under the measure ‘Sustainable development of urban areas’  of 
Priority 9 of the OP). Here, it is relatively straightforward to monitor and 
assess  achievements  against  project  indicators,  but  what  about 
effectiveness and impact?   

o  Several  integrated  place-based  strategies  are  covered  in  a  single 
programme  priority,  often  related  to  sustainable  urban  development 
(e.g. Plovdiv, Nicosia, Centre-France-Comte).  

o  Other  case  study  strategies  cover  several  priorities  within  an  OP  (e.g. 
The Hague combines funding from three priority axes in the OP). Here, 
there  is  scope  to  incorporate  strategy  assessment  as  part  of  the 
thematic evaluation of OP priorities required by the Commission.  

o 

In one case, Brussels, the OP is seen as the strategy. Again, assessment 
of  the  strategy  can  be  part  of  individual  thematic  evaluations  and/or 
form  a  section  of  the  interim  and  ex-post  evaluations  of  the  OP  as  a 
whole. 

o  Finally,  there  are  strategies  that  are  supported  by  multiple  OPs  (e.g. 
Zagreb  is  funded  by  2  OPs).  In  such  cases,  there  is  the  possibility  to 

5 European Commission and European Investment Bank (2013) Methodologies for Assessing 
Social and Economic Performance in JESSICA. Final Report, December 2013, p8. 

7 

                                           
 
Assessing the Performance of Integrated Territorial Development Strategies 

address  broader  evaluation  questions  in  geographic  or  thematic  terms. 
However,  there  are  specific  evaluation  challenges  related  to  timing  and 
coordination  across  programmes  that  are  complicated  further  when 
these involve different funds which each have their own regulations for 
monitoring  and  evaluation  (e.g.  5  OPs  contribute  to  the  Brno  strategy 
and  three  ESI  funds  are  involved  –  ERDF,  ESF,  CF)  or  different  CP 
categories  of  region  (e.g.  more-developed  regions  and  less-developed 
regions in Prague).  

Table  1 :  Position  of  integrated  place-based  strategies  in  the  architecture  of 
Cohesion policy 

‘Project’ 

Projects 

A small number of 
projects in an 
Operational 
Programme 

Vejle  

Tartu  

‘Programme’ 

Programme 
Priority 

Single programme 
priority, often related 
to sustainable urban 
development ; or 
several priorities 
within an OP. 

Plovdiv, Nicosia, 
Centre-France-Comte 

Operational 
Programme  

OP is seen as the ITI 

‘Multiple’ 

Multiple OPs 

Strategies that are 
supported by multiple 
programmes 

Brussels 

Zagreb, Brno, Prague 

Straightforward to 
monitor and assess 
achievements against 
project indicators but 
what about 
effectiveness, impact? 

Scope to incorporate 
strategy assessment 
as part of the 
thematic evaluation of 
OP priorities required 
by the Commission. 

Part of individual 
thematic evaluations 
or part of interim and 
ex-post evaluations of 
the OP as a whole. 

Address broader 
evaluation questions 
in geographic or 
thematic terms. 

  Relationship  with  domestic  interventions.  It  is  also  important  to  note 
variation  in  the  relationship  between  the  ITI  and  domestic  strategies  and 
interventions.  

o 

In western and northern parts of Europe, the introduction  of integrated 
urban  approaches  is  not  novel  and  represents  a  continuation  of 
domestic  practices  albeit  with  some  changes.  Several  case  study 
strategies build on or are embedded in existing domestic initiatives (e.g. 
ITI  Limburg,  SUD  Nordhausen,  SUD  Lille).  From  an  evaluation 
perspective,  the  benefits  of  having  a  strategy  embedded  in  a  broader 
domestic  strategy  are  apparent  in  the  availability  of  a  set  of  dedicated 
indicators  and  datasets.  Moreover,  there  is  often  strong  capacity  and 
experience in the evaluation of territorially integrated initiatives. On the 
other  hand,  there  are  issues  with  reconciling  the  differing  demands  of 
domestic and EU dimensions in terms of the timing of reports, the type 
of data required, and in some cases geographical coverage. How do you 
disaggregate CP input into the achievements of domestic interventions?  

8 

 
Assessing the Performance of Integrated Territorial Development Strategies 

o  Elsewhere,  particularly  in  Central  and  Eastern  Europe,  integrated 
territorial initiatives represent a new approach to strategic planning and 
investment  (e.g.  Kaunas,  Katowice,  Brno,  Prague).  In  these  cases,  a 
benefit  for  evaluation  is  that  the  disaggregation  of  CP  achievements 
from domestic interventions and assessments of added value should be 
clearer.  On  the  other  hand,  ITIs  involve  new  ways  of  working,  with 
multiple goals relating to behaviour and outcomes, and there are issues 
concerning the limited capacity and experience in evaluating these new 
approaches, the availability and quality of data, etc. 

2.4 

Capturing added value 

Another challenge is how the ‘added value’ of  these initiatives be assessed: what are 
the  additional  benefits  of  taking  a  territorially 
integrated  approach  to  the 
implementation  of  ESI  Funds.  Figure  2  sets  out  some  broad  headings,  based  on 
analysis of the strategies. 

Figure 2: Potential added value of territorial and urban strategies  

 

In terms of funding and resources there is potential for increased awareness of 
opportunities  to  pool  different  funds,  to  draw  in  new  resources  and  create 
multiplier effects in the territory; examples are the formation of links with the 
private sector that can facilitate private funding for specific, innovative types of 
actions.  In  Vejle  SUD,  the  expected  added  value  is  that  it  will  help  to  build  a 
common  basis  for  public-private  partnership  and  in  so  doing  strengthen 
cooperation on sustainable urban development. 

  One main aspect of added value of SUD and non-SUD ITI is improved visibility 
and integrity of territories, creating stronger functional coherence and linkages 
that can be potentially durable beyond projects. Compared to ‘simple’ projects, 
SUD  and  non-SUD  ITI  gives  territories  the  opportunity  to  exchange  knowlege 
and  promote  learning.  The  development  and  implementation  of  integrated 
development  strategies  allows  stakeholders  to  identify  and  achieve  common 
objectives through multiple actions in various sectors. 

9 

 
 
 
 
 
 
 
Assessing the Performance of Integrated Territorial Development Strategies 

  Potential  added  value  is  recognised  in  developing  policy  governance  and 
building  capacity.  Territorial  provisions  can  address  inefficiencies  caused  by 
fragmentation.  The  involvement  of  local  authorities  in  the  design  and 
implementation  of  strategies  is  credited  by  implementing  authorities  with 
creating  the  potential  for  building  trust  and  reducing  rivalry,  competition  and 
duplication of projects.   

  SUD  and  non-SUD  strategies  can  also  increase  administrative  capacity.  In 
several  cases,  the  development  of  SUD  and  non-SUD  ITI  arrangements  are 
associated  with  the  establishment  of  bodies  at  the  local  level  to  fulfil 
implementation tasks.  Increasing the role of local authorities, NGOs and other 
sub-national bodies involved in managing and implementing ESI Funds can, in 
the  longer  term,  help  to  strengthen  capacities  for  implementing  territorial 
development.   

It is important to note that the realisation of this added value is influenced by Member 
State and EU institutional and regulatory contexts and the scale of funding allocations. 
Some  areas  have  long-established  traditions  of  working  with  integrated  urban 
strategies and have limited ESI funding allocations. In these cases, the ‘distinct’ added 
value may be limited and mainly confined to reinforcing existing trends. Conversely, in 
areas  with  large  ESI  allocations  and  weaker  traditions  of  integrated  urban 
development, levels of added value could be significant. 

2.5  Addressing different audiences 

Each  strategy  must  consider  the  different  arenas  in  which  the  knowledge  will  be 
disseminated. This includes audiences from EU to local levels, and within this there is 
varied  interest  in  strategies  depending  on  scale  and  content.  In  establishing  an 
assessment  approach,  it  is  important  to  consider  who  the  information  is  targeting. 
Different stakeholders will benefit from different types of information, with evaluative 
knowledge  on,  for  example,  policy  synergies,  of  particular  note  for  national  and  EU-
level stakeholders, and  more  operational knowledge, which is of value to project and 
programme  actors.  In  addition,  perceptions  of  programme  and  project  achievements 
vary  depending  on  who  is  asked,  with  views/interpretations  varying  between  EU, 
national, regional and local levels (see Figure 3).  

Figure 3 : Interpretations of achievements 

EU 

•reinforcing 'urban dimension' in Cohesion policy  
•horizontal integration, across policy areas 
•vertical integration, across levels of government 
•contribution to Europe 2020 strategy 

National 

•synergies 
•avoidance of duplication 
•opportunity to achieve economic of scale clustering projects  

Regional & Local 

•benefits and drawbacks of delegated reponsibilities 
•build community capacity 
•development based on local assets 
•stimulate innovation  
•improved capacity for change 

10 

 
 
 
 
 
 
 
Assessing the Performance of Integrated Territorial Development Strategies 

Despite this broad variation under various headings, it is possible to identify common 
themes that the strategy assessments ideally should cover. 

  Assessments  of 

integration, 

and 
implementation  responsibilities  of  institutions  at  different  levels,  and  in 
different policy fields; different types of stakeholders that are integrated in the 
strategy;  the  extent  to  which  resources  /  funding  are  pooled,  for  instance 
taking  into  account  the  contribution  of  different  programmes,  EU  funds, 
domestic resources; the scope and combination of the instruments involved. 

the  management 

concerning 

  Assessments  of  territoriality,  concerning  the  varied  spatial  scales  at  which 
the  instruments  are  implemented;,  potentially  effects  outside  the  territory 
covered  by  the  strategy;  and  citizen/stakeholder  involvement  in  design  and 
implementation. 

  Assessment  of  achievements,  concerning  performance  of  the  strategy  at 
project,  OP,  national  and  European  levels,  with  evaluation  according  to 
different dimensions: ‘softer’ and ‘harder’ results; long- and short-term results, 
efficiency  (i.e.  timely  implementation  of  projects)  and  utility  (completion  of 
projects with high strategic quality).  

11 

 
 
 
 
 
Assessing the Performance of Integrated Territorial Development Strategies 

12 

 
 
Assessing the Performance of Integrated Territorial Development Strategies 

3.  EXISTING METHODOLOGIES  

3.1 

Evolution of Cohesion policy evaluation  

Measuring the effectiveness of integrated place-based strategies takes place within the 
context  of  Cohesion  policy,  which  has  a  well-established  monitoring  and  evaluation 
framework.  For 
results-oriented 
the  2014-20  Cohesion  policy  programmes, 
programming  aims  to  ensure  that  programmes  have  clearly  specified  objectives,  a 
for  effective 
strong 
implementation  of  the  Funds,  and  clear  and  measurable  milestones  and  targets  to 
ensure progress is made. This approach sets the framework for evaluation of Cohesion 
policy interventions (see Figure 4).   

logic,  appropriate  conditionality  provisions 

intervention 

Figure 4: Intervention logic 

Source :  European  Commission  (2014)  Guidance  document  on  monitoring  and  evaluation, 
European  Cohesion  Fund  and  European  Regional  Development  Fund,  concepts  and 
recommendations, March 2014. 
http://ec.europa.eu/regional_policy/sources/docoffic/2014/working/wd_2014_en.pdf 
(downloaded 10/01/18). 

These efforts are in line with the EC ‘Better Regulation’ (see Figure 5), which seeks to 
ensure that:   

  decision-making is open and transparent;  
 

citizens and stakeholders can contribute throughout the policy and law‑ making 
process;  

  EU actions are based on evidence and understanding of the impacts; and 
 

regulatory burdens on businesses, citizens or public administrations are kept to 
a minimum. 

13 

 
 
 
 
Assessing the Performance of Integrated Territorial Development Strategies 

Figure 5: Steps taken towards better regulation 

Strengthening 
preparation 

Increasting 
copperation 
between 
institutions 

Improving 
consultation 

Ensuring quality 

Making sure laws 
and regulations 
are fit for purpose 

Source:  Based  on  European  Commission  web-site  Better  regulation :  why  and  how 
https://ec.europa.eu/info/law/law-making-process/better-regulation-why-and-how_en 
(downloaded 10/01/18). 

In  seeking  improved  targeting  and  capture  of  the  role  and  impact  of  policy 
interventions,  the  ‘results-orientation’  has  become  embedded  in  European  policy-
making  and  interventions.  In  this  context,  the  main  tasks  in  relation  to  evaluation 
during the programme period (Art. 54 and Art. 56 CPR)6 include the following.7 

  Drafting  Evaluation  Plans  (for  each  programme  or  for  more  than  one 
programme),  to  be  submitted  for  approval  to  the  Programme  Monitoring 
Committee  (PMC)  within  one  year  from  the  adoption  of  the  programme  (Arts. 
114.1 and 110.2 CPR).  The content and suggested structure of the Evaluation 
Plans  are  illustrated  in  detail  in  the  European  Commission’s  ‘Guidance 
Document  on  Evaluation  Plans’.8  The  Plans  can  be  reviewed  and  amended  by 
the PMG during the entire lifecycle of programmes.  

 

Implementing  the  evaluations  foreseen  in  the  Evaluation  Plans.  These 
evaluations can be of at least two types: 

o 

implementation  evaluations,  focused  on  how  a  programme  is  being 
managed and delivered; 

6 Ex ante evaluation falls outside the scope of this paper. 
7 Polverari L (2015) ‘The monitoring and evaluation of the 2014-20 Cohesion policy 
programmes’ IQ-Net Thematic Paper 36(2), European Policies Research Centre, University of 
Strathclyde, Glasgow. 
8 European Commission (2015) Guidance document on Evaluation Plans. Terms of Reference for 
Impact Evaluations. Guidance on Quality Management of External Evaluation, February 2015 
available online at : 
https://www.portugal2020.pt/Portal2020/Media/Default/Docs/AVALIACAO/Evaluation_plan_guid
ance_en%20Fev2015.pdf  (downloaded 10/01/18). 

14 

 
 
                                           
Assessing the Performance of Integrated Territorial Development Strategies 

o 

impact evaluations assessing the effectiveness, efficiency and impact of 
programmes ; these should be carried out so as to ensure that ‘at least 
once  during  the  programming  period,  an  evaluation  shall  assess  how 
support  from  the  ESI  Funds  has  contributed  to  the  objectives  of  each 
priority’ (Art. 56.3 CPR). 

The intervention logic applicable to Cohesion policy programmes extends to integrated 
place-based  strategies  and  the  result  orientation  also  is  a  key  principle.  Currently, 
arrangements for measuring the effectiveness of integrated place-based strategies are 
linked  to  common  output  indicators.  Guidance  on  evaluation 
integrated 
programmes is also relevant to integrated strategies.  

for 

There  is  a  range  of  methodologies  that  can  be  applied  to  the  assessment  of  these 
strategies.  These  vary  in  terms  of  the  data  sources  drawn  on,  data-gathering 
techniques,  and  the  role  of  stakeholders  in  the  process.  Each  approach  has  its  own 
benefits and challenges (see Table 2). 

The  Commission  guidance,  for  example,  states  that  the  evaluation  of  integrated 
approaches could benefit from:9  

  evaluating  different  components  of  integrated  programmes  separately  to 
establish their effectiveness and subsequently evaluate a combination of two or 
more interventions;  

  applying theory-based evaluation that assesses the intervention logic and how 
the  different  components  in  an  integrated  strategy  fit  with  each  other  and 
whether synergies are likely to occur; and  

 

in cases where interventions are large scale, methods such as macro-economic 
modelling or counterfactual approaches may be appropriate.  

Alongside these evaluation methodologies, it is also important to note specific impact 
analysis  tools  and  techniques  that  can  be  of  practical  value  in  assessments  of  the 
effectiveness of integrated territorial investments. These include the following.10 

  Territorial  Impact  Analysis,  which  produces  a  logical  framework  for 
assessing the territorial consequences of policy, providing insights into complex 
development  processes.  Many  analytical  methods  for  impact  analysis  exist, 
including econometric modelling and input-output models. Often they assume a 
level  of  precision,  focusing  on  the  use  of  metric  data,  but  non-metric 
measurement  of  data  can  also  be  applied.  Surveys  of  beneficiaries  and 
stakeholders  can  gauge  the  impact  of  strategies  under  different  thematic 
headings,  such  as  the  experience  of  business  occupiers  of  redeveloped  sites 
and  the  quality  of  life  of  inhabitants  of  specific  territories  covered  by  the 
strategy.  For  instance,  it  is  possible  to  survey  inhabitants  of  the  territory 
covered  by  the  strategy  according  to  quality-of-life  criteria  in  order  to  set 
baselines  and  subsequently  measure  impact.  In  Centre-France-Comte,  a  3-
month,  survey-based  study  was  produced  on  inhabitants’  quality  of  life. 
Indicators  and  targets  were  then  developed  on  the  basis  of  inhabitants’ 
expectations.  Of  course,  there  are  still  challenges  in  gauging  contribution  and 
attribution: even if perception surveys show that people  feel that their quality 
of life has improved, how can it be vertified that it was down to the strategy?  

  Cost-effectiveness  analysis,  which  relies  on  the  development  of  metrics 
(such  as  cost  per  job,  cost  per  area  or  dwelling  regenerated,  etc.)  to  allow 

9 European Commission (2014) op. cit. 
10 European Commission and European Investment Bank (2013) op. cit. 

15 

                                           
Assessing the Performance of Integrated Territorial Development Strategies 

comparison  of  policy  scenarios.  In  some  Member  States,  such  techniques  are 
important  parts  of  domestic  policy  analyses  and  integrated  place-based 
strategies may be able to draw on them for assessments.   

It is important to note that for aggregating the results of these impact analyses, there 
must  be  EU-level  guidance  and  work  on  establishing  cost-effectiveness  benchmarks 
for integrated territorial development and indicators such as quality of life that can be 
applied across different territories covered by the strategy.  

Table 2: Methodological approaches for evaluating the effectiveness of 
integrated place-based strategies 
Design 
approaches 

Description 

Challenges 

Basis  for  the 
causal 
inference 

Type of 
information 
produced 

Experimental  Counterfactuals; 
the co-presence 
of cause and 
effects. 

Top down 
analysis using 
surveys, 
statistics. 

Theory-
based 

Identification/ 
confirmation of 
causal chains 

Process –
tracing based 
on literature 
review, 
monitoring 
data, 
interviews, 
focus groups 
etc. 

Case-based 

Comparison 
across and 
within cases of 
combinations of 
causal factors. 

Meta-
evaluation 

Accumulation 
and aggregation 
of results of 
multiple studies. 

Finding a robust 
control group 
requires 
comprehensive, 
good quality data. 

Risks simplifying 
reality; may 
exclude issues 
that cannot be 
expressed in 
theories; 
challenge to keep 
pace with new 
theory models. 

Limited scope to 
test external 
validity and 
generalise 
findings. 

Relies on the 
quality of studies 
included. No new 
data is produced, 
difficult to 
conduct (scope to 
incorporate a 
range of biases). 

Quantitative 
estimate of 
impact by 
identifying 
‘policy off’ 
position. 

Explains why 
an instrument 
does (not) 
work. 

Focuses on 
understanding 
of context-
specific 
variables that 
explain 
causality. 

Tests or 
confirms 
results, can 
combine 
findings to 
estimate 
typical 
impacts, can 
capture 
impacts not 
picked up by 
smaller 
studies 

Source: based on M. Ferry and S. Davies (2015), ‘Impact evaluation of regional policy business 
support:  what  is  the  evidence?’  EoRPA  Paper  15/6,  European  Policies  Research  Centre, 
University of Strathclyde, Glasgow. 

16 

 
 
 
 
Assessing the Performance of Integrated Territorial Development Strategies 

3.2 

Evaluation of other urban development initiatives 

The development of a methodology for measuring the effectiveness of territorial tools 
in  Cohesion  policy  does  not  occur  in  a  vacuum.  The  literature  on  urban  development 
evaluation  provides  various  examples  of  established  practices  and  many  urban  areas 
in Europe have years of experience in terms of using these methodologies.  

 provides  some  examples  of  methodologies  that  have  been  developed  in  relation  to 
integrated territorial approaches.  

Table  3:  Review  of  methodologies  for  assessment  of  integrated  urban 
approaches 

Territorial Assessment 
Colantonio and Dixon 
(2009)11 

European Commission 
(2015)12 

URBACT (2013)13 

URBACT (2015)14   

Key elements 

Develops methods for measuring social sustainability, 
including measures for softer aspects of social 
sustainability, such as well-being, happiness and 
neighbourhood satisfactions. Stresses importance of 
linking local projects to city-wide development plans.  

Provides local government actors and stakeholders 
with a concise guide to the best currently available 
indicator tools for sustainable cities, focusing on the 
environmental dimension. 

Provides practical tools for engaging with local 
stakeholders, developing action plans and building 
capacity. Includes a self-assessment to measure 
progress with the Local Action Plan which specifically 
includes elements that measure the integrated 
approach (i.e. balance from an economic, social and 
environmental point of view). 

Emphasises a learning approach to evaluation, 
involving iterative and cyclical measurement and 
analysis of results as well as processes. Stresses the 

11 Colantonio, A. & Dixon, T. (2009) Measuring Socially Sustainable Urban Regeneration in 
Europe, Final Report. Oxford Institute for Sustainable Development (OISD), Oxford Brookes 
University, Oxford.  
12 European Commission (2015) Science for Environment Policy, In-depth Report: Indicators for 
sustainable urban development November 2015 Issue 12. 
http://ec.europa.eu/environment/integration/research/newsalert/pdf/indicators_for_sustainable
_cities_IR12_en.pdf (downloaded 10/01/18). 
13 URBACT (2013) The URBACT II Local Support Group Toolkit, June 2013. 
http://urbact.eu/sites/default/files/urbact_toolkit_online_4.pdf (downloaded 10/01/18). 
14 URBACT (2015) New Concepts and Tools for Sustainable Urban Development in 2014 – 2020, 
Synthesis Report September 2015. 

http://urbact.eu/sites/default/files/synthesis_report_urbact_study.pdf (downloaded 10/01/18). 

17 

                                           
 
 
 
 
Assessing the Performance of Integrated Territorial Development Strategies 

Dawson et al. (2013)15 

Xing et al. (2009)16  

EIB (2013)17  

need for capacity building at the local level.  

Explores the challenges and opportunities of urban 
integrated assessment through four perspectives: 
quantified integrated assessment modelling, climate 
change adaptation and mitigation; green and blue 
infrastructure; and urban policy and governance. 

Reports on the Urban Development Sustainability 
Model (UD-SAM) which considers sustainability of 
cities within their broader geographical context.  

Develops an overall performance measurement 
framework that allows measurement of Jessica-type 
financial instruments introduced in an urban context 
with the aim of achieving a coherent approach to 
assessing their non-financial performance and 
improving their effectiveness.  

15 Dawson RJ, Wyckmans A, Heidrich O, Köhler J, Dobson S and Feliu E (2014) Understanding 
Cities: Advances in integrated assessment of urban sustainability, Final Report of COST Action 
TU0902, Centre for Earth Systems Engineering Research (CESER), Newcastle, UK. ISBN 978-0-
9928437-0-0 
16 Yangyany X, Horner R, El Haram M and Bebington J (2009) A framework model for assessing 
sustainability impacts of urban development, Accounting Forum 33(3): 209-224. 
17 European Commission and European Investment Bank (2013) op. cit. 

18 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
                                           
 
 
 
Assessing the Performance of Integrated Territorial Development Strategies 

4.  EMERGING APPROACHES TO ASSESSMENT IN TERRITORIAL STRATEGIES 

Integrated  urban  and  territorial  development  strategies  are  developing  different 
monitoring  systems,  indicators  sets  and  assessment  strategies  for  implementing  the 
2014-2020  territorial  provisions.  The  following  points  are  all  raised  in  the  main  Final 
Report,18  but  are  worth  reiterating  with  reference  to  the  development  of  a 
methodology for measuring effectiveness.  

4.1  Monitoring systems  

A  common  approach  is  for  strategy  monitoring  to  take  place  as  part  of  the 
broader  OP  system.  Thus,  implementing  authorities  apply  programme  systems  to 
monitor project progress against selected programme indicators and targets.  

Most  strategies  also  have  arrangements  for  monitoring  the  progress  of  the 
strategy as a whole. Examples include: 

  Tartu  SUD:  projects  selected  by  the  local  urban  authorities  include  specific 
quantitative  outcomes  for  each  project.  The  results  of  these  projects  are  fed 
into the monitoring system of the OP priority axis. 

  Finnish  Six  Cities  SUD:  the  management  group  and  the  IB  are  collectively 
responsible  for  monitoring  the  strategy.  Meetings  are  organised  every  six 
months to discuss the progress of funding and other indicator data.  

  Kaunas :  the  City  Municipal  Administration  is  responsible  for  strategy 
monitoring, and the overall process is overseen by the Ministry of the Interior.  

Urban  and  local  authorities  are  developing  monitoring  arrangements  to 
ensure that progress can be assessed not just in terms of the contribution to 
programme  targets,  but  also  in  relation  to  the  territory  covered.  In  some 
instances,  this  has  included  the  establishment  of  new  monitoring  structures  or 
capacities. 

  Timisoara  SUD:  local  authorities  can  choose  to  set  up  an  autonomous 
monitoring  system  beyond  those  in  place  for  Cohesion  policy.    The  strategy 
sets out indicators, which are developed not just to be in line with the relevant 
OP, but also to measure results and impacts at the territorial level. This system 
is  being  completely  managed  at  local  level  and  is  independent  from  the 
monitoring of the funding OP. 

  Vienna:  considerable  emphasis  is  placed  on  monitoring  the  strategy’s 
implementation  progress.  The  Article  7  element  will  be  monitored  in  the 
context  of  the  ERDF  OP  monitoring.  However,  in  addition,  a  scoping  project 
‘SMART.MONITOR’  was  carried  out  between  September  2015  and  October 
2016.  The  project  was  funded  by  the  Austrian  Ministry  for  Transport, 
Innovation  and  Technology  and  involved  external  partners  experienced  in 
monitoring.  The 
in  October  2016  provided 
recommendations for the development of the monitoring process in practice. 

report  published 

final 

In some cases, the progress of the strategy is monitored through established 
systems  used  for  the  overarching  domestic  strategies  to  which  they 
contribute. Even where strategies are not nested in a wider domestic framework, in 
some  cases,  there  are  plans  to  draw  on  broader  monitoring  systems  to  assess 
progress within the specific territorial context. 

18 Van der Zwet et al (2017) op. cit. 

19 

                                           
Assessing the Performance of Integrated Territorial Development Strategies 

  Lille  SUD:  results  will  be  measured  in  terms  of  their  contribution  to  the  city 
contract, not as part of the OP. Monitoring and evaluation (through indicators) 
will  be  based  on  relevant  ERDF  IPs  applied  to  the  ITI.  However,  impact 
evaluations are planned and are expected to focus on the impact of the ITI on 
the  city  contract,  and  its  added  value  (in  other  words,  the  leverage  effect  of 
ERDF). 

  Cornwall  and  Isles  of  Scilly  regional  ITI:  an  ITI  Board  has  been  established, 
comprising  local  partners  and  representatives  from  the  voluntary,  public  and 
private sectors. The ITI Board is also responsible for project monitoring (as the 
Operational Programme Board does not oversee individual projects unless they 
are  of  significant  size).  Monitoring  outputs  and  impact  of  the  ITI  will  also  be 
achieved  through  the  use  of  Cornwall  Council’s  broader  economic  and  social 
indicator  data,  such  as  GVA  data,  which  –  it  is  anticipated  –  will  help  to 
understand  better  the  effectiveness  and  added  value  of  the  ITI  as  its  delivery 
progresses. 

4.2 

Indicator sets 

The OP indicator sets a basic source for measuring the effectiveness of strategies. Of 
course,  all  strategies  with  EU  funding  must  report  progress  against  indicators  in  the 
source  OPs.  However,  the  relationship  between  OP  and  integrated  place-based 
strategy  indicators  varies,  depending  on  their  size,  content  and  implementation 
arrangements.  

As  has  been  noted,  many  strategies  base  their  indicator  sets  entirely  on  OP 
indicators.  In  effect,  monitoring  of  the  strategy  is  integral  to  the  monitoring  of  the 
OP.  

On  the  other  hand,  there  are  cases  where  additional,  strategy-specific  indicators  are 
added to OP indicators. 

  Nordhausen SUD : the output and result indicators  of the ERDF OP will be the 
main indicators  used  to  assess  the  effectiveness  of  interventions  implemented 
under  the  strategy.  The  selection  of  the  indicators  included  in  the  OP  was  a 
consultative  process,  also  involving  the  municipalities,  which  aimed  to  identify 
a  small  number  of  workable  and  effective  indicators.  Experience  from  the 
previous  programme  period  showed  that  the  inclusion  of  a  large  number  of 
indicators  resulted  in  the  process  becoming  unworkable  and  not  necessarily 
accurate in assessing effectiveness.  

  Katowice  (Central  Sub-region)  SUD:  monitoring  and  evaluation  are  based  on 
broader arrangements for the regional OP. However, there are also specific ITI-
related ‘strategic’ indicators, linked to each priority and measure and aiming to 
measure  effectiveness  in  the  specific  territory  covered  by  the  ITI  (e.g. 
percentage  of  the  population  covered  by  ITI).  This  has  produced  a  large 
indicator set. 

For  some  strategies,  only  quantitative  indicators  are  used.  In  others, 
qualitative  indicators  were  found  to  be  a  valuable  complement  to  existing 
quantitative  indicators.  Examples  of  commonly  used  indicators  are  set  out  below 
(Box  1).  Indicator  numbers  are  commonly  kept  to  a  minimum,  in  order  to  reduce 
administrative  burdens  and  complexity  (e.g.  the  Hague).  The  scale  of  the  strategy 
interventions  also  heavily  determines  the  role  and  focus  of  indicators.  However,  in 
order  to  capture  the  specificities  and  targeted  results  of  strategies  in  some  cases  a 
wider range of indicator-type is applied. Three broad categories can be identified. 

a)  OP  quantitative  indicators.  Here,  the  relevant  quantitative  OP  indicators 
are applied to the strategies (Centre-Franche-Comté Metropolitan pole, Elblag, 
Nicosia,  Six  Cities),  some  of  which  are  urban-specific,  e.g.  ‘businesses 
20 

Assessing the Performance of Integrated Territorial Development Strategies 

cooperating with cities in an innovation environment’ and ‘innovation platforms’ 
(Six  Cities).  Adaptations  can  be  made  to  reflect  the  scope  and  scale  of  the 
strategies.  For  example,  in  Pécs  the  indicators  and  milestones  defined  in  the 
Operational  Programme  were  broken  down  and  determined,  proportionate  to 
the funding for each city. 

b)  Strategy-specific  quantitative  indicators.  Core  quantitative  indicators 
are supplemented by additional measures in many cases, which can be:  

  developed  locally  (Aurillac)  to  capture  more  localised  impacts/the  specific 
territory  of  the  strategy  (Katowice)  or  reflect  local  authority  competencies 
(Centre-Franche-Comté Metropolitan Pole); 

  developed to pick up on more specific thematic elements (Cascais); 
  developed to capture longer-term impacts (Brno); and 
  based on existing strategies and systems, e.g. the Brno City strategy, using 
a  system  of  over  80  indicators  for  more  than  five  years  (Brno);  or 
adaptation of a system of indicators developed for the city (Malaga). 

c)  Qualitative  indicators.  Given  the  importance  of  ‘softer’  or  less  tangible 
impacts  of  the  strategies,  the  development  of  qualitative  indicators  is  a 
prominent feature of many monitoring systems.  

 

 

 

 

In  some  cases,  although  no  specific  qualitative  indicators  are  used, 
reporting  systems  can  allow  more  qualitative  assessments  to  be  reflected. 
For example, within an annual progress report, as well as reporting data, a 
qualitative assessment of interventions is made (Azul).  
In  others,  although  no  qualitative  indicators  have  been  set,  they  may  be 
introduced  in  future.  For  example,  for  the  Patras  strategy,  municipal 
officials  are  examining  ways  of  using 
for 
communication actions to measure public opinion on SUD interventions and 
receive feedback. 
‘Softer’  areas  of  intervention  (e.g.  social  and  human  capital)  will  involve 
special  surveys  and  research  questionnaires  already  used  for  the  City 
Strategy (Brno); 
Intangible  results  (e.g.  levels  of  satisfaction  of  residents  living  in  areas 
covered by the SUD strategy) (Cascais, Porto). 

technical  assistance 

Box 1 : Examples of frequently used indicators  

  area accessible from TEN-T in 45 minutes (Brno) 
 
 
  area  of  regenerated  open  spaces  and  regenerated  public  buildings  (Aurillac, 

length of road (Cascais) 
share of public transport within total passenger transport (Brno) 

Cascais)  

  vacancy rate within city centres (Aurillac)  
  population living in areas with integrated urban development strategies (Cork) 
 
 
 
 
 
 

levels of satisfaction of residents living in areas covered (Cascais) 
increase in population (Kaunas) 
increased new business registrations per 1,000 inhabitants (Kaunas)  
increased household incomes (Kaunas)  
reduced air pollution (Kaunas) 
improvement  in  the  social,  economic  and  physical  conditions  in  selected  urban 
centres, based on an urban development index (Cork)  
increased non-private-car commuting levels in the designated urban centres (Cork) 
 
  evolution  of  inhabitants’  perception  of  the  enhancement  of  their  environment 

(Centre-Franche-Comté Metropolitan pole) 

21 

 
 
 
Assessing the Performance of Integrated Territorial Development Strategies 

4.3 

Emerging evaluation approaches 

According  to  evaluation  plans,  evaluation  of  the  startegies  will  usually  take  place  as 
part  of  overall  programme  evaluation.  Although,  there  are  no  specific  evaluations  for 
the strategy, the assumption is that the strategy is an integral part of the OP, so that 
assessments  of  programme  effectiveness  will automatically  cover  the  performance  of 
the strategy.   

In some cases, the strategy will be assessed as part of specific OP evaluations, based 
on a specific thematic objective or priority axis that is directly related to the strategy. 

  Aurillac  SUD:  the  strategy  will  be  evaluated  as  part  of  the  evaluation  of  the 
urban  axis  of  the  OP  by  the  managing  authority.  This  theory-based  impact 
evaluation is expected to start in 2019 for ten months, and it will evaluate the 
impact  and  the  efficiency  of  strategies  on  urban  sprawl,  as  well  as  on  the 
development  of  networks  of  local  actors  and  coordination  between  funds 
allocated to urban development. 

In other cases, the strategies will be subject to specific evaluations in parallel with OP 
evaluation. 

  Cascais  (SUD):  the  strategy  will  be  subject  to  a  mid-term  evaluation in  2019, 
seeking to identify possible execution deviations from the programmed targets 
and results, and the main implementation constraints and adjustment needs. A 
final  evaluation  will  focus  on  the  contribution  of  results  to  the  strategic 
objectives  and  on  the  formulation  of  recommendations  to  inform  future 
interventions.  Furthermore,  the  Lisbon  OP  Evaluation  Plan  will  also  take  the 
strategy  into  account.  All  strategies  will  be  subject  to  the  ‘Evaluation  of  the 
Impact  of  Public  (ESIF  co-financed)  Urban  Regeneration  and  Revitalisation 
Policies’,  among  other  things  evaluating  first  achievements  of  new  SUD 
provisions.  

  The  operations  contributing  to  the  Azul  regional  ITI  (non-SUD)  will  be 
evaluated  within  each  of  the  contributing  regional  OPs.  Regional  authorities 
include a specific section devoted to  ITI in each annual implementation report 
for each OP. A specific evaluation of the ITI Azul will also be carried out at the 
central level in 2020 by the DG for EU Funds, which is the managing authority 
of all ERDF OPs 2014-20 in Portugal. 

As  this  analysis  has  highlighted,  there  is  variation  in  the  types  of  approach  adopted. 
Broadly  speaking,  three  approaches  can  be  identified  (see  First,  there  is  a  ‘light 
touch’  approach  (e.g.  Vejle,  Plovdiv,  Prague).  This  is  often  used  where  funding  for 
the  strategy  is  relatively  low  and/or  where  experience  or  capacity  in  evaluating 
integrated  territorial  interventions  is  limited.  In  these  cases,  measures  of  strategy 
achievements  are  part  of  standard  OP  monitoring  and  evaluation  procedures. 
Monitoring of strategy implementation is based solely on broader arrangements for the 
OP.  Output  and  result  indicators  used  for  monitoring  the  OP  are  applied  with  targets 
set according to the proportion of funds allocated to the projects in the strategy (i.e. if 
the  strategy  has  been  allocated  a  percentage  of  OP  funding,  it  must  account  for  the 
same proportion of indicator targets in the relevant priority or measure. Project-level 
indicators (e.g. km of constructed NMV roads, number of childcare places) will produce 
detailed, specific quantitative outcomes for each project. More qualitative processes of 
evaluation  are  also  possible  where  there  are  a  limited  number  of  projects  and  close 
contact  with  project  leaders  and  those  involved  with  the  implementation  of  the 
strategy  is  feasible.  Nevertheless,  the  scope  to  assess  integrated  territorial  impact  is 
limited. Often, there are no explicit plans for dedicated evaluations of the strategy.  

22 

 
Assessing the Performance of Integrated Territorial Development Strategies 

Table 3). 

First,  there  is  a  ‘light  touch’  approach  (e.g.  Vejle,  Plovdiv,  Prague).  This  is  often 
used  where  funding  for  the  strategy  is  relatively  low  and/or  where  experience  or 
capacity  in  evaluating  integrated  territorial  interventions  is  limited.  In  these  cases, 
measures of strategy achievements are part of standard OP monitoring and evaluation 
procedures.  Monitoring  of  strategy  implementation  is  based  solely  on  broader 
arrangements for the OP. Output and result indicators used for monitoring the OP are 
applied with targets set according to the proportion of funds allocated to the projects 
in  the  strategy  (i.e.  if  the  strategy  has  been  allocated  a  percentage  of  OP  funding, it 
must  account  for  the  same  proportion  of  indicator  targets  in  the  relevant  priority  or 
measure.  Project-level  indicators  (e.g.  km  of  constructed  NMV  roads,  number  of 
childcare places) will produce detailed, specific quantitative outcomes for each project. 
More  qualitative  processes  of  evaluation  are  also  possible  where  there  are  a  limited 
number of projects and close contact with project leaders and those involved with the 
implementation  of  the  strategy  is  feasible.  Nevertheless,  the  scope  to  assess 
integrated territorial impact is limited. Often, there are no explicit plans for dedicated 
evaluations of the strategy.  

Table 3 : Planned evaluation approaches 

Approach 

1.‘Light Touch’ 
Approach 

Strategy 
Scale 
Type 

and 

Funding for the 
strategy is low 
and/or where 
experience or 
capacity in 
evaluating 
integrated 
territorial 
interventions is 
limited. 

2.‘Standard’ 
Approach 

Strategies with 
a relatively 
substantial 
budget.  

Approach 

Overview 

Measures of strategy 
achievements are 
part of standard OP 
monitoring and 
evaluation 
procedures. 

Broader OP 
monitoring and 
evaluation 
procedures adapted 
or supplemented to 
create specific 
arrangements for 
assessing 
achievements  

Indicators in these 
cases can be revised 
to match data 
sources available in 

23 

- Scope to assess overall 
integrated territorial 
impact is limited. 

 - Project-level 
indicators can offer 
quantitative outcomes 
for each project.  

 - Qualitative processes 
of evaluation also 
possible where there are 
a limited number of 
projects and close 
contact with project 
leaders. 

- More sophisticated 
evaluation 
methodologies are being 
planned.  

- Challenges: in some 
cases, this approach is 
producing very large 
indicator sets that 
complicate monitoring 
and evaluation; in other 
cases, there are 
challenges in 
disaggregating ITI 

 
 
 
 
 
Assessing the Performance of Integrated Territorial Development Strategies 

the territory covered 
by the strategy. 

effects from those of 
domestic interventions. 

3.‘Comprehensive’ 
Approach 

Significant 
budgets and 
are part of 
relatively 
complex 
programme 
architectures.  

Dedicated system 
for monitoring and 
evaluating, based on 
indicators of 
contributing OPs,  
but with other 
indicators included 
to cover the 
territorial dimension. 

- Clear benefits of 
comprehensive 
approach (assessment 
of effectiveness of the 
strategy from a 
territorial perspective, 
including ‘harder’ and 
‘softer’ dimensions in 
the analysis of added 
value, etc).  

- However, significant 
outlay of resources.  

Second, a ‘standard’ approach (e.g. Nicosia, Katowice) is associated with strategies 
with  a  relatively  substantial  budget.  In  the  majority  of  cases,  broader  OP  monitoring 
and  evaluation  procedures  are  adapted  or  supplemented  to  create  specific 
arrangements  for  assessing  the  achievements  of  strategies.    Alongside  common  OP 
indicators,  special  output  and  result  indicators  are  adapted  to  strategy  interventions, 
in  order  to  focus  on  the  territory  covered  by  the  strategy  and  to  improve  clarity  on 
what  is  being  measured  and  the  measurement  units  used.  Indicators  in  these  cases 
can  be  revised  to  match  data  sources  available  in  the  territory  covered  by  the 
strategy,  making  measurement  more  feasible  and  more  reflective  of  the  anticipated 
result  of  the  integrated  interventions  (e.g.  Nicosia).  In  Katowice  ITI,  for  example, 
output  and  result  indicators  are  drawn  from  the  OP,  but  there  are  also  ITI-specific 
‘strategic’  indicators  included  to  measure  the  effectiveness  of  the  ITI  in  the  specific 
territory it covers.  It should be noted that this has produced a large indicator set.  In 
the  Nordhausen  strategy,  experience  from  the  previous  programming  period  showed 
that  the  inclusion  of  a  large  number  of  indicators  resulted  in  the  process  becoming 
unworkable and not necessarily accurate in assessing effectiveness.  

The standard approach can also be identified in cases that draw on existing domestic 
strategies  and  initiatives  to  support  the  measurement  of  strategy  achievements.  In 
Limburg,  for  instance,  the  ITI  forms  a  sub-section  of  the  Strategic  Action  Plan  for 
Limburg  (SALK),  funded  through  ERDF  and  ESF  OPs.  Progress  of  SALK  is  measured 
twice  a  year  by  the  Flemish  Government  with  indicators  that  monitor  the  socio-
economic  development  in  Limburg  at  the  macro  level  (e.g.  conditions  for  economic 
growth  and  the  business  environment).  The  Lille  strategy  contributes  to  the  City 
Contract (Contrat de Ville, the main instrument for domestic urban policy). On the one 
hand,  projects  in  the  strategy  will  be  monitored  and  evaluated  along  with  other 
projects in the OP. However, assessments of the territorial results of the strategy will 
be measured in terms  of its contribution to the City Contract  and its added value (in 
other words, the leverage effect of  ERDF in the Contract). The methodology is yet to 
be  defined,  but  should  be  based  on  a  sample  of  projects.  In  these  cases,  there  are 
clear efforts to capture and assess the territorial dimension of the strategy.  

More  sophisticated  evaluation  methodologies  are  being  planned  (e.g.  TBIE  in 
Katowice),  but  there  are  challenges:  in  some  cases,  this  approach  is  producing  very 
large  indicator  sets  that  complicate  monitoring  and  evaluation;  in  other  cases,  there 
are  challenges  in  disaggregating  the  strategy  effects  from  those  of  domestic 
interventions. 

Lastly,  a  ‘comprehensive’  approach  is  associated  with  a  limited  number  of  cases, 
involving  strategies  that  have  significant  budgets,  are  part  of  relatively  complex 
programme  architectures  (e.g.  involving  multiple  programmes  and  combining  EU 

24 

 
Assessing the Performance of Integrated Territorial Development Strategies 

funds)  and  where  territorially  integrated  investment  approaches  are  new  and  the 
potential added value is high. In these  cases,  a dedicated system for  monitoring and 
evaluating the strategy is established, based on indicators of contributing OPs but with 
other indicators included to cover the territorial dimension and to better quantitatively 
indicate the long-term added value of the strategy.  

A  prominent  example  is  Brno  ITI,  where  the  results  of  the  strategy  and  the 
contribution to particular OP objectives will be measured with the help of an indicator 
system  compulsorily  set  for  the  strategy.  This  combines  indicators  drawn  from  the 
contributing  OPs  and  those  drawn  from  the  domestic  strategy  of  Brno  City,  using  a 
system of over 80 indicators for more than five years. ‘Hard’ results will be measured 
by the set indicators (e.g. area accessible from TEN-T by 45 minutes; share of public 
transport  on  the  total  passenger  transport).  ‘Softer’  areas,  including  added  value  in 
terms  of  new  forms  of  cooperation,  strengthened  social  and  human  capital,  etc.,  will 
be measured through special surveys and questionnaire research that is already used 
in assessments of the Brno City strategy. The benefits of this comprehensive approach 
are  clear  in  terms  of  assessing  the  effectiveness  of  the  strategy  from  a  territorial 
perspective, including ‘harder’ and ‘softer’ dimensions in the analysis of added value, 
etc.  Such  an  approach  involves  significant  outlay  of  resources  that  can  be  justified 
where  strategy  budgets,  anticipated  results  and  added  value  are  high,  but  it  may  be 
less feasible in other contexts.    

5.  OPTIONS FOR FUTURE ASSESSMENTS OF EFFECTIVENESS 

Looking forward to options for assessing the effectiveness of integrated territorial and 
urban  strategies  in  the  future,  it  is  possible  to  highlight  some  basic  principles  and 
ideas.  

  First  it  is  important  to  recognise  the  need  for  differentiation  in  the  size  and 
complexity  of  indicators  sets,  in  line  with  variation  in  budgets  and  scope  of 
strategies. 

  Second,  it  is  also  crucial  to  capture  the  results  of  territorially  integrated 
approaches  in  line  with  the  logic  of  the  intervention.  The  aim  is  to  define 
synthetic  or  integrated  indicators  that  can  assess  combined  actions.  For 
example,  under  the  heading  of  urban  mobility,  output  indicators  could  include 
integrated  ticketing,  bike  lanes,  trams,  etc.  and  the  overall  result  indicator 
could be a modal shift away from car transport in the territory. 

  The  third  principle  is  added  value.  In  many  cases,  these  territorial  initiative 
represent  new  or  innovative  ways  of  doing  things  and  the  reasons  for 
introducing  them  include  institutional  and  operational,  as  well  as  physical 
change.  This  highlights  the  need  for  a  combination  of  ‘hard’  and  ‘soft’ 
indicators. 

5.1 

Indicators 

The  importance  of  assessing  the  impact  and  results  of  the  strategies  is  clear.  The 
challenges and opportunities involved have been discussed in previous sections of this 
report  –  covering  the  sectoral  and  territorial  dimensions,  capturing  different 
geographies,  disaggregating  the  results  of  the  strategy  from  the  results  of  other 
interventions, including ‘softer’ value-added, such as new cooperative approaches, and 
new principles of governance. In line with the results orientation of EU investments it 
is  expected  that  the  strategies  and  their  interventions  should  be  developed  with 
results in  mind.  Policy-makers,  stakeholders  and  beneficiaries  require  knowledge  and 
understanding  of  the  progress  and  effectiveness  of  the  strategies;  transparency  and 

25 

 
 
 
 
Assessing the Performance of Integrated Territorial Development Strategies 

accountability have to be ensured. It is important to gather information with which to 
better  understand  the  role  of  the  interventions  and  an  evidence  base  comparable  to, 
and where possible compatible with, other interventions. 

The  following  indicators  are  presented  as  examples  of  the  types  of  measures  that 
could be used. The number and combination will vary linked to the budget, territorial 
coverage  and  thematic  focus  of  the  specific  strategy.  For  example,  developing 
additional tailored indicators is time-consuming, can be costly, and adds to the already 
considerable administrative burdens that stakeholders face.  

‘Hard  indicators’.  Sustainable  urban  development  is  a  broad  multi-dimensional 
concept and can cover a wide range of interventions.  Table 5 sets out key dimensions 
of sustainable urban development that integrated place-based strategies may address, 
as  well  as  specific  elements  of  these  dimensions.  A  limited  number  of  broad  ESIF 
programme  indicators  may  not  be  sensitive  enough  to  capture  the  specific  focus  and 
areas of impact of the strategies. In such cases, it may be necessary to look at more 
tailored indicators.  

Taking these areas of activity as a starting point, Table 5 also sets out examples of the 
types  of  indicators  that  could  be  useful,  many  of  which  are  or  could  be  linked  to  the 
common  indicators  used  for  ESIF  programmes,  with  some  examples  from  case  study 
strategies where they are used. The focus is on results indicators, as output indicators 
will be strongly linked to specific strategy aims and priorities. It should be noted that 
these  indicators  taken  in  isolation  have  a  sectoral  focus  and  therefore  do  not  reflect 
the territorially integrated nature of the interventions. Therefore, it is important that in 
any assessment of the strategies these comparatively ‘one dimensional’ measures are 
considered alongside the qualitative measures which are also proposed  in this report. 
As has been argued, ’hard’ programme indicators alone are unlikely to capture added 
value  associated  with  the  design  and  implementation  of  integrated  place-based 
strategies.  Elements  of  added  value  can  be  found  under  a  range  of  headings  (see 
section  added  value  in  Final  Report).  The  extent  to  which  the  approach  delivers 
positive change is a key issue for monitoring and evaluation exercises.  

Table 5: Elements of Sustainable Urban Development  

Element 

Heading 

Indicator example 

Social 
Wellbeing  

Health 

Safety 

Local or civic 
identity/sense of 
place 

Access to 
affordable 
housing and 
services 

Access to public 
and open spaces 

Access to 
transport 

Economic 
Opportunity  

Diversified 
economy 

Transport 
infrastructure 

Coordinated 
land use 

Coordinated 

  Access to key local services 
  % people affected by poor housing 
  % people affected by poverty, unemployment,  
 
  % of population with access or using recreation space 
 
Population perceptions of quality of life (e.g. CFC Pole) 
  Housing  stock  affordable  to  low-  and  very-low-income 

community participation in planning 

residents 

  Rehabilitated  housing  in  urban  areas  (ESIF  Common 

 

Indicator) 
Population  covered  by  improved  health  services  (ESIF 
Common Indicator) 

  Number of active neighbourhood organisations 
  Students in higher education (e.g. Timisoara) 
  % early school-leavers 

  Population  living  in  areas  with  integrated  urban 
development  strategies  (ESIF  Common  Indicator, 
e.g. Cork) 

  Employment  increase  in  supported  areas  (e.g. 

Nitra SUD) 

  Share  of  persons  living  in  households  with  very 

low work 

  No.  of  enterprises 

receiving  support  (ESIF 

26 

Assessing the Performance of Integrated Territorial Development Strategies 

growth plans 

Common Indicators) 

Environment 
Quality  

Access to capital 
and credit 

Access to 
education, jobs 
and training 

Efficient, 
planned land 
use 

Efficient 
resource use 

Pollution 
reduction and 
management 

Climate change 
mitigation, 
adaptation and 
resilience 

Low-carbon 
transport 

Diverse and 
protected 
natural 
environment 

  % new enterprise start-ups (e.g. Kaunas) 
  Public  or  commercial  buildings  newly  built  or 
(ESIF  Common 

in  urban  areas 

renovated 
Indicator) 
Increase 
to 
visitor 
cultural/heritage sites (e.g. Egnatia Odos)  

numbers 

key 

in 

 

  Length and use of new or upgraded infrastructure 

(e.g. Debrecen) 

  GHG  reduction  (ESIF  Common  Indicator,  e.g. 

Timisoara) 

  Decrease  in  annual  primary  energy  consumption 
of  public  buildings  (ESIF  Common  Indicator,  e.g. 
Torino) 

  Total passenger miles by private car 
  Miles  travelled  (or  trips)  taken  by  sustainable 

modes 

  Bike lanes in roadways 
  Population commuting via bus 
  Vehicles, buses, etc with hybrid or alternative fuel 
  Households  within  reach  of  public  transport  (e.g. 

Brno) 

  Designated critical habitat protected 
  Waste management  
  Vulnerability to flooding  
  Proportion  of  city’s  land  area  designated  as 
recreational  and  green  spaces  to  the  total  land 
area (e.g. Berlin) 

  Open space created, rehabilitated  in urban areas 

(ESIF Common Indicator) 

  % of waste recycled  
  % renewable energy used municipal operations 
  Amount  of  carbon  dioxide  emissions  in  metric 

tons 

  %  brownfield/infill  development  as  a  percentage 

of total development (e.g. Aurillac) 

Source:  EPRC,  adapted  from  Lynch  A  J,  Andreason  S,  Eisenman  T,  Robinson  J,  Steif  K,  and 
Birch E L (2011) Sustainable Urban Development Indicators for the United States. Report to the 
Office of International and Philanthropic Innovation, Office of Policy Development and Research, 
U.S.  Department  of  Housing  and  Urban  Development.  Philadelphia:  Penn  Institute  for  Urban 
Research.19 

 ‘Soft indicators’ and ‘added value’. 

Some  quantitative  indicators  can  be  used  to  assess  added  value,  although  these  are 
often  quite  general,  for  instance:  the  amount  of  people  in  areas  that  are  affected  by 
strategies  or  the  number  of  beneficiaries  of  related  projects;  or  the  number  of  new 
structures or systems (e.g. for coordination, strategy design or implementation tasks) 
established.  Inevitably,  exploring  added  value  involves  some  assessment  of  ‘softer’ 
impacts  and  these  headings  will  often  be  based  on  qualitative  approaches.  Here,  the 
aim  is  to  assess  less  tangible  aspects  of  added  value  (strategic  integration, 
coordinated implementation, territoriality, capacity-building and knowledge exchange). 
For instance, the added value of an ITI may be low in terms of capacity building where 

19 Data sources for these indicators include: National, regional and local statistics and studies 
Eurostat, European Social Progress Indicators, European Environment Agency, European Quality 
of Institutions Index, http://www.oecdbetterlifeindex.org 

27 

 
                                           
Assessing the Performance of Integrated Territorial Development Strategies 

existing capacities are strong but it may be higher if it adds a new territorial focus or 
draws in new partners.  Table 6 sets a proposed approach based on the main report’s 
categories of added value.   

Table 6 :  Examples of ‘soft’ indicators - added value 

Strategic integration  
Coordinated implementation 
Territorial focus 
Partnership, capacity building 
Exchange of knowledge, experience 

Contribution of territorial provision  

Low  Modest  High   Very High 

None 
 x 

 x 

 x 

 x 

 x 

Source:  EPRC,  based  on  European  Commission  (2009)  Promoting  sustainable  urban 
development in Europe: achievements and opportunities, April 2009. 
http://ec.europa.eu/regional_policy/sources/docgener/presenta/urban2009/urban2009_en.pdf 
(downloaded 10/01/18). 

Thus,  for  the  development  of  indicators  and  targets,  it  is  possible  to  identify  the  key 
tasks  of dealing with complex geographies, numerous administrative boundaries, and 
the  combination  of  ‘hard’  and  ‘soft’  impacts.  Lessons  can  be  drawn  from  other  ESI- 
funded  initiatives  that  face  similar  challenges.  For  instance,  in  European  Territorial 
Cooperation  programmes  an  interesting  approach  is  the  establishment  of  ‘tailored’ 
indicators and baselines for a programme. These are put together by a focus group of 
stakeholders and experts under different headings. An example would be ‘changes in 
institutional capacity’ and indicators could relate to the introduction of new structures, 
tools  or  processes.  Similar  approaches  could  used  under  thematic  headings.  There  is 
then a periodic review of progress towards targets by experts. The advantages of this 
approach is that it increases the scope for tailored indicator sets that can integrate or 
synthesise  different  territorial  dimensions  and  ensure  that  these  are  relevant  and 
realistic given the scale and scope of the strategy.  

5.2 

Evaluation 

Turning  to  evaluation,  again  it  is  possible  to  highlight  the  value  of  similar  principles: 
the  need  for  proportionality;  and  the  importance  of  mixed  methodologies  where  the 
balance of qualitative and quantitative methods reflects the situation ‘on the ground’.  

Each evaluation will also have to incorporate different types of knowledge, generated 
through a range of questions and drawing on different sources of information. 

  Strategic  knowledge.  Measuring  the  effectiveness  of  territorial  provisions 
includes the generation of strategic knowledge. This includes an assessment of 
the  identification  of  needs.  As  has  been  discussed  in  relation  to  the  wider 
evaluation debate, it is crucial that the knowledge developed is set in context. 
Contextual  specificities  play  a  central  role  in  understanding  the  role  and 
contribution  of  integrated  interventions.  Important  geographical,  cultural, 
economic,  social  and  political  differences  shape  how  and  to  what  extent  the 
concepts  and  tools  are  applied  and  their  added  value.  Assessments  must 
consider  the  logic  of  intervention  and  strategic  priorities  that  have  been 
identified  in  the  strategies  against  the  socio-economic  indicators  for  the 
territory  covered.  This  part  of  a  methodology  for  measuring  effectiveness  can 
involve  documentary  evidence  assessment  based  on  analysis  of  the  strategies 
alongside domestic strategic documents (at national or sub-national level) and 
EU/MS socio-economic datasets. The assessment can be  complemented with a 

28 

 
 
 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
Assessing the Performance of Integrated Territorial Development Strategies 

more qualitative assessment by expert stakeholders on whether the challenges, 
logic  and  objectives  are  relevant.  This  is  to  an  extent  static  knowledge,  but  it 
facilitates  policy-learning  and  can  inform  changes  during  the  course  of  policy 
implementation. Key questions include: 

o 

Is the strategy really relevant to the operations being considered? Is the 
strategy a broad list of potential actions or does it provide specific detail 
on how and where the funds will be allocated? 

o  Does  the  prioritisation  in  the  strategy  fit  with  the  specific  needs  of  the 
territory?  Is  this  justified  in  the  strategies  in  an  explicit  detailed  way? 
Why  are  certain  TOs  included  in  the  strategies  –  economic,  social, 
environmental aspects? 
Is the geographical coverage of the strategy (e.g. city, area of city, city-
region, etc.) consistent with the strategic priorities? 

o 

o  How clear and coherent is the intervention logic? 
o  Are  arrangements  for  achieving  synergies  between  projects,  funds, 

stakeholders, etc. clearly outlined? 

  Operational knowledge. This relates to the management and implementation 
of  the  intervention,  with  a  view  to  identifying  and  addressing  challenges, 
building  on  best  practice  and  maximising  efficiency  and  effectiveness.  On  the 
one hand, this concerns basic components of Cohesion policy management and 
implementation  systems,  assessing  arrangements 
financial 
absorption,  the  quality  of  monitoring  data,  issues  in  terms  of  project  pipeline, 
resourcing/capacity,  quality  of  indicators,  communication  arrangements,  audit 
issues,  state-aid  issues,  etc.  However,  particularly  for  integrated  place-based 
strategies,  it  is  important  for  evaluations  to  ask  whether  the  appropriate 
stakeholders  are  involved  in  the  design  and  implementation  of  the  strategy. 
This dimension of measuring effectiveness can combine documentary research 
with qualitative, interview-based approaches to assess the depth and quality of 
inputs from various actors. Specific questions could include:  

to  ensure 

o  Who is making an input into the design and implementation of the 

strategy?  

o  Are ‘grass roots’ actors involved or is it just professional meeting-

attenders (e.g. chambers of commerce versus SMEs).  

o  What is the involvement of non-public authority partners (especially 

private sector and civil society)? What is the territorial coverage (e.g. 
neighbouring municipalities)?  

o  What is the level of participation of 

communities/residents/microbusinesses, etc. (rather than institutions 
and agencies)?  

o  How is this input organised, i.e. according to vertical or horizontal 

principles – consultation or ‘bottom-up’ input?  

o  How  realistic  is  the  scale  of  resources  in  relation  to  the  indicators  and 

targets set in the strategy?  

o  Are  the  proposed  actions  realistic  given  the  level  of  administrative 

capacities available for implementation?  

  Evaluative  knowledge.  This  forms  the  basis  of  a  systematic  assessment  of 
the  worth  or  merit  of  the  interventions  contained  in  the  strategy.  As  well  as 
examining  project  and  programme-generated  data  and  assessments  against 
fixed  targets,  it  is  possible  to  build  a  more  in-depth  understanding  of 

29 

 
 
 
Assessing the Performance of Integrated Territorial Development Strategies 

achievements through focusing on added value. On the one hand, assessments 
will  include  a  review  of  the  performance  of  interventions  contained  in  the 
strategies according to the indicators and targets set and drawing on data from 
programme  monitoring  systems.  These  will  be  based  on  Cohesion  policy 
common  indicators  for  Operational  Programmes,  additional  indicators  and 
targets  developed  to  capture  specific  effects  in  the  territories  covered  by 
strategies. This would involve analysis of the performance of projects according 
to  indicators  and  targets  set  specifically  for  the  strategy  in  programmes.  It 
could  also  involve  comparison  of  the  performance  of  projects  under  a 
integrated  place-based  strategies  to  other  projects  in  a  priority  or  programme 
not  covered  by  the  strategy,  particularly  those  covering  the  same  Thematic 
Objectoves.  In  cases  where  the  strategy  contributes  to  domestic  strategies, 
comparison  between 
the  performance  of  EU  and  non-EU-supported 
interventions  may  also  be  possible.    Depending  on  the  scale  of  operations 
involved,  this  assessment  could  be  based  on  a  range  of  methodologies  e.g. 
theory-based  evaluation.  Integrated  place-based  strategies  cover  different 
combinations  of  thematic  objectives  depending  on  the  Member  States: 
transport;  environment;  climate  change;  education  systems;  employment  and 
information and communication technologies. The key question here is: 

o  Has  integration  and  the  territorial  approach  lead  to  stronger  financial 

and physical performance?  

  The  value  of  a  qualitative  dimension  in  evaluation  approaches. 
Generally,  the  value  of  a  strong  qualitative  approach  is  apparent  given  the 
specific features of integrated interventions and the importance of less tangible 
effects  in  the logic  of intervention.  While it is important  to  identify  and  record 
quantifiable outputs and results, ‘not everything that counts can be counted’.20 
Integrated  place-based  strategies  have  significant  scope  to  provide  value  that 
is  not  immediately  apparent  in  statistical  analyses.  Thus,  qualitative  and 
participative evaluation methodologies will yield non-numerical information but 
help to recognise the uniqueness of different strategy settings and capture the 
effects  of  the  strategy  on  the  behaviour  of  actors  and  organisations  in  the 
territory.  Assessments of many of these aspects would, therefore, rely mainly 
on qualitative approaches.  

 

Interviews and focus groups to draw out the opinions of stakeholders on the 
quality  and  sustainability  of  the  activities  undertaken  in  the  design  and 
implementation of the strategies thus far.  

  Potential  indicators  of  this  type  of  added  value  could  cover:  the 
establishment  of  collaborative  structures  or  processes  (networks,  fora); 
whether  these  are  new  or  linked  to  existing  arrangements;  whether  they 
are ‘one-off’ consultative initiatives designed solely to feed into the strategy 
or  are  designed  as  longer-term  arrangements;  the  quality  of  relationships 
between  stakeholders  (e.g.  the  dominance  of  horizontal  or  vertical 
organisational  principles);  and  the  impact  of  these  arrangements  on  the 
design  and  implementation  of  the  strategy  (e.g.  how  they  are  involved  in 
resource-allocation  decisions).  It  would  be  important  to  cover  these  issues 
in the evaluation research.  

  The fundamental logic of the territorialisation agenda is the Methodology for 
Measuring  the  Effectiveness  of  Territorial  Provisions  that  development 
initiatives benefit from concrete input from local networks. Thus, if there is 
the  design  and 
evidence  of  strong,  concerted 

involvement 

in 

20 Cameron W B (1963) Informal Sociology: A Casual Introduction to Sociological Thinking. New 
York: Random House. 

30 

 
 
 
                                           
Assessing the Performance of Integrated Territorial Development Strategies 

implementation  of  the  strategies,  and  that  these  are  sustainable 
arrangements  for  collaborative  working,  it  could  be  assumed  that  this  will 
boost their long-term effectiveness.  

  Key questions would include: 

o  Has  the  design  and  implementation  of  the  strategy  mobilised  new 
for  territorial  development 

institutional  resources 

financial  or 
initiatives?   

o  Has 

it  built  administrative  capacities  (skills,  confidence  and 
capabilities  of  actors  involved,  changing  organisational  cultures, 
structures, systems, tools)?  

o  Has  it  strengthened  social  capital  (networks,  norms,  relationships, 
values  and  informal  sanctions  that  shape  the  quantity  and  co-
operative quality of interactions)? 

6.  CONCLUSIONS 

The  preceeding  sections  described  the  opportunities  and  challenges  of  implementing 
effective,  efficient  and proportional  approaches  to  measuring  the  effectiveness  of  the 
integrated  place-based  strategies.  The  vital  importance  of  policy  accountability, 
transparency, and learning is noted, as are the challenges in capturing the potentially 
varied hard and soft impacts and influences of integrated territorial interventions.   

The  strategies  are  implementation  tools,  not  objectives  in  themselves.    They  must 
contribute  to  the  achievement  of  the  thematic  objectives,  investment  priorities  and 
specific  objectives  set  out  in  the  contributing  programmes.  Thus,  financial  and 
indicator data are traced back to contributing priority axes and specific indicators can 
be  used  at  the  management  level  to  assess  the  implementation  of  individual 
strategies.21  

However,  as  comparatively  new  approaches,  it  is  important  that  measures  of 
effectiveness also take into account the more operational aspects of the strategies and 
issues related to the process of implementing integrated actions are also recognised.   

The framework set out in Figure 6 draws together the key components of assessment 
approaches for territorial and urban strategies, based on an analysis of the strategies 
in  their  current  form,  taking  into  account,  the  current  reporting  requirements, 
assessment arrangements, data availability etc.  

Beyond  this,  it  is  important  to  reiterate  some  of  the  key  principles  that  should  be 
considered in developing plans to assess the achievements of these strategies. 

First,  assessments  should  have  have  a  strong  results  focus,  i.e.  do  the  strategies 
have  a  strong  interventions  logic?  are  logics  being  effectively  pursued?  have  results 
been  achieved?  However,  assessments  should  also  look  more  widely  at  the  role, 
influence and implementation of the strategies, which may also have additional results 
and benefits, which are not reflected, in ‘hard’ indicators.  

Second,  approaches  to  monitoring  should  be  built  around  the  key  output  and 
result  indicators  set  out  for  the  relevant  ESIF  funds,  as  interventions  should 
contribute  to  the  objectives  of  the  corresponding  investment  priorities  and  the 
objectives  of  the  integrated  place-based  strategies.  However,  these  basic  indicators 

21 Svanfeldt C (2013) Integrated Territorial Approaches, European Commission presentation, 
Bucharest 11 October 2013. 

31 

 
 
 
                                           
Assessing the Performance of Integrated Territorial Development Strategies 

should  be  complemented  by  relevant  programme/strategy  specific  indicators,  and,  in 
particular  qualitative indicators  and  asessments.  The  available  measures  and  insights 
can be used in combination to build an understanding of contributions to the territory, 
wider national and EU impacts and results focussing on key areas of added value, e.g. 
(Annex  1  sets  out  a  framework  for  developing  different  types  of  knowledge  through 
assessments, with associated methods, questions and data sources).  

Third,  there  is  a  need  for  proportionality.  The  huge  variation  in  size  in  terms  of 
population  covered,  thematic  focus,  budget,  geographic  scale  and  implementation 
approach,  means  that  there  must  be  ‘tailoring’  in  how  the  framework  is  applied  in 
terms of scale and scope and implementation approach (Annex 2 sets out a framework 
for differentiating between ‘light touch’, ‘standard’ and ‘comprehensive’ approaches). 

Fourth,  flexiblity  should  be  a  guiding  principle,  reflecting  the  varying  characteristics 
of 
thematic  content,  stages  of 
development, administrative capacities and urban development contexts. 

the  strategies,  differing  geographic  scales, 

Fifth, assessment should recognise the importance of ‘user-friendly’ methods and 
outputs – well defined in terms of data generation and functions for  monitoring and 
evaluation; tailored to capture a range of qualitative and quantitative knowledge; and 
taking into account the range of potential audiences. 

Finally,  it  is  important  to  set  realistic  goals.  It  is  important  to  be  attentive  to  what 
type  of  knowledge  can  be  generated,  given  the  complexity  of  the  subject  and  the 
resources and timescale involved.   

Figure 6: Integrated Framework 

Step 1 

Strategic 

Needs clearly 
expressed and 
justified? 

Step 2 

Operational 

Management & 
Implementation  

Step 3 

Evaluative 

Results 

Within an existing 
ex-ante evaluation 
process, or as part of 
a complementary 
analysis 

Documentary 
analysis  of  strategy 
documents 
(i.e. 
description of partner 
involvement 
in  the 
strategies)  

Mixed methods : 
Interview or 
Qualitative and 
questionnaire on 
Quantitaive analysis  
partner engagement 
and involvement 

ESIF Common 
Indicators 
(examples) 

Programme/ 
Strategy Specific 
Indicators 
(examples) 

32 

Qualitative 
Indicators 

(examples) 

 
 
 
 
 
 
 
Assessing the Performance of Integrated Territorial Development Strategies 

37 

persons  Population living in 

areas with integrated 
urban development 
strategies 
Open space created 
or rehabilitated in 
urban areas 
Public or commercial 
buildings built or 
renovated in urban 
areas 
Rehabilitated housing 
in urban areas 

38 

39 

square 
metres 

square 
metres 

40  Housing 

units 

See Box 1 for 
widely used 
indicators and 
Table 5 for other 
examples 

- Strategic 
integration 
- Territorial 
focus 
- Integrated 
governance 
- Partnership, 
capacity-
building 
- Knowledge 
exchange 

Strateg
ic 

Integrated gov  

Partnership 
and capacity 
areas 

Innovation 

33 

 
 
 
 
 
 
 
 
 
 
 
Assessing the Performance of Integrated Territorial Development Strategies 

Annex 1 : Framework for generating different types of knowledge 
Questions 

Rationale  

Timing 

Methods 

Data Sources 

Ensure 
objectives can 
be met, cost-
effectiveness 
and scope to 
maximise 
synergies 

c
i
g
e
t
a
r
t
S

Ex ante 

•  Clear  rationale  for  the  operations  being 
considered?  

•  Do  proposed  actions  fit  with  the  specific 
needs of the territory? 

• Are policy synergies adequately considered? 

•  Is  the  geographic  coverage  consistent  with 
the strategic priorities? 

•  Are  targets  in  line  with  the  resources 
available? 

 -  Within  an  existing  ex-ante 
evaluation process, or as part of a 
complementary analysis 

 - 
of 
Assessment 
complementarities,  contrasts  and 
relevant 
with 
overlaps 
programmes and strategies 

 -  Qualitative  assessment  by 
expert  stakeholders  on  whether 
and 
logic 
challenges, 
the 
objectives are appropriate  

Documentary sources  

socio-economic 
available, 

National  or  sub-national  levels  and 
datasets, 
EU/MS 
where 
e.g. 
http://urban.jrc.ec.europa.eu/?ind=g
dpppstrend&ru=metro&s=1&c=1&m=
0&f=1&p=IE001L1&swLat=49.32512
199104001&swLng=-
22.1923828125&neLat=57.26716357
153586&neLng=5.537109374999999 

http://database.espon.eu/db2/resour
ce?idCat=43 

•  What  is  it  realistic  to  expect  future 
it 
assessments 
necessary  to  develop  additional  benchmark 
measures?  

‘measure’/assess; 

to 

is 

34 

 
 
 
Assessing the Performance of Integrated Territorial Development Strategies 

Focus on 
effectiveness 
and efficiency 
of 
management 
and 
implementatio
n with a view 
to addressing 
any 
challenges, 
building on 
best practice 
and 
maximising 
results  

l
a
n
o
i
t
a
r
e
p
O

• Who is making an input into the design and 
implementation of the strategy?  

• Are genuine ‘grass roots’ actors and 
stakeholders involved? At what stages?  

• Are a range of actor-types and relevant 
territories involved?  

• How has partner participation been 
organised? If engagement has been a 
challenge, have measures been pursued to 
build engagement? 

• Has the strategy built on/complemented 
existing partnership approaches, or 
established new territorially–based 
partnerships?  

 -    Linked  to  a  thematic/priority 
review 
programme 
interventions;  or  separate  ITI-
specific review  

of 

Documentary  analysis  of  strategy 
documents (i.e. description of partner 
involvement in the strategies)  

 - Documentary research   

interview-based 
 -  Qualitative, 
approaches  to  assess  the  depth 
and  quality  of  inputs  from  various 
actors 

Interview or questionnaire on partner 
engagement and involvement 

35 

 
 
 
 
Assessing the Performance of Integrated Territorial Development Strategies 

 -  Intervention  output  and  results 
data  

- Interviews  

for 

the 

-  Where  appropriate  data  are 
relevant 
available 
geographic  scale  and 
thematic 
focus  of  the  strategy,  measures 
such  as  cost  per  job  maintained 
could be considered 

-  Identification  of  soft/hard;  long-
term/short-term 
using 
quantitative 
qualitative 
measures, 
realist 
evaluation, contribution analysis 

and 
including 

results 

 -  Focus  groups  to  draw  out  the 
on 
opinions 
qualitative results 

stakeholders 

of 

- 
Cross-programme/policy 
assessments,  drawing  on  evaluative 
evidence 
policy 
instruments  where  synergies  have 
been pursued 

related 

on 

 -  Data-gathering  and  analysis  of 
relevant results and output indicators 
across programmes 

    - Where possible, results 
assessment  would ideally involve 
statistical/quantitative analysis of 
cost-effectiveness, cost efficiency and 
impacts against established indicators  

  - Where necessary, territorially  
targeted assessments of, e.g., 
changes in key indicators such as 
quality of life 

What works, 
how and why? 
Critical 
assessment of 
to what 
extent, how 
and why 
territorial 
approach has/ 
has not 
delivered 
results 

e
v
i
t
a
u
l
a
v
E

• Does integration and the territorial 
approach lead to stronger results?  
• Has the approach facilitated synergies with 
other initiatives? 

• Has the design and implementation of the 
strategy built administrative capacities (skills, 
confidence and capabilities of actors involved, 
changing organisational cultures, structures, 
systems, tools)?  

• Has the approach strengthened social 
capital (networks, norms, relationships, 
values and informal sanctions that shape the 
quantity and co-operative quality of 
interactions)?  

Ex post 

36 

 
 
 
 
 
 
  
  
 
Assessing the Performance of Integrated Territorial Development Strategies 

Annex 2: Assessment methodologies 

‘Project’ (e.g. 
covering ltd no. of 
operations)  

Small budget 

‘OP’ (e.g. 
covering multiple 
priorities) 

Medium budget 

‘Multiple’ (e.g. 
covering multiple 
OPs) 

Large budget 

Light-touch 

Use of existing  
evaluation and 
monitoring indicators 
and approaches  

Standard 

Comprehensive 

Tailored arrangements 
included, e.g. additional 
indicators, targeted 
element in programme 
evaluations, inclusion of 
qualitative asessment of 
integrated approach  

Dedicated evaluation; 
cross-cuttting analysis 
across programmes, 
funding instruments, 
qualitative and 
quantitative approaches 

Est. 

New 

Est. 

New 

Est.  

New 

37 

 
 
 
 
 
Assessing the Performance of Integrated Territorial Development Strategies 

38 

 
HOW TO OBTAIN EU PUBLICATIONS 

Free publications: 

•  one copy: 

via EU Bookshop (http://bookshop.europa.eu); 

•  more than one copy or posters/maps: 

from the European Union’s representations (http://ec.europa.eu/represent_en.htm);  
from the delegations in non-EU countries 
(http://eeas.europa.eu/delegations/index_en.htm);  
by contacting the Europe Direct service (http://europa.eu/europedirect/index_en.htm) 
or calling 00 800 6 7 8 9 10 11 (freephone number from anywhere in the EU) (*). 

(*)  The information given is free, as are most calls (though some operators, phone boxes or hotels may 
charge you). 

Priced publications: 

•  via EU Bookshop (http://bookshop.europa.eu). 

Priced subscriptions: 

•  via one of the sales agents of the Publications Office of the European Union 

(http://publications.europa.eu/others/agents/index_en.htm). 

 
 
 
 
 
 
 
K
N
-
0
1
-
1
8
-
2
1
8
-
E
N
-
N

[

C

a

t

a

l

o

g

u

e

n

u

m

b

e

r

]

  doi: 10.2776/96617