COMMISSION STAFF WORKING DOCUMENT EVALUATION of the Research and Technological Development (RTD) infrastructures and activities supported by the European Regional Development Fund (ERDF) in the period 2007-2013
Date: 2023-03-24
Institutions: Directorate-General for Regional and Urban Policy, European Commission
Subjects: EU research policy, European Regional Development Fund, innovation, investment, operational programme, research and development, research infrastructure, research staff, technological change

EN 

  EN 

   EUROPEAN COMMISSION  Brussels, 24.3.2023  SWD(2023) 71 final  COMMISSION STAFF WORKING DOCUMENT  EVALUATION         of the        Research and Technological Development (RTD) infrastructures and activities supported by the European Regional Development Fund (ERDF) in the period 2007-2013 {SWD(2023) 72 final}  
 
 
Table of contents 

1. 

INTRODUCTION ................................................................................................................................ 5 

1.1. 

1.2. 

1.3. 

Context and purpose ................................................................... 5 

Scope of the evaluation .............................................................. 5 

Structure of the SWD .................................................................. 7 

2.  WHAT WAS THE EXPECTED OUTCOME OF THE INTERVENTION? ....................................... 7 

2.1. 

Background and needs ................................................................ 7 

ERDF priorities in the RTD field ............................................................................................... 7 

Investments needs and regional differences .......................................................................... 8 

2.2. 

Expected outcome .................................................................... 10 

Infrastructure investments .................................................................................................... 10 

R&D projects and related activities ....................................................................................... 14 

3.  HOW HAS THE SITUATION EVOLVED OVER THE EVALUATION PERIOD? ....................... 16 

3.1. 

3.2. 

Projects and beneficiaries ......................................................... 16 

Key achievements ..................................................................... 20 

R&D personnel and researchers............................................................................................ 21 

Scientific output .................................................................................................................... 23 

Technological development and innovation ......................................................................... 25 

Students and tertiary attainments ........................................................................................ 28 

3.3. 

Contribution analysis ................................................................ 30 

Impacts and causality ............................................................................................................ 30 

External factors ..................................................................................................................... 31 

4. 

EVALUATION FINDINGS ............................................................................................................... 34 

4.1. 

To what extent was the intervention successful and why? ...... 34 

Effectiveness .......................................................................................................................... 34 

Efficiency ............................................................................................................................... 36 

Coherence ............................................................................................................................. 37 

4.2. 

How did the EU intervention make a difference? ..................... 39 

EU added value ...................................................................................................................... 39 

4.3. 

Is the intervention still relevant? .............................................. 39 

Relevance .............................................................................................................................. 39 

5.  WHAT ARE THE CONCLUSIONS AND LESSONS LEARNED? ................................................. 40 

1 

 
 
 
5.1. 

5.2. 
periods 

5.3. 

Conclusions ............................................................................... 40 

Policy response in the 2014-2020 & 2021-2027 programming 
 ................................................................................................... 43 

Questions for further studies .................................................... 46 

ANNEX I. PROCEDURAL INFORMATION ............................................................................................. 49 

1. 

2. 

3. 

4. 

Lead DG and Decide reference ................................................. 49 

Organisation and timing ............................................................ 49 

Evidence and sources ................................................................ 49 

External expertise ..................................................................... 49 

ANNEX II. METHODOLOGY AND ANALYTICAL MODELS USED ................................................... 50 

1. 

2. 

3. 

4. 

5. 

General methodology ............................................................... 50 

Limitations and mitigation strategy .......................................... 53 

List of the sample of 53 operational programmes .................... 54 

Overview of core and common indicators on RTD ................... 56 

Econometric analysis ................................................................. 57 

ANNEX III. EVALUATION MATRIX ....................................................................................................... 59 

ANNEX IV. STAKEHOLDERS CONSULTATION - SYNOPSIS REPORT ............................................ 66 

1. 

2. 

Strategy ..................................................................................... 67 

Objectives .............................................................................................................................. 67 

Tools ...................................................................................................................................... 67 

Stakeholder participation ...................................................................................................... 67 

Delivery and results ................................................................... 68 

Roadmap consultation .......................................................................................................... 68 

Interviews .............................................................................................................................. 68 

Seminar.................................................................................................................................. 71 

ANNEX V. LIST OF REFERENCES .......................................................................................................... 83 

2 

 
 
 
 
 
Glossary 

Term or acronym 

Meaning or definition 

CCI 

EC 

ERDF 

ESF 

ESFRI 

ESIF 

EQ 

EU 

EU13 

EU15 

EUR 

FP7 

GDP 

H2020 

HEI(S) 

ICT 

Character Code Identifier 

European Commission 

European Regional Development Fund 

European Social Fund 

European Strategy forum on Research Infrastructures 

European Structural and Investment Funds 

Evaluation Question 

European Union 

The following Member States: Czechia, Estonia, 
Cyprus, Latvia, Lithuania, Hungary, Malta, Poland, the 
Slovak Republic, Slovenia, Romania, Bulgaria, and 
Croatia. 

The following Member States: Austria, Belgium, 
Denmark, Finland, France, Germany, Greece, Ireland, 
Italy, Luxembourg, the Netherlands, Portugal, Spain, 
Sweden, and the United Kingdom. 

Euro (currency) 

Seventh Framework Programme 

Gross Domestic Product 

Horizon 2020 

Higher Education Institution(s) 

Information and Communication Technology 

I3 Instrument 

Interregional Innovation Investments Instrument. 

MA(s) 

MS 

Managing Authority(ies) 

Member State(s) 

3 

 
 
 
NUTS 

OP(s) 

R&D 

RDI 

RTD 

RTO 

S3 

SME 

SWD 

ToC(s) 

Nomenclature des Unités Territoriales Statistiques 

Operational Programme(s) 

Research and Development 

Research Development and Innovation 

Research and Technological Development 

Research and Technology Organisation 

Smart Specialisation Strategies 

Small and Medium-Sized Enterprise 

Staff working document 

Theory of Change(s) 

4 

 
 
1. 

INTRODUCTION 

1.1.  Context and purpose 

The general objective of this SWD is to report on the ex-post evaluation of investment 
in  Research  and  Technological  Development  (RTD)  infrastructures  and  activities 
supported  by  the  European  Regional  Development  Fund  (ERDF)  in  the  period 
2007-2013. To carry out the evaluation, the European Commission, Directorate-General 
for  Regional  and  Urban  Policy,  contracted  CSIL  (Centre  for  Industrial  Studies),  in 
partnership with Prognos AG and Technopolis Consulting Group Belgium. The contract 
started on 1 September 2019 and the final report was published on 10 December 20211. 
This SWD is based on that work. 

1.2. 

Scope of the evaluation 

In total, for the period 2007-2013 around EUR17 billion of ERDF resources (5% of the 
total  allocation)  were  invested  through  215  Operational  Programmes  in  projects 
supporting RTD2 infrastructure, competence centres and activities in Member States and 
regions.  Based  on  reporting  by  Managing  Authorities  on  categories  of  expenditure, 
around EUR11 billion  of that sum went  to support for infrastructure and around EUR6 
billion to support for activities.  

This  evaluation  focused  on  53  Operational  Programmes  (OPs)  selected  by  the 
European Commission out of the total of 215 OPs funded by the ERDF. These cover 18 
of the 28 Member States involved, and a substantial amount of RTD investments (with 
EUR14.64 billion in funding, i.e. about 85% of the EU total funding for the relevant 
themes) under diverse contexts. Within the selected OPs, the evaluation study focused on 
two categories of ERDF expenditure (01 – RTD activities in research centres, and 02 – 
RTD infrastructure and centres of competence in a specific technology)3. Public support 

1 

https://ec.europa.eu/regional_policy/en/information/publications/evaluations/2021/evaluation-of-
investments-in-research-and-technological-development-rtd-infrastructures-and-activities-supported-
by-the-european-regional-development-funds-erdf-in-the-period-2007-2013. 

2  The expression ‘Research and Technological Development (RTD)   is  a  reference  to  the  2007-2013 
categories of intervention, reflected in the title of the evaluation and more generally to EC policy lines 
for the funding period under analysis. By contrast, the expression ‘Research and Development’ (R&D) 
derives from the EC 2020 strategy, Eurostat indicators definition, and the most common international 
statistical  and  policy  reference  for  the  sector. Therefore,  despite  partial  overlap,  the  two  expressions 
are used throughout the SWD with two distinct meanings. 

3  Categories  of  ERDF  expenditure  01  and  02  were  deliberately  analysed  later  than  the  main  ex-post 
evaluation 2007-2013, as further explained in Annex II.2. More specifically, they include: support for 
RTD  activities  in  research  centres  (e.g.,  scientific  R&D  activities;  collaborative  research  activities; 
support  for  the  internationalisation  of  research  activities;  development  of  researchers  and  other 
personnel involved in  R&D activities; support for technology-transfer activities; and the valorisation 
of research results), support for RTD infrastructures, and support for competence centres in a specific 
technology. 

5 

 
 
                                                           
for investment in firms directly linked to research and innovation was not covered by the 
evaluation4. 

Different levels of analysis were considered: 

  OP level: this level analyses the strategies and policy mixes put forward by the 
OPs,  their  linkage  with  specific  contexts  and  their  linkages  with  other  policies. 
This was done in a more general way for the 53 OPs and in a more specific way 
for the OPs analyses within seven case studies 

  Country-level:  this  level  reviews,  in  detail,  the  use  of  policy  instruments  for 
RTD  in  different  national  contexts  (including  relevant  OPs),  as  well  as  the 
rationale underpinning the policy mix. It was addressed in seven case studies; 

  Instrumental  level:  this  level  analyses  the  theories  of  change  (chains  of  effect 
and mechanisms) of selected types of intervention, allowing the identification of 
‘lessons  learnt’  and  providing  evidence  to  support  debates  and  future  policy 
considerations;  

  Project  and  beneficiary  level:  this  level  of  analysis  explores  and  describes  in 
detail  the  diversity  of  RTD  projects  funded  under  Cohesion  Policy  in  the  53 
representative OPs, covering 18 Member States. It also classifies them according 
to well-defined typologies. 

Looking  at  different  levels  of  analysis  yielded  complementary  benefits,  providing 
different perspectives of analysis.  

Figure 1.  Representative sample of 53 Operational Programmes and Member States 

4  Nonetheless,  evidence  from  the  evaluation  revealed  that,  in  a  limited  number  of  cases,  support  for 

RTD activities in SMEs was mistakenly encoded under codes 01 or 02. 

6 

With Major Project 
 
                                                           
Source: Ex-post evaluation report (2021) 

1.3. 

Structure of the SWD 

In compliance with Better Regulation Guidelines, the remaining sections are organised as 
follows: 

  Section  2  describes  the  expected  outcome  of  the  intervention.  In  particular,  2.1 

focuses on ERDF priorities, while 2.2 describes the intervention logic. 

  Section  3  outlines  how  the  situation  evolved  over  the  evaluation  period.  More 
specifically,  3.1  describes  the  actual  funding  allocation,  3.2  reports  on  the  key 
achievements, 3.3 reconstructs the causal links, and 3.4 examines external factors. 
  Section  4  reports  on  the  evaluation  findings.  The  section  is  organised  by 
evaluation  criteria  (effectiveness,  efficiency,  coherence,  EU  added  value, 
relevance). 

  Section  5  summarises  the  main  conclusions,  policy  considerations,  and  open 

questions. 

  The  Annexes  provide  complementary  information  on  procedures  (Annex  I), 
methodology  (Annex  II),  evaluation  matrix  (Annex  III),  costs  and  benefits 
(Annex IV), and stakeholder consultation (Annex V). 

2.  WHAT WAS THE EXPECTED OUTCOME OF THE INTERVENTION? 

2.1.  Background and needs 

ERDF priorities in the RTD field 

The Europe 2020 strategy adopted in 2010 put research, development and innovation at 
the  top  of  the  EU  agenda  for  smart,  sustainable  and  inclusive  growth.  In  that  period, 
R&D expenditure was increasing across the EU but was still considerably below the 3% 
target  of  investment  in  R&D  as  a  share  of  gross  domestic  product  (in  2007,  this  share 
ranged  from  a  maximum  of  3.35%  for  Finland  to  less  than  1%  in  most  Central  and 
Eastern European countries, the only exceptions being Czechia with 1.31% and Estonia 
with 1.07%)5.  

Since then, despite a yearly increase of 1% since 2000, R&D expenditure in the EU has 
remained  below  the  set  target.  Its  share  of  world  expenditure  on  R&D  has  declined 
compared  with  the  EU’s  global  competitors.  Only  Germany,  Denmark  and  Cyprus 
reached  their  targets,  with  the  greatest  progress  reported  by  Poland,  Greece,  Estonia, 
Hungary and Czechia (European Commission, 2020).  

To  contribute  to  the  overarching  goal  of  making  the  EU  a  leading  knowledge-based 
economy,  the  Community  strategic  guidelines  for  2007-13  emphasised  the  need  to 
increase  and  improve  investment  in  research  and  technological  development  (RTD).6 
Priority was allocated to the support of existing poles of excellence, making better use of 

5 

6 

Source: Eurostat. 

https://eur-lex.europa.eu/EN/legal-content/summary/cohesion-policy-in-support-of-growth-and-jobs-
community-strategic-guidelines-2007-13.html. 

7 

 
                                                           
existing potential and avoiding the excessive spatial  dispersion  of resources.  Increasing 
private  and  public  investment  in  RTD  and  innovation  also  encouraged  partnerships 
across the different regions of the EU. Meanwhile, creating and exploiting a larger pool 
of high-quality research talent in Europe was identified as a key strategy. In line with the 
systemic approach to innovation, RTD infrastructures, competence centres, and activities 
were  considered  necessary  to  help  structure  the  scientific  community  and  contribute  to 
the construction of an efficient research and innovation ecosystem. The role of research 
infrastructures  as  key  to  enhancing  national  and  regional  RTD  capacities  was  also 
emphasised. The development of pan-European research infrastructures and their impact 
on the regional ecosystems were seen as a key driver of economic growth.  

R&D investments  were  closely linked to  the objective of fostering regional  innovation. 
The report on ‘Creating an Innovative Europe’7 stressed the key role of the regional level 
to  foster  an  innovative  Europe  and  underlined  the  choice  to  focus  a  large  share  of 
Cohesion  Policy  resources  for  2007-2013  on  the  Innovation  priority.  About  EUR86.4 
billion,  nearly  25%  of  the  total  allocated  budget,  went  towards  innovation  in  the 
broader  sense,  including  research  centres  and  infrastructure,  technology  transfer  and 
innovation  in  firms,  the  development  and  diffusion  of  information  and  communication 
technologies, and human capital development. These investments represented more than 
a  tripling  of  absolute  financial  resources  dedicated  to  innovation  and  R&D  compared 
with the previous period (2000-2006). The amount  also  greatly  exceeded the budget  of 
the  7th  Framework  Programme  for  Research  (EUR50.5  billion)  and  the  Framework 
Programme for Competitiveness and Innovation (EUR3.6 billion). 

Although  still  significant,  looking  at  only  the  RTD  component  of  the  overall  financial 
resources, the figure is much lower. In the period 2007-2013, around EUR17 billion of 
ERDF resources (almost 5% of the total ERDF allocation) was invested through 215 
OPs in projects supporting RTD infrastructure, competence centres and activities in EU 
Member States and regions (codes 01 and 02). More than EUR11 billion (65.5% of the 
total)  was  allocated  to  research  infrastructure  support  (code  02)  and  around  EUR5.8 
billion (34.5% of the total) to research activities support (code 01).  

Investments needs and regional differences 

The analysis of the 53 OPs (documentary review complemented by direct interviews with 
the  Managing  Authorities)  shows  that  the  53  OPs  reviewed  generally  did  not  refer  to 
specific,  individual  rationales  of  RTD  investments,  but  they  recognised  multiple 
investment needs.  

Although different formulations were used in programming documents, two consisted of 
the  key  barriers  identified  by  policymakers.  On  the  one  hand,  there  was  the  lack  of  a 
critical  mass  of  infrastructure  endowments  and  research  capacities  to  enable  the 
production  of  top-class  research.  On  the  other,  there  was  the  need  to  increase  the 
industrial relevance of the regional science base by linking existing or emerging poles 
of  scientific  excellence  to  areas  of  industrial  strength.  While  the  lack  of  density  of 

7 

http://ec.europa.eu/invest-in-research/pdf/download_en/aho_report.pdf. 

8 

 
                                                           
research capacities seemed more relevant  for the EU13 countries, the asserted needs of 
the EU15 MSs related more commonly to better science-industry linkages.  

Figure 2.  Rationale for RTD investment by type of country – share of EU13 and EU15 
OPs mentioning ‘need’   

Note: Each OP can be associated with more than one need. The percentage was calculated as the number of 
EU13 (or EU15) OPs mentioning need(s) over the total number of EU13 (or EU15) OPs. The total number 
of EU13 OPs was 16, while the total number of EU15 OPs was 37.  

Source: Ex-post evaluation report (2021), based on a review of the OP programming documents 

The diagnosis reflected in the programming documents is not surprising. It mirrors fairly 
accurately the situation at the beginning of the programming period, as highlighted by the 
cluster  analysis  and  the  literature  and  policy  documents  (European  Commission,  2009; 
Radosevic and Lepori, 2009). For all the Central and Eastern European countries that 
joined the EU in 2004 and 2007, the 2007-2013 Operational Programmes represented the 
first systematic set of interventions addressed in the research field. They were facing 
the  most  considerable  challenges:  the  public-research  system  was  fragmented,  with 
research  and  higher  education  systems  being  split  between  academies  of  science  and 
universities;  research  technologies  were  obsolete;  collaboration  with  industry  was 
minimal,  due  to  a  lack  of  strategic  awareness  of  the  importance  of  innovation  among 
companies,  and  due  to  insufficient  orientation  of  research  towards  industry  needs.  The 
main problem was linked to education and research infrastructures, which were in poor 
condition  and  outdated.  Therefore,  they  failed  to  contribute  to  top-level  research  and 
educational  activities, 
their  attractiveness  for  companies  and 
thereby 
international research networks.  

limiting 

Compared  with  EU13  Member  States,  in  the  selected  Western  European  countries, 
regional  innovation  systems  were,  overall,  relatively  more  mature.  They  had  a 
stronger network of universities and research centres and some already existing structures 
to  favour  technological  diffusion  and  science-industry  collaboration,  such  as  clusters, 
poles  of  excellence,  and  science-and-technology  parks.  Nonetheless,  there  were 
numerous differences across both countries and regions, both in terms of perceived RTD 
needs and designed strategic approaches. On the one hand, Germany was one of the  10 
most  research-intensive  economies  worldwide,  with  only  limited  development  needs  in 

9 

0%20%40%60%80%100%Lack of sufficient physical (science and technology)infrastructureInsufficient business capabilityProblems in the interaction and collaboration among actors inthe innovation systemRisk aversion of the private sectorHuman capital and skills deficienciesInadequacies in research institutionsWeak or failing framework conditionsEU13EU15 
 
certain areas of R&D in convergence regions. Here, traditionally strong SMEs played a 
notable  role  in  innovation,  and  there  was  intensive  historical  cooperation  between  the 
science and enterprise sectors. On the other hand, Portugal and Italy were still follower 
countries.  Italy,  in  particular,  suffered  from  significant  regional  disparities  beyond  a 
generalised weak research system at the national level. This problem also stemmed from 
a long tradition of scarce mobilisation of financial resources for research. 

Figure 3.  RTD regional contexts: results of a cluster analysis - 2007 

Source: Ex-post evaluation report (2021) 

These agglomeration effects were not surprising. Indeed, widespread empirical evidence 
shows that research and innovation activities tend physically and spatially to agglomerate 
in specific geographical areas. Some studies (e.g., Falk, Hölzl and Leo, 2007) highlight 
how  the  nationally  fragmented  nature  of  the  public  research  system  can  contribute  to 
innovation gaps between regions. This fragmentation does not enable the exploitation of 
scale effects and/or efficiency in RTD investments (particularly for large infrastructures) 
that would otherwise derive from the concentration of resources on selected priorities. 

2.2.  Expected outcome 

Infrastructure investments 

Infrastructure  investments  were  selected  as  a  result  of  a  top-down  approach,  with 
national  authorities  undertaking  needs  assessments,  and  national  or  EU  roadmap 
exercises,  as  part  of  the  implementation  of  the  European  Research  Area  (ERA).  The 
extent  to  which  the  needs  assessment  was  reflected  in  the  selection  of  funded 
infrastructures varied. The roadmap exercises followed a largely top-down approach and 
identified  national  infrastructure  of  pan-European  relevance,  contributing  to  the 
construction of the ERA and the identification of European investment priorities by the 
European  Strategy  forum  on  Research  Infrastructures  (ESFRI).  For  example,  the  ELI 
Beamlines  Facility  was  planned  as  part  of  the  European  roadmap  of  next-generation 
major research facilities that the ESFRI Forum prepared (ESFRI Roadmap) 

10 

 
 
Exceptions in infrastructure planning  and selection were  represented by the  ICT-based 
infrastructures in higher education institutions (HEIs), and the Major Project analysed 
in the German case study. The former was the result of a thorough bottom-up and top-
down analysis. 

The  funded  infrastructure  projects  differed  in  their  underlying  rationales.  A  more  in-
depth  analysis  of  funded  projects  and  relative  beneficiaries  showed  a  wide  variation  in 
their design and implementation, reflecting different objectives.  

In  the  Seventh  Framework  Programme  of  the  European  Community  for  research, 
technological development and demonstration activities (FP7)8, and later within Horizon 
2020  Regulation  (EU)  No  1291/2013  of  11  December  2013,  the  term  ‘research 
infrastructure’  referred  to  ‘facilities,  resources,  and  services  used  by  the  research 
communities  to  conduct  research  and  foster  innovation  in  their  fields.  They  include 
major scientific equipment  (or sets of instruments), knowledge-based resources such as 
collections, archives and scientific data, e-infrastructures, such as data- and computing 
systems and communication networks, and any other tools that are essential to achieve 
excellence in research and innovation’9.  

This category of interventions accounted for the largest share of ERDF resources in the 
OPs  under  consideration,  in  line  with  the  fact  that  most  OPs  indicated  infrastructural 
failures  as  one  of  the  key  investment  needs  to  be  tackled  by  the  OP  in  question.  The 
specific  nature  of  the  projects  funded  ranged  from  support  for  new  or  reconstructed 
infrastructure, such as buildings, plants or laboratories, to investment in research-related 
equipment, such as lab instruments, machinery or highly specialised apparatus, as well as 
supporting  infrastructure.  These  instruments  addressed  the  lack  of  sufficient  or  modern 
physical and technological infrastructure, an essential component in fostering knowledge 
creation.  

Among many of the newer Member States of the EU, there was a research infrastructure 
gap that impacted the effectiveness of their R&D capabilities. As highlighted by some of 
the case studies, research infrastructures were underfunded. They were thus not providing 
sufficient capacity and quality for researchers, as research equipment was outdated or not 
in  line  with  modern  research  standards.10  This  policy  intervention  aimed  at  upgrading 
existing infrastructure and equipment and replacing obsolete or outdated instances 
of these. In so doing, it sought to develop new research capacities that aimed to match the 

8  Decision  No  1982/2006/EC  of  the  European  Parliament  and  the  Council  of  18  December  2006 
concerning  the  Seventh  Framework  Programme  of  the  European  Community  for  research, 
technological development and demonstration activities (2007-2013) (OJ L 412, 30.12.2006, p. 1). 

9  Article 2 (6) of Regulation (EU) No 1291/2013 of 11 December 2013 of the European Parliament and 
of  the  Council  of  11  December  2013  establishing  Horizon  2020  -  the  Framework  Programme  for 
Research  and  Innovation  (2014-2020)  and  repealing  Decision  No  1982/2006/EC  (OJ  L  347, 
20.12.2013, p. 104-173). 

10  Case Study Report: Czechia - Evaluation of investments in Research and Technological Development 
(RTD) infrastructures and activities supported by the European Regional Development Funds (ERDF) 
in the period 2007-2013. 

11 

 
                                                           
level  of  quality  and  research  excellence  at  European  and  international  levels.  The 
intervention  was  not  seen  as  an  end  in  itself  but  rather  regarded  as  a  mechanism  that 
would improve the quality of research and the innovative capacity of economies. 

In  terms  of  the  intentions  and  expectations  declared  by  the  OP  or  the  Major  Project 
applications, the establishment of new research infrastructures was expected to contribute 
to three categories of impact:  

  Scaling  up  research.  The  goal  of  many  Major  Projects  was  to  achieve  more 
ambitious research projects with respect to research status in the region at that time. 
In  some  cases,  the  objective  was  even  more  ambitious,  i.e.,  to  promote  world-class 
research  with  international  standards  of  excellence  in  a  specific  topic.  The  goal  of 
scaling up research was complemented by the necessity of scaling up facilities (new 
research  buildings  and  equipment)  and  supporting  their  cost.  In  response  to  these 
costs,  several  Major  Projects  were  expected  to  attract  increasing  volumes  of 
international  research  grants  (both  private  and  public),  increase  the  number  of 
research contracts, and to build up knowledge-sharing networks.  

  Increasing collaboration with industry. Most of the research infrastructure projects 
funded  in  the  2007-2013  period  were  conceived  with  clear  attention  being  given  to 
facilitating a potentially positive spillover to the business environment and thus (also) 
possible benefits in terms of knowledge transfer, patenting and commercialisation of 
innovation.  

  Seeding  early-stage  researchers.  Some  major  infrastructural  projects  indicated 

educational activities and the training of early-career researchers as a priority. 

Figure 4.  Generalised (hypothesised) Theory of Change of infrastructure investments for 
research 

Source: Ex-post evaluation report (2021) 

ICT-based infrastructures are another identified subgroup of the broader infrastructure 
investment  for  research.  Under  the  influence  of  the  increasing  importance  of  data 
digitalisation and open science as means to improve the accessibility, interoperability and 
re-use  of  scientific  publications  and  data,  ICT-based  infrastructures  provide  digital-

12 

 
  
 
based  services  and  tools  for  data  and  computing-intensive  research  in  virtual  and 
collaborative  environments.  These  environments  offer  services  and  tools  that  support 
whole research cycles and ease or foster the movement of scientific data across scientific 
disciplines. Moreover, open-data spaces can be created, and scientific workflows can be 
improved  by  connecting  data  sets  from  diverse  disciplines.  Researchers  also  have  the 
opportunity to link with high-performance computational systems, and they can therefore 
improve the overall capacity and scope of their research (Thanos, 2010). 

By their nature, ICT-based infrastructures are meant to support researchers in their work 
and serve various research communities. Generally, the literature suggests that ICT-based 
infrastructure  investments  positively  affect  innovation  capacities.  This  is  because  this 
infrastructure  enables  researchers  to  handle  big  data  sets  efficiently  through  the  use  of 
high-performance computational systems.  Together with  the improved dissemination of 
scientific  output  and  exchange  with  other  researchers,  this  means  that  innovative 
capacities increase (Thanos, 2010). 

Figure 5.  Generalised (hypothesised) Theory of Change of infrastructure investments for 
research 

Source: Ex-post evaluation report (2021) 

The ICT-based infrastructures identified across the OPs can be categorised as follows: 

  rojects  responding  to  advanced  computing  needs:  computing  grids  and 
infrastructures/centres, supercomputers, computing servers, and similar phenomena; 

  Projects  to  manage  more  effectively  data  creation,  storage  and  access:  these 
include  data  digitisation,  data  storage  centres,  open-data  infrastructures,  and  ICT 
network systems; 

  E-infrastructures to deliver e-services (mainly in the health domain) and to connect 

existing resources to a central hub for research and development; and 

  Projects  related  to  investments  in  other  IT  infrastructure  or  equipment  for 

research, which do not fall under any of the previous subtypes.  

13 

 
 
R&D projects and related activities 

Collaborative  R&D  projects  support  research  activities  carried  out  jointly  by  either 
research institutions/HEIs themselves or in collaboration with partners in private industry 
(referred  to  as  science-industry  collaborations).  The  latter  is  widely  considered  an 
essential driver of knowledge-based economies and societies.  

A collaboration project is defined as a project where at least two partners participate in 
the  design  of  the  project  itself,  contribute  to  its  implementation,  and  share  the  risk  and 
output of the project (European Commission, 2006). Collaborative efforts funded by this 
policy  intervention  had  various  aims,  ranging  from  addressing  industrially  relevant  or 
societal challenges to stimulating technological advancement in specific areas to boosting 
international  cooperation  by  conducting  internationally  competitive  high-quality  R&D 
activities. 

Figure 6.  Generalised (hypothesised) Theory of Change of collaborative R&D projects 

Source: Ex-post evaluation report (2021) 

The immediate outcomes of the policy intervention were aimed at increasing the number 
of  joint  projects  and  activities,  both  between  R&D  institutions  themselves  and  in 
collaboration  with  private-sector  partners.  Moreover,  researchers  across  supported 
research institutions, and those in private enterprises, were expected to develop skills and 
competencies that would increase their scientific-technological knowledge. The research 
pursued  by  these  collaborative  entities  would  be  expected  to  lead  to  technological 
advancement  and  thereby  facilitate  the  improvement  or  invention  of  new  products, 
processes or services. These immediate outcomes (during the project) were expected to 
be  followed  by  more  intermediate  outcomes  (after  the  project),  such  as  the  economic 
benefits  stemming  from  the  commercial  valorisation  of  R&D  results.  Moreover,  an 
enhanced knowledge transfer between science and industry partners was to be expected. 

Individual  research  projects  were  supported  in  various  scientific  fields  to  strengthen 
the  region’s  scientific  and  technological  capacity  in  question.  This  policy  intervention 
consisted of support for existing research fields, for which applications were investigated, 
and support for more ‘exploratory’ or ‘foundational’ research, which targeted areas that 

14 

 
 
had  great  potential  for  innovation  but  were  untapped.  To  some  extent,  the  intervention 
also  sought  to  improve  knowledge  and  technology  transfer  into  the  industry,  which 
would involve the economic valorisation of new scientific or technological products and 
processes. 

The  focus  of  ERDF  support  shifted  significantly  in  the  last  two  programming  periods. 
However, this kind of R&D support was a central feature of the Lisbon National Reform 
Programmes  (2006)  since  large  disparities  between  the  EU  Member  States  and  regions 
were  observed,  and  a  persistent  gap  existed  at  the  global  level  compared  with 
competitors.  Early-stage  (foundational)  and  exploratory  research  often  do  not  have 
specific,  predetermined  commercial  applications  and  rather  serve  to  generate  new 
knowledge and further develop innovative skills in research institutions; this is crucial for 
long-term, rather than immediate, R&D results. The fact that this type of research carries 
high  risks  and  is  therefore  of  reduced  interest  to  private-sector  investment  is  well 
documented in the literature (European Commission, 2017a). The uncertainty relating to 
the return of investment, and the sunk costs involved in ensuring a critical mass in terms 
of knowledge and skills accumulation, which is often a precondition for any meaningful 
R&D  results,  would  otherwise  induce  underinvestment  in  such  forms  of  research  and 
innovation: this, therefore, underscores the significance of sufficient public investment in 
these areas. 

Figure 7.  Generalised (hypothesised) Theory of Change of individual R&D projects 

Source: Authors on the cross-case study analysis performed in Task 4 

The  classification  process  identified  a  residual  category  of  interventions  (accounting 
for  less  than  3%  of  the  total  ERDF  expenditure  for  RTD)  that  could  not  be  defined  as 
either R&D projects or infrastructure investments but were still relevant in the RTD field. 
They included the following. 

  Internationalisation  of  research,  mainly  including  projects  related  to  promoting 
international  collaboration  among  HEIs  and  RTOs,  or  international  mobility 
programmes for scientists and students. 

15 

 
 
  Capacity building for research11, including projects addressing the development of 
researchers  involved  in  R&D  activities  (including  support  for  PhD  programmes)  or 
activities improving the capacity of institutions to raise funds, increase international 
visibility, or better manage the research process. 

  Science  dissemination  to  the  general  public,  including  projects  to  increase  public 

engagement in and awareness of science.  

  Intellectual  Property  Protection  Instruments,  a  very  specific  policy  instrument 
implemented by  the Polish OP  for the  Innovative Economy,  benefiting those RTOs 
and HEIs that had previously implemented an industrial R&D project, supported by 
the same OP. 

  Operating  subsidy,  providing  generic  support  for  the  functioning  of  research  and 
technology  centres (either science and  technology  parks or RTOs), with  no specific 
reference to implementing any RTD  activity or infrastructure. These responded to a 
very  broad  and  ‘unspecified’  logic,  with  no  precise  outputs  and  results  expected 
beyond the operation of the research institute itself. 

3.  HOW HAS THE SITUATION EVOLVED OVER THE EVALUATION PERIOD? 

3.1.  Projects and beneficiaries 

The analysis of expenditure mapped more than 20,000 projects, half of which occurred in 
Spain  alone12.  Altogether,  they  accounted  for  almost  EUR14.9  billion,  which  is  2% 
higher than the allocated amount reported in the Final Implementation Report. The ERDF 
contribution  was  typically  provided  in  the  form  of  non-repayable  grants.  Private  co-
funding  was  provided  only  for  15.7%  of  the  almost  8  000  projects  for  which  this 
information is available. The average duration of projects was three years.  

Projects  funded  infrastructure  investments,  R&D  activities  and  other  types  of 
activity13. A residual share was either not classifiable or should not have been classified 

11 

Individual and collaborative R&D projects could also include a capacity-building component, although 
this was not the main objective of the project. 

12  Spain  had  an  untypical  monitoring  system  that  lacked  any  project-level  identification  code.  This 
prevented the aggregation of all expenditure data at project level. In the report, approximate data and 
information  on  the  Spanish  programmes  are  provided.  However,  to  preserve  the  accuracy  and 
reliability  of  the  rest  of  the  data,  they  are  not  considered  when  producing  aggregate  project-level 
statistics. 

13  The  comparison  between  the  types  of  intervention  attributed  to  individual  projects,  and  the  official 
category of expenditure under which the same project was coded by the Managing Authorities, reveals 
that a certain share of projects had been miscoded, and that it was indeed necessary to reclassify the 
projects  under  a  new  taxonomy  to  have  a  more  precise  distinction  between  investments  for  RTD 
activities  and  RTD  infrastructures.  More  specifically,  around  9%  of  projects  and  19%  of  ERDF 
contributions  coded  as  01  referred,  in  fact,  to  infrastructure  investments.  When  one  considers  the 
projects  coded  as  02,  it  is  found  that  83%  of  these  projects  and  94%  of  the  respective  ERDF 
contribution  had  been  properly  coded,  with  the  remaining  share  being  research  projects,  other  RTD 
activities  (e.g.  capacity-building  activities  delivered  to  research  centres)  or  other  non-RTD-related 
investments  (e.g.,  infrastructure  investment  in  incubator  centres  to  support  entrepreneurship  and 
innovation, but not RTD activities). 

16 

 
                                                           
under  codes  01  and  02.  To  further  distinguish  the  logic  of  interventions  among  the 
different  projects,  these  main  types  were  further  split  into  10  different  types  of 
intervention. The largest share of ERDF expenditure was concentrated on support for 
infrastructure investment (72% of total  expenditure),  with  infrastructure investment 
for research absorbing more than half of ERDF expenditure (57%). 

In terms of numbers, the most common types of intervention were R&D projects (55% of 
the  total).  A  smaller  share  of  projects  (6%)  and  expenditure  (3%)  was  allocated  to 
implementing  other  RTD  activity.  A  residual  portion  (3%  of  the  projects  and  ERDF 
expenditure) fell into a fourth category, namely activities that were not strictly related to 
RTD and should instead have been classified under different codes. 

Figure 8.  Types of funded projects, number and million EUR 

Source: Ex-post evaluation report (2021), based on monitoring data from the 46 OPs (excluding Spain) 

Funded projects are, on average, of a significant financial scale: they received an average 
ERDF  contribution  of  EUR1.3  million,  while  24  Major  Projects  accounted  for  10%  of 
total  ERDF  expenditure  in  the  reported  OPs  (i.e.,  EUR1.49  billion).  Infrastructure 
investments  for  education  activities  cost,  on  average,  much  more  than  other  types  of 
investment.  

17 

 
 
The  funded  research  was  predominantly  conducted  in  the  fields  of  Engineering  and 
Technology,  Natural  Sciences  and  Medical  and  Health  Sciences.  The  vast  majority  of 
projects  and  expenditure  pertained  to  applied  research,  generally  with  a  possible 
industrial application. 

Figure 9.  Type of research, total ERDF contribution (million EUR) 

Source: Ex-post evaluation report (2021), based on monitoring data from the 46 OPs (excluding Spain) 

Most beneficiaries (almost 77%) are publicly owned organisations. Higher Education 
Institutes  (HEIs)  and  Research  and  Technological  Organisations  (RTOs)  account  for 
nearly 88% of the sample of lead beneficiaries and received nearly 83% of total ERDF 
contribution. Enterprises comprised 4.5% of total direct beneficiaries, 70% of these being 
SMEs.  They  played  the  role  of  partners  in  collaborating  R&D  projects,  typically  with 
HEIs  and  RTOs.  There  were  recurrent  beneficiaries:  on  average,  each  body/institution 
received funding under four or five projects. 

18 

 
 
Figure 10. Type of lead beneficiaries, number and million EUR 

Source: Ex-post evaluation report (2021), based on monitoring data from the 46 OPs (excluding Spain) 

A  total  of  about  4  000  different  institutions  (including  businesses)  can  be  identified 
among the almost 24 000 lead beneficiaries (9 973 for the 46 OPs and about 13 000 for 
the  7  Spanish  OPs).  When  excluding  the  7  Spanish  OPs,  the  total  falls  to  about  2  000 
different institutions (including almost 580 HEIs, more than 720 RTOs, and nearly 470 
businesses).  Nonetheless,  data  on  the  total  ERDF  contribution  shows  that  more  than 
13%  of  the  ERDF  support  for  RTD  provided  to  lead  beneficiaries14  was 
concentrated  on  10  institutions,  and  more  than  20%  on  20  institutions,  with  the 
Fraunhofer-Gesellschaft  Institute  in  Munich  (Germany)  receiving  more  than  2%  of  the 
total. The other institutions where the largest share of ERDF funding was provided were, 
conversely,  located  mainly  in  Czechia  and  Poland,  the  largest  recipient  countries 
alongside Germany. 

14  Excluding the seven Spanish OPs, the total ERDF contribution to lead beneficiaries amounted to EUR 

11.7 billion. 

19 

 
 
                                                           
Figure 11.  ERDF expenditure on RTD investments by NUTS2 regions in the 
targeted OPs 

Note: expenditure of national or multi-regional OPs was regionalised based on the location of project 
beneficiaries 

Source: Ex-post evaluation report (2021), based on monitoring data from the 46 OPs (excluding Spain) 

Overall,  evidence  on  expenditure  on  RTD  infrastructures  and  activities  highlights 
patterns of territorial concentration as a result of eligibility rules on the one hand and 
territorial  concentration  of  RTD  capacities  on  the  other.  More  than  50%  of  ERDF 
expenditures in the selected OPs was concentrated in Poland, Germany and the Czechia, 
while 70% was concentrated in convergence regions and 64% in urban areas.  

3.2.  Key achievements 

The monitoring system set up by the European Commission provided a first set of data 
on  ERDF  investment  achievements.  In  compliance  with  Council  Regulation  (EC)  No 
1083/2006  laying  down  general  provisions  on  the  ERDF,  the  ESF  and  the  Cohesion 
Fund15, indicators were reported by the Managing Authorities to assess the achievements 
of each OP. Annex II(4) presents the core and common indicators associated with RTD 
investments across the 53 OPs, along with their initial target values. As confirmed by the 
analysis  carried  out  in  the  case  studies,  monitoring  systems,  in  general,  provided  a  fair 
amount  of  quantitative  evidence  regarding  instrument  activity  and  output  performance. 
They were much weaker in the context of long-term achievements. 

The monitoring indicators, which were not specific for the categories of expenditure 01 
and  02  but  referred  to  the  overall  OP,  showed  that  the  total  number  of  research  jobs 
created  across  the  53  OPs  amounted  to  20  500.  Meanwhile,  the  total  number  of  RTD 

15  https://eur-lex.europa.eu/legal-content/en/ALL/?uri=CELEX%3A32006R1083. 

20 

 
 
                                                           
projects  supported  was  over  51  000,  and  the  number  of  cooperation  projects  between 
businesses and research institutions was 15 000. As compared with the initial target, the 
degree  of  achievement  was  mixed,  and  most  of 
the  OPs  demonstrated  an 
overachievement  for  some  indicators  and  underachievement  for  others.  Nonetheless,  as 
already  noted  in  previous  evaluation  studies16,  monitoring  indicators  present  some 
shortcomings when it comes to assessing OP effectiveness. 

In general terms, the quality and robustness of ERDF RTD performance measurement 
systems  and  accompanying  indicators  were  found  to  be  weak.  The  amount  of  data 
available for evaluators to assess selected instruments at this level was limited. 

In some cases,  the  evidence from the monitoring  indicators could  be complemented by 
mid-term and ex-post evaluations conducted by the Managing Authorities or by external 
independent  studies.  Impact  evaluations  on  individual  policy  instruments  or 
programmes  carried  out  at  the  national/regional  level  are  not  implemented  on  a 
systematic  basis.  Nonetheless,  these  studies,  when  available,  can  provide  richer  and 
more qualitative information on effectiveness. The scope of the evaluation studies varies 
to a great extent (e.g. they may focus only on selected policy interventions and not on the 
overall  policy  mix  for  RTD).  Equally  variable  are  the  adopted  methodologies 
(counterfactual  impact-evaluation  studies,  qualitative  case  studies,  macroeconomic 
modelling, etc.). Overall, this evidence is not sufficient to provide conclusive information 
about overall OP effectiveness in addressing pre-existing RTD barriers. 

Finally,  stakeholder  perceptions  regarding  achievements  were  collected  during  the 
seminar held as part of the evaluation. Although reflecting the views of a limited number 
of  invited  programme  managers  and  representatives  of  beneficiaries,  they  provide  an 
initial picture of the relevant achievements of the 2007-2013 ERDF support for RTD. 

Therefore,  the  2021  evaluation  filled  the  gaps  by  making  use  of  other  sources,  as 
described below and detailed in the annexes to this SWD. The analysis led to identifying 
the following four categories of achievement. 

R&D personnel and researchers 

The  availability  of  more  and  better-skilled  R&D  personnel  and  researchers  in  a  region 
improves  its  R&D  capacities  and  makes  it  possible  to  increase  both  the  volume  and 
quality of knowledge production. Between 2007 and 2017, the average growth rate in the 
number  of  R&D  personnel  and  researchers  in  the  target  regions  was  40%,  and  the 
average  regional  ERDF  expenditure  on  infrastructures  and  R&D  projects  in  HEIs  was 
about  EUR35  million.  The  econometric  analysis  shows  that  ERDF  investments  in 
research  infrastructures  and  individual  R&D  projects  in  HEIs  contributed  to  an 
increase  in  the  number  of  R&D  personnel  and  researchers  at  the  regional  level. 
Such  a  relationship  was  valid  on  average,  i.e.  without  any  statistically  significant 

16  See 

for 

instance  WP0  of 

the  2007-2013  Ex  Post  Evaluation  of  ERDF  and  CF 
https://ec.europa.eu/regional_policy/policy/evaluations/ec/2007-2013_en  and  WP1  of  the  2014-2020 
Ex Post Evaluation of ERDF and CF   https://ec.europa.eu/regional_policy/policy/evaluations/ec/2014-
2020_en. 

21 

 
                                                           
differences  between  the  EU13  and  EU15  or  between  convergence  and  competitiveness 
regions17.  

The mechanisms through which this achievement was made possible are described in the 
case  studies.  One  channel  of  effects  was  that  triggered  by  individual  R&D  projects. 
Evidence indicates how funded projects facilitated an increase in the qualification levels 
of  researchers  and  enabled  the  training  of  young  scientists,  as  evidenced  by  the 
completion of around 70 PhDs as a direct result of the funded projects in Saxony. 

The main channel of effects, however, lay through infrastructure development. Evidence 
illustrates  how  support  for  infrastructure  investment  in  research  contributed  to  the 
creation  or  modernisation  of  public  R&D  facilities  and  essential  scientific 
equipment;  the  evidence  suggests  that  newly  purchased  equipment  created  more 
respectable  research  environments  and 
increased  research  operating  standards. 
Infrastructure  projects,  such  as  buildings,  needed  time  to  be  constructed  before  they 
could  generate  useful  outcomes,  while  acquiring  research  equipment  and  simpler 
infrastructure (such as laboratories) had more immediate effects. Beneficiaries confirmed 
that the access to modern research equipment contributed to the elimination of previous 
handicaps  that  had  prevented  research  institutions  from  participating  in  research 
endeavours.  Consequently,  the  research  potential  and  capacities  of  beneficiaries 
institutions  were  perceived  to  have  increased  significantly,  rendering  them  more 
attractive and competitive and gaining increased interest from students and researchers. 

Increased  research  capacities  in  universities  also  led  to  better  awareness  of,  and 
interest  in,  industrial  research  needs  and  opportunities.  In  Poland,  the  number  of 
enterprise  R&D  personnel  with  doctoral  degrees  increased  significantly.  The  ERDF 
policy intervention supporting cross-sectoral R&D collaboration played a crucial role in 
redirecting  the  career  pathways  of  many  scientists,  who  became  aware  of  industrially 
oriented  R&D  research  and  the  possibility  of  pursuing  such  research  in  private 
enterprises18. 

The creation or  modernisation of ICT-based infrastructure tackled the urgent  need 
for updated ICT infrastructure. For some institutions, basic ICT infrastructure elements, 
such  as  passive  network  components  or  phone  systems,  remained  at  the  level  of  the 
1990s.  In  other  cases,  some  network  elements  (especially  active  components)  were  so 
outdated that no service was available from the suppliers. In Poland, a total of 59 projects 
funded 2 167 new or rebuilt laboratories. In Czechia, more than EUR24 million of ERDF 
funding  was  directed  at  new,  developed,  or  modernised  ICT  information  infrastructure 
for R&D, including repositories and storage capacities, optical networks, network optical 
elements,  licences  or  databases.  These  outputs  ensured  a  qualitative  shift  in  existing 
services or, in some cases, the establishment of new ones. 

17 

In order to test the validity of this result, the analysis tested a set of interaction terms (first) between 
the  dummy  variable  ‘EU13’  and  the  ERDF  policy  variable,  (second)  between  the  dummy  variable 
‘Convergence region’ and the ERDF policy variable, and (third) between the Type of Territory dummy 
variables and the ERDF policy variable. Interaction terms were never statistically significant.  

18  See the Polish case study.  

22 

 
                                                           
Although ICT-based infrastructure projects need time to be constructed before generating 
useful  outcomes,  certain  ICT  infrastructures  could  be  swiftly  improved.  Evidence  from 
the case studies suggests that the building or modernisation of ICT-based infrastructure 
created  more  professional  research  environments  and  increased  the  quality  of 
education. 

Scientific output 

Scientific output is traditionally measured by the number of scientific publications. In the 
period 2007-2017, scientific publications  almost  doubled in  volume  across  the EU as  a 
whole.  However,  the  EU13  regions  experienced  a  higher  growth  rate  as  compared 
with  regions  in  the  EU15  (145%  against  96%),  highlighting  an  ongoing  catching-up 
process.  In  particular,  some  regions  in  Romania,  Slovakia  and  Czechia  experienced 
values higher than 400%19. Investments in research infrastructure and individual projects 
in HEIs have contributed to this achievement. 

Evidence  indicates  that  the  ERDF  investments  significantly  contributed  to  the 
catching-up  process  of  the  EU13  regions  in  terms  of  scientific  output.  This  was 
partially due to the magnitude of the investments in HEIs (EUR43 million in the EU13 
against  EUR29  million  in  the  EU15  regions)  and  certain  other  factors  (national  public 
expenditure  in  particular).  The  analysis  also  suggests  that  lagging  regions  performed 
better than more developed regions in terms of growth in the number of publications, but 
this  happened  regardless  of  the  ERDF  investments  in  the  type  of  instrument  under 
scrutiny20.  The higher  growth rate recorded by the EU13 regions  was mainly  explained 
by their poorer starting  positions with  respect  to  more developed  regions. For instance, 
the Romanian region of Centru recorded 101 scientific publications in 2007 and 1 116 in 
2017,  with  an  extraordinary  growth  rate  in  the  period  under  assessment.  Indeed,  the 
regression  analysis  shows  a  negative  and  statistical  significance  of  the  coefficient 
associated  with  the  initial  number  of  scientific  publications  in  2007,  pointing  to 
‘diminishing  returns  to  scale’  concerning  the  evolution  of  the  number  of  publications 
over time. 

The  regression  analysis  indicates  a  positive  and  statistically  significant  relationship 
between  ERDF  support  for  research  infrastructure  and  HEIs,  on  the  one  hand,  and  the 
growth rate in the number of scientific publications on the other: on average, the higher 
the  ERDF  expenditure,  the  higher  the  growth  rate  in  the  number  of  scientific 
publications in the period 2007-2017. 

19  The t-test suggests that the mean difference between the EU13 and EU15 was statistically significant 
at 1% level. In our sample, there were four NUTS2 regions with an extremely high rate of growth in 
the  number  of  scientific  publications.  They  were:  RO12  –  Centru  (growth  rate=1,005%);  SK03  - 
Stredné Slovensko (883%); RO31 - Sud – Muntenia (723%); and CZ08 – Moravskoslezsko (549%).  

20  To  test  whether  the  ERDF  policy  variable  was  more  effective  in  lagging  regions,  we  tested  a  set  of 
interaction  terms  between  the  dummy  variable  ‘EU13’  and  the  ERDF  policy  variable,  between  the 
dummy  variable  ‘Convergence  region’  and  the  ERDF  policy  variable,  and  between  the  Type  of 
Territory  dummy  variables  and  the  ERDF  policy  variable.  Interaction  terms  were  never  statistically 
significant. 

23 

 
                                                           
Evidence  from  the  case  studies  helps  to  qualify  the  findings  on  scientific  output.  One 
channel  of  effects  stemmed  from  individual  R&D  projects  consolidating  existing 
knowledge or expanding it to new research fields. Individual R&D projects played a 
significant role in contributing to the increase of scientific and technological output since, 
in  some  regions,  they  were  the  main  source  of  support  for  individual  R&D  projects 
during  this  period.  The  immediate  outcomes  of  individual  R&D  projects  included  an 
increase  in  the  scientific  and  technological  capacity  of  researchers,  allowing  them  to 
increase  existing  expertise  and  develop  new  areas  of  inquiry.  The  reinforcement  of 
teams with more human resources, and the acquisition of essential materials and services, 
ensured  that  the  beneficiaries  could  improve  conditions  to  develop  their  main  research 
lines.  In  Portugal,  almost  all  projects  (over  99%)  funded  under  the  assessed  policy 
instrument  supporting  individual  R&D  projects  produced  publications,  in  contrast  to 
Germany, where it was estimated that around 76% of the funded projects did so. These 
results echoed broader national trends. In Portugal, for instance, from 2005 to 2014, the 
contribution to published knowledge (i.e. in the form of publications) more than doubled, 
with an average annual growth rate of 11%. 

Another  channel  of  effects  arose  from  investments  in  upgraded  research  facilities  and 
equipment. Beneficiaries confirmed that new infrastructure, modern equipment, and first-
class  instruments  attracted  students,  researchers,  and  professors,  both  from  the  country 
and from abroad21. The analysed  ICT-based infrastructure project, for example,  led to a 
profound shift in quality in provided services. It operated and developed the national e-
infrastructure  for  science,  research  and  education,  encompassing  a  computer  network, 
computational grids, data storage and a collaborative environment in a ‘radical’ way. The 
newly  developed  infrastructure  lifted  the  system  towards  the  wider  European  level  and 
opened a gateway to European cooperation. 

Another positive implication of increased scientific expertise and  international visibility 
was that funded entities  were  more  empowered to apply for ambitious  projects and 
also  demonstrated  improved  chances  of  obtaining  third-party  funds  from  both 
federal and international (EU) sources. 

However,  the  econometric  analysis  found  that  there  was  no  statistically  significant 
relationship  between  ERDF  investments  in  HEIs  and  scientific  excellence.  When 
looking at the 2007-2017 rate of growth in the share of regional publications in both the 
top  25% and the top  10% by  citations worldwide, the analysis suggested that scientific 
excellence  was  mainly  driven  by  the  level  of  R&D  development  in  the  region  in  the 
initial  year  (2007),  measured  by  the  level  of  regional  Gross  domestic  expenditure  on 
R&D in that year, and by long-term investments in R&D in the following decade. This 
finding  is  in  line  with  the  analysis  presented  by  the  European  Commission  (2020), 
indicating evidence of a catch-up process among the eastern regions in terms of quantity 
of scientific output, but not yet in terms of quality, a process that may take longer. The 
production of high-quality publications currently remains concentrated in some western 
and northern EU regions. 

21  See the Czech case study. 

24 

 
                                                           
Technological development and innovation 

The reconstructed theories of change indicated that, among the expected impacts of some 
policy instruments, there was the ambition to support the development of innovation and 
technological production capacity in the region as a consequence of improved scientific 
capacity and production. Policy instruments more directly addressing this objective were 
those  implemented  by  RTOs,  science  parks,  competence  centres,  cluster  organisations 
(both infrastructure and R&D projects) and science-industry collaborative projects. 

the  econometric  analysis  confirms 

When  looking  at  the  hard  technological  output,  as  measured  by  patenting  and  public-
private  co-publications22, 
the  results  already 
highlighted in the case studies.  In the period under scrutiny, there is no statistically 
significant relationship between the ERDF support provided to policy instruments 
targeting  applied  scientific  activities  and  science-industry  collaborations  (either 
infrastructure  investments  or  activities  implemented  by  RTOs,  enterprises  or  centre  of 
excellence)  and  the  growth  rate  of  technological  output.  This  holds  true  even  when 
considering  different  model  specifications  with  different  context  variables  and 
controlling  for  the  average  ERDF  investment  in  that  policy  instrument  (e.g.  amount  of 
ERDF  expenditure  per  number  of  beneficiaries).  This  finding  emerges  against  a  more 
general  pattern  of  dynamic  patenting  activity  in  EU  regions  in  the  period  under 
assessment  and  against  some  convergence  patterns  of  eastern  regions  (European 
Commission, 2020). The average value of the growth rate in the number of patents in the 
period 2008-16 in the sample of regions was 48%: four regions experienced an increase 
in the number of patents higher than 500% (RO12 – Centru; PL34 – Podlaskie; RO22 – 
Sud-est;  PL43  –  Lubuskie).  By  contrast,  a  set  of  regions,  located  mainly  in  Germany, 
Italy, the UK, Czechia and Belgium, recorded negative growth rates. The case studies on 
selected policy instruments confirmed the anecdotal evidence that pointed to limited 
uptake of the scientific results of supported R&D projects, with poor results in terms 
of improved innovation and technological development. 

Implementation issues were, however, limited, and overall, the projects were successfully 
completed. Immediate outcomes included an increased number of joint projects between 
R&D institutions and collaborations with private-sector partners. The evidence suggests 
that  the  funded  R&D  projects  played  a  significant  role  in  increasing  scientific  and 
technological  knowledge  and  competencies  among  beneficiaries  since  many  private 
companies  gained  access  to  new  ideas  and  became  aware  of  new  technologies.  At  the 
same time, through their collaboration with industry partners, research centres were also 
able  to  explore  business  partners’  needs  and  develop  skills  needed  for  industrially 
oriented and applied R&D. 

Results,  however,  may  remain  far  too  distant  from  an  industrial  application.  This 
issue  may  apply  in  a  number  of  cases  when  fundamental  research  was  the  focus  of 
funded activities. Although this applies to a limited share of projects, it can be part of a 

22  The evaluation used i) the growth rate in the number of patents between 2008 and 2016; (ii) the growth 

rate in the number of public - private co-publications in the period 2008-2015. 

25 

 
                                                           
broader situational explanation. In many cases, research results were simply not relevant 
for  the  industry,  or  no  follow-up  projects  had  been  intended.  In  the  Polish  case  study, 
scientific  partners  reported  limited  interest  among  industry  partners  due  to  changes  in 
business strategies that made the R&D results useless; alternatively, limitations related to 
insufficient  technological  readiness were combined with  a lack of funding  necessary  to 
continue  the  technological  development.  Consequently,  as  of  2013,  R&D  results  had 
been successfully implemented by companies in only four cases for the policy instrument 
under assessment (Re-Source, 2014). 

Another  explanation  for  the  lack  of  relationship  between  ERDF  support  and 
technological  output  is  that  of  time  lag.  As  the  literature  points  out  (see,  for  example, 
Finardi, 2011 and Bastianin et al., 2021), it may take some time for research activities to 
generate a technological output, so the results  of research activities implemented in  the 
2007-2013 programming period may be visible far beyond 2017. This may be even truer 
regarding infrastructure development. Infrastructure investments supported by the ERDF 
may  have  facilitated  the  setup  of  a  suitable  environment,  enabling  more  intensive  and 
productive  exchanges  between  science  and  industry.  This  phenomenon,  however,  takes 
time  to  be  consolidated  and  to  deliver  results23.  While  an  increase  in  collaboration 
between  scientific  institutions  and  commercial  partners  can  be  observed  from  the 
evidence, the transfer of results from  R&D projects  to  external  users for  economic and 
social valorisation occurred to a more limited degree. - 

The  most  convincing  explanation  regarding  the  lack  of  relationship  between  ERDF 
support  and  the  growth  of  patents  is,  however,  the  consideration  that  ERDF  support 
alone may not have been enough: other supporting factors were probably necessary for 
a noticeable shift in technological capacity. Rodriguez Pose (2020) noted that innovation 
in the EU regions is linked to four main factors: investment in R&D; population density; 
a  higher  share  of  population  with  tertiary  education;  and  governmental  quality.  The 
econometric  analysis  confirms  this  hypothesis  and  indicates  that  the  main  drivers  of 
patent growth were the R&D investments carried out by firms and the maturity of 
the  R&I  industry  system  in  the  region,  as  suggested  by  the  initial  level  (in  2008)  of 
regional  R&D  investments  in  the  business  sector.  Thus,  the  ERDF  contribution  to 
technological production may be more indirect. One viable hypothesis is that the ERDF 
policy  instrument  positively  influenced  the  R&D  expenditure  variation  in  the  business 
sector and, through this channel, had an indirect impact on patents. Overall, private R&D 
expenditure  still  increased  in  all  convergence  regions,  reaching  the  target  as  set  by  the 
OP. In 2007, private R&D as a proportion of GDP amounted to 0.22%, while in 2015, it 
reached 0.32%, and by 2017, it had fallen back slightly to 0.31%. 

Thus, the econometric analysis also tested whether ERDF policy instruments showed an 
indirect link with the growth rate in the number of patents through the R&D expenditure 
in  the  business  sector.  Nonetheless,  the  regression  analysis  rejects  any  statistically 
significant correlation between ERDF policy instruments and increases in business 
R&D expenditure in the target regions during the decade 2007-2017. While this is more 

23  See for example the Italian, Czech and Romanian case studies. 

26 

 
                                                           
disappointing  than  the  previous  results  regarding  patents,  there  are  also  explanations 
available  here.  Other  factors  can  play  a  more  direct  and  significant  role  in  triggering 
business R&D expenditure, including ERDF support to business R&D (outside the scope 
of  the  present  evaluation),  other  forms  of  direct  support  to  business  R&D,  or  more 
contextual  factors.  Among  the  latter,  the  role  of  the  economic  crisis  should  not  be 
overlooked, and this will be discussed in the next section. In addition, building trust and a 
positive  relationship  between  science  and  industry  might  not  be  achieved  via  a  single 
project;  more  effort  is  needed.  In  Poland,  the  case  study  describes  that  few  follow-up 
R&D  contracts  or  joint  R&D  initiatives  between  industry  and  scientific  partners  were 
pursued  upon  completion  of  the  funded  projects.  Most  interviewed  representatives  of 
both  scientific  and  industrial  organisations  confirmed  that  no  joint  R&D  projects  were 
carried out with the same partner following the conclusion of projects. Polish businesses 
gradually  reduced  their  funding  of  external  R&D  projects  performed  by  scientific 
organisations for reasons concerning legal challenges related to intellectual property and 
technology  transfers  –  challenges  that  themselves  resulted  from  overly  bureaucratic 
frameworks.  Scientific  partners  also  pursued  different  objectives  upon  project 
completion,  focusing  more  on  publications  and  broad  knowledge  dissemination  rather 
than commercial developments. 

Hence,  the  contribution  of  ERDF  support  for  R&D  collaborative  projects  to  the 
competitiveness of the regions may occur through more indirect and, especially, ‘softer’ 
effects than through an observable shift in patent or business R&D. Improvements in the 
competitiveness  of  local  industry  may  come  from  improvements  in  R&D  management, 
increased  interest  in  the  pursuit  of  innovation  strategies  and  a  greater  understanding  of 
new technologies and international technological trends, rather than from the successful 
commercialisation  of  the  R&D  results  of    projects24.  Another  more  indirect  link  to 
innovation may have occurred through the effect on the education sector and the share of 
the population with tertiary education, as discussed earlier. 

The role of ERDF support in fostering behavioural change among the target beneficiaries 
has  already  been  highlighted  by  previous  evaluations  (European  Commission,  2016b). 
The Romanian case study points to a ‘soft’ impact, such as ‘paving the connecting road’ 
between large and powerful industries operating in Romania, alongside profound changes 
in  the  organisational  culture  and  management  style  of  scientific  research  organisations 
and higher education institutions inherited from the previous economic regime. 

Following this line, the econometric analysis tested correlations between the ERDF and 
‘softer’ innovation outcomes. The analysis suggests that ERDF support25 positively and 
significantly  correlated  with  the  growth  rate  of  EU  trademark  applications  in  the 
period  2007-2015.  Interestingly,  in  addition  to  ERDF  support  for  RTD  investments, 
additional drivers were the level of ERDF expenditure on business support and R&D 

24  See the Polish case study. 

25  The  analysis  considered  the  expenditure  on  infrastructure  for  research  and  in  collaborative  R&D 

projects. 

27 

 
                                                           
expenditure in the business sector (as a percentage of GDP)26. Although not conclusive 
evidence in itself, this finding seems to confirm the impression that the role of the ERDF 
is more related to behavioural changes and less to technologically intensive innovation. 

The ERDF role in triggering behavioural changes among its beneficiaries is supported by 
evidence from the case studies. It is indicated in some instances as, perhaps, one of the 
most lasting effects of ERDF support in the field of RTD. There is widespread consensus 
that the ERDF played a key role in strengthening the capacities of beneficiaries and that 
this has lasted well beyond the period during which they benefited directly from ERDF 
support.  These  effects  have  also  rippled  into  others,  such  as  the  capacity  to  engage  in 
collaborative research activities or participate in international research programmes. Still, 
it is worth noting that these factors mainly concern public organisations such as Higher 
Education Institutions or Public Research Organisations. 

Students and tertiary attainments 

Even  though  this  was  not  the  primary  objective  of  the  policy,  a  selected  number  of 
countries  implemented  policy  instruments  supporting  infrastructure  and  investment  for 
education to enhance Research and Innovation capacities by attracting more students and 
improving  tertiary  attainments  in  the  region.  The  observed  growth  rate  in  the  share  of 
tertiary-educated  people  in  the  target  regions  was,  on  average,  7%  in  the  period  2007-
2017, from a minimum of -5% in the German region of Chemnitz (DED1) to a maximum 
of 18% in the region of Prague (CZ01). 

In  the  evaluation  sample  of  NUTS2  regions,  only  29  regions  (out  of  104)  invested  in 
R&D infrastructure for education. Of these 29, 21 were in the EU13. The Polish regions 
of  Mazowieckie  (PL12)  and  Pomorskie  (PL63),  together  with  Estonia  (EE00),  were 
noteworthy  for  allocating  a  total  of  more  than  EUR100  million  to  that  instrument. 
Evidence shows that  ERDF investments in education infrastructures contributed to 
increasing  the  RTD  capacity  of  beneficiary  regions  by  supporting  the  increase  in 
tertiary  population  attainment  and  the  increase  in  people  employed  in  science  and 
technology. 

The  econometric  analysis  points  to  a  positive  and  statistically  significant  correlation 
between the ERDF policy instrument and the growth rate in the number of tertiary-
educated people (as a percentage of the population aged 25-64) in the period 2007-2017. 
The same positive relation holds for the growth  rate in  the number of tertiary-educated 
people employed in science and technology in the same period. 

The evidence of the case studies offers a description of the mechanisms underpinning this 
achievement.  Beneficiaries  underscored  the  high  degree  of  satisfaction  about  the 
intervention, as it allowed universities, for the first time, to invest in the development of a 
holistic  teaching  and  learning  environment.  The  renovated  and  newly  constructed 

26  Other  context  variables  such  as  the  type  of  R&D,  type  of  territory  or  membership  of  EU13/EU15 
played  no  role  in  shaping  the  relationship  between  the  ERDF  support  and  the  growth  rate  in  the 
number of EU trademark applications. 

28 

 
                                                           
buildings  enabled  institutions  to  accommodate  new  equipment,  creating  a  better 
environment and attracting new students and researchers. 

Improved  infrastructure  had  an  impact  on  the  quality  of  life  at  supported 
universities. For example, in Czechia, thanks to numerous investments, the city of Brno 
became a valuable alternative to Prague in terms of opportunities to gain a high-quality 
tertiary education, especially in certain fields. Investments helped dramatically transform 
the  appearances  of  HEIs  in  some  regional  university  cities,  with  positive  effects 
spreading  to  local  communities  (e.g.,  further  upgrades  of  the  physical  environment  in 
particular city districts)27. 

Poland  invested  the  highest  amount  in  terms  of  ERDF  funding  to  the  development  of 
educational facilities. In the supported region, there was a steady and significant growth 
in  student  choices  of  strategic  (i.e.  OP-supported)  faculties  from  the  academic  year 
2008/2009  until  2015/2016,  while  the  total  number  of  students  decreased  in  the  same 
period. One evaluation (EGO, 2013) indicated that the new infrastructure was used by a 
significant share of students attending HEIs who participated in the programme (32% of 
students  overall  and  83%  of  students  of  faculties  directly  supported  by  the  project). 
Moreover, the evidence suggests that the improved conditions and teaching environments 
helped  attract  students  and  researchers  from  abroad,  as  evidenced  by  the  increase  in 
international  students  in  Poland  and  higher  levels  of  cooperation  with  international 
partners.  This  has  contributed  to  an  increase  in  the  internationalisation  index  of  Polish 
HEIs, from 0.5% in 2005 to 5.63% in 2017. 

The  degree  of  achievement  from  this  type  of  instrument  also  depends  on  factors 
other than the ERDF.  The econometric analysis shows that a positive and statistically 
significant  contribution  to  the  growth  in  the  number  of  tertiary-educated  people  (and 
employment in science and technology) was achieved as compared with the initial level 
(in  2007).  As  for  the  growth  rate  (2007-2017)  of  R&D  in  the  business  sector,  regions 
with  a  more  advanced  industrial  fabric  and  higher  R&D  in  the  business  sector 
experienced higher growth rates in the number of tertiary-educated people employed in 
science  and  technology.  While  ERDF  support  for  RTD  can  increase  the  supply  of 
researchers,  demand-side  effects  related  to  the  absorption  capacity  of  local  labour 
markets are crucial for the final success of such measures. 

While there is plenty of evidence pointing to the fact that education has been improved, it 
is  uncertain  whether  these  improvements  are  in  line  with  the  needs  of  the  labour 
market and potential employers. There are also questions regarding the extent to which 
improved education has led to higher employability and increased recruitment. In Poland, 
for  instance,  a  recent  peer  review  of  Poland’s  Higher  Education  and  Science  system 
(European Commission, 2017b) suggests that the employment of recent higher education 
graduates is above the EU average, but there are growing concerns about labour market 
mismatches.  In  2015,  the  employment  of  recent  tertiary  graduates  in  Poland  stood  at 
85.1%,  compared  with  the  EU  average  of  81.9%.  Nonetheless,  a  substantial  and 

27  See the Czechia case study. 

29 

 
                                                           
increasing  number  of  tertiary  education  graduates  are  in  medium  or  low-skilled  jobs, 
(European  Commission,  2016a). 
pointing 
Nonetheless,  the  extent  of  ‘over-qualification’  remains  significantly  below  the  EU 
average, as evidenced by several studies (Cedefop, 2015). 

labour-market  skills  mismatches 

to 

Important issues in the research labour market were also found in Romania, although it 
should be borne in mind that the country did not invest in education facilities but rather 
in infrastructure for research in universities. Even though there was a slight increase from 
25.4% in 2008 to 28.2% in 2016, human resources in science and technology remained 
lower  than  the  EU  average.  One  reason  for  this  development,  as  illustrated  by  a  recent 
JRC  report  (2018),  was  that  staff  in  HEIs,  as  well  as  in  the  education,  research  and 
medical systems, remained strikingly underpaid in comparison with EU28 averages and 
with  other  categories  of  personnel  (e.g.  judges,  local  administration,  police,  army)  in 
Romania. 

3.3.  Contribution analysis 

Impacts and causality 

Evidence  from  the case  studies shows that the  great  majority  of instruments  completed 
their  planned  activities  and  delivered  both  intended  outputs  and  immediate  outcomes 
(e.g. projects and grants were provided, buildings and new research infrastructures were 
developed,  collaborative  projects  were  completed).  This  was  also  reflected  in  the  high 
level  of  disbursements  of  ERDF  funds,  which  indicated  that  activities  and  work 
originally scheduled under Operational Programmes were successfully executed. 

The  great  majority  of  intended  outputs  and  immediate  outcomes  identified  in  the 
reconstructed Theories of Change were duly observed by the case study teams. Instances 
in which projects originally selected to receive ERDF support were then cancelled due to 
poor  performance  or  other  implementation  challenges  were  extremely  limited.  Italy  is 
perhaps the most notable exception to this general finding, given that various output and 
immediate-outcome  goals  have  been  only  partially  met,  given  the  delays  in  ERDF 
funding execution. 

The  causal  link  is  then  diluted  along  the  pathway  towards  impacts.  When  one 
considers effects that were observed far beyond the life cycle of the policy intervention, it 
must  be  acknowledged  that  various  other  factors  –  not  directly  related  to  the  policy 
intervention  of  2007-2013  –  exerted  a  significant  influence.  These  other  factors 
combined  their  effects  with  the  ones  produced  by  the  ERDF  in  many  ways.  Such 
contributing  factors,  preconditions  and  risks,  and  the  way  they  influenced  the  causal 
pathways, are discussed in the next  section. This dilution effect may also be the reason 
why evidence regarding intermediate outcomes and impacts is more limited. 

The main positive outcomes identified in the case studies relate to improvements in the 
scientific capacity and performance of funded beneficiaries. This is visible through an 
increase  in  students  and  graduates,  an  increase  in  R&D  personnel  and  researchers,  and 
the  higher  production  of  research  outputs  such  as  publications.  It  is  also  confirmed, 
however,  by  more  qualitative  aspects  not  immediately  captured  by  indicators.  These 

30 

 
aspects relate to the increase in scientific standing and visibility (also internationally) of 
the beneficiaries, increased participation in international research collaboration networks 
and  projects,  and  reported  increased  capacities  to  plan,  execute  and  manage  research 
projects and infrastructures in accordance with international best practices and standards. 

Most  important,  however,  is  the  relatively  limited  existence  of  evidence  and  data 
pointing to any uptake in private-sector innovation as a result of the work performed 
based on ERDF support for RTD investments. The case studies only identified a handful 
of instances in which data, knowledge or infrastructures stemming from ERDF projects 
either directly or indirectly benefited the private sector, thereby developing innovations 
that  were  later  introduced  to  the  market.  Even  though  private-sector  innovation  was  a 
common intended outcome of many of the instruments analysed, in hindsight, this does 
appear  to  be  one  of  the  major  blind  spots  of  ERDF  support  for  RTD  during  the  2007-
2013 period. 

External factors 

More  than  90%  of  OPs  saw  a  reprogramming  of  the  planned  expenditures.  The 
reprogramming did  not  necessarily imply a change in  the overall  intervention  logic but 
potentially pointed to  a financial reshuffle among interventions with the same logic. 
Interestingly, there was a sharp difference between the reasons mentioned by the EU15 
and  EU13  OPs,  with  the  former  being  mainly  related  to  the  external  shock  of  the 
economic crisis. Conversely, the latter were more influenced by the need to better target 
the  funds  to  the  instruments  considered  more  attractive  by  potential  beneficiaries,  thus 
showing a better performance in terms of funds absorption. Since this was the first full 
programming  period  for  those  countries,  it  is  reasonable  to  conclude  that  planning  and 
administrative capacities were still less developed in the EU13 than in the EU15. 

In  the  period  under  assessment,  the  economic  crisis  negatively  affected  investments  in 
RTD infrastructure and activities. The impact of the crisis across EU regions was uneven. 
It is possible to identify a core continental area wherein RTD efforts were affected only 
to a limited extent by the crisis or where they could recover relatively quickly. This was 
true  of  Germany28,  most  of  Poland29,  and  neighbouring  regions  such  as  Slovakia.  By 
contrast,  one  can  discern  a  more  peripheral  area  that  was  impacted  the  most:  Ireland, 
Spain, parts of Italy, Greece, Cyprus, Lithuania, Latvia and Estonia. 

In 14 of the 53 OPs analysed, not all of which were hit by the economic recession,  the 
crisis  determined  a  change  in  the  OP  policy  mix  for  RTD.  In  most  of  the  observed 
cases,  the  Managing  Authorities  decided  to  increase  the  resources  allocated  to  RTD 
investment  (both  R&D  projects  and  infrastructures),  considering  this  as  a  means  to 

28  Compared  with  other  EU  MS,  Germany  showed  a  relatively  fast  economic  recovery  and  was 

nicknamed an ‘engine of growth’ in the years after the financial and economic crisis. 

29  Poland’s economic growth was not as significantly affected by the slowdown as the other EU countries 
to  which  it  was  linked,  as  the  country  benefited  from  an  influx  of  foreign  investments,  the 
internationalisation  of  businesses,  changes  in  skills  and  human-resource  availability,  the  growing 
importance of innovations in the economy, and reforms of the science sector. 

31 

 
                                                           
sustain  business  development  and  innovation  and  ensure  job  creation  and  economic 
competitiveness  in  the  long  run  (two  Spanish  OPs,  the  three  Portuguese  OPs  and  the 
Slovenian  OP).  This  is  in  line  with  what  the  literature  suggests,  indicating  that 
maintaining  or  even  increasing  investment  levels  remains  crucial  during  economic 
downturns30. In a minority of cases (especially the Italian OP for the southern regions, the 
Latvian OP and, to a lesser extent, the UK West Wales and the Valleys OP), resources 
allocated  to  RTD  interventions  were  cut  and  moved  to  direct  business  support  to 
mitigate the impact of economic recession, especially on SMEs, in a countercyclical role. 

Although different reactions to the crisis were adopted in different Member States, there 
was  a  generalised  decrease  in  national  public  expenditure  for  RTD  investments, 
which the ERDF partially compensated31. 

Looking  at  broader  RTD  investments  beyond  the  ERDF,  public  R&D  expenditure 
during  the  period  remained  highly  concentrated  in  those  countries  where  most  regions 
with high R&D intensity were located (i.e. Finland, Germany and France). Nevertheless, 
a generally positive trend was recorded, primarily in the newest Member States, such as 
Czechia, Estonia, Lithuania and Slovakia. 

In terms of the regional strategies that were potentially complementary to the ERDF OPs, 
special mention should be made of the Smart Specialisation Strategies (S3). Even if these 
strategies  were  drafted  towards  the  end  of  the  programming  period  and  examined  in  a 
recent  study32  covering  2014-2020,  clear  synergies  and  mechanisms  to  ensure 
coherence between the S3 and the ERDF 2007-2013 OPs were detected in some OPs. 
Full  synergy  between  the  ERDF  and  the  S3  was  ensured  when  they  adopted  coherent 
sectoral priorities. In some countries, especially the EU13, the investment carried out in 
the 2007-2013 period directly informed the selection of priorities for the next S3. In other 
cases, the S3 influenced the design and implementation of the OP. 

Generally  speaking,  linkages  between  ERDF  and  non-ERDF  support  for  RTD  were 
visible  through  the  co-financing  requirements  established  by  the  ERDF.  Beneficiaries, 
and public entities, in particular, made use of regional or national financial resources to 
ensure they complied with ERDF co-financing rules and obligations. 

Evidence collected from the case studies indicated a significant degree of coordination 
across interventions carried out under the ERDF. This applied to coordination across 
different  ERDF  OPs  (national  and/or  regional)  and  the  different  axes,  measures,  and 
instruments implemented under individual OPs.  

Within  individual OPs,  beyond investments  for  RTD infrastructure  and  activities, other 
interventions  potentially  contributed  to  strengthening  the  regional  RTD  systems.  This 

30  Pellens,  Maikel,  et  al.  (2018).  ‘Public  investment  in  R&D  in  reaction  to  economic  crises  -  a 

longitudinal study for OECD countries.’ 

31  See in particular the Italy, Czechia and Estonia case studies.  

32  https://ec.europa.eu/regional_policy/en/information/publications/studies/2021/study-on-prioritisation-

in-smart-specialisation-strategies-in-the-eu. 

32 

 
                                                           
was  particularly  so  as  concerns  the  objective  of  improving  the  systematic  relations  of 
regional  actors.  In  the  context  of  those  additional  resources,  expenditure  for  these 
interventions  represented  a  significant  portion  of  the  ERDF  contribution  in  the  period 
under evaluation (more than EUR26 billion)33. 

Case studies show that there  was no overlapping or duplication of efforts between 
the  two  types  of  instruments  in  terms  of  design  and  implementation.  A  typical 
distinction was made in terms of target beneficiaries: while instruments supported by 01 
and  02  categories  of  expenditure  mainly  targeted  research  providers  to  improve  their 
capacities, the other codes primarily reflected the targeting of SMEs, with the principal 
aim of supporting innovation processes. 

Beyond ERDF measures funded under the same OP, expenditures were also planned in 
other  OPs  but  targeted  the  same  territories,  notably  when  regional  ones  complemented 
national  OPs.  In  many  cases,  the  case  studies  found  that  sufficient  complementarities 
across ERDF support actions and programmes in the field of RTD played a key role 
in achieving the expected results, either as necessary preconditions or as key supporting 
factors34. 

Synergies  with  the European Social  Fund  (ESF)  consisted in  funding training activities 
directly  related  to  RTD  projects  by  applying  the  principle  of  complementarity  between 
Structural Funds (Article 34 of Regulation (EC) 1083/2006). This allowed the use of the 
ERDF to finance actions that fell within areas of intervention established by the ESF, up 
to a maximum of 10% of the Community contribution of the priority axis. 

The  analysis  of  regional  strategies  shows  positive  examples  of  the  combination  of 
ERDF  and  ESF  funding.  Not  surprisingly,  synergy  was  particularly  strong  in  the 
regions where the ERDF emphasised infrastructure investment in tertiary education, as in 
Poland,  Estonia  and  Slovakia.  Complementary  training  programmes  funded  by  the 
ESF were implemented primarily in Poland. 

The  in-depth  analysis  carried  out  with  the  case  studies  further  expands  on  this  point. 
Several  instances  have  been  identified  in  which  national  and  regional  authorities 
developed  coordination  to  ensure  that  ERDF  and  ESF  operations  contributed  to 
achieving common goals. 

33 

It  is  worth  noting  that  Work  Package  13,  ‘Geography  of  Expenditure’,  of  the  Ex-post  evaluation  of 
Cohesion Policy programmes 2007-2013, in the  analysis of the ERDF contribution to RTD, actually 
referred not only to codes 01 and 02, but also to codes 03, 04, 07 and 09. Similarly, the categorisation 
of  ERDF  expenditure  for  the  2014-2020  programming  period  aggregated  these  six  codes  under  the 
same umbrella category of ‘Research and development and innovation’, because of the strong linkages 
between them.  

34  See the Czechia and Estonia case studies.  

33 

 
                                                           
4.  EVALUATION FINDINGS 

4.1.  To what extent was the intervention successful and why?  

Effectiveness 

The main achievement of ERDF support for RTD investment in the period 2007-2013 is 
a  positive  and  significant  contribution  to  the  observed  improvement  of  R&D 
capacities in the target regions, particularly in EU13 regions. Evidence shows that ERDF 
investments  aimed at  modernising education facilities are positively correlated  with the 
growth  rate  in  the  number  of  tertiary-educated  people  and  the  growth  rate  of  tertiary-
educated people employed in science and technology, in 2007-2017 and within the target 
regions.  The  growth  rate  in  the  share  of  tertiary-educated  people  in  the  target  regions 
was, on average, 7% in the period under assessment. Investments supported by the ERDF 
have contributed to this trend by improving the conditions and teaching environments of 
the target universities, which  have  attracted more students,  not  only nationally but also 
from abroad. 

Evidence  from  the  case  studies  shows  that  renovated  and  newly  constructed  buildings 
enabled institutions to accommodate new equipment, which created a better environment 
to attract new students and researchers. Evidence further shows that regions with a more 
advanced  industrial  fabric  and  higher  R&D  in  the  business  sector  experienced  higher 
growth  rates  in  the  number  of  tertiary-educated  people  employed  in  science  and 
technology.  Problems  in  absorption  capacity  of  the  labour  market  and  labour  market 
mismatches were specifically reported for Poland but may also apply to other countries. 

R&D capacities were further improved in terms of the number of R&D personnel 
and researchers, with an average growth rate in the target regions of 40% between 2007 
and 2017. ERDF investments in research infrastructures and individual R&D projects in 
HEIs  positively  contributed  to  the  increase  in  the  number  of  R&D  personnel  and 
researchers  at  the  regional  level.  Infrastructure  investments  also  contributed  to  the 
creation or modernisation of public R&D facilities, including ICT-based infrastructures, 
which  increased  the  potential  and  capacity  of  the  beneficiary  institutions  and  created 
more  attractive  and  better  performing  research  and  education  environments.  Individual 
projects allowed researchers both to increase existing expertise and to develop new areas 
of inquiry. 

Evidence  also  points  to  a  positive  and  statistically  significant  relationship  between 
ERDF support and the growth rate in the number of scientific publications. While a 
catching-up  process  in  scientific  production  is  particularly  visible  in  the  EU13  (145% 
growth  in  the  volume  of  publications  between  2007  and  2017),  it  is  reasonable  to 
suppose  that  ERDF  investments  in  HEIs  have  significantly  contributed  to  this  process. 
Conversely,  no  relationship  is  found  regarding  the  quality  of  scientific  production  (as 
evidenced by the growth rate in the number of regional scientific publications in the top 
25% of most cited publications), which may take longer to catch up. 

More  limited,  however,  was  the  capacity  of  funded  projects  to  generate  economic 
benefits from the commercial valorisation of R&D results and improve the knowledge 

34 

 
transfer  capacities  and  mechanisms  from  scientific  to  industry  partners.  No  statistically 
significant  relationships  are  found  in  the  econometric  analysis  between  ERDF  support 
and  the  growth  rate  of  technological  outputs,  which  confirms  the  limited  uptake  of 
research results observed in the case studies. 

the  case  studies  shows 

technological  outputs.  Evidence  from 

The ERDF was not successful in stimulating business R&D, which is the main driver 
of 
that  some 
implementation  issues  were  reported  for  collaborative  R&D  instruments  and  projects, 
despite  reports  of  generally  high  interest  from  beneficiaries.  Evidence  collected  in  the 
field  shows  that  science-industry  collaborations  did  not  lead  to  systematic  follow-up 
projects. This was because of a lack of trust or resources, or because of legal problems 
related  to  intellectual  property  rights  and  technology  transfer  procedures.  Expected 
results in terms of consolidation of research partnerships showed limited sustainability in 
the  long  run.  Nonetheless,  some  positive  results  were  reported  in  terms  of  softer 
innovation aspects measured by the growth rate of EU trademark applications, especially 
in  those  regions  with  higher  ERDF  expenditures  in  business  support.  Positive  effects 
were  also  reported  regarding  the  managerial  capacity  of  research  institutions  and  the 
enhancement of their research and innovation capacities. 

Overall,  there  is  evidence  that  the  ERDF  contributed  to  building  and  modernising 
R&D  infrastructure  in  EU  regions,  especially  those  lagging  behind.  This  process  of 
upgrading  and  improving  RTD  capacities  is  especially  evident  in  the  EU13,  where  the 
ERDF  contributed  to  filling  the  chronic  investment  gap  they  had  suffered.  ERDF 
investments  in  2007-2013  supported  the  creation  of  favourable  conditions  to  conduct 
research  of  international  quality,  helping  less  developed  EU  regions  to  bring  their  R&I 
systems closer to EU standards. Evidence from the cluster analysis of RTD performance 
in  the  EU  regions  shows  that  half  of  these  regions  did  not  change  their  relative  RTD 
performance  in  the  ten  years  since  the  start  of  the  programming  period.  Although 
performance  improvements  were  concentrated  in  economically  stronger  regions,  some 
transition regions also saw a catch-up dynamic. Evidence of this evaluation suggests that, 
particularly in certain EU13 regions, the ERDF has positively contributed to this catch-up 
process of RTD capacities, thus contributing to reducing disparities between EU regions 
in performing quality research. Thanks to the combined effects of ERDF investments and 
favourable framework conditions (including national support to RTD investments), some 
of  the  EU13  regions  were  well  equipped  to  conduct  more  and  better-quality  research, 
contributing to strengthening the EU’s RTD capacity. 

The  ERDF  was  less  effective  in  facilitating  the  coordination  and  interactions 
between  all  the  actors  involved  in  the  innovation  ecosystem,  thus  adding  to  system 
failures.  In  particular,  there  is  no  evidence  of  an  improvement  in  the  science-industry 
relationship, which is one of the possible explanations of the observed limited knowledge 
transfer and innovation uptake. Moreover,  while there is  a dominant  scale effect of the 
policy,  where  existing  systems  performed  better  or  maintained  a  stable  performance, 
limited or even no  evidence is  available about a  transformation  of the regional system, 
for example in the extent and nature of the science-industry links. Indeed, the policy did 
not  move  towards  a  structural  transformation  in  how  knowledge  is  produced, 

35 

 
disseminated and exploited. Ultimately,  the ERDF was less effective in  transferring the 
increased research capacity into more competitive territories and regional economies that 
were  addressing  system  failures.  If  this  may  have  come  true  in  a  longer  time  horizon, 
capitalising on the investments made with the Smart Specialisation Strategies, it should 
be verified in future evaluation studies. However, it is important that future evaluations 
take a more systemic point of view in the assessment: first, by mapping regional systems 
and their investment needs; and second, by assessing the appropriateness of the observed 
trajectory of systemic change. 

Finally, the 2021 evaluation on which this SWD is based highlighted the importance of 
underlying factors for impact generation in the implementation of R&D funding - a 
point widely discussed in the literature. In particular, synergy and complementarity with 
existing  funding  sources  were  not  always  well  exploited.  Moreover,  administrative 
failures  and  legal  constraints  exposed  the  implementation  to  delays,  uncertainties, 
rejections  and,  indeed,  financial  stress  in  a  field  where  timing,  long-term  commitments 
and clear rules are crucial incentives for the collaboration of engaged actors. 

Efficiency 

Financial  concentration  is  often  seen  as  a  desirable  outcome  of  policy  action  and  an 
indication  of  efficiency.  Evidence  collected  on  funded  projects  and  beneficiaries 
highlights  a  concentration  pattern  on  economically  stronger  territories,  sectors  and 
leading  institutions.  Reflecting  existing  agglomeration  effects  of  R&D  activities  and 
capacities, RTD investments funded by the ERDF followed concentration patterns: more 
than 50% of mapped funds were invested in Poland, Germany and Czechia, while 
70% were directed to convergence regions and 64% to urban areas. ERDF support 
for RTD was overall sufficiently concentrated to lead to upgrades in both the quality of 
research  infrastructure  and  research  management  capacities  in  most  of  the  countries 
under  investigation.  However,  its  role  as  ‘game-changer’  or  ‘needle  mover’  of  RTD 
performance in beneficiary countries and regions was strongly related to the importance 
of  the  ERDF  in  the  overall  national  and  regional  RTD  policy  mix.  As  already 
highlighted, fund concentration on increasing efficiency may lead to a ‘winner-take-all’ 
dynamic that needs  to  be addressed with  policy  action to  reduce  gaps between winners 
and losers. 

The  evaluation  also  looked  at  the  efficient  use  of  financial  resources  in  the 
management and implementation of interventions, strictly linked to the administrative 
capacity  issue.  The  administrative  and  managerial  capacities  of  both  programme 
for  effective  public  spending.  Some 
managers  and  beneficiaries  are  crucial 
implementation  issues,  mainly  related  to  limited  administrative  capacity  or  an  unclear 
legal  framework,  were  reported  especially  for  collaborative  R&D.  Uncertainties  in  the 
interpretation  and  application  of  rules,  especially  for  what  concerns  State  aid  rules, 
caused  delays  and  generated  confusion  and  adjustments  during  the  implementation 
process. 

It  is  not  surprising  that  administrative  capacity  can  improve  the  effectiveness  of 
supported  instruments.  However,  specifically  for  RTD,  the  capacity  to  ensure  selected 

36 

 
projects’ high scientific quality, and their timely selection and funding, are identified as 
key elements. Ensuring a timely and smooth project assessment and selection procedure 
can  improve  the  quality  at  entry  of  funded  projects  and  increase  the  probability  of 
success. A successful regional RTD system requires intensive and successful interactions 
among  many  different  players,  alongside  multiple  stakeholders  and  behavioural 
incentives. 

Wider  aspects  of  value  for  money  were  outside  the  scope  of  this  evaluation,  given  the 
scale and heterogeneity of the funded interventions, making it unpractical to measure and 
value the produced output in a systematic and comprehensive way. 

Coherence 

ERDF support for RTD was implemented as part of a wider policy mix, including other 
ERDF  policies  and  other  national,  regional  and  EU  initiatives.  They  all  somehow 
contributed  to  improving  R&D  performance  in  the  EU  regions.  Thus,  their  respective 
roles  and  potential  synergy  were  carefully  considered.  The  role  of  the  ERDF  differed 
significantly  across  regions  and  countries  in  terms  of  financial  weight  and  strategic 
coherence. 

One of the key factors affecting the long-term sustainability of projects was revealed to 
be the long-lasting strategic and financial commitment to investment priorities, both for 
private and public organisations. In this regard, the ERDF played a countercyclical role 
in  many  regions,  representing  a  ‘safety  belt’  for  many  beneficiaries.  One  of  the 
evaluation  findings  is  that,  in  some  countries,  ERDF  funding  in  the  period  2007-2013 
prevented  the  erosion  of  R&D  systems  in  a  time  of  severe  cuts  in  public  funding  for 
education  and  research,  given  the  economic  downturn  of  2008.  Thus,  it  played  a 
significant  substituting role in  those countries that  were most severely hit  by the crisis. 
Conversely, this may have prevented or delayed the restructuring of some national R&D 
systems, thereby losing the opportunity specifically to promote the most relevant systems 
or to achieve excellent growth. 

Above  all,  the  crisis  impacted  firms’  financial  capacity  and  resources  to  undertake 
investments,  with  a  risk  of  limited  fund  absorption,  especially  for  those  OPs  that 
allocated large resources to collaborative research projects. This underscored the need for 
adaptive strategies to cope with changing socio-economic contexts. Evidence shows that 
continued  public  investment  in  research  institutions  is  key.  It  enables  for  follow-up 
projects to take place that strengthen existing capacities and encourage the development 
of new ones. Hence, the long-term alignment of the ERDF with national and regional 
RTD strategies becomes a crucial element of success. Resilience in strategy design and 
implementation  is  also  a  crucial  element  of  success  in  combining  different  policy 
instruments and funding schemes. 

Coherence with other forms of ERDF support (internal coherence, i.e. coherence with 
other ERDF measures in the same OP, or ERDF support for RTD by other OPs targeting 
the same territories) was generally high. There was robust coordination among different 
OPs and between different priority axes within the same OP, clearly considering possible 
synergies and complementarity of respective roles. 

37 

 
The  ERDF  policy  mix  for  RTD  was  also  generally  coherent  with  regional  and 
national  RTD  strategies,  especially  regarding  a  strategic  alignment  of  priority  sectors 
and  scientific  fields.  In  some  countries,  when  the  prevailing  rationale  was  to  improve 
science-industry collaboration, RTD strategies supported by the ERDF were often closely 
linked to objectives of industrial competitiveness. The ERDF role in shaping national and 
regional policies was stronger in those countries where it represented a significant share 
of national or regional R&D expenditure, and therefore mainly in convergence regions. 

Despite  a  high  strategic  alignment,  however,  there  was  often  a  tacit  division  of  goals 
between  local  and  ERDF  policies  and  instruments  in  more  operational  terms. 
Coordination was mainly driven by co-financing obligations in these cases, and there was 
a  notable  effort  to  avoid  overlapping.  In  some  cases,  a  lack  of  political  stability  and 
related  long-term  commitment,  and  the  absence  of  financial  predictability  for  national 
RTD strategies, prevented a stronger alignment. In some regions, in the aftermath of the 
economic crisis in 2008 and subsequent years and shrinking national public expenditures, 
this also meant the substitution of national funds by ERDF resources. 

Good  synergy  was  reported  with  the  ESF,  with  specific  reference  to  support  in  the 
higher education sector. Here, the combination of ERDF and ESF funding resulted in a 
limited number of positive examples that nevertheless showed significant promise. 

ERDF  and  EU  Research  and  Innovation  Framework  Programmes  were  seen  as 
serving related but essentially different purposes. The former mainly provided funds 
to  ensure  the  enabling  conditions  to  carry  out  excellent  scientific  work  (through 
infrastructure investment) and to support applied research benefiting local R&I systems; 
the  latter  provided  funds  for  excellent,  EU-wide  research  activities,  primarily  in 
fundamental  research. Nonetheless,  despite  ambitions to  build on relative strengths  and 
implement  projects  with  coordination  between  the  two  funds  regarding  selected  target 
areas  or  beneficiaries,  no  specific  arrangements  were  implemented  to  facilitate  or 
promote  active  synergy.  No  special  coordination  mechanisms  were  put  in  place  to 
implement  programmes  and  specific  instruments,  mainly  because  the  two  funds  still 
followed  different  implementation  mechanisms  (e.g.  the  selection  method  for  the 
interventions  and  the  object  of  these  interventions).  It  is  not  yet  clear  if  this  lack  of 
coordination  has  had  some  adverse  impact  on  the  overall  performance  of  the 
programmes. 

Clarity about the rules of the game, shared within the common RTD space by science 
and  industry  partners  and  regulating  their  respective  roles  and  responsibilities  while 
providing  the  most  appropriate  incentives  for  successful  partnerships,  proved  to  be 
decisive.  Here,  the  perception  of  State  aid  rules  was  reported  as  the  most  problematic 
factor  in  many  countries.  They  are  among  the  main  challenges  hampering  a  more 
intensive  and  effective  involvement  of  businesses  in  the  funded  projects  and  follow-up 
activities. The role and influence of State aid were more evident in the implementation of 
policy  instruments  rather  than  in  their  design.  According  to  interview  respondents, 
unclear interpretation of State aid rules generated uncertainty regarding the eligibility of 
business enterprises to benefit directly from the ERDF and the extent to which the private 

38 

 
sector  could  be  involved  as  users  of  ERDF-funded  infrastructures35.  This  likely  had 
implications  on  the  effectiveness  of  RTD  instruments,  especially  those  that  targeted 
science-industry  collaboration.  Although  State  aid  provisions  have  been  later  adjusted 
and  revised  to  cope  with  perceived  challenges,  the  question  of  coherence  between 
cohesion and competition policy remains open. 

4.2.  How did the EU intervention make a difference? 

EU added value 

In the broader policy mix, MAs recognise that  the main EU added value was a scale 
effect  produced  by  accessing  a  considerable  quantity  of  financial  resources.  This 
holds true especially in the EU13, where ERDF 2007-2013 programmes represented the 
first  systematic  set  of  interventions  addressed  to  the  research  field  after  years  of 
underinvestment and limited political priority. 

A missed opportunity was the lack of the systematic promotion of interregional or 
international  research  collaborations  as  a  potential  EU  added  value.  Partnerships  of 
collaborative R&D projects were mainly regional or, albeit only in selected cases, multi-
regional within the same country. 

EU-wide  effects  were  not  among  the  intended effects  of  funded  instruments.  Thus, 
the  contribution  of  ERDF  support  to  them  was  more  indirect.  It  occurred  through  the 
development  of  EU-level  research  communities  in  specific  fields,  enabling  the 
construction  or  upgrading  of  strategic  infrastructures  of  pan-European  relevance  (as 
confirmed  by  the  later  inclusion  into  the  ESFRI  roadmap)  and  also  supporting  the 
internationalisation  of  research  communities.  It  helped  structure  and  consolidate  a 
European  Research  Area  by  promoting  the  achievement  of  EU  standards  in  RTD 
capacities and production, and this can be claimed to have been the main EU added value 
of ERDF support to RTD investments in the period 2007-2013. 

4.3. 

Is the intervention still relevant? 

Relevance 

The evidence collected through the case studies indicates that, in most cases, the ERDF 
support for RTD was relevant, meaning that it addressed the most pressing needs of 
expansion and modernisation of national RTD systems. It addressed, in particular, the 
huge infrastructure gap of Central and Eastern European countries. However, the ERDF 
support also reflected the need to improve science-industry collaboration, mainly in more 
advanced  regions.  Overall,  ERDF  support  to  RTD  investments  concentrated  on 
interventions on the supply side, mainly focused on strengthening RTD capacities rather 
than on improving the performance of regional RTD systems as a whole. 

35  State  aid  rules  are  per  definitions  applicable  to  undertakings  to  ensure  a  level-playing  field  in  the 
market  and  in  order  not  to  crowd-out  private  investment.  On  the  other  hand,  public  support  for 
infrastructure  used  for  scientific/educational  purposes  can  and  in  many  cases  does  fall  outside  the 
control of State aid rules. 

39 

 
                                                           
Project  selection  was  conducted  by  managing  authorities  based  on  a  mixed  approach. 
While infrastructure investments were typically the result of a top-down approach guided 
by  national  road  mapping  exercises,  research  projects  followed  a  more  bottom-up 
approach,  responding  to  the  needs  of  regional  scientific  communities  within  a  well-
identified set of scientific and technological priority fields. 

Looking at the targeting strategies, it is important to highlight that the dominant approach 
was  more  functional  than  place-based.  ERDF  prioritisation  strategies  targeted 
territories/institutions/sectors with significant potential or comparative advantages. 
When  the  distribution  of  ERDF  support  was  not  geographically  driven  by  eligibility 
criteria, such as in national OPs in Central and Eastern Europe, ERDF expenditure was 
mainly concentrated in urban areas, economically stronger sectors and more competitive 
institutions and organisations. This approach was justified by the need to ensure critical 
mass, fund absorption and knowledge externalities in more mature territories. Only in a 
few cases was there a balance between the choice of international or national excellence 
and territorial cohesion. However, overall, the evaluation found a lack of precise context-
specific considerations, including territorial imbalances, in the design of the policy. This 
evaluation however could not provide conclusive evidence about whether the pursuit of 
excellence  objectives  was  made  at  the  expense  of  cohesion  and  whether  may  have 
contributed to increasing the territorial divide. 

The  observed  targeting  strategy  based  on  excellence  confirms  what  is  already  noted  in 
the literature as ‘innovation paradox’, i.e. that regions with a stronger need to invest in 
RTD seem to be those with a comparatively lower capacity to absorb funding than 
more  mature  regions.  The  adoption  of  the  Smart  Specialisation  Strategy  in  the  period 
2014-2020 is expected to have offered a workable solution to the innovation paradox and 
a platform to guide the design of RTD policies in lagging regions. It also possibly offered 
a more solid theoretical framework to assess the relevance of the adopted policy mix in 
the different territories, which should be based on a thorough mapping and prioritisation 
of regional vocation and potential. Future evaluations should shed light on the extent to 
which this new approach effectively supported the design and implementation of place-
based RTD investments, especially in less developed regions. It should also point to the 
extent  to  which  this  approach  has  facilitated  the  understanding  of  needs,  capacities, 
motivations and interests of the different actors in the system, an aspect that appeared to 
be weak in the design of the 2007-2013 ERDF programmes. 

5.  WHAT ARE THE CONCLUSIONS AND LESSONS LEARNED? 

5.1.  Conclusions 

The findings of this study indicate that the selected 2007-2013 ERDF programmes, and 
their  associated  strategies,  were  well  justified  overall  and  backed  by  a  relatively  solid 
rationale.  Positive  achievements  were  reported,  in  particular,  by  policy  instruments 
targeting  HEIs.  Overall  ERDF  support  has  positively  contributed  to  the  catch-up 
process  of  RTD  capacities  in  certain  regions,  thus  contributing  to  reducing  disparities 
between  EU  regions  in  performing  quality  research.  This  was  mainly  delivered  by 

40 

 
infrastructure  investments  targeting  the  upgrade  and  modernisation  of  existing 
facilities and the construction of new ones. This applied especially to those EU13 regions 
lagging behind in terms of scientific and technological capacities and standards. Support 
for R&D projects, both individual and collaborative, was important in developing know-
how in new scientific fields or existing areas with great potential. As a result, there were 
observable  effects  on  scientific  production  and  capacity,  as  witnessed  by  some  key 
RTD  performance  indicators:  the  increase  in  the  volume  of  scientific  publications;  the 
increase  in  the  number  of  people  with  a  tertiary  education  employed  in  science  and 
technology; and the growth rate in the number of R&D personnel and researchers. 

The  main  drawback  reported  by  the  study  relates  to  the  lack  of  observable  long-
term  impacts,  as  far  this  concerns  the  use  of  research  results  for  technological 
development  and  innovation.  Evidence  shows  that  the  observed,  improved  scientific 
knowledge  did  not  translate  into  technological  development  and  innovation  and 
ultimately  did  not  increase  regional  competitiveness.  Certainly,  the  economic  crisis 
played  a  role  in  reducing  the  capacities  of  both  public  and  private  organisations  to 
maintain long-term commitments to research and innovation strategies. Still, this was not 
the  only  reason.  The  analysis  of  the  complex  causal  packages  underpinning  both  the 
success of individual instruments and the full policy mixes shows that a combination of 
factors  had  to  be  guaranteed.  The  evidence  shows,  in  particular,  that  synergy  and 
complementarity  with  existing  funding  sources  were  not  always  well  exploited, 
which  led  to  specific  regulatory  measures  to  overcome  the  issue  in  the  2014-2020  and 
2021-2027 programming periods. Moreover, administrative failures and legal constraints 
exposed  implementation  to  delays,  uncertainties,  rejections  and,  for  some  beneficiaries, 
financial  stress  in  a  field  where  timing,  long-term  commitments  and  clear  rules  were 
crucial factors for the successful collaboration of engaged actors. 

Based  on  this,  the  evaluation  identified  a  list  of  issues  that  should  be  considered  when 
designing and implementing RTD policy. 

The  preparatory  phase  includes  the  needs  assessment  for  the  RTD  landscape  and 
the prioritisation process. It should be based on an in-depth understanding of existing 
RTD  actors  (i.e.  their  capacities  and  expectations,  their  willingness  and  incentive  to 
engage  in  know-how  transfer,  and  their  territorial  distribution)  and  the  national 
institutional  and  legal  framework  (i.e.  administrative  capacities,  legal  constraints,  and 
policy framework). Specific points of attention should be the following. 

  Long-term  commitment  to  public  and  private  investments  benefits  from  clarity 
regarding  the  legal  framework.  In  line  with  their  recent  efforts,  national  and  EU 
authorities should guarantee that legislation regarding public procurement, State aid, 
and  other  important  regulations  is  sufficiently  clear  and  perceived  as  such,  so  that 
they lead to a smooth implementation of RTD investments. Clear and effective State 
aid rules are important in ensuring that businesses are eligible for public funding and 
encouraged  to  participate  in  collaborative  R&D  projects.  Administrative  burdens 
related  to  public  procurement  should  be  minimised,  and  rule  changes  should  be 
avoided  to  reduce  project  delays.  Legal  constraints  and  other  framework  conditions 

41 

 
that may prevent appropriate pay levels for researchers are also important obstacles to 
consider. 

  Equally importantly, a clear policy strategy delineating a long-term commitment to 
public  investment  in  R&D  should  be  established,  communicated  and  maintained 
over  time,  thereby  reducing  fluctuations,  particularly  in  times  of  crisis.  This  should 
include a plan to coordinate the various support programmes in the field of RTD in 
the region and country, to ensure an effective alignment and complementarity of all 
funding mechanisms. In this manner, the logic of supporting RTD investments in the 
long journey for research and innovation, with appropriate coordination, instead of a 
clear  separation  of  competencies  that  may  lead  to  fragmentation,  can  facilitate 
follow-up investments. A specific role for the ERDF in the broader policy mix should 
be designed and acknowledged.  

  Programme-management  units  within  MAs  should  be  appropriately  staffed  and 
trained.  Implementing  RTD  investment  support  is  a  demanding  task  that  requires 
managerial  and  entrepreneurial  capacity.  When  these  are  not  already  in  place, 
especially in less developed regions benefiting from large funding, swift restructuring 
within  responsible  administrations  should  be  carried  out,  with  dedicated  units 
equipped with the necessary staff and competencies. 

 

Investment  prioritisation  and  targeting  should  be  informed  by  an  in-depth 
understanding  of  the  system  failures  affecting  the  regional  RTD  ecosystem.  This 
should look particularly at existing relationships between science-industry partners in 
the  region  and  the  drivers  that  can  foster  an  environment  enhancing  their 
collaboration  and  better  diffusion  of  R&D  results.  RTD  supply-side  investments 
should be combined with due consideration of demand-side absorption capacities and 
constraints. The absorption capacity of the local labour market or the business sector 
of trained researchers and advanced technological services offered as a result of the 
planned investments should be considered. Technology transfer offices, or permanent 
collaboration  platforms  such  as  competence  centres  or  cluster  organisations,  can  be 
promoted  in  those  cases  where  there  are  possible  mismatches  between  the  research 
supply  and  actual  local  demand.  Demographic  change  can  have  an  impact  on  the 
territorial  absorption  potential  of  RTD  capacities.  For  example,  the  emigration  of 
students  and  researchers  due  to  unfavourable  general  conditions  can  dilute  the 
expected local impact of RTD investments and result in a brain drain. 

  To improve the sustainability of supported investments, in the design phase, there is 
the need to enhance the focus on market orientation of research activities to support 
smart economic transformation. 

  Possible trade-offs between excellence and territorial inequality can emerge in the 
targeting strategy. In a place-based approach, such trade-offs should be addressed by 
better considering the local  relevance of RTD investments  to  the territorial  context, 
avoiding  the  promotion  of  investments  motivated  by  the  pursuit  of  scientific 
excellence but unrelated to the local business sector and technological capacities. 

42 

 
During programme implementation, it is necessary to ensure a transparent and timely 
selection  procedure,  and  clear  and  prompt  funds  disbursement  to  prevent  delays  and 
ensure  that  high-quality  projects  are  implemented  and  produce  sustainable  results. 
Positive conditions should be guaranteed and consistently and robustly maintained.   

  Instrumental  support  from  advisory  and  support  services  may  be  useful  in 
improving  the  engagement  of  stakeholders  and  ensuring  that  good-quality  projects 
are  prepared.  This  would  also  avoid  that  selection  criteria  promoting  excellence 
concentrate the funds in a few leading institutions. MAs and implementing bodies are 
encouraged to guarantee a high level of commitment and willingness to assist in the 
application  process.  Capacity  building  is  also  essential  in  developing  awareness  of 
industry  needs  and  providing  the  capacity  to  transfer  knowledge.  Communication 
channels  can  be  activated  to  present  and  explain  R&D  results  that  may  have 
commercial potential. 

  Administrative  procedures  for  fund  disbursement  should  be  kept  as  simple  as 
possible  to  reduce  the  administrative  burden  on  beneficiaries  and  the  impact  on 
timely beneficiary payments. 

 

In selecting infrastructure investments, due attention should be paid to ensure that 
sufficient,  highly  qualified  R&D  and  ICT  staff  can  be  employed  in  the  new 
infrastructure. 

  For long-term financial sustainability, any beneficiary infrastructure must  develop a 
business model specifying a balanced source of funds without relying unduly on an 
individual  source.  It  is  also  essential  to  maximise  the  revenue-generating  capacity 
arising from services offered to industrial partners. 

  Private partners’ lasting commitment and interest regarding research activities and 
collaboration  with  science  partners  should  be  promoted  and  maintained.  Measures 
should be taken to prevent this commitment and interest from being undermined by 
external shocks or unfavourable contextual conditions. 

5.2.  Policy response in the 2014-2020 & 2021-2027 programming periods 

The 2014-2020 programming period brought forward a new legislative framework for the 
five  ESI  Funds.  For  instance,  the  ‘thematic  concentration  of  funds’  and  the 
delineation  of  11  thematic  objectives  represented  a  key  element  in  the  reform  of  the 
ESI Funds. This principle and the related regulatory requirements sought to align the ESI 
Funds with the Europe 2020 strategy to integrate the ESI Funds in the broader EU policy 
framework.  It  also  aimed  to  increase  added  value  by  concentrating  the  funds  on  fewer 
priorities  across  all  Member  States  and  regions.  The  first  four  of  these  thematic 
objectives36 constituted key priorities for the ERDF and helped guide ERDF investments 

36  1) Strengthening research, technological development, and innovation; 2) Enhancing access to, and use 
and quality of information and communication technologies (ICT); 3) Enhancing the competitiveness 
of small and medium-sized enterprises (SMEs); 4) Supporting the shift towards a low-carbon economy 
in all sectors. 

43 

 
                                                           
in  research  and  innovation.  As  such,  these  priorities  aimed  at  addressing  some  of  the 
shortcomings outlined in this evaluation. 

Furthermore,  special  provisions  were  developed  to  encourage  Member  States  to  pursue 
the complementary use of ESI Funds and other EU instruments, and in particular Horizon 
funding,  thereby  emphasising  synergy.  These  conditions  ensured  that  Member  States 
pursued the strategic identification of priorities and made better use of the possibility to 
combine support from different instruments to finance individual operations. 

Moreover, Member States were tasked to develop Smart Specialisation Strategies (S3) 
in entrepreneurial discovery processes so that ESI Funds could be used in a more targeted 
way, taking into account different challenges, resources and capacities of each region and 
Member State. For 2014-2020, over 180 smart specialisation strategies were developed, 
with  both  the  Council  of  the  EU  and  the  European  Parliament  highlighting  the  need  to 
further  build  on  this  approach37.  A  recent  study38  confirms  that  such  concentration  and 
prioritisation  efforts  have  largely  been  effective  and  are  based  on  an  objective  data-
driven identification process that involves key stakeholders from the private, public and 
research sectors. 

Among the most ambitious innovations, a notable one is also the Interregional Innovation 
Investments  (I3)  Instrument.  It  aims  at  supporting  interregional  innovation  projects  in 
their  commercialisation  and  scale-up  phases  giving  them  the  tools  to  overcome 
regulatory and other barriers and bring their project to investment level. 

Upon starting a new programming period (2021-2027), the 11 thematic objectives were 
reduced  to  five  policy  objectives  (PO).  While  investments  in  all  objectives  will  be 
supported, the ERDF will pursue the twin transition and prioritise spending that promotes 
a  more  competitive  and  smarter  Europe  (PO1),  in  particular  under  specific 
objectives  1.1  and  1.4,  and  a  greener,  low-carbon  economy  (PO2).  This  will  also 
guide  investments  that  will  support  development  of  research  and  innovation  capacities 
and  uptake  of  advanced  technologies,  foster  innovation  in  small  and  medium-sized 
enterprises, and promote digitalisation and digital connectivity and development of skills 
for smart specialisation, entrepreneurship and industrial transition39. 

The achievement of the goals under these new priorities will benefit from guidance and 
synergies  with  the  broader  EU  strategic  context  and  in  particular  the  European  Green 
Deal. The policy framework for the smarter Europe objective is  set  out  in  a number of 
strategic  documents  such  as  the  Smart  Specialisation  Communication,  A  renewed 

37  Council  Conclusions  2016  on  'A  more  R&I  friendly,  smart  and  simple  Cohesion  Policy  and  the 
European  Structural  and  Investment  ESI  Funds  more  generally'  (24  June  2016),  and  European 
Parliament  resolution  of  13    September  2016  on  Cohesion  Policy  and  R&I  Strategies  for  Smart 
Specialisation (2015/2278(INI)). 

38  https://ec.europa.eu/regional_policy/en/information/publications/studies/2021/study-on-prioritisation-

in-smart-specialisation-strategies-in-the-eu. 

39  European  Regional  Development  Fund  –  Funding  Priorities.  European  Commission: 

https://ec.europa.eu/regional_policy/en/funding/erdf/#2. 

44 

 
                                                           
European Agenda for Research and Innovation, and A New Industrial Strategy for 
Europe40. The policy consequence for ERDF investments is that a stronger emphasis will 
be put on market orientation and capability building to help less developed region catch 
up,  specifically  by  concentrating  investments  both  geographically  and  in  terms  of 
beneficiaries  and  focusing  on  higher  technology  readiness  levels  for  more  immediate 
commercial  application.  Specific  focus  will  also  need  to  be  paid  to  digitalisation  as  an 
innovation  enabler cutting across all areas, the specific location of SMEs and start-ups, 
interregional cooperation, and access to high-capacity telecommunication networks. 

40  Key  references  also  include:  An  SME  strategy  for  a  sustainable  and  digital  Europe,  European  Skills 
Agenda  for  sustainable  competitiveness,  social  fairness  and  resilience,  the  Renewed  EU  agenda  for 
higher education, Shaping Europe’s digital future and the Communication on a new ERA for research 
and innovation, and the Strategy on Connectivity for a European Gigabit Society. 

45 

 
 
 
                                                           
5.3.  Questions for further studies 

One of the key aspects of the SWD was the presentation and discussion of the rationales 
underpinning ERDF interventions, concerning both individual policy instruments and the 
whole  RTD  set  of  investments  within  a  broader  policy  mix.  The  2021  evaluation  on 
which  this  SWD  is  based  was  descriptive  on  this  point,  reporting  the  various  claimed 
justifications, usually based more on an ex-post reconstruction of interviewed programme 
managers. A more  comprehensive and conclusive judgement about  the most credible 
rationales of the different strategies cannot be provided. 

To this end, it would be particularly interesting to check to what extent  introducing the 
Smart  Specialisation  approach  in  the  2014-2020  programming  period  may  have 
provided  a  solid  logic  to  assess  the  rationale  of  ERDF  support  to  RTD  investments.  A 
longer-term analysis or a back-to-back approach of evaluations of different programming 
periods  would  better  account  for  longer-term  effects,  including  the  link  between 
infrastructure investment and innovation, and policy development of cohesion policy. It 
would  also  help  assessing  whether  the  stand-alone  approach  to  investments  has 
effectively  been  overcome,  absorbing  research  infrastructure  into  the  ecosystem 
approach. 

Linked to the previous point, there is the need to reflect better a ‘system perspective’ in 
evaluating RTD investments under cohesion policy. One of the main barriers for RTD 
investments  recognised  by  the  literature  and  indicated  by  MAs  was  system  failures, 
particularly the suboptimal interaction of the main actors of regional RTD systems. The 
study highlighted the importance of understanding the needs, capacities, motivations and 
interests belonging to the different actors of the system. In this sense, an analysis of the 
broader  system  involved  in,  and  affected  by,  RTD  policies  in  the  different  regions  is 
necessary to understand the impact or desired impact of both individual instruments and 
overall investment strategies. 

Moreover, very limited evidence could be gathered by this evaluation about the capacity 
of  the  ERDF  to  transform  the  regional/national  systems  of  actors  structurally.  The 
evaluation  could  instead  offer  conclusions  about  a  dominant  scale  effect  of  the  policy, 
where existing systems performed better or maintained a stable performance but did not 
move  towards  a  structural  transformation  in  how  knowledge  is  produced,  disseminated 
and  exploited.  The  problem  may  be  due  to  the  fact  that  systems  needed  different 
interventions  or  that  there  were  not  enough  synergies  between  the  ones  designed.  The 
Smart  Specialisation Strategy approach may have provided  already an improvement on 
this  point  in  the  2014-2020  programming  period.  The  strategy  focused  on  bringing 
together  different  stakeholders  and  on  the  importance  of  mapping  needs,  for  both 
enterprises and research providers as part of RTD regional ecosystems, as a starting point 
in the formulation of the regional strategy. 

Also, because of the smart specialisation approach, future evaluation should possibly take 
a more systemic point of view, first, by mapping regional systems and their investment 
needs and, second, by assessing the appropriateness of the trajectory of systemic change 
to  which  the  ERDF  has  contributed  or  facilitated.  A  starting  point  is  provided  by  the 

46 

 
categorisation  of  activities  by  theme  performed  in  this  evaluation,  which  proved 
successful for subsequent aggregated analysis. 

The  policy  response  for  2014-2020  and  2021-2027  introduced  further  aspects  that  will 
need  specific  assessment.  Among  these,  particular  importance  will  be  played  by  the 
evaluation  of  effectiveness  and  efficiency  for  the  full  range  of  financial  instruments 
implemented and for investments in different technology readiness levels s. 

As  highlighted  in  the  conclusions,  implementation  capacity  remains  a  problem 
hampering  the  smooth  and  efficient  delivery  of  public  investment  programmes, 
especially  in  countries  such  as  Italy  and  Romania.  This  issue  is  particularly  relevant 
when dealing with RTD investments where the efficient  engagement of different  actors 
is  a  determinant  success  factor. 
and  stakeholders  of  the  regional  ecosystems 
Administrative capacity has been addressed for a long time as one of the main areas for 
improvement in the delivery of cohesion policy, but it will deserve further attention in the 
2021-2027 period. Future evaluations should dedicate enough attention to considerations 
on 
in  the 
is  noticeable 
administrative capacity of regions and the role that the ERDF may have had to facilitate 
such improvements. In addition, any evaluation should consider the extent to which the 
design  of  regional/national  strategies  and  individual  policy  instruments  were  taken  into 
account  and  whether  they  anticipated  possible 
to 
administrative capacity issues. 

implementation  failures  due 

improvement 

the  extent 

long-term 

to  which 

there 

The SWD illustrates how the added value of cohesion policy in the RTD field, intended 
as  the  production  of  EU-wide  effects  and  catching-up  phenomena  of  regional  RTD 
performances  to  EU  standards,  comes  as  a  sort  of  unexpected  and  unplanned  effect  of 
funded  interventions.  In  particular,  added  value  does  not  appear  in  the  reconstructed 
ToCs  as  an  explicitly  intended  effect.  Despite  this,  it  constitutes  a  relevant  and  visible 
effect  of  the  policy  that  MAs  should  better  recognise  and  pursue  as  from  the 
programming phase. MAs seem to have a better awareness of the financial scale effect of 
the  contribution  of  EU  funds  to  regional  and  national  RTD  investments  rather  than  the 
more  strategic  effect,  which  has  to  do  with  setting  standards  of  performance  and 
promoting common approaches to RTD policies. It would be interesting to investigate to 
what extent, in the following programming period, the EU added value of RTD policies 
was an integral aspect of the intervention logic of both individual policy instruments 
and the policy mix as a whole. 

The  above  questions  will  be  addressed  in  the  ex  post  evaluation  of  Cohesion  policy 
programmes 2014- 2020 financed by the European Regional Development Fund (ERDF) 
and the Cohesion Fund (CF). 

A  relatively  blind  spot  of  this  evaluation  remained  the  relation  between  ERDF  support 
and the European Research Framework Programmes’ support to RTD investments. 
This evaluation described the existing alignment of policy objectives and implementation 
procedures,  concluding on  minor synergy  between the two funding streams, which was 
tackled  through  specific  regulatory  provisions  for  the  following  programming  periods 
and  will  be  evaluated  in  the  future.  For  data  quality,  however,  information  on 

47 

 
beneficiaries collected by this evaluation could not be systematically matched with data 
on  beneficiaries  of  the  Research  Framework  Programmes.  The  relation  with  ERDF 
support  is  tackled  by  a  study  on  the  ‘Analysis  of  key  parameters  of  Smart 
Specialisation  Strategies  (S3)’41,  which  examines  the  synergies  between  the  ERDF 
and Horizon 2020. This will not only include a matching of Horizon 2020 funding areas 
and  S3  priorities,  but  also  an  analysis  of  data  covering  individual  operations  and 
beneficiaries,  building  on  the  more  comprehensive  dataset  available  for  the  2014-2020 
ERDF and a newly built project database for Horizon 2020. 

41  https://op.europa.eu/en/publication-detail/-/publication/3026007b-8be2-11ed-999b-01aa75ed71a1. 

48 

 
                                                           
ANNEX I. PROCEDURAL INFORMATION 

1.  Lead DG and Decide reference 

The evaluation was led by the European Commission's Directorate-General for Regional 
and Urban Policy, DG REGIO Unit B2: Evaluation and European Semester. 

Decide entry: PLAN/2018/4614. 

2.  Organisation and timing 

The  evaluation  roadmap  was  published  on  8t  November  2018  for  a  four-week  period 
until 6 December 2018 for stakeholders and the general public to provide feedback. 

An  Interservice  Steering  Group  was  set  up  comprising  EC  services:  SG,  BUDG, 
CNECT, COMP and RTD. Four meetings of the Interservice Steering Group were held 
on:  23 October 2019, 23 April 2020, 18 January 2021, and 13 July 2021. 

3.  Evidence and sources 

Evidence  was  gathered  from  a  variety  of  sources,  including:  operational  programmes, 
implementation reports by the Member States, Major Projects documentation, monitoring 
systems  of  the  managing  authorities,  previous  evaluations  undertaken  by  the  Member 
States,  literature  review,  interviews  with  managing  authorities,  stakeholders,  final 
beneficiaries  and independent  experts, data sources at  NUTS2 level  including Eurostat, 
Patstat and Web of Science. 

Annex VI includes a full list of bibliographic references. 

4.  External expertise 

The evaluation was supported by a study run by a consortium led by CSIL – Centre for 
Industrial  Studies,  in  partnership  with  Prognos  AG  and  Technopolis  Consulting  Group 
Belgium. The service contract started on 1st of September 2019 and the final report42 was 
delivered on the 1st of December 2021. 

42  See:  https://ec.europa.eu/regional_policy/en/information/publications/evaluations/2021/evaluation-of-
investments-in-research-and-technological-development-rtd-infrastructures-and-activities-supported-
by-the-european-regional-development-funds-erdf-in-the-period-2007-2013  

49 

 
 
 
                                                           
ANNEX II. METHODOLOGY AND ANALYTICAL MODELS USED 

1.  General methodology 

The  proposed  methodological  approach,  which  is  further  detailed  in  the  ex-post 
evaluation report43 at the basis of this SWD, stems from the ambition to build a theory-
based impact evaluation in the specific context of RTD infrastructures and activities. In 
particular,  the  role  of  the  “theory”  behind  the  supported  interventions  (i.e.  the  overall 
rationale and expected supporting factors, pre-conditions and risks) was the starting point 
informing  all  the  evaluation  activities,  from  the  literature  review  to  the  projects  and 
beneficiary mapping until the case studies and the econometric analysis. In this way, the 
study  went  beyond  assessing  what  had  happened  (i.e.,  the  direct  effects  of  the  ERDF 
support for RTD), and it also tried to provide answers about why and how the observed 
effects had occurred. 

For  the  assessment  of  the  degree  of  effectiveness  of  selected  policy  instruments,  the 
study followed the approach of Contribution Analysis (CA) (Mayne, 2011), a specific 
form  of  theory-based  evaluation  that  focuses  on  ‘causal  relationships  and  explanatory 
(European 
conclusions  between  observed  changes  and  specific 
Commission, 2013). ‘Theories of Change’ (ToC) are central to this approach. The aim 
is  to  provide  evidence  to  reduce  uncertainty  rather  than  to  define  links  between 
interventions  and  effects.  This  approach  relies  on  assumptions  that  should  be  made 
visible as both requirements for and limits to our evaluation. 

interventions’ 

The  principles  of  CA  were  used  to  guide  the  collection  and  processing  of  evidence  to 
provide a judgment on the effectiveness of selected policy instruments in the seven case 
studies.  However,  the  adopted  approach  went  beyond  the  standard  CA  approach  in 
several  ways,  adapting  to  the  specificity  of  the  evaluation  and  the  need  to  be 
operationalised in a manageable yet rigorous way that could be implemented consistently 
by  all  country  experts  performing  the  field  analysis.  One  specific  element  expanding 
beyond  the  standard  CA  was  considering  that  an  intervention  works  as  part  of  a 
broader ‘causal package’, comprising the intervention outputs, a set of support factors, 
preconditions  and  possible  risks  or  threats.  The  assessment  of  the  effectiveness  of 
individual policy instruments was therefore designed along with three main steps:  

  Assessing what has changed in the performance of the beneficiaries of ERDF;  

  Assessing the extent to which the ERDF has contributed to the observed changes;     

  Assessing  how  and  via  which  mechanisms  or  contextual  factors,  outcomes  and 

results materialise.  

43  https://ec.europa.eu/regional_policy/en/information/publications/evaluations/2021/evaluation-of-

investments-in-research-and-technological-development-rtd-infrastructures-and-activities-supported-
by-the-european-regional-development-funds-erdf-in-the-period-2007-2013 

50 

 
                                                           
This evaluation was guided by a set of Evaluation Questions corresponding to several 
evaluation criteria (see Annex III). The evaluation approach relied on a combination of 
different methods to provide comprehensive answers to the questions. In particular: 

  The selection of 53 representative OPs by the European Commission out of the 
total of 215 OPs funded by the ERDF and covering 18 Member States out of 28 
as well as a substantial amount of RTD investments (with EUR 14.64 billion of 
contribution, i.e., about 85% of the EU total funding for the relevant themes). 
Within  the  selected  OPs,  the  evaluation  study  focused  on  two  categories  of 
ERDF  expenditures  (01  –  RTD  activities  in  research  centres,  and  02  –  RTD 
infrastructure and centres of competence in a specific technology)44. 

  A  mapping  of  ERDF  expenditures  by  types  of  projects  and  beneficiaries 
funded by 53 OPs provided a fine-grained description of what was funded on the 
ground and the logic underpinning the mix.  

  An analysis of strategies and objectives pursued by the 53 OPs to understand 

the funded instruments' motivations and rationales.  

  A cluster analysis described the RTD performance of EU regions in the period 

under assessment.  

  A literature review identified prior theories of change regarding ERDF support 

for RTD investments and their expected results.  

  Seven  case  studies  at  Member  State  (MS)  level  assessed,  in  detail,  a  selected 
number  of  OPs  and  their  most  representative  policy  instruments  (a  total  of  21 
policy instruments were assessed with a contribution analysis). This was done 
to  build  more  precise  theories  about  the  implemented  instruments  and  collect 
evidence  on  observed  outcomes  and  results  and  the  conditions  for  their 
materialisation.  Primary  evidence  collected  via  the  case  studies  came  from  200 
interviews  with  managing  authorities,  stakeholders,  final  beneficiaries  and 
independent experts.  

  A cross-case analysis, at the levels of four types of intervention, aggregated and 
generalised  the  theories,  and  results  duly  crystallised  in  terms  of  what  works, 
where, and according to which mechanisms.  

  A  seminar  with  stakeholders  and  experts  allowed  the  discussion  of  preliminary 

results;  

  An econometric analysis, employing a multivariate regression and using the data 
collected  from  funded  projects  and  beneficiaries,  allowed  the  testing  of  a  set  of 
hypotheses about the contribution of the various types of instruments to a set of 

44  More  specifically,  they  include:  support  for  RTD  activities  in  research  centres  (e.g.,  scientific  R&D 
activities;  collaborative  research  activities;  support  for  the  internationalisation  of  research  activities; 
development of researchers and other personnel involved in R&D activities; support for technology-
transfer  activities;  and  the  valorisation  of  research  results),  support  for  RTD  infrastructures,  and 
support for competence centres in a specific technology. 

51 

 
                                                           
regional  outcomes.  The  multivariate  analysis  permits  one  to  isolate  the 
contribution of specific ERDF types of instrument to specific regional outcomes 
from  other  potential  factors  (these  ‘other’  factors  might  include,  e.g.,  regional 
socio-economic  start  conditions,  other  R&D  policies  beyond  the  ERDF 
interventions,  etc.).  The  analysis  was  not  meant  to  provide  an  estimation  of  the 
size of the observed effects, which was not conceivable given the available data45, 
but  to  offer  a  complement  to  the  qualitative  analysis  of  the  case  studies.  In 
combination  with  the  understanding  of  the  causal  mechanisms  explored  in  the 
case studies, the estimated statistical significance expanded and corroborated their 
findings on the contribution of ERDF interventions to some observed changes.    

Figure 1 - Map of selected case studies and policy instruments 

Source: Ex-post evaluation report (2021)   

The combination of different methods was instrumental in collecting a comprehensive set 
of  evidence.  Their  interconnection  was  built  at  a  sequential  level,  building  on  the 
evidence gained from the previous step and expanding in the directions indicated by the 
evaluation  questions.  While  some  analysis  methods  were  selected  because  of  their 
appropriateness in answering some of the evaluation questions (case studies, cross-cases 
analysis,  econometric  analysis),  others  were  necessary  as  a  starting  point  for  further 
analysis  (mapping  of  projects  and  beneficiaries,  literature  review,  analysis  of  OP 
strategies). The overall logic went from a general overview to the specific assessment of 
individual cases, then expanded to a more general level.  

45 

In  particular,  lacking  a  control  group  of  non-treated  regions,  a  pure  counterfactual  method,  such  as 
propensity  score  matching,  regression  discontinuity  design  or  difference  in  difference,  could  not  be 
carried out.  

52 

 
 
                                                           
Figure 2 - Methodological framework: a combination of methods 

Source: Ex-post evaluation report (2021) 

2.  Limitations and mitigation strategy 

Despite  the  huge  amount  of  data  and  evidence  collected  via  the  evaluation  activities, 
some limitations remain. Among these, the most significant are as follows:  

the 

  The  scope  of  the  evaluation  is  limited  to  investments  supported  under 
expenditure codes 01 and 02, which were not included in the ex-post evaluation46. 
target  OPs,  other  codes  of  expenditure  also 
Nevertheless,  within 
synergistically  contributed  to  the  overarching  objective  of  strengthening  the 
regional RTD systems.  This  was true, in  particular,  as  far as  the contribution of 
ERDF concerns the objective of bridging the gap between the provision and the 
use of research results. The distinction between ERDF expenditure codes in this 
area is rather artificial, also prone to possible miscoding on the part of the MA. 
Focusing merely on codes 01 and 02 may have limited the capacity of the present 
evaluation  to  explore  the  contribution  of  ERDF  to  this  strategic  objective. 
Nonetheless,  this  evaluation  could  partially  address  this  gap  by  building  on  the 
results of a previous evaluation focused on ERDF support for SME research and 
innovation (European Commission, 2016b). 

  As  discussed  in  section  3  of  the  SWD  and  in  Annex  II.4,  the  monitoring 
indicators collected by MAs have limitations both in terms of coverage of the 
actions supported and in  terms  of quality (i.e. consistency and  gaps).  Therefore, 

46  The Commission, in work package 2 "Small and medium sized enterprises, innovation, ICT", assessed 
the effects of ERDF support on research and innovation in micro, small and medium-sized enterprises 
(SMEs)  and  outlined  the  mechanisms  and  context  features  underlying  these  effects.  However  it  was 
limited  to  SME  support  and  did  not  include  specific  investments  on  Research  and  Technological 
Development (RTD) infrastructures and activities, as these investments, due to their scope and nature, 
take longer to implement (many of them were not finalised by the time when the ex-post evaluations of 
2007-2013 period were carried out) and yield results even later. 

53 

 
 
                                                           
the evaluation filled the gaps by making use of additional indicators. Expanding 
the list of such indicators in further evaluations could further increase the strength 
of similar analyses. 

  Evidence on outcome and impact at the level of individual policy instruments is 
mostly  qualitative  and  based  on  stakeholder  perception.  Respondents'  possible 
optimism biases were mitigated by triangulating among different stakeholders and 
by complementing primary evidence with secondary data sources.  

  A theory-based impact evaluation was performed on 21 individual instruments, as 
implemented  in  seven  MS.  This  was,  therefore,  a  limited  sample  of  all 
implemented  instruments.  By  construction,  the  contribution  analysis  aimed  to 
provide  evidence  to  reduce  uncertainty  regarding  the  mechanisms  underpinning 
the  observed  changes,  rather  than  to  define  links  between  interventions  and 
observed  changes.  This  approach  relies  on  certain  assumptions  as  both 
requirements  for  and  limits  to  our  evaluation.  A  standardised  approach  to 
producing  the  contribution  analyses  and  assessments  and  to  reporting  on 
preconditions,  supporting  factors  and  risks,  was  designed  and  promoted  among 
the case-study authors. Nevertheless, some degree of discretion may remain in the 
way judgements are derived, both in terms of the evidential base (particularly, the 
balance  between  evidence  generated  by  the  case  studies  and  the  existing 
literature) and the interpretation of such evidence and causal mechanisms.  

  The  poor  quality  of  some  beneficiary  data  (lack  of  partner  beneficiaries  for 
collaborative  projects,  duplication  or  missing  data)  prevented  a  more  in-depth 
analysis, and it also obstructed systematic matching with external databases (for 
instance, matching with  the CORDIS database could  be carried out  only for the 
selected OPs of the case studies).   

  The evaluation did not collect information on compliance costs (nor direct or 
indirect)  and  enforcement  costs  due  to  the  fact  that  this  information  is  not 
relevant in  case of the evaluated interventions which are investments. Similarly, 
no data was collected on simplification and burden reduction which was not in 
the  scope  of  this  evaluation.  Consequently,  this  SWD  does  not  contain  “Annex 
IV.  Overview  of  benefits  and  costs  and,  where  relevant,  table  on  simplification 
and  burden  reduction”  as  envisaged  by  Tool#49  of  the  Better  Regulation 
Toolbox. 

Despite these limitations, the triangulation of data sources and extensive discussions with 
country  experts,  senior  advisors,  and  stakeholders  enabled  the  team  to  gather  robust 
conclusions  for  most  evaluation  questions.  Some  open  questions  remain;  they  are 
discussed in the last chapter of the report.  

3.  List of the sample of 53 operational programmes 

The list of the 53 Operational Programmes covered by the study is presented in the Table 
below. 

Table 1 - List of 53 Operational Programmes 

54 

 
Country 
BE 
BE 

CZ 
CZ 
CZ 
DE 
DE 
DE 

DE 
DE 
DE 
DE 

DE 
DE 

EE 

ES 
ES 
ES 
ES 
ES 
ES 

ES 

FI 
FR 
FR 
FR 
FR 
FR 
FR 

FR 
HU 
IE 
IT 

IT 

LT 
LV 
PL 
PL 
PL 
PL 

PL 

PL 

CCI 
2007BE161PO001 
2007BE162PO003 

2007CZ161PO004 
2007CZ161PO012 
2007CZ162PO001 
2007DE161PO001 
2007DE161PO002 
2007DE161PO003 

2007DE161PO004 
2007DE161PO007 
2007DE162PO001 
2007DE162PO003 

2007DE162PO004 
2007DE162PO007 

2007EE161PO001 

2007ES162PO002 
2007ES162PO004 
2007ES162PO005 
2007ES162PO006 
2007ES162PO010 
2007ES16UPO001 

2007ES16UPO003 

2007FI162PO001 
2007FR162PO001 
2007FR162PO011 
2007FR162PO015 
2007FR162PO016 
2007FR162PO017 
2007FR162PO020 

2007FR162PO021 
2007HU161PO001 
2007IE162PO002 
2007IT161PO006 

2007IT162PO002 

2007LT161PO002 
2007LV161PO001 
2007PL161PO001 
2007PL161PO002 
2007PL161PO003 
2007PL161PO010 

2007PL161PO011 

2007PL161PO013 

Name of the OP 
Programme opérationnel 'Convergence' Hainaut - FEDER 
Programme opérationnel 'Compétitivité régionale et emploi' - 
Wallonie (hors Hainaut) - FEDER 
OP Podnikání a inovace 
OP Výzkum a vývoj pro inovace 
OP Praha Konkurenceschopnost 
Operationelles Programm EFRE Thüringen 2007 bis 2013 
Operationelles Programm EFRE Brandenburg 2007-2013 
Operationelles Programm EFRE 2007 - 2013 Mecklenburg- 
Vorpommern 
Operationelles Programm EFRE Sachsen 2007-2013 
Operationelles Programm EFRE Sachsen-Anhalt 2007-2013 
Operationelles Programm EFRE Bayern 2007 - 2013 
Operationelles Programm EFRE Schleswig-Holstein 2007-
2013 
Operationelles Programm EFRE Berlin 2007-2013 
Operationelles Programm EFRE Nordrhein-Westfalen 2007-
2013 
Operational Programme for the Development of Economic 
Environment 
Programa Operativo FEDER del País Vasco 
Programa Operativo FEDER de Madrid 
Programa Operativo FEDER de La Rioja 
Programa Operativo FEDER de Cataluña 
Programa Operativo FEDER de la Comunitat Valenciana 
Programa Operativo FEDER de Investigación, Desarrollo e 
innovación por y para el beneficio de las Empresas - Fondo 
Tecnológico 
Programa Operativo FEDER de Economía basada en el 
Conocimiento 
Itä-Suomen EAKR-toimenpideohjelma 2007-2013 
Programme opérationnel FEDER AQUITAINE 
Programme opérationnel FEDER HAUTE-NORMANDIE 
Programme opérationnel FEDER LORRAINE 
Programme opérationnel FEDER PAYS DE LA LOIRE 
Programme opérationnel FEDER NORD PAS-DE-CALAIS 
Programme opérationnel FEDER PROVENCE ALPES COTE 
D´AZUR 
Programme opérationnel FEDER MIDI-PYRENEES 
Economic Development Operational Programme 
Southern and Eastern Operational Programme 
Pon Ricerca e competitivita' - Riprogrammazione - 30 ottobre 
2014 
Por Emilia Romagna FESR Versione approvata dal Comitato di 
Sorveglianza 2015 
2007-2013 m. Ekonomikos augimo veiksmų programa 
Entrepreneurship and Innovations 
Program Operacyjny Innowacyjna Gospodarka, 2007-2013 
Program Operacyjny Infrastruktura i Środowisko 
Program Operacyjny Rozwój Polski Wschodniej 2007-2013 
Małopolski Regionalny Program Operacyjny na lata 2007-
2013 
Regionalny Program Operacyjny Województwa 
Mazowieckiego 
Regionalny Program Operacyjny Województwa 
Podkarpackiego 

55 

 
Country 
PT 
PT 
PT 
RO 

SI 

SK 
UK 
UK 

UK 

UK 

UK 

Source: Author 

CCI 
2007PT161PO001 
2007PT161PO002 
2007PT161PO003 
2007RO161PO002 

2007SI161PO001 

2007SK16UPO001 
2007UK161PO002 
2007UK161PO003 

2007UK162PO001 

2007UK162PO008 

2007UK162PO009 

Name of the OP 
PO Factores de Competitividade 2007-2013 
PO Regional do Norte 2007-2013 
PO Regional do Centro 2007-2013 
Sectoral Operational Programme Increase of Economic 
Competitiveness 
Operativni program krepitve regionalnih razvojnih 
potencialov za obdobje 2007 - 2013 
OP Research and Development 
West Wales and the Valleys ERDF Convergence programme 
Cornwall and the Isles of Scilly ERDF Convergence 
programme 
Lowlands and Uplands of Scotland ERDF Regional 
Competitiveness and Employment programme 
Northwest England ERDF Regional Competitiveness and 
Employment Operational Programme 
Yorkshire and Humberside England ERDF Regional 
Competitiveness and Employment programme 

4.  Overview of core and common indicators on RTD 

In  the  framework  of  the  past  2007-2013  Cohesion  Policy,  monitoring  and  evaluation 
activities were asked to be performed by regulation 1083/2006 (Art 37.c). The relevance 
of the indicators is “to make it possible to measure the progress of each Priority Axis of 
the Operational  Programme in  relation  to  the baseline situation and the achievement of 
the  targets”.  For  each  result  and  output  indicator,  the  baseline  and  target  values  were 
required to be established by the Managing Authorities. The progress towards achieving 
targets  established  for  each  indicator  has  to  be  reported  by  the  Member  States  in  the 
Annual Implementation Report (as required by Art 29 of the EC Reg. 1083/2006) and the 
financial execution tables.  

While  result  and  output  indicators  were  mandatory,  impact  indicators  were  strongly 
encouraged but  not  formally required. Member States were free to  identify  and use the 
most  appropriate  indicators  according  to  Programme  objectives  and  the  programming 
period's strategic focus. While providing the Member States with this responsibility, the 
European Commission also strongly encouraged them to use a limited number of “core 
indicators” (output and result), aggregated and compared at the EU level.47 Out of the full 
list of 56 core indicators indicated by the European Commission, three are directly and 
explicitly referred to research and technological development (RTD), i.e.: 

 

 

 

‘number of RTD projects (4)’ 

‘number of cooperation projects enterprises – research institutions (5)’  

‘research jobs created (preferably 5 years after projects start) (6)’.  

47  The  concept  of  core  indicators  was  first  used  in  the  EC  Working  Document  No.  2,  providing 
guidelines on the setting up of indicator systems for monitoring and evaluation. European Commission 
(2006),  Working  Document  No.  2:  Indicative  guidelines  on  evaluation  methods:  monitoring  and 
evaluation indicators, DG Regio, Brussels. 

56 

 
                                                           
Additionally, during the 2014-2020 programming period, the Commission recommended 
using  an  additional  46  common  indicators,  some  of  which  had  already  been  used  by 
many OPs in the previous programming period. The common indicators relevant to the 
RTD investments are: 

 

 

 

 

‘number of new researchers in supported entities (24)’,  

‘number  of  researchers  working  in  improved  research  infrastructure  facilities 
(25)’, 

‘number of enterprises cooperating with research institutions (26)’  

‘private  investments  matching  public  support  in  innovation  or  R&D  projects 
(27)’.  

In addition to these, each OP included various programme-specific indicators selected by 
the  Managing  Authorities  to  keep  track  of  the  achievements  concerning  the  specific 
objectives set in the programme. 

Previous evaluation studies have stressed that the achievement indicators of the regional 
monitoring systems have some limitations: they are generally not available at the level of 
individual projects or policy instruments and could refer to projects funded under other 
categories of expenditure than 01 and 02; the target indicators can be flawed making the 
comparison with the actual achievement indicators not fully reliable; it is not possible to 
compare  the  programme  specific  indicators  across  different  OPs;  being  focused  on  the 
programme  outputs  and  results  (and  not  on  the  impact),  the  achievement  indicators  are 
not sufficient for a complete evaluation of effectiveness.  

Despite this, these indicators are a source of evidence (even the only one for some OPs) 
to have some signs of what has been reported as having been achieved by the OPs. 

5.  Econometric analysis 

The  ex-post  evaluation  report  employed  multivariate  regression  analysis  to  support  the 
findings  in  the  main  report.  The  multivariate  analysis  permits  one  “to  isolate”  the 
contribution  of  the  ERDF  types  of  instruments  in  the  scope  of  the  study  to  specific 
regional  outcomes  from  the  other  potential  factors  (e.g.,  regional  socio-economic 
conditions,  other  R&D  policies  beyond  the  ERDF  instruments,  etc.)  influencing  those 
outcomes (see box below). 

The regression equation estimated is as follows:  

Y_i=α+βX_i+γZ_i+ε_i 

Where: 

  Y_i indicates the outcome (dependent) variable we want to explain in the NUTS2 
region i (for instance, the growth rate in the number of scientific publications); 

  X_i is the ERDF type of instrument (policy variable) we are interested in;  

  Z_i is a vector of controls, i.e. variables that can influence the outcome variable 
beyond  the  ERDF  instrument  (for  instance,  the  Gross  domestic  expenditure  on 

57 

 
R&D in the region);  

  ε_i is an i.i.d error term.  

  α,β, and γ represent the parameters (coefficients) of the model to be estimated and 
measure  the  correlation  between  the  regressors  (X_i,Z_i)  and  the  outcome 
variable. If the ERDF policy variable had an impact on the outcome variable, then 
the coefficient β in the equation above is expected to be positive and statistically 
significant. α is the constant of the model. 

Specifically, the  ex-post evaluation  tested  a set  of hypotheses  about  the  contribution  of 
three ERDF types of instruments implemented in the 2007-2013 programming period to 
a  set  of  regional  outcomes.  The  ERDF  types  of  instruments  examined  (see  the  ex-post 
evaluation report for the rationale behind their selection) were: 

  ERDF expenditure in infrastructures for research and individual R&D projects in 

universities / Higher Education Institutions (HEIs);  

  ERDF expenditure in infrastructures and investments for education; 

in 

  ERDF  expenditure 

infrastructures  for  research  and 

in  private-public 
collaborative R&D projects that involve enterprises. This instrument is targeted to 
Research  and  Technology  Organisations  (RTOs),  Enterprises,  Clusters;  Science 
Parks, Consortia, Competence and/or excellence centres. 

As explained in detail in the ex-post evaluation report, each ERDF type of instrument had 
its  specific  instrument  logic,  its  policy  objectives  and  targeted  different  beneficiaries. 
Accordingly,  depending  on  the  instrument  and  related  hypotheses,  several  regression 
analyses with different dependent (outcome) and control (context) variables were run. 

Building on the findings of the case studies, the following hypotheses were selected for 
testing: 

  HP1: Regions, especially lagging ones, improved their basic research capacities 
by supporting universities' infrastructure investments and research projects. This 
contributed  to  an  increase  in  the  R&D  personnel  in  the  region  and  an 
improvement in scientific production. This process has been further supported by 
an  increase  in  public  expenditures  in  R&D  but  did  not  immediately  improve 
scientific excellence. 

  HP2:  Lagging  regions  investing  in  infrastructure  for  education  attracted  more 
students and improved the tertiary attainment in the region. This contributed to an 
increase in employees in science and technology, but only in those regions with 
an already advanced industrial fabric. 

  HP3:  Regions  investing  a  larger  share  of  funds  in  RTO,  science-industry 
collaborations  or  centres  of  excellence  (either  infrastructures  or  activities) 
experienced an increase in the level of patent applications and/or other intellectual 
property  rights  and  public-private  co-publications.  This  holds  true  in  those 
regions with a high concentration of funds among beneficiaries and regions with a 
more mature R&I system. 

58 

 
The ex-post evaluation  assembled a database at  the regional NUTS2 level  from several 
sources, including Eurostat, Patstat, Web of Science to test the hypotheses. The sample 
includes 104 EU regions (NUTS2) covered by selected Operational Programmes (OPs) 
in  the  scope  of  the  evaluation:  46  OPs  with  the  highest  ERDF  expenditure  in  the 
Cohesion Policy themes of expenditures 01 and 02.48 

Hypotheses are verified employing regression analysis. The main variables entering the 
models vary according to the specific hypothesis and type of instrument under scrutiny49.  

Annex IV to the ex-post evaluation reports on the full results. It is organised into three 
main  sections,  each  one  devoted  to  the  analysis  of  one  hypothesis.  Tables  with  related 
statistics and econometric results are reported, and, at the end of each section, the main 
findings are summarised. 

ANNEX III. EVALUATION MATRIX 

RELEVANCE 
The relevance criterion has to do with the relationship between the needs to be tackled and the objectives of the 
overall ERDF strategies and related policy instruments. More specifically, it touched on aspects of the design of 
the programmes and identified whether there was a mismatch between the ERDF mix of policy instruments for 
RTD and the barriers to research and technological development identified by the programmes. 
Questions/sub-
questions 

Conclusions 

EQ 1: What are the 
interventions 
supported by ERDF? 

including 

›  ERDF  support  in  RTD  investments  funded  mainly  infrastructure 
construction  and  modernisation, 
improve 
education and research activities. These investments took the lion’s 
share  of  ERDF  contributions.    In  total,  excluding  Spain,  55%  of 
projects  financed  R&D  projects,  both  individual  and  collaborative. 
internationalisation  of  research  and 
Other 
capacity  building 
for  science  and 
dissemination, intellectual property rights instruments and operating 
subsidy (see Section 3.1). 

interventions  were 

for  research, 

investments 

ICT, 

to 

Source of 
evidence 
›  Mapping  of 
projects  and 
beneficiaries 
(Task 1)  

Level of 
analysis  
 ›  OP  and 
level 
MS 
(46 
OPs 
and 17 MS) 

48  Depending on the  way  how the  regional and national expenditure  monitoring systems are organised, 
project-level  data  were  not  available  for  all  the  OPs.  In  some  cases,  they  had  to  be  derived  from 
processing the list of beneficiaries or data on individual tranches of payments. Significant efforts were 
devoted to this activity, and it was eventually possible to build a consistent database of projects for 46 
out  of  53  OPs.  In  the  specific  case  of  Spain,  the  peculiarities  of  the  monitoring  systems  and 
particularly  the  lack  of  any  project-level  identification  code,  prevented  from  aggregating  all 
expenditure  data  at  the  level  of  projects.  In  the  report,  approximate  data  and  information  on  the 
Spanish  programmes  are  provided.  Still,  they  are  not  considered  when  producing  aggregate  project-
level statistics to preserve the accuracy and reliability of the rest of the data. 

49 

In  the  analysis,  the  variable  of  interest  –  i.e.,  the  ERDF  instrument-  is  labelled  as  “ERDF  policy 
variable”. 

59 

 
 
                                                           
RELEVANCE 
The relevance criterion has to do with the relationship between the needs to be tackled and the objectives of the 
overall ERDF strategies and related policy instruments. More specifically, it touched on aspects of the design of 
the programmes and identified whether there was a mismatch between the ERDF mix of policy instruments for 
RTD and the barriers to research and technological development identified by the programmes. 
Questions/sub-
questions 

Conclusions 

Source of 
evidence 
›  Mapping  of 
projects  and 
beneficiaries 
(Task 1)  
› OP analysis 
(Task 1) 

Project 
(46 

Level of 
analysis  
› 
level 
OPs) 
›  OP  level 
and  MS  (53 
OPs and 18 
MS) 

› 
Documentary 
and literature 
review (Task 
2) 
› OP analysis 
(Task 1) 
› Case 
studies (Task 
3) 

›  OP  level 
and  MS  (53 
OPs and 18 
MS) 
›  PI 
(21 PIs) 

level 

EQ 2 How is ERDF 
support divided 
between the different 
types of intervention 
and forms of 
financing?  

EQ 3: What is the 
underlying rationale, 
also considering the 
role of ERDF support 
in the policy mix? 

› The largest share of ERDF expenditure, more than EUR 9 billion 
(72%  of  total),  was  concentrated  on  support  for  infrastructure 
investments, with infrastructure investments for research absorbing 
more  than  half  of  ERDF  expenditure  (57%).  R&D  projects,  both 
individual and collaborative, represented the most common type of 
intervention in terms of the number of projects, but only 23% of the 
total  ERDF  expenditure.  A  smaller  share  of  projects  (6%)  and 
expenditure (3%) was allocated to the implementation of other sorts 
of RTD activity, while a residual portion (3% of ERDF expenditure) 
funded  activities  that  were  not  strictly  related  to  RTD  and  should 
instead  have  been  classified  under  different  codes  (see  Section 
3.1). 
›  The  selected  53  OPs  supported  more  than  20,000  projects, 
almost half of which were in Spain. Almost all projects were funded 
through non-repayable grants (see Section 3.1). 
›  The  majority  of  beneficiaries  were  publicly  owned  organisations. 
About  4,000  different  institutions  can  be  identified  among  the 
almost  24,000  lead  beneficiaries  (about  2,000  different  institutions 
including  almost  580  HEIs,  more  than  720  RTOs,  nearly  470 
enterprises, when excluding Spain) (see Section 3.2). 
Overall: 
› The rationale for public intervention in RTD stems from the need 
to  tackle  multiple  investment  needs.  According  to  the  literature, 
such  needs  refer  to  specific  market  mechanisms  preventing  long-
term investments in RTD because of the indivisible, inappropriable 
and  uncertain  nature  of  research  (market  failures)  as  well  as  key 
deficiencies in the actors producing research and in overall systems 
(systemic failures) (see Section 2.1).  
›  According  to  the  OP  analysis,  some  systemic  failures  were  the 
main  rationale  for  RTD  support.  The  need  to  tackle  infrastructure 
gaps and failures and to facilitate improvements in science-industry 
collaboration  comprised  the  logical  foundation  of  the  interventions 
of  most  of  the  programmes  (87%  and  79%  respectively),  despite 
the  wide  variety  of  territorial  contexts  and  research  systems. 
Differences  were  instead  observed  in  the  way  the  OPs  translated 
their  strategic  approaches  into  policy  mixes.  Similar  territorial 
contexts saw the adoption of different combinations of instruments 
(see Section 2.4).  
By policy instrument: 
›  Infrastructure  investments  for  research  aimed  at  addressing  the 
lack  of  sufficient  or  modern  physical  and 
technological 
infrastructure,  an  essential  component  in  fostering  knowledge 
creation (see Section 3.3.1). 
›  Infrastructure  investments  for  education  were  geared  more 
towards improving university education facilities than towards RTD 
laboratories (see Section 3.3.2). 
› 
infrastructures  aimed  at  providing  digital-based 
services  and  tools  for  data  and  computing-intensive  research  in 
virtual and collaborative environments (see Section 3.3.3). 
› Collaborative projects had various aims, ranging from addressing 
industrially relevant or societal challenges, stimulating technological 
advancement 
international 
in  specific  areas,  and  boosting 
cooperation  by  conducting  internationally  competitive  high-quality 
R&D activities (see Section 3.4.1). 
›  Individual  research  projects  had  the  objective  to  strengthen  the 
scientific  and  technological  capacity  of  the  supported  region  (see 
Section 3.4.2). 

ICT-based 

60 

 
 
Conclusions 

RELEVANCE 
The relevance criterion has to do with the relationship between the needs to be tackled and the objectives of the 
overall ERDF strategies and related policy instruments. More specifically, it touched on aspects of the design of 
the programmes and identified whether there was a mismatch between the ERDF mix of policy instruments for 
RTD and the barriers to research and technological development identified by the programmes. 
Questions/sub-
questions 
EQ 4: Is ERDF 
support based on 
research demand 
(bottom-up), or does 
it focus on the 
availability of 
support services and 
infrastructure and 
gaps in these (top-
down)? 

›  In  project  selection,  infrastructure  investments  largely  followed  a 
top-down  approach  guided  by  national  road-mapping  exercises 
(see  Section  3).  R&D  projects,  conversely,  primarily  followed  a 
bottom-up  approach  within  well-identified  priority  scientific  or 
technological fields (see Section 4).  

Source of 
evidence 
› OP analysis 
(Task 1) 
› 
Case 
studies (Task 
3) 

Level of 
analysis  
›  OP  level 
and  MS  (53 
OPs and 18 
MS) 

›  Mapping  of 
projects  and 
beneficiaries 
(Task 1)  
› 
Case 
studies (Task 
3) 

›  OP  level 
and  MS  (9 
OPs  and  7 
MS) 

›  OP  level 
and  MS  (53 
OPs and 18 
MS) 
›  PI 
(21 PIs) 

level 

› Mapping of 
projects and 
beneficiaries 
(Task 1)  
› OP analysis 
(Task 1) 
› Case 
studies (Task 
3) 
› Cross-case 
studies 
analysis 
(Task 4) 
› Seminar 
(Task 5) 

EQ 5: How was 
investment targeted 
in respect of 
geographical areas 
and sectors: to those 
with significant 
potential or 
comparative 
advantage or those 
in difficulty or 
lagging behind? 

EQ 6: Did the ERDF 
interventions match, 
or respond to, the 
policy challenges? 

›  While  broader  territorial  targeting  strategies  were  driven  by 
eligibility criteria  targeting lagging  territories,  targeting strategies  of 
specific  OPs  (especially  nationally)  or  instruments  did  not  include 
an  explicit  geographical  component  and  were  rather  ‘territorially 
agnostic’.  Overall,  the  funds  were  mainly  addressed  to  strengthen 
existing territorial excellence, even in countries with strong regional 
disparities in RTD (such as Romania) (see Sections 3.7 and 5.2.1). 
› In many cases, ‘target priority’ sectors and technologies reflected 
regional  specialisation  and  were  identified  by  existing  policy 
strategies and documents, either at national or regional levels (see 
Sections 3.7 and 5.2.1). 
›  In  most  cases,  the  target  OPs  funded  beneficiaries  with  a 
competitive  advantage,  with  a  high  concentration  within  individual 
beneficiary  organisations  leading  institutions  in  their  field  (see 
Sections 3.7 and 5.2.1). 
Overall: 
› Despite differences in terms of policy challenges between Central 
and Eastern European countries on the one hand and Western EU 
countries on the other, the key strategic objectives pursued by the 
selected OPs and related PIs were to fill the infrastructure gap and 
to  improve  the  systematic  interaction  among  regional  actors.  This 
was in line with the main systemic failures identified by the different 
OPs (see Section 3.6). 
›  ERDF  support  for  RTD  responded  to  the  main  policy  challenges 
and  external  challenges,  such  as  the  economic  crisis.  It  helped 
public  research  infrastructures  and  businesses  to  withstand  the 
crisis  by  providing  a  significant  source  of  funds,  sometimes 
palliating  a  decrease  in  national  public  support.  This  holds  true 
particularly  in  those  countries  and  regions  most  severely  affected 
by the crisis (see Section 5.1.1).  
By policy instrument: 
›  Infrastructure  investments  for  research  and  education  upgraded 
existing  infrastructure  and  equipment  and  replaced  obsolete  or 
outdated  instances  in  Higher  Education  Institutions  and  RTD 
organisations  (see  Section  3.3.1  and  Romania,  Czechia,  Estonia 
and Poland Case Study).  
›  ICT-based  infrastructures  established  or  improved  computing 
grids, data-storage centres, open-data infrastructures, ICT network 
systems  and  e-infrastructures  (see  Section  3.3.1  and  the  Czechia 
and Germany Case Study). 
›  Collaborative  projects  consisted  of  projects  between  R&D 
institutions themselves and with private-sector partners carrying out 
research  activities  mainly  with 
innovation 
potential  (see  Section  3.4.1.  and  Romania,  Poland  and  Italy  Case 
Study). 
› 
Individual  research  projects  consisted  of  both  early-stage 
(foundational)  and  exploratory  research  aimed  to  generate  new 
knowledge and develop innovative skills in research institutions and 
projects with a predetermined commercial application (see Section 
3.4.2 and the Portugal and Germany Case Study). 

technological  and 

61 

 
 
 
 
 
RELEVANCE 
The relevance criterion has to do with the relationship between the needs to be tackled and the objectives of the 
overall ERDF strategies and related policy instruments. More specifically, it touched on aspects of the design of 
the programmes and identified whether there was a mismatch between the ERDF mix of policy instruments for 
RTD and the barriers to research and technological development identified by the programmes. 
Questions/sub-
questions 

Source of 
evidence 

Level of 
analysis  

Conclusions 

Overall assessment 
on RELEVANCE 

ERDF  support  for  RTD  was  overall  relevant.  The  combinations  of  policy  instruments  were 
designed to respond to a wide range of needs, mainly related to RTD capacities, identified in the 
programming  documents  and  confirmed  by  the  literature  and  the  cluster  analysis.  More 
specifically, the ERDF supported massive investments to address infrastructural gaps and, to a 
lesser extent, tackled difficulties in the interactions of the innovation system actors.  
The  majority  of  RTD  interventions  were  geared  at  supporting  excellence  objectives,  targeting 
more  advanced  territories,  stronger  sectors  and  best  performing  institutions  within  eligible 
territories.  Although  this  approach  was  justified  by  the  need  to  ensure  critical  mass,  fund 
absorption  and  knowledge  externalities  in more mature territories, the  question  of  whether  this 
approach  may  have  contributed  to  increasing  the  territorial  divide  vis-à-vis  lagging  regions 
remains open. 

EFFECTIVENESS 
Effectiveness assesses the extent to which selected policy instruments have successfully achieved 
or  progressed  towards  the  stated  objectives  and  delivered  the  expected  outputs,  outcomes,  and 
impacts. Effectiveness assessed the extent to which: 

 

 

selected policy instruments have been successful, also in combination with other EU and 
national  support  for  RTD,  in  achieving  or  progressing  towards  the  stated  objectives  and 
delivered the expected results; 
the ERDF policy mix for RTD has been effective in improving RTD performances of funded 
regions. 

The criterion also analysed the main factors influencing the effectiveness of RTD interventions. 
Questions/sub-
questions 

Conclusions 

of 

EQ 7: Have research 
projects achieved 
their intended 
objectives? 

EQ  8:  How  effective 
the  different 
were 
groups 
ERDF 
of 
interventions  for  RTD 
infrastructure 
and 
activities,  and  how 
they  were  combined 
with 
RTD 
other 
support? 

for 

ICT-based 

investments 

Infrastructure 

infrastructures  enabled 

› 
research  and  education 
contributed  to  the  creation  or  modernisation  of  public  R&D 
facilities  (including  universities),  which  in  turn  increased  the 
potential  and  capacity  of  the  beneficiary  institutions  and  created 
more  ‘respectable’  research  and  education  environments,  thus 
attracting  new  students  and  researchers  (see  Section  4.2,  4.3 
and the Czechia, Estonia, Poland and Romania Case Study).  
› 
the  higher  storage, 
computational  and  information  capacities  of  R&D  institutions, 
thus  improving  the  availability  of  scientific  information  resources 
and  keeping  up  with  the  always  faster  progressing  digitisation 
(see the Czechia and Germany Case Study). 
›  Collaborative  projects  generally  boosted  cooperation  between 
science  and  industry  actions,  thus 
favouring  a  knowledge 
exchange  process,  but  more  limited  evidence  is  available 
regarding  the  capacity  of  funded  projects  to  generate  economic 
benefits obtained from the commercial valorisation of R&D results 
(see Section 4.5 and Romania, Poland and Italy Case Study). 
› Individual research projects helped develop high-level scientific 
activities and consolidate relevant knowledge in the scientific and 
technological  system  (see  Section  4.4.  and  the  Portugal  and 
Germany Case Study). 
›  Infrastructure  investments  targeting  HEIs  had  a  significant 
positive  effect  on  the  number  of  tertiary-educated  persons  and 
employment  in  science  and  technology,  especially  in  those 
regions with more advanced industrial fabric and with higher R&D 
in  the  business  sector  (see  Section  4.2  and  4.3)  and  where 
continued  public 
technology  and 
innovation  in  combination  with  ERDF  support  was  ensured  (see 
the Estonia and Portugal Case Study). 
While  the  level  of  collaboration  has  generally  improved  or 
remained stable, the role of industry actors in the uptake of RTD 
results  has  not  significantly  changed.  ›  Collaborative  R&D 
projects were less effective in improving the interactions between 
the  different  actors  of  the  RTD system.  This  happened  although 
collaborative  R&D  projects  were  combined  with  infrastructure 
investments  which  should  improve  the  attractiveness  of  RTD 
for  science-industry 
the  groundwork 
institutions  and 

in  science, 

investment 

lay 

62 

Level  of 
analysis  
›  PI 
(21 PIs) 

level 

Source 
evidence 
› Mapping of 
projects and 
beneficiaries 
(Task 1)  
› Case studies 
(Task 3) 
›  Cross-case 
studies 
analysis  (Task 
4) 
› Seminar 
(Task 5) 
›  Econometric 
analysis  (Task 
6) 

›  PI 
(21 PIs) 

level 

› Mapping of 
projects and 
beneficiaries 
(Task 1)  
› Case studies 
(Task 3) 
›  Cross-case 
studies 
analysis  (Task 
4) 
› Seminar 
(Task 5) 
› Econometric 
analysis (Task 
6) 

 
 
 
 
 
EFFECTIVENESS 
Effectiveness assesses the extent to which selected policy instruments have successfully achieved 
or  progressed  towards  the  stated  objectives  and  delivered  the  expected  outputs,  outcomes,  and 
impacts. Effectiveness assessed the extent to which: 

 

 

selected policy instruments have been successful, also in combination with other EU and 
national  support  for  RTD,  in  achieving  or  progressing  towards  the  stated  objectives  and 
delivered the expected results; 
the ERDF policy mix for RTD has been effective in improving RTD performances of funded 
regions. 

The criterion also analysed the main factors influencing the effectiveness of RTD interventions. 
Questions/sub-
questions 

Source 
evidence 

Conclusions 

Level  of 
analysis  

of 

Is 

EQ  9: 
specific 
associated 
certain 
interventions? 

there  a 
impact 
with 
of 

types 

did 

EQ  10:  To  what 
extent 
the 
support 
generate 
additional  innovation 
in 
or  output 
the 
entities 
supported 
and 
and 
development 
in  the 
regions? 

growth 

EQ  11:  Which  were 
underlying 
the 
impacts 
for 
factors 
generation? 

Overall assessment 
on EFFECTIVENESS 

to 

level 

from  an 

›  PI 
(21 PIs) 

the  magnitude  of 

remained  unfeasibly  distant 

level 
›  OP 
and  MS  (53 
OPs  and  18 
MS) 

› Mapping of 
projects and 
beneficiaries 
(Task 1) 
› Case studies 
(Task 3) 
›  Cross-case 
studies 
analysis  (Task 
4) 
› Seminar 
(Task 5) 
›  Econometric 
analysis  (Task 
6) 

collaborations  (see  Section  4.4  and  Italy,  Romania  and  Poland 
Case Study). 
› Infrastructure investments and individual projects contributed to 
an increase in the number of R&D personnel and researchers at 
the  regional  level;  infrastructure  investments  for  education  were 
also  key  to  increasing  the  number  of  students  and  tertiary 
attainments. Those infrastructure investments targeting HEIs also 
contributed  to  an  increase  in  scientific  outputs.  Specifically, 
ERDF  infrastructure  investments  significantly  contributed  to  the 
catching-up  process  of  the  EU13  regions  in  terms  of  scientific 
the 
output.  This  was  partially  due 
investments  in  HEIs  and  certain  other  factors  (national  public 
expenditures  in  particular).  Lagging  regions  performed  better 
than  more  developed  regions  in  terms  of  growth  in  publications 
(see Section 4.3). 
›  Collaborative  R&D  projects  contributed  to  the  increase  of 
scientific and technological knowledge and competencies among 
beneficiaries.  However,  even  when  results  were  produced,  they 
generally 
industrial 
application (see Section 4.4). 
› Overall, while ERDF support played a role in the modernisation 
of  RTD  systems  by  supporting  the  creation  of  more  advanced 
and  competitive  research  systems,  it  was  less  effective  in 
translating this increased research capacity into more competitive 
territories  and 
regional  economies.  The  cluster  analysis 
highlighted  that  a  decade  after  the  launch  of  the  2007-2013 
programming  period,  half  of supported  regions  had  not changed 
their relative RTD performance. Performance improvements were 
concentrated in stronger regions, but some transition regions also 
saw  a  catch-up  dynamic.  In  those  regions  where  the  economic 
crisis  hit  more  profoundly,  the  ERDF  support  for  RTD  did  not 
result  in  a  leap  in  RTD  performance;  however,  it  played  a 
countercyclical  role,  representing  a 
for  many 
beneficiaries (see Section 7.3).  
›  A  number  of  contextual  factors  played  a  crucial  role  (as  pre-
conditions, supporting factors or risks) in explaining what worked 
and  how.  Long-lasting  strategic  and  financial  commitment  to 
investment  priorities,  both  for  private  and  public  organisations, 
was key as it allowed for follow-up projects to take place. Clarity 
about  the  ‘rules  of  the  game’,  shared  within  the  common  RTD 
space  by  science  and  industry  partners,  was  decisive  for 
successful 
implementation. 
and 
for 
Administrative  and  managerial  capacities  were  crucial 
effective  public  spending  as  they  ensured  the  high  scientific 
quality of selected projects and their timely selection and funding 
(see Section 5). 
ERDF support for RTD effectively contributed to the consolidating and modernisation of existing 
RTD  systems,  also  favouring  a  catch-up  process  of  EU13  countries  on  EU  RTD  standards.  It 
was  less  effective  in  transforming  the  knowledge  base  of  regional  economic  systems  and 
ultimately improving target territories' long-term competitiveness. In less developed regions, the 
consequences of the economic crisis were more severe, and ERDF support has certainly played 
a  countercyclical  role.  Regarding  the  effectiveness  of  specific  policy  instruments,  while 
infrastructure  investments  and  individual  R&D  projects  generally  matched  their  intended 
objectives  and  intermediate  results,  collaborative  projects  were  not  always  effective  in 
consolidating  the  role  of  industrial  partners  in  the  RTD  system  and  their  uptake  of  research 
results.  In  this  regard,  the  lack  of  continued  public  funding  and  administrative  and  managerial 
capacities issues have had a negative impact on effectiveness. 

› Cluster 
analysis (Task 
1) 
› Case studies 
(Task 3) 
›  Cross-case 
studies 
analysis  (Task 
4) 
› Seminar 
(Task 5) 

› Case studies 
(Task 3) 
›  Cross-case 
studies 
analysis  (Task 
4) 
› Seminar 
(Task 5) 

›  OP 
level 
and  MS  (9 
OPs  and  7 
MS) 

‘safety  belt’ 

partnerships 

effective 

63 

 
 
 
 
 
 
COHERENCE 
Coherence was assessed from three perspectives:  

 

 

 

internal  coherence, which  focused on  assessing whether  different  interventions  under  the 
same ERDF OP or across different ERDF OPs within the same region/country were coherent 
and complementary; 
external  coherence,  which  focused  on  determining  whether  ERDF  support  to  RTD 
infrastructures  and  activities  were  coherent  and  complementary  with  respect  to  other  EU 
and regional/national policies (including the EU Research Framework Programmes); 
influence of EU State Aid rules on the choice of interventions under ERDF. 

of 

Source 
evidence 
› Documentary and 
literature review 
(Task 2) 
› Case studies (Task 
3) 

of 

Level 
analysis  
level 
›  OP 
and  MS 
(9 
OPs  and  7 
MS) 

›  OP 
level 
and  MS  (53 
OPs  and  18 
MS) 

› OP analysis (Task 
1) 
› Documentary and 
literature review 
(Task 2) 
› Case studies (Task 
3) 

level 
›  OP 
(9 
and  MS 
OPs  and  7 
MS) 

› Documentary and 
literature review 
(Task 2) 
› Case studies (Task 
3) 
› Cross-case studies 
analysis (Task 4) 

Sub-questions 

Conclusions 

EQ  12:  Did  EU  State 
influence 
Aid  rules 
the 
of 
choice 
interventions? 

EQ  13: What was  the 
the  policy 
role  of 
mix's  links  with  the 
Research  Framework 
Programmes 
(FP7 
and Horizon2020)?  

EQ  14:  What  is  the 
mix  of  RTD  policy 
that  MS 
measures 
in  the 
implemented 
period 
(including 
regulatory  incentives 
or  national  schemes 
if  they  play  a  role  in 
the 
programmes)? 
What  is  the  role  of 
ERDF  in  this  policy 
mix? 

result 

findings 

that  would 

›  The  influence  of  State  Aid  was  more  evident  in  the 
implementation  of  policy  instruments  rather  than  in  their 
design.  Managing  Authorities  adapted  their  instruments  to 
avoid  any  potential  conflicts  with,  or  infringement  of,  such 
rules, for instance, by limiting the involvement of the private 
sector  or  by  selecting  projects  that  could  not  produce 
in  a  competitive 
factual 
advantage 
the 
limited 
for  certain  companies.  This 
possibility  of  involving  the  private  sector  either  as  a  direct 
beneficiary  or  as  users  of  funded  infrastructures,  with 
negative  impacts  on  the  uptake  of  research  results  and 
(see 
strengthening  of  science-industry  partnerships 
Section 5.2.2). 
›  The  need  to  ensure  coherence  with  State  aid  rules 
proved to be a challenge in some countries because of the 
unclear  interpretation  and  changes  in  legislation  over  the 
period. The limited degree of alignment and even conflicts 
between competition and cohesion policy was the weakest 
aspect of coherence in the 2007-2013 programming period 
(see Section 5.2.2). 
› Despite a very high level of coherence in terms of overall 
policy  goals,  the  level  of  synergy  between  ERDF  RTD 
support  for  RTD  investments  and  the European  Research 
and  Innovation  framework  programme  was  found  to  be 
limited.  The  two  funds  were  conceived  as  being  highly 
complementary,  but  they  followed  different  rationales  and 
operational  arrangements 
that  somehow  hampered  a 
systematic and intended combination of funds (see Section 
5.1.4). 
›  Despite  there  was  no  active  strategy  for  combining  the 
two  sources  of  funds,  matching  data  from  the  ERDF 
beneficiaries  mapped  by 
this  study  with  FP7/H2020 
beneficiaries  from  the  CORDIS  database  shows  that  the 
proportion  of  ERDF  beneficiaries  that  also  benefited  from 
FP7/H2020,  at 
in  some  countries,  was  rather 
significant (see Section 5.1.4). 
›  ERDF  support  for  RTD  was  generally  coherent  with 
industrial 
national 
competitiveness  strategies.  Strategic  coherence  in  priority 
sectors  and  scientific  fields  was  particularly  strong,  but 
there  was  a  lack  of  long-term  commitment  regarding 
specific  RTD  strategies  and  the  related  political  stability 
and  predictability  of  national  policies.  In  more  operational 
terms, there was a general tacit division of goals between 
local and ERDF policies and instruments, with coordination 
mainly driven by co-financing obligations and a great effort 
towards avoiding overlapping (see Section 5.1.2). 
›  There  was  a  significant  degree  of  coordination  across 
interventions  carried  out  in  the  framework  of  ERDF.  This 
applied 
to  coordination  across  different  ERDF  OPs 
(national and/or regional) and the different axes, measures, 
and 
framework  of 
individual OPs (see Section 5.1.3). 
›  There  are  also  positive  examples  of  the  combination  of 
ERDF and ESF funding. Synergies were particularly strong 
in  the  regions  where  the  ERDF  emphasised  infrastructure 
investment in tertiary education, as in Poland, Estonia and 
Slovakia (see Section 5.1.5). 

implemented 

both  RTD 

instruments 

strategies, 

least 

and 

the 

in 

64 

 
 
 
 
COHERENCE 
Coherence was assessed from three perspectives:  

 

 

 

internal  coherence, which  focused on  assessing whether  different  interventions  under  the 
same ERDF OP or across different ERDF OPs within the same region/country were coherent 
and complementary; 
external  coherence,  which  focused  on  determining  whether  ERDF  support  to  RTD 
infrastructures  and  activities  were  coherent  and  complementary  with  respect  to  other  EU 
and regional/national policies (including the EU Research Framework Programmes); 
influence of EU State Aid rules on the choice of interventions under ERDF. 

Sub-questions 

Conclusions 

EQ  15:  To  what 
extent 
are 
interventions 
to 
organised 
their 
maximise 
effects, 
combined 
considering 
the 
different  underlying 
goals? 

Overall assessment 
on COHERENCE 

of 

Source 
evidence 
› Documentary and 
literature review 
(Task 2) 
› Case studies (Task 
3) 

of 

Level 
analysis  
level 
›  OP 
and  MS 
(9 
OPs  and  7 
MS) 

targeted 

Interventions 

funded  with 

different  EU 

and 
› 
national/regional funds were mainly implemented in light of 
a  separation  of  objectives  and  approaches,  with  more 
attention  to  avoiding  overlapping  than  building  on  relative 
strengths and maximising the joint effects.  
›  While  instruments  supported  by  01  and  02  ERDF 
categories  of  expenditure  mainly 
research 
providers to improve their capacities, the other expenditure 
codes  primarily  reflected  the  targeting  of  SMEs,  with  the 
principal  aim  of  supporting  innovation  processes  (see 
Section 5.1.3). 
research  projects  was 
›  EU-level  cooperation  on 
undertaken  mainly  through  the  FP7/H2020  programme, 
while  ERDF  was  essentially  seen  as  an  instrument  for 
regional and national cooperation (see Section 5.1.4). 
›  RTD  programmes  supported  by  the  ERDF  were  often 
closely linked to objectives of industrial competitiveness, as 
indicated  by 
the  ERDF 
interventions  for  RTD  and  those  of  national  and  regional 
strategies  for  cluster  development,  business  innovation, 
support.  The  ERDF  was  also  instrumental  in  supporting 
national and regional strategies of economic conversion or 
transition from an industrial economy towards a diversified 
economy.  Despite  a  high  strategic  alignment,  however, 
there  was  often  an  implicit  division  of  goals  between  local 
and ERDF policies and instruments (see Section 5.1.2).  
ERDF  support  for  RTD  was  strongly  in  line  with  other  support  policies,  such  as  Framework 
Programmes and other ERDF and national/regional support. Therefore, the coherence of policy 
strategic objectives was relatively high with the FP7, ERDF support for business innovation, ESF 
support,  and  national  RTD  support.  However,  synergies  and  coordination  between  ERDF 
support for RTD and other types of RTD funding were not always ensured in practice. No active 
strategy for combining  different sources  of  funds  was  generally implemented,  except  for  ERDF 
and  ESF  OPs.  Significant  challenges  were  reported  in  terms  of  coherence  between  cohesion 
and  competition  policies.  Uncertainties  in  the  rules  or  risk-aversion  interpretation  of  the  legal 
framework limited the involvement of private businesses in implemented projects and hampered 
their capacity to use the services provided by funded research infrastructure.  

links  between 

the  strict 

Conclusions 

EFFICIENCY 
Efficiency did not address the wider aspect of value for money considerations but concentrated on 
two aspects.  Firstly,  it  assessed the scale  of  funding and  the  use  of  financial  resources  under  the 
angle of the concentration of ERDF funds to ensure critical mass. Second, it also explored the issue 
of administrative capacity and speed in funds absorption in selected case studies.  
Questions/sub-
questions 
EQ 16.1: Was the 
funding sufficiently 
concentrated on 
making a perceptible 
difference to 
pursuing policy 
objectives (including 
when combined with 
other instruments or 
sources of support)? 

› There was a concentration pattern on stronger territories, sectors 
and leading institutions (see Section 5.2.1). 
›  ERDF  support  for  RTD  was  overall  sufficiently  concentrated  to 
lead  to  upgrades  in  both  the  quality  of  research  infrastructure  and 
research  management  capacities  in  most  of  the  countries  under 
investigation. Its role as “game-changer” or “needle mover” in terms 
of  RTD  performance  in  beneficiary  countries  and  regions  was 
strongly  related  to  the  importance  of  ERDF  in  the  overall  national 
and regional RTD policy mix. In cases where ERDF represented a 
limited  share  of  the  total  national  R&D  spending,  but  where  there 
was  a  high  regional  concentration  of  this  spending  (e.g.,  in 
convergence regions), its role was crucial to develop critical mass in 
specific  areas,  sectors  and  types  of  beneficiary  organisations  (see 
Section 5.2.1 and the Czechia, Romania, Poland, Italy Case Study). 

Source of 
evidence 
› Case studies 
(Task 3) 
›  Cross-case 
studies 
analysis  (Task 
4) 

Level  of 
analysis  
› OP level 
and  MS 
(9  OPs 
and 
7 
MS) 

65 

 
 
 
 
 
 
EQ 16.2: Were there 
sources of 
inefficiencies in the 
way funds were 
managed and 
disbursed? 

Overall assessment 
on EFFICIENCY 

to 

related 

implementation 

issues,  mainly 

Some 
limited 
administrative  capacity  or  unclear  legal  framework,  were  reported 
especially  for  collaborative  R&D.  Delays  in  project  selection  and 
funds  disbursements,  especially  in  Italy  and  Romania,  which 
impacted  the  capacity  of  funded  projects  to  generate  benefits. 
Uncertainties 
interpretation  and  application  of  rules, 
especially  regarding  State-aid  rules,  caused  delays  and  generated 
confusion and adjustments during the implementation process. 
ERDF support for RTD helped reach critical mass in most of the target territories, especially in 
convergence  regions  where  it  represented  the  main  funding  source  of  RTD  investments.  It 
should also be highlighted that some inefficiencies in the implementation of ERDF interventions 
impacted their effectiveness.  

› Case studies 
(Task 3) 
›  Cross-case 
studies 
analysis  (Task 
4) 

› OP level 
PI 
and 
level 
(9 
OPs  and 
21PIs) 

the 

in 

EU ADDED VALUE 
The EU added value criterion analysed the beneficial impacts attributed to EU intervention, over and 
above what could reasonably have been expected and achieved from the action of Member States at 
the national and/or regional level. Moreover, it provided evidence on the potential EU- wide effects of 
ERDF interventions in the field of RTD.  
Sub-questions 

Source of evidence  Level  of 

Conclusions 

18: 

What 
EQ 
additional 
value 
results  from  the  EU 
intervention 
compared 
to  what 
could  have  been 
achieved  by  MS  at  a 
national, 
regional 
and local level? 

EQ  19: What was  the 
the 
impact 
of 
interventions 
on 
cooperation  between 
regions  and  Member 
States within the EU? 
EQ 
the 
20:  Did 
interventions achieve 
any  other  EU-wide 
effects? 

Overall assessment 
on EU ADDED 
VALUE 

analysis  
›  OP  level 
and  MS 
(53  OPs 
18 
and 
MS) 

international 

interregional  or 

›  Case  studies  (Task 
3) 
› Seminar (Task 5) 

› OP analysis (Task 1)  
›  Case  studies  (Task 
3) 
› Seminar (Task 5) 

›  The  main  EU  added  value  recognised  by  MAs  was  a 
scale  effect  produced  by  the  access  of  a  considerable 
quantity  of  financial  resources,  especially  in  the  EU13, 
where ERDF 2007-2013 programmes represented the first 
systematic  set  of  interventions  addressed  to  the  research 
field  after  years  of  underinvestment  and  limited  political 
priority (see Section 7.1). 
› Another aspect highlighting the EU added value effect of 
ERDF  support  lies  in  the  ambition  of  supporting  research 
infrastructures  of  EU  scientific  and  research  relevance 
operating at EU level standards (see Section 7.2). 
 ›  A  missed  opportunity  was  the  lack  of  the  systematic 
promotion  of 
research 
collaborations as a potential EU added value. Partnerships 
of  collaborative  R&D  projects  were  mainly  regional  or, 
albeit only in selected cases, multi-regional within the same 
country (see Section 7.2). 
›  EU-wide  effects  were  not  among  the  directly  intended 
effects  of  funded  instruments.  Thus,  the  contribution  of 
ERDF  support  to  them  was  more  indirect,  and  it  occurred 
through 
research 
communities in specific fields, enabling the construction or 
upgrading  of  strategic  infrastructures  of  pan-European 
relevance  (as  the  later  inclusion  into  the  ESFRI  roadmap 
confirms)  and  also  supporting  the  internationalisation  of 
research communities (see Section 7.2). 
ERDF support for RTD produced a scale effect that would not have been achieved by national 
sources  alone,  especially 
from 
underinvestment  in  the  field  of  RTD  policy  (all  EU13  countries  and  convergence  regions). 
Despite not specifically intended at producing EU-wide effects, ERDF support for RTD indirectly 
contributed  to  the  development  of  EU-level  research  communities.  It  helped  structure  and 
consolidate  a  European  Research  Area  by  promoting  EU  standards  in  RTD  capacities  and 
production. This can be claimed to have been the main EU added value of the ERDF support to 
RTD investments in the period 2007-2013.  

›  Case  studies  (Task 
3) 
› Seminar (Task 5) 

›  OP  level 
and  MS 
(53  OPs 
18 
and 
MS) 

›  OP  level 
and  MS 
(53  OPs 
and 
18 
MS) 

regions  generally  suffering 

those  countries  and 

of  EU-level 

development 

the 

in 

ANNEX IV. STAKEHOLDERS CONSULTATION - SYNOPSIS REPORT  

In  compliance  with  the  Better  Regulation  guidelines,  this  Synopsis  Report  outlines  the 
delivery and results of the consultation activities organised in the context of the ex-post 
in  Research  and  Technological  Development  (RTD) 
evaluation  of  investments 

66 

 
 
 
 
 
infrastructures  and  activities  supported  by  the  European  Regional  Development  Funds 
(ERDF) in the period 2007-2013. 

The  first  section  of  this  Synopsis  Report  outlines  the  objectives,  tools,  and  stakeholder 
participation. The second section analyses the delivery and results by type of consultation 
activity: roadmap consultation, interviews, and seminar. 

1.  Strategy 

Objectives 

The general objectives of the consultation activities were to: 

  Disseminate the activity and stimulate stakeholder participation in the evaluation 

process; 

  Ensure  transparency  of  the  process;  enhance  accountability  and  transparency  of 

the European Commission activities; 

  Stimulate  stakeholders  to  respond  to  the  enquiries  and  the  other  knowledge 

gathering activities to fill information gaps; 

  Gather  general  public  end  expert  stakeholders  opinions  on  the  principal 

evaluation findings; and to 

  Publicise the main evaluation findings. 

Tools 

The  stakeholders  targeted  by  the  consultation  activities  were  first  mapped  by  and  then 
matched to different consultation tools. In particular, three sets of activities with different 
purposes were organised: 

  The  evaluation  roadmap  was  published  by  the  EC  in  order  to  gather  feedback 

from the general public; 

  Semi-structured  and  in-depth  interviews  were  implemented  by  the  contractor  in 

order to gather OP-level evidence; and 

  A  seminar,  divided  into  an  expert  workshop  and  a  validation  webinar,  was 

organised by the contractor in order to discuss. 

Stakeholder participation 

A summary of the final stakeholder participation figures is provided in the table below. 

Table 2 – Stakeholder reach by activity 

Activity 

Key numbers 

Roadmap consultation 

1 respondent, Wallonia (Belgium) government 

Interviews 

103  stakeholders  (mainly  Managing  Authorities  and  Intermediate 
Bodies) – Semi-structured interviews 

67 

 
200  stakeholders  (43  Managing  Authorities,  135  direct  and  final 
beneficiaries,  14  EU/national/regional  authorities  involved  in  the 
oversight  of ERDF funds and 8 other  relevant  stakeholders, such as 
business associations, etc.) – In-depth interviews 

65  participants,  including  EC  officers,  representatives  of  Managing 
Authorities and direct/final beneficiaries – In-depth expert workshop 

200  participants  from  a  broader  audience  of  interested  RTD  policy 
stakeholders  from  Member  States,  pan-European  institutions,  and 
European Commission officials – Validation workshop 

569  stakeholders  reached  (individual  stakeholders  might  have  taken 
part in more than one activity) 

Seminar 

TOT 

Source: Author   

2.  Delivery and results 

Roadmap consultation 

After  internal  consultation  of  EC  services,  the  evaluation  roadmap  was  published  for 
feedback  on  the  official  EC  website.50  The  roadmap  aimed  to  inform  citizens  and 
stakeholders  about  the  Commission's  work  to  allow  them  to  provide  feedback  and  to 
participate effectively in planned consultation activities.  

The  roadmap  was  open  for  public  feedback  over  the  period  08  November  2018  -  06 
December 2018. However, there only was one reply, coming from the Wallonia regional 
government of Belgium. 

Interviews 

Semi-structured interviews 

A total number of 103 interviews with the Managing Authorities were to collect further 
information  and  critically  discuss  the  OP  logic,  beyond  the  intentions  expressed  in  the 
programming documents. 

The interviewees were selected in order to insure a balanced representation of the 53 Ops 
included  in  the  analysis.  A  list  of  the  interviewees  is  provided  in  the  table  below, 
reporting only the OP code and name in order to preserve anonymity. 

Table 3 - List of semi-structured interviews conducted 

OP code and name 
2007BE161PO001 - 
Programme opérationnel 'Convergence' Hainaut - 
FEDER 
2007BE162PO003 - 

50  https://ec.europa.eu/info/law/better-regulation/have-your-say/initiatives/1983-R&D-infrastructure-&-

activities-investment-2007-13-evaluation_en 

68 

 
                                                           
Programme opérationnel 'Compétitivité régionale et emploi' - 
Wallonie (hors Hainaut ) - 
FEDER 
2007CZ161PO004 - 
OP Podnikání a inovace 
2007CZ161PO012 - 
OP Výzkum a vývoj pro inovace 
2007CZ162PO001 - 
OP Praha Konkurenceschopnost 
2007DE161PO001 - 
Operationelles Programm EFRE Thüringen 2007 bis 2013 
2007DE161PO002 - 
Operationelles Programm EFRE Brandenburg 2007-2013 
2007DE161PO003 - 
Operationelles Programm EFRE 2007 -2013 Mecklenburg- Vorpommern 
2007DE161PO004 - 
 Operationelles Programm EFRE Sachsen 2007-2013 
2007DE161PO007 - 
Operationelles Programm EFRE 2007-2013 in Sachsen-Anhalt 
2007DE162PO001 - 
Operationelles Programm EFRE Bayern 2007 - 
2013 
2007DE162PO003 - 
Operationelles Programm EFRE Schleswig-Holstein 2007-2013 
2007DE162PO004 - 
Operationelles Programm EFRE Berlin 2007-2013 
2007DE162PO007 - 
Operationelles Programm EFRE Nordrhein-Westfalen 2007-2013 
2007EE161PO001 
Operational Programme 'Development of Economic Environment' 
2007ES16UPO001 - 
Programa Operativo FEDER de Investigación, Desarrollo e innovación por y para el beneficio de las 
Empresas - Fondo Tecnológico 
2007ES16UPO003 - 
Programa Operativo FEDER de Economía basada en el Conocimiento 
2007ES162PO002 –  
Pais Vasco – PO FEDER Pais Vasco 
2007ES162PO004 – 
Programa Operativo FEDER de Madrid 
2007ES162PO004 - 
Programa Operativo FEDER de Madrid 
2007ES162PO005 - 
Programa Operativo FEDER de La Rioja 
2007ES162PO006 - 
Programa Operativo FEDER de Cataluña 
2007FI162PO001 - 
Itä-Suomen EAKR-toimenpideohjelma 2007-2013 
2007ES162PO010 - 
PO FEDER Comunitat Valenciana 
2007FR162PO001 - 
Programme opérationnel FEDER AQUITAINE 
2007FR162PO011 - 
Programme opérationnel FEDER HAUTE-NORMANDIE 
2007FR162PO015 - 
Programme opérationnel FEDER LORRAINE 
2007FR162PO016 - 
Programme opérationnel FEDER PAYS DE LA LOIRE 
2007FR162PO017 - 
Programme opérationnel FEDER NORD PAS-DE-CALAIS 

69 

 
2007FR162PO020 - 
Programme opérationnel FEDER PROVENCE ALPES COTE D´AZUR 
2007FR162PO021 - 
Programme opérationnel FEDER MIDI-PYRENEES 
2007HU161PO001-  
Gazdaságfejlesztési Operatív Program (GOP) 2007-2013                                 
2007IE162PO002 - 
Southern and Eastern Operational Programme 
2007IT162PO002 -  
Por Emilia Romagna FESR 2007-2013 
2007IT161PO006 - 
PON Ricerca e Competitività 
2007LT161PO002 - 
2007-2013 m. Ekonomikos augimo veiksmų programa 
2007LV161PO001 - 
Operational Programme Entrepreneurship and Innovations“ 
2007PL161PO001 - 
Program Operacyjny Innowacyjna Gospodarka, 2007-2013 
2007PL161PO002 - 
Program Operacyjny Infrastruktura i Środowisko 
2007PL161PO003 - 
Program Operacyjny Rozwój Polski Wschodniej 2007-2013 
2007PL161PO010 - 
Małopolski Regionalny Program Operacyjny na lata 2007-2013 
2007PL161PO011 - 
Regionalny Program Operacyjny Województwa Mazowieckiego 
2007PL161PO013 - 
Regionaln Program Operacyjny Województwa Podkarpackiego 
2007PT161PO001 - 
Programa Operacional Fatores de Competitividade (COMPETE) 2007-2013 
2007PT161PO002 - 
PO Regional do Norte 2007-2013 
2007PT161PO003 - 
PO Regional do Centro 2007-2013 
2007RO161PO002 - 
Sectoral Operational Programme Increase of Economic Competitiveness 
2007SI161PO001 - 
Operativni program krepitve regionalnih razvojnih potencialov za obdobje 2007 - 
2013 
2007SK16UPO001 - 
Operačný program Výskum a vývoj 
2007UK161PO002 - 
West Wales and the Valleys ERDF Convergence programme 
2007UK161PO003 - 
Cornwall and the Isles of Scilly ERDF Convergence programme 
2007UK162PO001 - 
Lowlands and Uplands of Scotland ERDF Regional Competitiveness and Employment programme 
2007UK162PO009 - 
Yorkshire and the Humber Regional Competitiveness and Employment Programme 
2007UK162PO008 - 
North West England ERDF Regional Competitiveness and Employment Operational Programme 

Source: Author   

In-depth interviews 

The following table presents the types of stakeholders we contacted and interviewed for 
each of the selected case studies. 

Table 4 - Stakeholders interviewed as part of the case studies 

70 

 
European/ national/ 
regional authorities 
involved in the 
oversight of ERDF 
funds 

Managing 
authorities 

Direct and 
final 
beneficiaries 

Total 
interviewees 

Other relevant 
stakeholders 
(e.g. business 
associations, 
other experts) 

2 

5 

6 

9 

2 

8 

11 

43 

18 

21 

23 

6 

24 

19 

24 

135 

/ 

5 

/ 

/ 

2 

1 

/ 

8 

20 

35 

31 

15 

32 

31 

36 

200 

Poland 

Czech 
Republic 

Romania 

Estonia 

Germany  

Italy 

Portugal 

/ 

4 

2 

/ 

4 

3 

1 

Total 

14 

Seminar 
In-depth expert workshop 

The  first  of  two  seminar  days  took  the  form  of  an  in-depth  expert  workshop,  which 
virtually hosted 65 participants – ranging from representatives of Managing Authorities, 
R&D and evaluation experts, as well as European Commission officials. The workshop 
took  place  on  May  4th,  2021  from  09.30  AM  to  1.30  PM  and  was  virtually  hosted  on 
Microsoft Teams. 

The  purpose  of  the  workshop  was  to  discuss,  deepen  and  validate  the  emerging 
findings  from  the  study.  It  also  served  as  an  opportunity  to  reflect  upon  possible 
recommendations  and  improvements  of  future  RTD  support  under  ERDF  in  the 
upcoming programming period. 

To  begin,  the  participants  were  invited  to  share  what  type  of  RTD projects  or  policy 
instruments were supported by the ERDF (2007 – 2013) in their region. In the form of 
a WordCloud, the responses can be observed in Figure 1. The text size of the response 
indicates the share of participants that provided that answer (i.e., the larger the size, the 
more participants provided that response).    

Figure 3 – Responses of participants: What type of RTD projects or policy instruments were supported by ERDF (2007 – 
2013) in your region?   

71 

 
 
 
Source: Author   

The  first  presentation  –  Mapping  the  funding  of  RTD  infrastructure  and  activities 
under  the  ERDF  2007-2013:  Taking  stock  of  funded  projects  and  beneficiaries 
(CSIL)  –  provided  an  overview  of  the  objects  of  inquiry  of  this  ex-post  evaluation,  by 
comprehensively  mapping  and  describing  the  funded  RTD  infrastructure  and  activities 
under the 2007 – 2013 ERDF (key findings of Task 1 of the project). As can be observed 
in Figure 2 below, the presentation highlighted the types of RTD projects funded, which 
ranged from individual and collaborative R&D projects, to infrastructure investments for 
research or education, to other types of projects, such as internationalisation of research 
and capacity building for research.  

Figure 4 – Types of RTD projects funded 

72 

 
 
Source: Author   

After  taking  stock  of  the  funded  projects  and  beneficiaries,  the  second  presentation  – 
RTD  support  from  ERDF  and  policy  mix  in  selected  case  studies  (Technopolis)  – 
outlined  the  case  studies  pursued,  which  covered  21  policy  instruments  across  seven 
Member  States.  The  presentation  discussed  the  key  insights  from  Task  3  of  this 
evaluation by highlighting the performance of ERDF support for RTD activities based on 
the  relevant  evaluation  criteria.  A  graphic  overview  of  the  findings  can  be  observed  in 
Figure 3 below. 

Figure 5 – Overall performance of ERDF support for RTD activities for the 2007 – 2013 period 

73 

7024312241751501381862593,0591,7793,581N/A4. Others - Non-RTD3.5. Operating subsidy3.4. Intellectual Property Protection instruments3.3. Science dissemination to the general public3.2. Capacity building for research3.1. Internationalisation of research2.3. ICT-based infrastructure2.2. Infrastructure investments for education2.1. Infrastructure investments for research1.2. Collaborative R&D projects1.1. Individual R&D projectsNumber of projects by type of intervention27.85M€332.66M€5.86M€16.44M€103.16M€161.75M€61.94M€472.75M€1,477.87M€7,387.80M€2,026.43M€892.75M€N/A4. Others - Non-RTD3.5. Operating subsidy3.4. Intellectual Property Protection instruments3.3. Science dissemination to the general public3.2. Capacity building for research3.1. Internationalisation of research2.3. ICT-based infrastructure2.2. Infrastructure investments for education2.1. Infrastructure investments for research1.2. Collaborative R&D projects1.1. Individual R&D projectsERDF contribution by type of intervention (Million €) 
 
Source: Author   

A  plenary  discussion  began  with  an  initial  polling  of  participants’  views  regarding 
significant  achievements  from  RTD  support  under  ERDF.  The  achievements  under 
inquiry, as well as the results of the poll, can be observed in the figures below.   

Figure 6 – Rating of achievements from RTD support under ERDF (2007-2013) 

74 

Criterion / Case study Relevance Coherence Effectiveness Efficiency EU added value Internal External Poland       Czech Republic        Romania       Estonia       Germany        Italy       Portugal         
 
 
 
Source: Author   

Upon  the  backdrop  of  the  above  polling,  RTD  contribution  stories,  which  entailed 
detailed  accounts  of  the  functioning  of  RTD  policy,  were  shared  by  participants  of  the 
expert  workshop.  This  allowed  for  participants  to  explain  in  more  depth  how  the  RTD 
funding  functioned  in  their  settings,  highlighting  both  the  achievements  gained  and 
challenges  encountered.  The  reflections  were  documented  live  using  the  online 
documentation tool, MIRO, an example of which can be observed in Figure 5 below.  

Figure 7 – Documentation of plenary discussion: RTD Contribution Stories 

Source: Author   

75 

 
 
 
The  insights  gained  from  this  exercise,  which  reflect  stakeholders’  perception  about 
supported RTD efforts in three Member States, are displayed below.  

Figure 8 - RTD Contribution Stories: Some insights from participants 

A third presentation – Preparation for the group discussion: The four types of policy 
interventions  and  the  respective  theories  of  change  in  Czechia,  Estonia,  Germany, 
Italy,  Poland,  Portugal,  and  Romania  (Prognos  AG)  –  followed,  which  outlined  the 
four  overarching  RTD  policy  interventions  under  inquiry  and  described  the  associated 
generalised theories of change. Moreover, the presentation highlighted certain important 
contextual factors (pre-conditions, supporting factors, risks) that either contributed to the 
effective  functioning  of  the  RTD  policy  intervention  or  limited  its  effectiveness.  These 
factors were further deepened upon and discussed in the group discussions that followed.  

Group  discussions  on  important  contextual  factors  for  individual  and  collaborative 
R&D projects as well as for R&D and ICT-based infrastructure investments were held in 
virtual  breakout  rooms.  Using  MIRO,  some  key  learnings  were  shared  by  the 
stakeholders and concurrently documented, which can be viewed below. 

Figure 9 - Documentation of group discussion: important contextual factors 

76 

 
 
Source: Author   

Some findings of the respective group discussions in regard to salient contextual factors 
for the implementation of RTD policy were as follows.  

Group 1: Collaborative & Individual R&D Projects 

  Quality  of  project  selection  procedure: 

in  Portugal,  National  Science 
Foundation  helped  identify  RTD  actors  most  capable  to  pursue  rigorous  R&D. 
This  ensured  that  highly  skilled  actors  are  involved,  with  better  chances  for 
internationalisation efforts and further acquisition of funding (and: higher impacts 
compared to other selection procedures). 

 

Important  to  create  permanent  collaboration  platforms  for  exchange  between 
industry  and  science  actors,  in  order  to  foster  sustainable  relationships  (e.g., 
competence centres or clusters) 

77 

 
 
 
  Capacity building:  develop  more of an awareness  for industry  needs.  Not only 
about  technology  development,  but  capacities  to  foster  knowledge  transfer. 
Currently  more  a  “science  push”  approach.  Suggestions 
for  better 
commercialisation pathways: 

o  Organise calls around challenges expressed by companies and use micro-

grants for testing 

o  Include  representative  of  a  Technology  Transfer  Office  in  each  research 

team from the start of the R&D process  

Group 2: RTD & ICT Infrastructure Projects 

  Road-mapping process before project call & selection helps ensure that projects 

matching strategic priorities get funds and reduces complexity in selection.  

  Top-down  vs  bottom-up  prioritisation:  mixed  approach  observed  in  multiple 

countries – indication of priorities is followed by proposals. 

o  In Estonia: small-scale investments bottom-up, big investments top-down 

 good compromise 

  Mission-oriented infrastructure is often public-private: potential conflict with 

state aid rules. The more application oriented, the greater the conflict.  

o  Uncertainty hampers business engagement in R&D.  

  Qualified human resources to operate new ICT infrastructure? Yes, relevant 

skills are available, but they need a sufficient salary.  

  Without  proper  IT  experts  running  ICT  systems,  infrastructure  yields  limited 

results.  

After a review of the key findings of the morning, the in-depth workshop finished with 
closing  words  by  the  European  Commission,  referring  to  the  succeeding  and 
complementary validation webinar that was to take place on May 7th, 2021.  

Validation webinar 

The  second  event  in  the  seminar  series  took  the  form  of  a  validation  webinar,  which 
was intended for a broader audience of interested RTD policy stakeholders from Member 
States,  pan-European  institutions,  and  European  Commission  officials.  The  workshop 
took place on May 7th, 2021 from 09.30 AM to 12.30 PM and was virtually hosted on 
Microsoft Teams. 

In total, more than 200 participants took part in the validation webinar. The purpose of 
the webinar was to present, reflect and validate the emerging findings of the ex-post 
evaluation.  It  was  also  used 
to  reflect  upon  possible  recommendations  and 
improvements of future RTD support under ERDF. While the in-depth expert workshop 
(see above) provided an opportunity to assess in detail the operation of ERDF supported 

78 

 
 
 
projects  during  the  2007  –  2013  funding  period,  the  validation  webinar  sought  to 
encourage more high-level strategic discussions on RTD policy. 

The  participants  were  welcomed  to  the  validation  webinar  by  Nicola  De  Michelis 
(former  Director  DG  REGIO),  who  set  the  scene  and  objective  of  the  morning.  The 
participants  were  then  invited  to  share  their  thoughts  on  what  strategic  objectives 
future  RTD  policy  under  ERDF  should  address.  In  the  form  of  a  WordCloud,  the 
responses can be observed in Figure 9. The text size of the response indicates the share of 
participants  that  provided  that  answer  (i.e.,  the  larger  the  size,  the  more  participants 
provided that response).    

Figure 10 – What strategic objectives should future RTD policy under ERDF address? 

Source: Author   

Upon this basis, the first presentation – Big Data - Mapping 2007-2013 ERDF support 
to RTD infrastructure and activities: Beneficiary and project dashboard (CSIL and 
DG REGIO) – offered a detailed mapping of the RTD infrastructure and activities under 
inquiry  in  the  ex-post  evaluation.  This  was  complemented  by  a  presentation  of  the 
ERDF  R&I  Database  on  the  ESIF  Open  Data  Portal,  which  is  maintained  by  DG 
Regio and gives access to data on financing and achievements of ESIF funds.  

The second presentation  –  Success  Stories and  Challenges in ERDF  RTD support  – 
experiences  from  seven  EU  MS  (Technopolis)  –  outlined  in  some  more  depth  the 
empirical basis and methodological approach for the ex-post evaluation and highlighted 
key  findings  from  the  seven  Member  State  case  studies  pursued,  according  to  the 
pertinent  evaluation  criteria  (relevance;  coherence;  effectiveness;  efficiency;  EU-added 
value).  The  presentation  engendered  some  lively  discussion  via  the  chat  function.  For 
instance, one participant pointed to the fact that for some Member States this was the first 
programming period and considered to what extent this may have had an influence on the 

79 

 
 
 
choice  of  R&D  investments.  Another  set  of  comments  underscored  the  concept  of 
coherence  and  the  relationship  between  ERDF  support  for  RTD  and  other  EU 
Framework  Programmes,  reflecting  on  the  objectives  that  each  funding  source  holds. 
Another participant reflected on the challenge of assessing effective interventions since 
the evaluation criteria remains at national levels in many cases, suggesting that an inter-
state  perspective  could  better  inform  the  evaluation  process.  Further  comments  and 
questions on how to improve the measurement of ERDF effectiveness followed, as well 
as reflections on the challenges of assessing long-term impacts in a contribution analysis 
approach.  Regarding  the  latter  point,  it  was  underscored  that  especially  infrastructure 
investments  are  to  be  regarded  as  long-term  investments.  As  such,  a  time-lag  is  to  be 
expected and improved RTD performance should be evaluated with this in mind.  

The  third  presentation  –  The  RTD  cookbook:  Success  factors  and  important 
considerations when designing RTD instruments (Prognos AG) – presented the RTD 
Cookbook, which serves  as a  decision-aid tool for policy practitioners  at  the start  of 
the upcoming programming period. Upon a comprehensive analysis of contextual factors 
(pre-conditions, supporting factors, risks), the RTD Cookbook identifies key ingredients 
(depicted  in  Figure  10)  that  can  help  avoid  common  pitfalls  in  the  design  and 
implementation of RTD policy.  

Figure 11 - RTD Cookbook: key ingredients for effective RTD policies 

Source: Author   

A  comprehensive  discussion  of  these  salient  factors  was  followed  by  live  interaction 
with  the  webinar  participants,  who  were  asked  to  provide  their  opinion  on  the 
significance of the identified contextual factors. The result of the polling is displayed 
in Figure 11 below.  

Figure 12 – Significance rating: contextual factors for effective RTD policies  

80 

 
 
 
Source: Author   

81 

 
 
 
 
 
 
It  is  important  to  once  again  underscore  that  the  webinar  was  particularly  enriched  by 
the  active  participation  of  the  participants  via  the  chat  function,  continuously 
offering insightful and important reflection questions or comments. Serving the purpose 
of a validation webinar, these helpful insights have been captured and will be considered 
and reflected upon when drafting the final report of this ex-post evaluation.  

The validation webinar concluded with a policy roundtable - “What works, why, and 
where in RTD policy?” – and a final discussion of the results. Facilitated by CSIL, the 
three  academic  experts  -  Prof.  Slavo  Radosevic,  Prof.  Dirk  Czarnitzki  and  Prof.  Elliot 
Stern – focused on the following key topics:  

  Main achievements of ERDF  

  Substitution role of ERDF during the economic crisis  

  Excellence vs. territorial cohesion  

  Coherence with other EU policies, in particular State Aid  

  Coherence with national policies  

  Administrative capacity and implementation failures  

  Methodological  considerations  about  impact  evaluation  of  ERDF  support  to 

RTD investments 

Some key insights from this discussion related to the tension between “local relevance” 
and “scientific excellence” of RTD support under ERDF, limited institutional capacities 
in  many regions,  and ongoing uncertainties of stakeholders  around “State Aid”. On the 
first point, it was underscored the crucial policy challenge to ensure the balance between 
R&D excellence and local relevance. RTD policy under ERDF should prioritise locally 
relevant  but  internationally  excellent  R&D.  As  this  often  may  not  be  possible,  the 
priority seems to be given to the second-best area, to islands of excellence but locally not 
necessarily relevant research. Considering the increasing interconnectivity through digital 
technologies,  however,  may  also  allow  us  to  think  about  more  distributed  RTD 
infrastructures in the future. With regards to institutional capacities, it was stressed that 
both administrative capacities of public authorities and also the institutional capacities of 
the  various  stakeholders  in  the  innovation  system  (universities,  enterprises,  regional 
development authorities, cluster organisations) need to be considered and that sufficient 
institutional  capacities  are  a  major  pre-condition  for  effective  policies  –  especially  in 
complex  policy  domains  such  as  RTD.  Finally,  regarding  ongoing  uncertainties  around 
State Aid it was put  forward that  – especially in the 2007-2013 period – it was often a 
very  significant  factor  in  generating  incoherence  due  to  the  missing  link  between  State 
Aid rules and the selection of RTD instruments implemented in the framework of ERDF. 
In  particular  due  to  the  perceived  complexities,  policymakers  have  deterred  in  advance 
from  considering  the  issue  of  science-industry  collaboration  and  the  role  of  industry 
commons  or  joint  public-private  RDI  infrastructure.  Funding  rules  limit  private  sector 
and  prevent 
in  newly  developed 
organisations' 
commercialisation of knowledge stemming from ERDF-supported research projects that 

infrastructure 

involvement 

82 

 
defy  the  purpose  –  ultimate  application  of  knowledge  in  innovation  processes.  Thus, 
effectively  the  State  Aid  rules  hindered  the  creation  of  'industry  commons'  or  RTD 
activities that cross public-private knowledge domains.  

Regarding  methodological  considerations  it  was  emphasised  by  the  experts  that  the 
approach to this ex-post evaluation is quite novel in the field of RTD support, mainly due 
to  its  scale,  its  cross-case  analysis,  and  the  strong  emphasis  on  the  role  of  contextual 
factors.  At  the  same  time,  it  was  highlighted  that  further  methodological  advances  are 
needed,  incorporating  agent-based  modelling  approaches,  or  combining  a  system-
dynamics  approach  with  theory-based  evaluation  techniques.  This  is  particularly 
important  in  the  field  of  RTD  support,  as  the  unit  of  analysis  typically  changes 
throughout  the  evaluation  exercise,  starting  from  individual  operations  and  ending  in 
innovation ecosystems towards the end of the evaluation. A final thought put forward by 
the experts was on the future use of theory-based evaluations and the learnings from such 
evaluations.  Thus,  it  was  suggested  to  use  the  concept  of  theories  of  change  more 
frequently in the design phase of RTD policy instruments (i.e., in an ex-ante fashion) and 
to  build  on  the  learnings  from  such  types  of  evaluations  to  strengthen  our  “foresight” 
capacities. The latter could  be a  cornerstone in  our thinking  about  increasing  resilience 
and prepare us on upcoming crisis.     

The  webinar  was  closed  by  the  European  Commission,  who  reflected  on  the  insights 
gained from the webinar and offered some concluding remarks. 

ANNEX V. LIST OF REFERENCES 

Aghion, P. (2006). "A primer on innovation and growth," Policy Briefs 233, Bruegel. 

Arnold, E. (2004). “Evaluating Research and Innovation Policy: A Systems World Needs 
Systems 
3–17. 
https://doi.org/10.3152/147154404781776509. 

Evaluations”, 

Evaluation 

Research 

(1): 

13 

Arrow, K.J. (1962). “Economic Welfare and the Allocation of Resources for Innovation”, 
in  Nelson  R.R.  (ed.),  “The  Rate  and  Direction  of  Inventive  Activity”,  Princeton, 
Princeton University Press, pp. 609-662. 

Barnes,  T.A.,  Pashby  I.R.  and  Gibbons  A.M.  (2006).  “Managing  Collaborative  R&D 
Projects Development of a Practical Management Tool”, International Journal of Project 
Management 24 (5): 395–404. https://doi.org/10.1016/j.ijproman.2006.03.003. 

Bastianin,  A.,  Castelnovo,  P.,  Florio,  M.  et  al.  (2021),  “Big  science  and  innovation: 
gestation  lag  from  procurement  to  patents  for  CERN  suppliers”.  They  Journal  of 
Technology Transfer (2021). https://doi.org/10.1007/s10961-021-09854-5  

Benneworth,  P.  &  Dahl  Fitjar,  R.  (2019).  “Contextualizing  the  role  of  universities  to 
regional  development:  introduction  to  the  special  issue”,  Regional  Studies,  Regional 
Science, 6:1, 331-338, DOI: 10.1080/21681376.2019.1601593. 

Bleda,  M.,  &  Del  Río  P.  (2013).  “The  Market  Failure  and  the  Systemic  Failure 
Rationales  in  Technological  Innovation  Systems.”  Research  Policy  42  (5):  1039–52. 
https://doi.org/10.1016/j.respol.2013.02.008. 

83 

 
Boschma,  R.  &  Gianelle,  C.  (2013).  “Regional  branching  and  smart  specialization 
policy”, JRC technical reports, S3 Policy Brief Series, issue 06/2104. 

Cedefop  (2015).  “Skills,  qualifications  and  jobs  in  the  EU:  the  making  of  a  perfect 
match?  Evidence  from  Cedefop’s  European  skills  and  jobs  survey”,  Luxembourg: 
Publications Office. Cedefop reference series; No 103. http://dx.doi.org/10.2801/606129. 

Cooke, P., Gómez Uranga M. and Etxebarria G. (1997). “Regional Innovation Systems: 
Institutional and Organisational Dimensions”, Research Policy 26(4-5):475-491. 

Crescenzi,  R.,  de  Blasio,  M.,  Giua,  M.,  (2018).  “Cohesion  Policy  Incentives  for 
Collaborative  Industrial  Research. The Evaluation  of a Smart  Specialisation Forerunner 
Programme”, Spatial Economics Research Centre, Discussion paper 231. 

Dahlin, K. & Behrens, D. (2005). “When Is an Invention Really Radical? Defining and 
Measuring Technological Radicalness”, Research Policy 34(5):717-737. 

Daigneau, W. A., Valenti, M.S., Ricciarini, S. Bender, S.O., Alleyne, N., Di Grappa, M., 
Duart, J.M., Lupiáñez, F., Ehrenzweig Sanchez, M.A. (2005). “Planning, Designing and 
Managing  Higher  Education  Institutions”,  PEB  Exchange,  Programme  on  Educational 
Building, https://doi.org/10.1787/541370867752. 

Davey, T., Galan Muros, V. Meerman, A., Orazbayeva, B. and Baaken, T. for European 
Commission, Directorate-General for Education, Youth, Sport and Culture (2018). “The 
State of University-Business Cooperation in Europe”, Final Report. 

De  Bruijn,  P.  &  Lagendijk,  A.  (2005).  « Regional  Innovation  Systems  in  the  Lisbon 
strategy”, European Planning Studies, 13, (8), 1153-1172 

Di  Cataldo,  M.,  Monastiriotis,  V.,  and  Rodríguez-Pose,  A.  (2020).  “How  ‘Smart’  Are 
Smart  Specialization  Strategies?”,  JCMS:  Journal  of  Common  Market  Studies, 
https://doi.org/10.1111/jcms.13156. 

EACE  –  Evaluation  Advisory  Central  Europe  (2018).  “Ex  post  evaluation  of  the 
programming  period  2007-2013  in  the  R&D  sector”,  Final  report  (in  Czech),  National 
Coordination Authority, Ministry of Regional Development. 

Ebbekink,  M.  and  Lagendijk,  A.  (2013).  “What’s  next  in  researching  cluster  policy: 
place-based  governance  for  effective  cluster  policy”,  European  Planning  Studies,  Vol. 
21, No. 5, pp.735–753. 

Edquist,  C. 
Organizations”. 

(1997).  “Systems  of 

innovation:  Technologies, 

Institutions  and 

EGO  (2013).  “Badanie  podsumowujące  realizację  Priorytetu  XIII  Infrastruktura 
szkolnictwa wyższego Programu Operacyjnego Infrastruktura i Środowisko (pakiet 2)". 

European  Strategy  Forum  on  Research  Infrastructures  (ESFRI)  (2017).  “Long-Term 
Sustainability  Working  Group.  Long-Term  Sustainability  of  Research  Infrastructures”, 
Milano: Università degli studi di Milano, Dipartimento di fisica. 

European Commission (2006). “Community Framework for State Aid for Research and 
Development and Innovation”. 

84 

 
European  Commission  (2008).  “European  Competitiveness  Report”,  DOI  10.2769  / 
65417. 

European  Commission  (2009).  “European  Innovation  Scoreboard  2009:  Comparative 
analysis of Innovation Performance”. 

European Commission, Directorate General for Regional Policy (2013). EVALSED: The 
resource for the evaluation of Socio-Economic Development. 

European  Commission  (2016a).  "Employment  and  Social  Developments  in  Europe 
2015", 
http://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=7859&furtherPubs=ye
s. 

European  Commission,  Directorate  General  for  Regional  Policy  (2016b).  “Ex  post 
evaluation  of  Cohesion  Policy  programmes  2007-2013,  focusing  on  the  European 
Regional Development Fund (ERDF) and the Cohesion Fund (CF)”, Final Report Work 
Package 2. 

European Commission  (2017a). “The Economic  Rationale for Public R&I  Funding and 
Its Impacts”. 

European  Commission,  Policy  Support  Facility  (2017b).  “Peer  Review  of  Poland’s 
Higher 
System”, 
and 
https://rio.jrc.ec.europa.eu/sites/default/files/report/PSF-
Peer_review_Poland__FINAL%20REPORT.pdf 

Education 

Science 

European Commission (2020). “Science, research and innovation performance of the EU 
2020 - A fair, green and digital Europe”. 

European Commission, DG Regional and Urban Policy (2021). “Study on prioritisation 
in Smart Specialisation Strategies in the EU”. 

European  Commission,  DG  Regional  and  Urban  Policy  (2021).  “Evaluation  of 
investments  in  Research  and  Technological  Development  (RTD)  infrastructures  and 
activities supported by the European Regional Development Funds (ERDF) in the period 
2007-2013”. 

European  Commission,  DG  Regional  and  Urban  Policy  (2021).  “Evaluation  of 
investments  in  Research  and  Technological  Development  (RTD)  infrastructures  and 
activities supported by the European Regional Development Funds (ERDF) in the period 
2007-2013”, Case study: Czech Republic. 

European  Commission,  DG  Regional  and  Urban  Policy  (2021).  “Evaluation  of 
investments  in  Research  and  Technological  Development  (RTD)  infrastructures  and 
activities supported by the European Regional Development Funds (ERDF) in the period 
2007-2013”, Case study: Estonia. 

European  Commission,  DG  Regional  and  Urban  Policy  (2021).  “Evaluation  of 
investments  in  Research  and  Technological  Development  (RTD)  infrastructures  and 
activities supported by the European Regional Development Funds (ERDF) in the period 
2007-2013”, Case study: Germany. 

85 

 
European  Commission,  DG  Regional  and  Urban  Policy  (2021).  “Evaluation  of 
investments  in  Research  and  Technological  Development  (RTD)  infrastructures  and 
activities supported by the European Regional Development Funds (ERDF) in the period 
2007-2013”, Case study: Italy. 

European  Commission,  DG  Regional  and  Urban  Policy  (2021).  “Evaluation  of 
investments  in  Research  and  Technological  Development  (RTD)  infrastructures  and 
activities supported by the European Regional Development Funds (ERDF) in the period 
2007-2013”, Case study: Poland. 

European  Commission,  DG  Regional  and  Urban  Policy  (2021).  “Evaluation  of 
investments  in  Research  and  Technological  Development  (RTD)  infrastructures  and 
activities supported by the European Regional Development Funds (ERDF) in the period 
2007-2013”, Case study: Portugal. 

European  Commission,  DG  Regional  and  Urban  Policy  (2021).  “Evaluation  of 
investments  in  Research  and  Technological  Development  (RTD)  infrastructures  and 
activities supported by the European Regional Development Funds (ERDF) in the period 
2007-2013”, Case study: Romania. 

European  Strategy  Forum  on  Research  Infrastructures  (ESFRI)  (2017).  “Long-Term 
Sustainibility  Working  Group.  Long-Term  Sustainability  of  Research  Infrastructures”, 
Milano: Università degli studi di Milano, Dipartimento di fisica. 

Finardi,  U.  (2011).  Time  relations  between  scientific  production  and  patenting  of 
knowledge: The case of nanotechnologies. Scientometrics, 89(1), 37–50. 

Florio,  M.  (2019).  “Investing  in  Science:  Social  Cost-Benefit  Analysis  of  Research 
Infrastructures”, Cambridge, MA: The MIT Press. 

Florio,  M.  and  Sirtori,  E.  (2016).  “Social  Benefits  and  Costs  of  Large  Scale  Research 
Infrastructures”, Technological Forecasting and Social Change, 112: pp. 65-78. 

Foray,  D.  (2014).  “Smart  Specialisation:  Challenges  and  Opportunities  for  Regional 
Innovation  Policies:  Opportunities  and  Challenges  for  Regional  Innovation  Policy”, 
London: Routledge. 

Foray,  D.  and  Van  Ark,  B.  (2007).  “Smart  specialisation  in  a  truly  integrated  research 
area is the key to attracting more R&D to Europe”, Knowledge Economists Policy Brief 
n° 1, October 2007. 

Foray,  D.,  David,  P.A.  and  Hall,  B.  (2009).  “Smart  specialisation  –  The  concept”, 
Knowledge Economists Policy Brief n° 9, June 2009. 

Foray,  D.,  Morgan,  K.  and  Radosevic,  S.  (2018).  “The  Role  of  Smart  Specialisation  in 
the EU Research and Innovation Policy Landscape”, Brussels: European Commission.  

Freeman,  C.  (1987).  “Technology  Policy  and  Economic  Performance:  Lesson  from 
Japan, Pinter Publisher. 

86 

 
Freeman,  C.  (1988).  “Japan:  A  New  National  System  of  Innovation?”  in  G.  Dosi,  C. 
Freeman,  R.  Nelson,  G.  Silverberg,  L.  Soete  (eds.),  Technical  Change  and  Economic 
Theory. London and New York: Pinter Publisher (pp. 330–348). 

Georghiou, L. (1993). “National Systems of Innovation. Towards a Theory of Innovation 
and  Interactive  Learning  Edited  by  Bengt-Ake  Lundvall  (Pinter  Publishers,  London. 
1992)  Pp.  Xiii  +  342,  £45,  ISBN  1-85567-063-1.”  Prometheus  11  (2):  291–291. 
https://doi.org/10.1080/08109029308629360. 

Griliches, Z. (1979). "Issues in Assessing the Contribution of Research and Development 
to Productivity Growth," Bell Journal of Economics, The RAND Corporation, vol. 10(1), 
pages 92-116, Spring. 

Guy, K. (2007). “Policy Mix Peer Reviews: Synthesis Report,” 79. 

Hallonsten,  O.  (2014).  “How  scientists  may  ‘benefit  from  the  mess’:  A  resource 
dependence  perspective  on  individual  organizing  in  contemporary  science”,  Social 
Science Information, 53(3):341-362. doi:10.1177/0539018414524037. 

Hospers,  G-J.  and  Beugelskijk,  S.  (2002).  “Regional  cluster  policies:  learning  by 
comparing?”, KYKLOS, Vol. 55, No. 3, pp.381–402. 

Jacoby,  N.  (2010).  “An  Evolutionary  Theory  of  Economic  Change,  Nelson  &  Winter 
(1982).” In De Taylor à Aujourd’hui (2010), l’Encyclopédie Multimédia En Sciences de 
l’organisation,  edited  by  E.  Friedberg,  CD-Rom.  R&O  multimédia.  https://hal.archives-
ouvertes.fr/hal-00605363. 

JIIP  (2017).  “Synergies  between  Framework  Programmes  for  Research  and  Innovation 
and European Structural and Investment Funds”. 

Joint  Research  Centre 
https://rio.jrc.ec.europa.eu/sites/default/files/riowatch_country_report/RIO_CR_RO_201
7_PUBSY_IDF.pdf 

“RIO  Country  Report  2017:  Romania”, 

(2018). 

Keraminiyage,  K.,  Amaratunga,  D.,  and  Haigh,  R.  (2009).  “Achieving  Success  in 
Collaborative  Research:  The  Role  of  Virtual  Research  Environments”,  Journal  of 
Information Technology in Construction, 14, 59-69. 

Landabaso, M., Oughton, C., and Morgan, K. (2002). ‘The innovation paradox’, Journal 
of Technology Transfer, 27, pp. 97-110. 

Laudel,  G.  (2002).  “What  Do  We  Measure  by  Co-Authorships?",  Research  Evaluation, 
Oxford University Press, vol. 11(1), pages 3-15, April. 

Lundvall,  B-Å.  (1992).  “National  Systems  of  Innovation.  Towards  a  Theory  of 
Innovation and Interactive Learning”. Pinter Publisher. 

Malerba,  F.  (1996).  “Public  Policy  and  Industrial  Dynamics:  An  Evolutionary 
Perspective”, 30. 

Martin,  B.  R.  (1996).  “The  Relationship  between  Publicly  Funded  Basic  Research  and 
Review”. 
Economic 

Performance: 

SPRU 

A 

87 

 
https://www.academia.edu/17547694/The_relationship_between_publicly_funded_basic
_research_and_economic_performance_a_SPRU_review. 

Mayne, J. (2011).  “Contribution  analysis:  Addressing cause  and effect”,  Evaluating  the 
Complex, pp. 53–96. 

Nelson,  R.  (1993).  “National  Innovation  Systems:  a  Comparative  Analysis”,  Oxford 
University Press. 

(1998).  “Redefining  Tertiary  Education”,  OECD  Publishing,  Paris, 

OECD 
https://doi.org/10.1787/9789264163102-en.  

OECD  (2014).  Science,  Technology  and  Industry  Outlook  2014,  OECD  Publishing, 
Paris. 

O’Kane,  M.  (2008).  “Review  of  the  National  Innovation  System”,  Canberra,  A.C.T.: 
Dept. of Innovation, Industry, Science and Research.  

Perspektywy  (2020).  “Foreign  students  in  Poland  2020  conference”,  Educational 
Foundation. 

Porter,  M.  E.  (1998).  "Clusters  and  the  New  Economics  of  Competition",  Harvard 
Business Review 76, no. 6: 77–90. 

Radosevic,  S.  &  Lepori  B.  (2009).  “Public  Research  Funding  Systems  in  Central  and 
Eastern  Europe:  Between  Excellence  and  Relevance:  Introduction  to  Special  Section”,  
Science and Public Policy 36 (9), 659-666. 

Re-Source  (2014).  “Ewaluacja  procesu  komercjalizacji  wyników  prac  B+R  oraz 
współpracy  jednostek  naukowych  z  przedsiębiorcami  w  ramach  I  osi  priorytetowej 
Programu Operacyjnego Innowacyjna Gospodarka (Poddziałanie 1.1.1 oraz Poddziałanie 
1.3.1)”, Raport końcowy. http://www.ewaluacja.gov.pl/media/24739/2_092.pdf 

Robin,  S.  &  Schubert,  T.  (2013).  “Cooperation  with  Public  Research  Institutions  and 
Success  in  Innovation:  Evidence  from  France  and  Germany”,  Research  Policy  42  (1): 
149–66. https://doi.org/10.1016/j.respol.2012.06.002. 

Rodríguez‐Pose,  A.  (2020).  "Institutions  and  the  fortunes  of  territories,"  Regional 
Science Policy & Practice, Wiley Blackwell, vol. 12(3), pages 371-386, June. 

Romer,  P.M.  (1990).  “Endogenous  Technological  Change”,  Journal  of  Political 
Economy,  98  (5),  Part  2:  71-102.Salter,  A.  J.,  &  Martin  B.  R.  (2001).  “The  Economic 
Benefits of Publicly Funded Basic Research: A Critical Review”, Research Policy 30 (3): 
509–32. https://doi.org/10.1016/S0048-7333(00)00091-3. 

Simmonds, P. (2016). “ISIS Lifetime Impact Study”. 

Smith,  K.  (2000).  “Innovation  as  a  Systemic  Phenomenon:  Rethinking  the  Role  of 
(1):  73–102. 
Policy”,  Enterprise 
https://doi.org/10.1080/146324400363536. 

Innovation  Management  Studies  1 

and 

88 

 
Technopolis Group & MIOIR (2012). “Evaluation of Innovation Activities. Guidance on 
methods  and  practices”,  Study  funded  by  the  European  Commission,  Directorate  for 
Regional Policy. 

Technopolis  Group,  Praxis  and  Institute  of  Baltic  Studies.  (2011).  “Euroopa  Liidu 
ja  arendustegevuse  ning  kõrghariduse 
tõukefondide  perioodi  2007‐2013 
meetmete rakendamise vahehindamine”, Tallinn: Government of Estonia. 

teadus‐ 

Thanos, C. (2010). “Global Research Data Infrastructures: The GRDI2020 Vision”. 

World  Bank  (2014).  “SABER  —  Systems  Approach  for  Better  Education  Results 
Education Management Information Systems”. 

89