Evaluation study on the relevance and internal coherence of Horizon 2020 and its policy mix : executive summary.
Date: 2023-10-27
Institutions: Directorate-General for Research and Innovation
Subjects: EU programme, EU research policy, Framework Programme for Research and Development, coronavirus disease, innovation, new technology, research and development, research policy, research results, scientific cooperation

Evaluation study on the relevance 

and internal coherence of  

Horizon 2020 and its policy mix 

Executive summary

Read the full publication 
Evaluation study on the relevance and internal coherence of Horizon 2020 and 
its policy mix 

EXECUTIVE SUMMARY 

The purpose of this study is to provide the Commission with the specific data and analyses needed to 
support  the  ex-post  evaluation  of  Horizon  2020  regarding  the  evaluation  criteria  'relevance'  and 
'internal  coherence' as  well  as  its  policy  mix.  Further,  the  study's  conclusions  and  recommendations 
shall  inform  the  implementation  of  Horizon  Europe,  and  potentially  future  framework  programmes 
(FP). 

The study covers the whole Horizon 2020 programme (2014-2020) and its policy mix defined as the 
set of activities, instruments, and types of actions used to implement Horizon 2020. It is performed in 
line with the Better Regulation guidelines as applicable to all EU expenditure programmes.  

The  focus  of  the  evaluation  is  on  the  relevance  and  internal  coherence  evaluation  criteria.  The 
'coherence'  criterion  splits  broadly  into  internal  and  external  coherence,  and  it  is  internal  coherence 
that is considered in this study. Evaluating 'effectiveness', 'efficiency' or 'EU added' value is not part of 
this assignment and will be addressed in related Horizon 2020 impact area studies. 

In the context of this specific study, and building on the corresponding baseline definitions in the Better 
Regulation  toolbox,  'relevance'  and  'internal  coherence'  of  the  policy  mix  and  the  processes 
established to ensure relevance and internal coherence are defined as follows: 

Relevance of the design: The programme's relevance to the needs, priorities, problems and issues 
for R&I to be addressed at the European level; the relevance of Horizon 2020 to the European political 
priorities; the responsiveness to emerging needs and priorities.  

Relevance  of  the  policy  mix:  The  degree  to  which  the  policy  mix  addresses  the  Horizon  2020 
objectives; the adequacy in addressing the needs of target groups. 

Relevance of processes: The extent to which processes supported relevance, for example different 
types of consultation and evidence used in Horizon 2020 programming. 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Executive Summary  
Evaluation study on the relevance and internal coherence of Horizon 2020 and its policy mix 

Internal  coherence  of  the  policy  mix:  The  translation  of  Horizon  2020  objectives  into  work 
programmes  and  calls,  and  coherence  across  the  programme  parts;  the  complementarity  and 
consistency of established and new instruments, activities, or actions within the policy mix. 

Internal  coherence  of  processes:  A  set  of  process  characteristics,  expected  to  support  the 
successful design of the programme. 

Methods  

Evidence was gathered during the period August 2021 to September 2022 using a mix of quantitative 
and qualitative methods. Data collection and analysis was structured in five phases. 

In  the  exploration  phase,  a  cross-cutting  basis  of  relevant  data  was  developed  based  on  European 
Commission  sources  (e.g.  extracts  from  eCORDA  database,  call  texts  etc.),  as  well  as  compiled 
through  the  analysis  of  documents  (Horizon  2020  programming  documents,  policy  documents, 
evaluation  studies,  expert  advice  etc.).  Subsequently,  first  qualitative  (performance  and  SWOT 
analysis,  exploratory  interviews)  and  quantitative  analyses  (programme  data  analysis,  text  mining) 
were performed, which provided orientation to the subsequent steps of the study. In the third phase, 
20  Horizon  2020  policy  mix  case  studies  were  complemented  by  ten  worldwide  comparison  case 
studies.  While  the  worldwide  comparison  case  studies  provided  inspirational  knowledge  on  leading 
practices,  they  were  not  used  as  benchmarks,  as  their  contexts  were  not  comparable  to  that  of 
Horizon 2020. For both Horizon 2020 policy mix case studies and worldwide comparison case studies, 
we  conducted  75  targeted  interviews  to  support  the  sense-making  on  the  cases.  In  the  synthesis 
phase, insights and conclusions from the study were assessed and validated in two policy workshops, 
which included targeted stakeholder consultations of altogether 63 participants. From the concluding 
meta-analysis we derived lessons learnt, areas for improvement and operational recommendations for 
Horizon Europe. 

Conclusions  on  key  ambitions  and  challenges  for  the  relevance  and  internal  coherence  of 
Horizon 2020 

The relevance of the design of Horizon 2020 

•  Horizon 2020 intervention was highly relevant given the needs, priorities, problems and issues for 
R&I  to  be  addressed  at  the  European  level.  All  three  rationales,  which  eventually  became  the 
cornerstones  of  Horizon  2020,  namely  i)  the  reinforcement  of  scientific  excellence,  ii)  the  turn 
towards  innovation,  and  iii)  the  more  political  and  impact-oriented  framing  of  the  thematic  top-
down  funding  programmes,  were  anticipated  and  brought  forward  by  research  communities.  In 
that way, the three-pillar structure of Horizon 2020 covered the major needs for R&I in Europe at 
the time when the programme was designed, while responding at the same time (more than ever) 
to political priorities.  

•  Horizon 2020 took place in an era of dynamic global change, having been conceived during the 
aftermath  of  the  2008/09  financial  crisis  and  extending  into  a  period  of  increasing  geopolitical 
uncertainty,  a  more  acute  awareness  of  the  unfolding  climate  crisis  and,  finally,  the  first  major 
pandemic of the 21st century. Overall, Horizon 2020 addressed these political priorities by adding 
them  to  the  work  programmes,  calls  and  funded  projects.  Likewise,  issues  and  needs  emerging 
from  within  the  research  and  innovation  (policy)  arena  during  the  Horizon  2020  implementation 
phases,  such  as  open  science  and  societal  impacts  of  research,  were  picked  up  in  work 
programmes.  

•  A limitation on the relevance of Horizon 2020 design was that the inclusion of emerging themes as 
additional  topics  in  the  work  programmes  was  not  carried  out  on  the  basis  of  a  strategic 
(refocusing,  redirecting)  approach.  However,  emerging  topics  –  coming  either  from  new  political 
priorities or bottom-up emerging needs from the research communities  – could be addressed by 
the budget, mostly by reallocating portions within the same programme heading. As these budget 
envelopes tended to be based on political priorities, the budgetary discretion of the EC was largely 
limited to shifts within the programme headings.  

2 

 
 
Executive Summary  
Evaluation study on the relevance and internal coherence of Horizon 2020 and its policy mix 

•  Two major events brought about new impulses to re-orient the last programming period of Horizon 
2020  and  give  directions  for  Horizon  Europe.  These  were  the  'Report  of  the  Independent  High 
Level Group on Maximising the Impact of EU Research & Innovation Programmes' (also called the 
'Lamy  Report')  following the interim evaluation  of Horizon 2020  in 2017 and the arrival of a new 
European Commission along with new priorities, in particular the European Green Deal, in 2019. 
The transformative change called for by the Green Deal was addressed by dedicated calls in the 
last year of Horizon 2020. 

•  The  EU  is  beginning  to  catch  up  in  artificial  intelligence  –  which  is  one  of  the  most  dynamic 
technology  fields  with  regard  to  patenting  and  start-up  activity  in  the  EU  –  yet,  there  still  is  a 
significant gap between itself and the US/China. While the EU remains one of the global leaders in 
AI  science,  it  still  lags  in  AI  innovation,  among  other  areas,  due  to  a  lack  of  available  big  data 
sources. 

•  Overall, the EU's industrial R&D investment in ICT, ICT services as well as in the health sector (in 
particular in biotechnology) has remained below that of its competitors for a number of years.   

The relevance of the Horizon 2020 policy mix 

•  Overall, the Horizon 2020 set of instruments, activities and types of action (the 'policy mix') proved 
well matched to its expected outputs and impacts, and thus found to be 'relevant'. In each of the 
three  main  programme  parts  (Excellent  Science,  Industrial  Leadership,  Societal  Challenges),  a 
specific policy mix was brought to bear, which corresponded to their specific objectives. 

More specifically:  

•  The ERC is regarded as a crucial building block of European scientific excellence. Together with 
other bottom-up schemes supporting frontier research, it is important from a responsiveness and 
preparedness  perspective,  exploring  a wide  range  of  novel  research  paths  and  thus  acting  as  a 
reservoir  of  possible  response  options  to  a  variety  of  emerging  needs.  The  development  of  the 
COVID-19  vaccine  by  BioNtech/Pfizer  is  one  of  the  most  remarkable  recent  examples  of  this 
preparedness: an ERC grant to Ugur Sahin and his team was among the grants that provided the 
relevant knowledge. 

•  The policy mix of the Industrial Leadership pillar put a strong focus on SMEs and was thus highly 
relevant  for  the  specific  objectives  of  strengthening  SME  innovation  performance  and  growth. 
Further,  all  programmes  of  the  Societal  Challenges  pillar  adopted  the  cross-cutting  policy  of 
strengthening SMEs, although the SME Instrument was used with varying intensity. The analysis 
of call texts reveals in this regard frequent mentions of 'SMEs' and 'technological innovation' in the 
Excellent Science pillar, which indicate a certain degree of relevance to (and of course coherence 
with)  the  Industrial  Leadership  pillar's  objective  of  fostering  industrial  competitiveness,  not  the 
least because of the ERC's Proof of Concept scheme. 

•  The availability of skilled staff and the ability to attract talent was the most important challenge for 
many  start-ups  and  growth-oriented  SMEs  across  the  EU  during  the  framework  programme. 
Actions  supporting  training,  capacity  building  and  access  to  talent  proved  to  be  particularly 
relevant  to  these  needs,  including  the  Marie  Skłodowska-Curie  Actions  in  the  Excellent  Science 
part and the Enterprise Europe Network and INNOSUP Actions in the Industrial Leadership pillar 
(About  10%  of  companies'  participations  in  Horizon  2020,  equivalent  to  6.500  in  total  numbers, 
was in MSCA).  

•  The  slow  uptake  of  mission-oriented  and  transformative  policy  approaches  in  the  Societal 
Challenge  programmes  was  identified  as  a  limitation  in Horizon 2020  – slow compared  to some 
leading Member States such as Sweden (which has included such policies since 2011), Germany, 
the Netherlands and UK (all since 2018). Over the course of Horizon 2020, Societal Challenges 
programmes have developed towards more innovation and market orientation, which is shown by 
the importance of the SME Instrument and Innovation Actions in the SC policy mix. Nevertheless, 
the policy approach remained traditional, as it continued to focus on technological innovation and 
market-based solutions. This hampered, for example, the participation of 'practice partners' such 
as  public  authorities  and  CSOs  (see  also  below).  Changes  in  the  direction  of  transformative 

3 

 
 
Executive Summary  
Evaluation study on the relevance and internal coherence of Horizon 2020 and its policy mix 

approaches  in  research  and  innovation  activities  started  only  in  the  last  year of  Horizon  2020  in 
response  to  the  European  Green  Deal  and  calls,  and  included  (i)  the  acknowledgement  that 
research  and  innovation  activities  should  aim  to  support  processes  of  social  change,  (ii)  a  more 
open  understanding  of  different  types  of  generalisation  processes  allowing  for  'location-based 
for  multi-disciplinarity 
innovation'  among  other  examples,  and  (iii)  a  more  explicit  call 
(strengthening  the  role  of  SSH)  and  trans-disciplinarity  (strengthened  role  for  'practice  partners' 
due to greater focus on research supporting the implementation of solutions). 

•  Throughout  Horizon  2020,  the  integration  of  SSH  in  multidisciplinary  projects  remained 

challenging, as they focused primarily on technological change. 

•  The  aim  of  Horizon  2020  to  increase  the  participation  of  CSOs  especially  in  the  Societal 
Challenges  pillar  proved  difficult  to  meet.  Although  their  participation  increased  as  compared  to 
FP7, CSOs often faced obstacles meeting participation  rules  and  assessment criteria  for project 
research  funding.  The  share  of  funding  was  even  lower  (4%)  than  their  numerical  share  of 
participation  (6%),  which  indicates  that  CSOs  seemed  to  generally  take  on  non-core  roles  in 
research  project  consortia  and,  rather,  participating  more 
in  research  communication, 
coordination,  and  results  dissemination/uptake  activities.  Although  the  participation  of  CSOs 
showed signs of increasing from work programme 1 to 3, the generally low numbers indicate that 
Horizon 2020 lacked coherence and understanding on how exactly they should be involved. The 
Green Deal calls, which gave more weight to implementation, generalisation and uptake, started 
to  fill  this  gap,  by  for  example  requiring  'practice  partners'  to  take  on  the  responsibility  of  local 
experimentation. 

•  The  number  of  participants  and  the  net  EU  contributions  requested  by  the  research  institutions 
show  a  very  uneven  distribution  between  EU-13  and  EU-15  countries.  The  largest  recipients  of 
funds  were  still  the  EU-15  countries  with  a  share  of  79%  of  the  requested  EU  net  contribution, 
followed  by  the  EU-13  countries  with  7%.  The  two  most  significant  barriers  to  widening 
participation  within  public  research  were:  1)  oversubscription,  which  often  led  to  self-exclusion, 
and 2) missing support mechanisms (both nationally and institutionally). 

•  The current study has identified challenges in terms of openness and entry levels for newcomers 
in  partnerships,  especially  if  coming  from  small  countries,  although  a  clear  distinction  must  be 
made  between  the  individual  partnerships.  This  was  a  concern  as  specific  topics  were  only 
covered by partnership activities, but not by Horizon 2020 work programmes. 

•  The  introduction  of  a  reference  to  the  TRL  classification  gradually  spread  to  encompass  the 
majority of the Horizon 2020 programmes (except actions such as MSCA, ERC funding, Access to 
Finance), and was used in about 20% of the calls. It spawned two interrelated shortcomings: as 
the practice evolved to introduce clearer specifications of the levels at which projects should start, 
and the expected levels that the projects should reach, this tended to support a linear intervention 
mode, which does not fit all programme parts and intervention logics. Second, in cases where a 
systemic  or  holistic  approach  was  expected,  and  societal  engagement  considered  key  for 
advancing the solution towards higher adoption levels (for example, topics in Societal Challenge 
5), TRLs limited the solution scope at the technology adoption/diffusion end.  

•  Several instruments and actions under Horizon 2020 proved to be very useful in responding to the 
COVID-19  pandemic,  for  example,  the  InnovFin  Infectious  Diseases  action  and  the  IMI 
partnership.  The  possibility  to  award  grants  without  a  call  for  proposals,  together  with  newly 
introduced  emergency  funds,  enabled  the  programme  to  respond  even  faster  to  the  COVID-19 
crisis compared to the previous Ebola and Zika outbreaks. 

How Horizon 2020 processes support the relevance of the programme 

•  The programming up to FP7 by and large responded to the needs and interests of the research 
communities. Yet, in cases of crises and new political priorities the responses were reactive. With 
Horizon 2020, strategic programming evolved into a tool that allowed policymakers to shape and 
direct political priorities.  

4 

 
 
Executive Summary  
Evaluation study on the relevance and internal coherence of Horizon 2020 and its policy mix 

•  Over the course of the three programming phases, this process was further refined, as were the 
different approaches to consultation and the gathering of evidence in the process. Advisory groups 
and  the  preparation  of  scoping  papers  was  seen  together  with  the  coordination  between  EC 
services as the major elements contributing to the expected benefits of the strategic programming 
process. These benefits included a better (and more prospective) response to new developments 
and  challenges  from  the  R&I  side  while  contributing  significantly  to  the  EU's  overall  policy 
objectives. 

•  Coordination  between  the  European  Commission  directorates  general  (DGs)  has  considerably 
improved  in  Horizon  2020  and  can  be  characterised  in  particular  in  its  last  phase  (and  the 
transition to Horizon Europe) in the following ways: (1) It is one of few co-created EC programmes 
at  the  time  and  involved  the  highest  number  of  DGs;  (2)  Before  Horizon  2020,  coordination  of 
framework  programmes  within  the  Commission  happened  primarily  at  cabinet  level,  but  with 
Horizon  2020,  DG  RTD  managed  to  establish  a  more  integrated  programming  approach  that 
included  operational-level  coordination  which  enriched  the  programming  process  and  supported 
increased  coherence  compared  to  FP7.  The  evaluation  study  notes  as  downsides  of  the  co-
creation  process  that  (1)  power  dynamics  between  DGs  were  evident  in  setting  the  WP-related 
decisions,  and  hindered  to  some  extent  the  benefits  of  widespread  consultations,  and  (2)  work 
programme  development  –  especially  the  updating  process  –  resembled  a  'collection  exercise', 
where  new  topics  were  added  to  the  work  programmes  and  little  consideration  given  to  refining 
down or removing topics deemed to be less effective. Co-creation was one of the major reasons 
the number of topics increased.  

•  Despite well noted approaches to improve the consultation among the stakeholder groups, several 
problematic issues remained. Most importantly, open consultation mechanisms were not designed 
in a way to include the distinct views of external stakeholders. The survey structure asked whether 
topics were relevant, but leaves no possibility for differentiated feedback. In its current structure, 
the survey is perceived as an instrument for legitimising the current EC priorities, not as one that 
can  open  the  discussion  and  bring  in  new  aspects.  In  addition,  the  stakeholder  consultation,  in 
particular  the  informal  channels,  favoured  the  dominant  R&I  stakeholders  (who  know  the  game 
and  have  the  resources  to  act),  which  put  newcomers  at  a  disadvantage,  especially  those 
stakeholders representing the end-users of research and innovation processes, in particular civil 
society. 

•  With the orientation towards impacts and the consecutive opening up of new consultation arenas 
(European  R&I  Days,  Green  Deal  public  consultation),  in  its  last  phase,  Horizon  2020  made 
notable  improvements;  introducing  societal  needs  and  societal  interests  more  directly  via 
consultation into the framework programme.  

•  Strategic  use  of  foresight  and  policy  feedback  (i.e.  interactions  of  funded  research  projects  with 
policy)  were  not  systematically  institutionalised  by  the  end  of  Horizon  2020,  despite  having 
significant  potential  to  inject  some  longer-term  thinking  into  the  Commission's  efforts  to  better 
orientate the framework programme. 

•  The SRIAs of partnerships fulfilled important roles during the FP, helping to orientate partners and 
guide  their  strategies.  The  increased  use  of  collaborative  methods  also  raised  the  level  of 
legitimation  –  and  thus  relevance  –  which  helped  balance  different  interests  between  various 
stakeholder  groups.  Although  SRIAs  followed  different  timelines  to  work  programmes,  some 
partnerships aligned more strategically with PCs than others because of overlapping memberships 
in partnership steering bodies and programme committees.  

•  The  reaction  to  COVID-19  and  the  mobilisation  of  budget  from  the  emergency  funds  in  Societal 
Challenge  1  (Health)  happened  rapidly.  It  was  concluded  that  the  emergency  procedure 
developed after Ebola and Zika outbreaks served the response to COVID-19 pandemic well.  

•  Although  the  amount  of  crisis  funding  mobilised  in  response  to  COVID-19  turned  out  to  be  too 
little, the existence of the emergency funds proved to be very useful in ensuring a swift reaction to 
the resulting crisis. 

5 

 
 
Executive Summary  
Evaluation study on the relevance and internal coherence of Horizon 2020 and its policy mix 

The internal coherence of the Horizon 2020 policy mix  

•  Horizon 2020 objectives were translated very well into work  programmes  and calls. Further, key 
objectives of Horizon 2020 were addressed across pillars, in particular the policy focus on SMEs 
and on Societal Challenges.  

•  Horizon  2020  integrated  newly  emerging  themes  exhaustively  in  the  calls.  Societal  Challenges 

programmes were the most prominent domain under which new themes were addressed.  

•  The Excellent Science pillar featured a coherent policy mix. The mono-beneficiary grants of MSCA 
and  ERC  were  the  cornerstones  of  this  policy  mix.  This  is  where  the  age  profile  of  the  MSCA 
fellows was complementary to ERC grantees, i.e. the MSCA fellows were all younger or at earlier 
stages in their career.  

•  At  the  same  time,  opportunities  for  collaborative  basic  research  decreased  with  the  Industrial 
Leadership  and  Societal  Challenges  programmes  focusing  on  higher  TRLs.  The  Future  and 
Emerging  Technologies  (FET)  programme  could  not  fully  compensate  for  this change,  as  it  was 
too narrow in scope and low in volume.  

•  Among  the new  instruments  and  types  of  actions,  in  particular  the  SME  Instrument  and  the  EIC 
Accelerator  Pilot,  which  encompassed  the  previous  SME  Instrument,  showed  complementarities 
with many other actions – in the Industrial Leadership and Societal Challenge pillar: (1) In terms of 
target  groups,  as  a  mono-beneficiary  grant,  it  complemented  the  FTI  pilot  scheme  and  the 
Eurostars programme, the aim and the scope of which were very similar; (2) The EIC was the only 
instrument  covering  almost  the  entire  TRL  spectrum,  and  it  supported  the  commercialisation  of 
innovations  across  all  sectors  and  technology  domains.  Thus,  the  introduction  of  European 
Innovation Council helped to improve the internal coherence of the framework programme. 

• 

Innovation procurement added a strategically important diversity to the Horizon 2020 policy mix as 
it  focused  on  innovation  used  to  solve  problems  and  allowed  high-end  research  to  play  into  the 
needs  of  the  public  sector.  However,  as  a  counterpart  to  the  framework  programmes'  overall 
supply-side approach, a shortage of capacity on the part of interested procurers was deemed to 
be  the  main  reason  for  poor  uptake.  In  practice,  supply-side  measures  dwarfed  demand-side 
efforts in Horizon 2020. 

•  Focus areas as a means to establish cross-pillar linkages had very little structuring effect on the 
coherence of processes within the EC. Focus areas were entirely perceived within the EC – led by 
DG RTD and aimed at boosting policy relevance and obtaining more buy-in from other DGs. This 
worked to some extent well, but there were no processes established to maintain interest among 
higher levels in the hierarchy. Further, budget rigidities prevented a true cross-over between work 
programmes,  the  selected  portfolios  were  not  systematically  followed  up,  and  systematic  linking 
between projects only developed in the last WP. 

How Horizon 2020 processes support the internal coherence of the programme 

•  The strategic programming process was developed during Horizon 2020 and there was a constant 
ambition to learn and to improve, with the effect that by the time of the third programming cycle 
many of the process parts had taken shape, and were carried on into the governance of Horizon 
Europe. 

•  Because of staff mobility policy, contracts ending, and other staff moves, EC units responsible for 
framework  programme  governance  could  not  sufficiently  accumulate  knowledge.  Horizontal 
mobility  was  floated  as  an  idea  to  avoid  silo-thinking  and  enable  learning  and  cross-fertilisation 
across  domains.  Today,  the  balance  between  continuity  and  mandated  staff  mobility  –  both  at 
managerial  and  operational  levels  –  remains  an  issue  for  knowledge  management  and  policy 
learning in DG RTD. 

•  Scouting  of  emerging  topics,  upcoming  needs  and  challenges  was  not  fully  enshrined  in  the 
strategic programming process in Horizon 2020, which meant it lacked (systematic) approaches to 

6 

 
 
Executive Summary  
Evaluation study on the relevance and internal coherence of Horizon 2020 and its policy mix 

deal with future uncertainties in highly volatile contexts and adequately cover societal needs and 
mechanisms.  

•  Consultation  mechanisms  used  in  the  (strategic)  programming  process  of  Horizon  2020  were 
lacking transparency on how the input from EC external stakeholders is used; feedback from the 
various  consulted  groups  needed  to  be  integrated  to  improve  the  outcomes  and  justify  the 
resource-intensive  consultation  procedures.  This  was  especially  true  for  the  open  consultations 
and  the  Advisory  Groups.  A  positive  development  was  observed  –  greater  transparency  and 
openness – as part of the consultation process for the Green Deal call; indeed, it was made public 
from the beginning and provided easier access for all stakeholders. 

•  The  drive  for  more  open  calls  compared  to  earlier  framework  programmes  revealed  different 
understandings  of  openness/prescriptiveness  among  those  involved  in  programming.  Evidence 
collected for this study suggests that the dominant understanding of these terms turned out to be 
the  following:  calls  should  be  prescriptive  in  specifying  intended  impacts,  but  open  in  defining 
approaches,  methods  or  technologies.  Against  the  backdrop  of  the  worldwide  comparison  case 
studies  performed  for  this  study,  it  was  found  that  this  approach  has  been  used  for  the  UK 
Strategic  Priorities  Fund  since  2018.  Further  evidence  suggests  that  the  approach  started  to 
become institutionalised in Horizon 2020. 

Recommendations  

The study offers two sets of recommendations following from the findings of this study and in response 
to  the  requirement  to  draft  recommendations  with  relevance  for  Horizon  Europe:  (1)  'quick  wins': 
operational recommendations, which should be addressed in the short term in the implementation of 
Horizon Eu-rope, and (2) areas for improvement requiring structural change. While acknowledging that 
structural change needs a mid- to long-term perspective for its implementation, the identified areas for 
improvement were highly relevant to Horizon Europe. We took into account the improvements already 
made  during  the  first  two  years  of  HEU  when  formulating  our  proposal  for  the  future.  These 
recommendations  have  been  discussed  and  were  further  substantiated  in  a  policy  workshop  with 
stakeholders and DGs involved in the framework programme. 

Quick wins to foster relevance 

•  Make processes of knowledge management and policy learning more relevant: Interactions 
between  FP-funded  research  and  policy  should  be  a  two-way  process  with  policy  DGs  actively 
engaging  with  the  agencies  to  absorb  the  knowledge  created  in  research  projects  and  to 
communicate  their  needs.  The  established  'feedback  to  policy'  framework  in  HEU  should  be 
evaluated in due time (in the long run, organisational change needs to take place: more resources 
are needed for targeted policy feedback. Greater and easier mobility of staff to and from agencies 
could be systematically encouraged as part of a career path in the EC). 

•  More  transparent  stakeholder  engagement:  Stakeholder  organisations  do  not  have  equal 
opportunities  to  get  their  voices  heard.  The  consultation  for  the  Green  Deal  call  was  different  in 
this regard, as the draft work programme was made public from the beginning and provided easier 
access  for  all  stakeholders.  This  open  model  should  be  the  standard  approach  for  organising 
consultations on draft work programmes in the future.  

•  Aiming  for  transformative  impact:  A  new  type  of  action(s)  should  aim  for  multi-  and  trans-
disciplinarity,  put  social  change  on  equal  footing  with  technological  innovation,  and  recognise 
different  generalisation  patterns.  Now,  in  Horizon  Europe's  cluster  programmes  and  missions, 
there  is  room  to  call  for  transformative  research.  A  general  call  for  more  involvement  of  social 
sciences and humanities (SSH) – although correct – will not be enough.  

•  Valuing  the  contribution  of  bottom-up  programme  parts  to  political  priorities  differently: 
The  approach  to  work  with  impact  pathways  in  Horizon  Europe  should  be  monitored  and 
evaluated  against  the  following  questions:  (1)  Do  these  tools  help  programmers  to  map  the 
specific  characteristics  of  each  part?  (2)  Are  they  useful  in  supporting  the  design  of  more 
transformative types of actions and activities? Further, for cross-cutting targets, like the spending 

7 

 
 
Executive Summary  
Evaluation study on the relevance and internal coherence of Horizon 2020 and its policy mix 

targets related to SDGs or climate change, it should be made clear that the different parts of the 
programme are expected to contribute to these targets to varying degrees.  

Quick wins to foster internal coherence 

•  New  collaborative  grant  scheme  for  the  excellence  programme  part:  It  is  recommended  to 
reconsider  the  option  of  opening  up  the  framework  programme  to  more  collaborative  research 
opportunities at lower TRL.  

•  Coordination  among  different  policy  domains  and  broader  policy  mixes:  With  Horizon 
Europe taking a mission-oriented approach and aiming for broader societal transformation, there 
arises  a  need  to  coordinate  the  research  and  innovation  missions  with  the  sectoral  policy 
approaches.  A  holistic  approach  requires  'systems  thinking'  to  analyse  mutual  inter-relations 
between  the  policy  mixes  within  the  FP  and  to  combine  instruments  beyond  R&I  funding  (e.g. 
regulation,  public  procurement).  Thus,  cross-policy  domain  structures  and  processes  should 
become part of the FP governance and policy cycle.  

•  Alignment  of  partnership  strategies  with  work  programme  development:  Double 
memberships  in  partnership  governing  boards  and  programme  committees  should  be  further 
developed and adopted by more partnerships. 

Areas, where structural change is needed to improve relevance 

•  A  true  crises  budget:  The  EU  might  want  to  think  about  a  'crisis  response budget  line'  for  any 
future situations warranting an immediate response, including the non-health related ones. Such a 
FP crisis budget line should be anchored in the programme in a way, that it can be purposed for 
other R&I priorities in case no major crisis happens. Further, an FP crisis budget should be part of 
a  broader  cross-policy  domain  crisis 
to 
react 
emergencies/situations in a holistic manner, beyond the mere R&I needs that a crisis creates.  

reaction  mechanism  and  budget 

to 

•  Focus on 'need owners': Following the insight that the inclusion of civil society actors in research 
projects turned out to be difficult, when they had no clear mandate or role in the implementation of 
the research results, it seems more straightforward to focus on those who have such a role (and 
thus  an  own  research  need).  One  way  to  approach  this  should  have  been  to  mobilise  so-called 
'need  owners'  (often  cities,  ministries  and  other  public  authorities  next  to  civil  society  actors)  for 
participation  in  these  projects.  The  aim  should  be  to  strengthen  their  role  the  closer  a  project 
moves  towards  implementation.  Thus,  an  'expanded  toolbox'  would  be  needed  to  stimulate  the 
participation  of  these  actors.  Accordingly,  participation  and  funding  rules  should  be  adapted  in 
order to adequately support the participation of 'need-owners'.  

•  More  selectivity  and  priority  setting:  Focus  and  selectivity  on  focal  topics  is  needed  in  work 
programme  development.  Selection  criteria  should  be  developed  in  the  strategic  programming 
process.  They  should  be based  on  a good  combination  of  external  advice  and  the  EC's  internal 
strategic intelligence. External advice could for example build on a more strategic use of foresight 
studies. 

Areas, where structural change is needed to improve Internal coherence 

•  Mandate to focus and (re)direct: Programme committees working jointly with expert committees 
could become a decision structure for implementing focus  and re-direction in work programmes. 
This  would  rebalance  the  current  decision  structures  by  upgrading  the  role  of  the  programme 
committees  and  strengthening  expert  advice  in  strategic  decision-making.  The  mechanism  to 
focus and re-direct should be accompanied by more budgetary flexibility. 

Disclaimer 

The information and views set out in this study are those of the authors and do not necessarily reflect the official 
opinion of the Commission. The Commission does not guarantee the accuracy of the data included in this study. 
Neither the Commission nor any person acting on the Commission's behalf may be held responsible for the use 
which may be made of the information contained therein. 

8 

 
 
 
European Commission 
Directorate-General for Research and Innovation 
Directorate RTD G — Common Policy Centre 
Unit RTD G2 — Common Programme Analysis & Regulatory Reform 
Contacts  Milena Isaković Suni, Xiména Rodriguez, Anne-Catherine Gridelet 
Email  

RTD-G2-SUPPORT@ec.europa.eu 
RTD-PUBLICATIONS@ec.europa.eu 

European Commission 
B-1049 Brussels