REGULATORY SCRUTINY BOARD OPINION REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL Ex post evaluation of Horizon 2020, the EU framework programme for research and innovation
Date: 2024-01-29
Institutions: Directorate-General for Research and Innovation, European Commission
Subjects: EU programme, EU research policy, Framework Programme for Research and Development, economic consequence, innovation, research and development, research expenditure (EU), scientific cooperation, social impact, sustainable development

EUROPEAN COMMISSION 

Brussels, 12.10.2023 
SEC(2024) 52 final 

REGULATORY SCRUTINY BOARD OPINION 

REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND 
THE COUNCIL 

Ex post evaluation of Horizon 2020, the EU framework programme for research and 
innovation 

{COM(2024) 49 final} 

{SWD(2024) 29 final } 
{SWD(2024) 30 final} 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
EUROPEAN COMMISSION 
Regulatory Scrutiny Board 

Brussels,  
Ares(2023) 

Opinion 

Title: Final evaluation of the Horizon 2020 Programme 

Overall 2nd opinion: POSITIVE 

(A) Policy context 

Horizon  2020,  the  eighth  framework  programme  (FP),  was  the  EU  research  and 
innovation program running from 2014 to 2020 with a budget of around EUR 80 billion.  

Its  successor  is  Horizon  Europe  (2021–2027).  Horizon  2020  promoted  collaboration 
between  scientists,  businesses,  and  other  stakeholders  to  foster  excellent  science, 
strengthen  EU’s  competitiveness/industrial  leadership  and  to  find  solutions  to  societal 
challenges.  

The evaluation examines the performance, the scientific, societal and economic impact of 
the implementation of Horizon 2020. 

(B) Summary of findings 

The Board notes the improvements made to the evaluation report in line with the 
first opinion and technical comments. 

The Board gives a positive opinion. The Board also considers that the report should 
further improve with respect to the following aspects:  

(1)  The  report  does  not  sufficiently  explain  the  underlying  assumptions  of  the 

Benefit Cost Analysis and uncertainty of the three models used. 

(2)  The  report  does  not  fully  account  for  the  impact  of  data  limitations  in  the 
conclusion chapter and does not adequately align some scoring with the outcome 
of the impact analysis. 

  ________________________________   
This opinion concerns a draft evaluation which may differ from the final version. 

Commission européenne, B-1049 Bruxelles - Belgium. Office: BERL 08/010. E-mail: regulatory-scrutiny-board@ec.europa.eu 

 
 
 
 
 
(C) What to improve 

(1) The  report  should  better  explain  the  underlying  assumptions  behind  the  derived 
Benefit-Cost-Ratio  rather  than  relying  on  a  reference  to  the  technical  annexes.  It 
should also include the administrative cost estimates in the quantification table under 
Annex IV. It should better place the average application cost estimates in relation to 
FP7, or comparable R&I programmes in the Member States or third countries. 

(2) The  report  should  better  explain  the  diverging  outcomes  of  the  three  quantitative 
models applied. Given the report’s finding that Europe’s overall competitive position 
has not fundamentally changed over the duration of Horizon 2020, while indicating 
that this does not imply that Horizon 2020 did not contribute to competitiveness, the 
report should be more balanced in concluding and scoring Horizon 2020’scontribution 
to boosting EU’s competitiveness.  

(3) The report should elaborate further on the data limitations related to having 59% of 
finalised projects. The fact that more projects have been finalised than under FP7 at 
the corresponding moment does not present a solid base for drawing firm conclusions. 
Even though the report claims that results will improve further on the basis of the IPR 
outputs of FP7 whose performance more than doubled 10 years after its end, this is 
only  a  small  part  of  the  Horizon  2020  programme,  and  it  is  not  clear  whether  this 
development  could  be  indicative  for  other  parts  of  the  programme,  too  (past 
performance does not guarantee to be repeated). In general, the conclusions should be 
more nuanced for specific parts of the programme. 

(4) Several  scores  allocated  in  the  conclusion  section  should  be  critically  reviewed  to 
better reflect the outcome of the impact analysis. For example, the objectives regarding 
the European Institute of Technology in terms of start-up creation were not reached 
the attributed score should be reviewed. Considering the discussion on the impact of 
H2020  on  GDP,  employment  and  overall  EU  competitiveness,  the  corresponding 
scoring should be more balanced. 

(5) The report should critically review some of the assumed causality links, such as that 
Horizon  2020  beneficiaries  invest  more,  recruit  more  and  grow  more  than  non-
beneficiaries. It should assess to what extent this is a direct consequence of the grant 
or a rather a result from selecting the most promising projects from most promising 
companies.  It  should  be  clearer  whether  successful  beneficiaries  continue  to  out-
perform non-beneficiaries after the competition of the project. 

(6) The report should better distinguish between what has been learned from the final (or 
mid-term)  Horizon  2020  evaluation,  and  be  more  explicit  on  which  lessons  have 
already  been  (partly)  taken  up  and  revised  in  the  successor  programme,  and  which 
remain relevant for future revisions in this policy area.  

2 

 
 
 
 
(D) Conclusion 

The DG may proceed. 

The  DG  must  take  these  recommendations  into  account  before  launching  the 
interservice consultation. 

Full title 

Final evaluation of the Horizon 2020 Programme 

Reference number 

PLAN/2022/785 

Submitted to RSB on 

15 September 2023 

Date of RSB meeting 

Written procedure 

3 

 
 
EUROPEAN COMMISSION 
Regulatory Scrutiny Board 

Brussels,  
Ares(2023) 

Opinion 

Title: Final evaluation of the Horizon 2020 Programme 

Overall opinion: NEGATIVE 

(A) Policy context 

Horizon 2020, the eighth framework programme (FP), was the EU research and innovation 
program running from 2014 to 2020 with a budget of around EUR 80 billion. Its successor 
is Horizon Europe (2021–2027). 

Horizon  2020  promoted  collaboration  between  scientists,  businesses,  and  other 
stakeholders  to  foster  excellent  science,  strengthen  EU’s  competitiveness/industrial 
leadership and to find solutions to societal challenges.  

The evaluation examines the performance, the scientific, societal and economic impact of 
the implementation of Horizon 2020.   

(B) Summary of findings 

The Board notes the complexity of the evaluation exercise given the large scope and 
size of the Programme and the significant analytical efforts. The Board notes as well 
the additional information  provided  in advance of  the  meeting  and  commitments  to 
make changes to the report. 

However,  the  Board  gives  a  negative  opinion,  because  the  report  contains  the 
following fundamental shortcomings:  

(1)  The  analysis  of  effectiveness  and  efficiency  is  often  inconclusive  as  regards  the  
value for money of the Programme, in general and for its individual components.  

(2)  The  conclusions  do  not  adequately  reflect  the  presented  analysis  and  do  not 
provide  a  clear  and  balanced  narrative  about  the  successes  and  failures  of  the 
Programme. 

(3)  The  lessons  learned  from  the  evaluation    do  not  provide  the  basis  for  future 

action and Programme adaptations. 

The Board considers that in its present form this report does not sufficiently respond 
to the mandate of the evaluation. 

  ________________________________   
This opinion concerns a draft evaluation which may differ from the final version. 

Commission européenne, B-1049 Bruxelles - Belgium. Office: BERL 08/010. E-mail: regulatory-scrutiny-board@ec.europa.eu 

 
 
 
 
 
 
(C) What to improve 

(1)  The effectiveness and efficiency analyses should be improved so that it is clear to what 
extent  Horizon  2020  has  been  successful  in  achieving  (or  progressing  towards)  its 
objectives  (overall  and  in  specific  parts)  and  whether  and  to  what  extent  the  benefits 
outweigh  the  costs.  The  report  should  improve  the  evaluation  of  the  impact  of  Horizon 
2020 on  all dimensions of competitiveness, productivity, and industrial leadership of EU 
industry  as  part  of  the  effectiveness  analysis.  It  should  be  clearer  on  the  extent  to  which 
Horizon  2020  increased  the  contribution  of  research  and  innovation  to  tackle  societal 
challenges.  When  discussing  leverage,  it  should  be  clearer  in  distinguishing  co-funding 
from  leverage  effects.  It  should  also  discuss  the  areas  of  possible  simplifications,  in 
particular  regarding  the  application  process  and  the  cost  for  applicants.  While  the  report 
acknowledges  that  the  Programme’s  monitoring  system  was  deficient,  which  led  to  data 
gaps, there should be increased efforts to better exploit the findings of the support studies 
and to  recreate  the  missing points of  comparisons  or  benchmarks  to  be able  to  assess the 
extent  to  which  the  objectives  have  been  achieved  and  at  what  cost.  The  assumptions 
underpinning  the  analysis  of  costs  and  benefits  should  be  clarified  as  to  whether  the 
assessment  considers  full  or  actual  spending,  given  that  only  59%  of  projects  have  been 
finalised.  

(2)  Once  the  effectiveness  and  efficiency  analyses  are  improved  in  line  with  the  points 
mentioned above, the report should bring all the quantified costs and benefits together in an 
overview  table  (differentiated  by  key  program  parts  or  instrument  where  possible)  and 
calculate  a  Benefit  Cost  Ratio.  The  analysis  of  the  overall  value  for  money  of  the 
Programme  and  its  specific elements  should  also  better  consider  the  non-quantified  costs 
and benefits. 

(3)  The  conclusions  of  the  report  should  answer  the  three  fundamental  questions  of  any 
evaluation, i.e. was the intervention successful and why, how did the EU intervention make 
a  difference  and  to  whom,  and  has  it  been  /  is  it  still  relevant.  In  its  current  form,  the 
concluding  section  does  not  adequately  reflect  the  key  findings  of  the  analysis,  not 
providing sufficient discussion of the shortcomings or underperformance and not making a 
clear assessment on the degree of achievement of the Horizon 2020 Programme’s specific 
and  general  objectives.  The  impact  of  data  limitations  resulting  from  a  deficient 
performance  monitoring  system  and  potentially  from  having  data  for  only  59%  of  the 
projects should be clearly qualified in the report. 

(4)  The lessons learned from the evaluation are not developed in the report and therefore 
have  little  use  for  future actions  and further  improving  the ongoing  programme  (Horizon 
Europe) or developing future EU research programmes. If there are lessons learned derived 
from  the  mid-term  evaluation  of  Horizon  2020  that  have  been  already  addressed,  they 
should  be  explicitly  identified  and  described.  The  lessons  learned  should  reflect  all  the 
elements  from  the  previous  analysis  where  any  shortcomings  were  identified  and  be 
coherent with the overall picture presented in the conclusions. In particular, any potential 
for  further  simplification  of  procedures,  introducing  efficiency  enhancing  measures  and 
improving  and  future-proofing  the  performance  monitoring  system  of  the  programme 
should be discussed. The lessons learned regarding the Programme’s capacity to contribute 
more  effectively  to  the  EU’s  overall  as  well  as  sectoral  competitiveness  should  be  also 
discussed. 

Some more technical comments have been sent directly to the author DG. 

2 

 
 
 
(D) Conclusion 

The  Board  advises  the  DG  not  to  launch  the  interservice  consultation  before 
substantially revising the report. 

The lead DG may resubmit to the Board a revised version of this report. 

Full title 

Final evaluation of the Horizon 2020 Programme 

Reference number 

PLAN/2022/785 

Submitted to RSB on 

17 May 2023 

Date of RSB meeting 

21 June 2023 

Electronically signed on 12/10/2023 15:44 (UTC+02) in accordance with Article 11 of Commission Decision (EU) 2021/2121

3