COMMISSION STAFF WORKING DOCUMENT EVALUATION Accompanying the document REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL Ex-post evaluation of Horizon 2020, the EU Framework Programme for Research and Innovation
Date: 2025-12-09
Institutions: Directorate-General for Research and Innovation, European Commission
Subjects: EU programme, EU research policy, Framework Programme for Research and Development, economic consequence, innovation, research and development, research expenditure (EU), scientific cooperation, social impact, sustainable development

EN 

  EN 

   EUROPEAN COMMISSION  Brussels, 9.12.2025  SWD(2024) 29 final/3 PART 2/2  CORRIGENDUM  This document corrects document SWD(2025) 29 final/2, part 2/2, of 3.12.2025:  - SWD reference year should read 2024 instead of 2025 and  - numbering of Figures 57 to 62 should read 56 to 61.   The text shall read as follows: COMMISSION STAFF WORKING DOCUMENT  EVALUATION  Accompanying the document REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL Ex-post evaluation of Horizon 2020, the EU Framework Programme for Research and Innovation {COM(2024) 49 final} - {SEC(2024) 52 final} - {SWD(2024) 30 final}  
 
 
ANNEX I. PROCEDURAL INFORMATION  

The ex post evaluation of the Horizon 2020 programme (Decide reference: PLAN/2022/785) has been developed under the lead of DG RTD, under the 
guidance of the interservice steering group (ISSG) composed of 29 DGs (AGRI, BUDG, CLIMA, CNECT, COMM, COMP, DEFIS, EAC, ECFIN, ECHO, 
EMPL, ENER, ENV, GROW, HOME, HR, IAS, INTPA, JRC, MARE, MOVE, NEAR, OLAF, OP, REGIO, SANTE, SG, SJ, TRADE) and 4 Executive 
Agencies (CINEA, EISMEA, ERCEA, HADEA) established in April 2022.  

The ISSG met in April 2022 to discuss the expectations of participating services, the draft call for evidence, the draft consultation strategy and the working 
methods of the ISSG. Following this ISSG the call for evidence was then published in July 2022 for four weeks and received 35 individual replies (presented 
in Annex 5). The ISSG met again in October 2022 to discuss the feedback received on the call for evidence and the draft questionnaire for stakeholder 
consultation.  

Following this ISSG meeting, the stakeholders’ consultation was launched on 1 December and closed on 19 February, having gathered 1 818 replies. The 
ISSG  met  for  the  third  time  in  February:  an  update  was  provided  about  the  late  advancement  of  external  evaluation  studies,  emerging  findings  were 
discussed and a draft evaluation (SWD) was presented and open for comments. Main messages from the upstream meeting with the Regulatory Scrutiny 
Board were shared with the member of the ISSG. A revised version of the evaluation (SWD) was open for comments in March by the Directorates General 
and agencies of the Horizon Europe governance and comments were integrated. The ISSG met on 3 May 2023 to discuss the draft final version of the 
evaluation SWD before submission to the Regulatory Scrutiny Board (RSB). 

This evaluation has been selected for scrutiny by the RSB. The outcome of the scrutiny was the issuance of a negative opinion, following a dedicated 
meeting on 21 June 2023. In the first negative opinion the Regulatory Scrutiny Board pointed at areas for improvement relating to: 

Strengthening the 
effectiveness and efficiency 
analysis 

•  Under Chapter 4.1.3 on economic impacts, a new section was added on ‘Improving Europe’s economic growth and competitiveness’, while the 
section on ‘Facilitating access to risk capital’ has been expanded. Under Chapter 4.1.2 on societal impacts, it is recognized that societal impacts 
take a longer time to become visible and the shortcomings of the monitoring system are recognised, noting that the conclusion is based not only 
on KPI data but also on qualitative evidence from the enclosed case studies. The section on ‘Promotion of gender equality in Horizon 2020’ has 
been completely revised by recreating the missing point of comparaison (FP7) and explaining the data limitations. The assessment period for 
GDP gains (cumulative and average annual) was changed from 2014-2030 to 2014-2040, to allow enough time for impacts to emerge as the last 
projects will only end in December 2028.  

•  Under Chapter 4.2 on efficiency, a Benefit Cost Ratio (BCR) of 5 to 1 is stated, referring to a newly created Section 5 in Annex 4 for calculations 
and further discussion of the BCR. It is clarified that the assessment considers actual, paid out amounts for the EU budget, with data frozen on 1 
January 2023. The quantification of time saved from faster grant agreements was corrected. A new section has been added on ‘What are potential 
areas for further simplification?’, covering lump sum funding (including expanded data from the pilot exercise), Seal of Excellence (moved from 
the External Coherence chapter) and the two-stage application process. 

•  Chapter 4.3.1 on internal coherence has been expanded. 

1 

 
 
Strengthening the cost 
benefit analysis 

Strengthening the 
conclusions 

• 

In Chapter 4.4 on EU added value, the section of leverage now explains that, as different types of participants have different funding rates, even 
within the same type of action, it is not possible to determine a priori the ‘expected’ direct leverage factor per type of action. There was no target 
set for leverage in Horizon 2020 and benchmarking against FP7 is not appropriate, because the two programmes have different types of action 
with different funding rates. Instead, benchmarking against two other programmes is included, with some limitations. In addition, in the final 
revision of the FP-wide counterfactual analysis, causality links were explained in more detail – additional information was added on the approach 
used, including analysis of only comparable companies and ensuring a sufficient number of lead and lag years in the FP-wide counterfactual 
analysis. 

The benefit cost ratio of 5:1 has been added in the main SWD. The overview table and further explanations are available in a new dedicated section in 
Annex 4. Quantification of costs and benefits has been privileged but non quantified benefits are also presented in Annex 4. In the final revision, 
information on the benefit cost ratio in the main report was extended (based on information previously in Annex 4). The estimate of the total 
beneficiaries’ administrative cost of Horizon 2020 was added to the main cost table in Annex 4.2 and given more visibility. The price base of the total 
benefit estimate used in the BCR calculation was changed to current prices. The BCR calculation table presents alternative values broken dwon by 
model used and assessment period. The central model output used for the calculation of the GDP gain (total benefits) was changed from Nemesis to 
Rhomolo, to present a more conservative estimate. Additional information on application costs found at Horizon 2020 interim evaluation was added 
in Annex 4 as a point of comparison.  

Conclusions have been re-organised and twice further elaborated by repeating some key statistical data from the evaluation findings. For Societal 
Challenges, conclusions explain the deficiencies of the monitoring system and longer timespan needed to contribute to societal challenges. Following 
improved analysis of gender aspects, this conclusion has been updated. New conclusions have been added on boosting Europe’s economic growth 
and competitiveness and the cost-benefit ratio. The main areas for improvement are identified: research infrastructure, dissemination and exploitation 
of R&I results and measures aimed at widening participation in the programme. In the final revision, a new score was added for the programme’s 
contribution to Europe’s competitiveness. Also the following note was deleted because data was available only for IPR: ‘It is expected that the 
programme will further improve its performance as ongoing projects come to an end, in line with IPR outputs from FP7, which more than doubled in 
2023 after the final evaluation of FP7 (2015)’. Lastly, a new sentence was added to further acknowledge uncertainties caused by the ongoing 
implementation of 41% of the projects, while informing that the cumulative implementation rates at the time of the evaluation were 99.99% for 
Horizon 2020 commitments and 87.84% for payments. Similar statement also added in the section on ‘How has the situation evolved in the reporting 
period’. 

Strengthening lessons 
learned 

The section on lessons learned has been expanded, reflecting all the elements from the previous analysis where any shortcomings were identified and 
coherent with the overall picture presented in the conclusions. An update has been provided on the main short-term and long-term suggestions for 
improvement made by the interim evaluation of Horizon 2020. In the final revisions, status was provided for three lessons learned from the interim 
evaluation, on the programme externalisation, feedback to policy and international cooperation. It was also clarified which lessons learned came from 
the interim evaluation and which ones from the final. Lastly, when response to a lesson learned was finalised, this was indicated, though most points 
will require continuous monitoring and follow up in future (Horizon Europe) evaluations. 

Following the negative opinion, the ISSG had a written consultation on the revised SWD between 25 August and 5 September. Based on this, the revised evaluation 
was resubmitted to the RSB on 15 September 2023 – receiving a positive opinion on 12 October 2023.  

2 

 
 
 
In the second, positive opinion the Regulatory Scrutiny Board pointed at further areas for improvement relating to: 

The report should better explain the underlying assumptions behind the derived Benefit-Cost-
Ratio  

Given the report’s finding that Europe’s overall competitive position has not fundamentally 
changed over the duration of Horizon 2020, while indicating that this does not imply that Horizon 
2020 did not contribute to competitiveness, the report should be more balanced in concluding and 
scoring Horizon 2020’scontribution to boosting EU’s competitiveness.  

The report should elaborate further on the data limitations related to having 59% of finalised 
projects. The fact that more projects have been finalised than under FP7 at the corresponding 
moment does not present a solid base for drawing firm conclusions. (past performance does not 
guarantee to be repeated). In general, the conclusions should be more nuanced for specific parts 
of the programme.  

Assumptions behind the Benefit Cost ratio expanded in the SWD 

Language nuanced regarding the direct contribution of Horizon 2020 in boosting 
EU competitiveness 

Data on finalised projects at the time of the evaluation could not be updated. 
59% is higher percentage at the time of the final evaluation of FP7 (50%). 
Timing of the evaluation could not be postponed but 59% of finalised projects is 
deemed a solid base. 

Several scores allocated in the conclusion section should be critically reviewed to better reflect 
the outcome of the impact analysis.  

Scoring double checked 

The report should critically review some of the assumed causality links, such as that Horizon 
2020 beneficiaries invest more, recruit more and grow more than nonbeneficiaries.  

Different analyses were run for this purpose covering different samples and 
using a diverse set of methods. An overview of those along with their caveats is 
provided in Annex 2. The results of the analyses agree in finding positive effect 
on firms’ employment and revenues growth. One of those, which tracks firms up 
to 5 years before and after their application for funding, found that beneficiary 
firms increase on average their employment level by 20% compared to non-
funded firms with high quality proposals, and their total revenues by about 30% 
in the years following the receipt of the first grant. These average effects are 
present even after 2.5 years (the average duration of a project in the sample).  

The report should better distinguish between what has been learned from the final (or mid-term) 
Horizon 2020 evaluation, and be more explicit on which lessons have already been (partly) taken 
up and revised in the successor programme, and which remain relevant for future revisions in this 
policy area. 

Language reinforced 

No exception from the usual procedural requirements of the better regulation guidelines was requested for this evaluation.  

This evaluation is based on evidence gathered via different channels and an overview is presented in Annexes 2 and 3. The main sources of evidence are 
internal analyses by the European Commission, analysis and reports by other European Institutions and by external evaluators who worked on twelve 
evaluation studies, carried out between 2021 and 2023, listed in Annex 2.  

The external evaluation studies operated under the steer of interservice groups composed of relevant Commission services which at the end of the studies 
agreed on the adequacy of the resulting final reports, with particular respect to their relevance, appropriate design, reliable data, sound analysis and reliable 
findings.  

3 

 
 
 
ANNEX II. METHODOLOGY AND ANALYTICAL MODELS USED  

The ex post evaluation of Horizon 2020 was coordinated by the Common Programme Analysis & Regulatory Reform Unit of the Commission’s Directorate-
General for Research & Innovation, with the support of: (i) a working group (the ‘MEAVE’ - Impact Monitoring, Evaluation and Analysis Virtual Entity) 
gathering together the R&I family DGs and Executive Agencies; (ii) and an interservice steering group comprising relevant Commission DGs. The ex post 
evaluation of Horizon 2020 started in 2022 and was guided by the Terms of Reference adopted by the Commission after a vote by the Member States’ 
Programme Committee1.  

The evaluation builds on: (i) a large amount of quantitative and qualitative evidence collected through a variety of methods described below; and (ii) a 
thorough evaluation analysis, applying triangulation of evidence from different sources, ensuring an objective and robust assessment.  

Main data sources 

The scope of the Horizon 2020 ex post evaluation includes all calls with a closure deadline on 31 December 2020 and grants signed by June 2022. Section 
3 provides data on how programme implementation evolved from its launch until 1 January 2023, when 41% of projects were still ongoing. Section 4 
provides an analysis of the programme based on triangulation of evidence predating 1 January 2023 (most external evaluation studies were carried out 
during 2022, with programme data extracted at the end of 2021).  

The analysis was based on the following data sources: 

•  The main source of data for the evaluation is the Common Research Data Warehouse (CORDA) Portal. The portal gathers data collected through 
different Commission tools, including policy monitoring at work programme level, data collected at proposal stage, grand agreement preparation 
and through continuous project reporting.  

•  Beyond CORDA, additional datasets were used. This was also to have comprehensive data on the whole framework programme, in particular for 
the different partnerships (such as European Institute of Innovation and Technology knowledge and innovation communities (EIT KICs) and joint 
undertakings (JUs)), for the Joint Research Centre (data and analysis provided by the JRC on its activities) and Innovation Radar data.  

•  Evidence and analysis conducted in the interim evaluation of Horizon 20202, the interim evaluation of the EIT3, and the independent High-Level 

Group on maximising the impact of EU research & innovation programmes (the ‘Lamy Group’)4.  

•  External datasets such as Scopus5, Orbis6, PATSTAT, Crunchbase, Dealroom, Pitchbook, Technote, and MAG/OpenAlex.  

1 C(2022) 7817. 
2 https://op.europa.eu/en/publication-detail/-/publication/bccdcce7-d8c9-11e8-afb3-01aa75ed71a1/  
3 https://op.europa.eu/en/publication-detail/-/publication/7415ff23-db2d-11e8-afb3-01aa75ed71a1  
4 https://ec.europa.eu/info/sites/default/files/conferences/sof/hlg_2017_report.pdf  
5 https://www.scopus.com/search/form.uri?display=basic#basic 
6 https://orbis.bvdinfo.com/version-202251/Orbis/Companies/Login?returnUrl=%2Fversion-202251%2FOrbis%2FCompanies  

4 

 
 
•  Monitoring  reports  of  Horizon  20207  and  statistical  data  mainly  from  the  Commission’s  internal  IT  Tools  (Horizon  Dashboard),  as  well  as 

Eurostat/OECD data. 

•  Extensive quantitative and qualitative analyses on specific aspects and objectives of Horizon 2020, conducted through 12 external evaluation studies 

by independent evaluation experts, selected using a transparent process and overseen by relevant Commission departments.  

Seven studies covered specific aspects of Horizon 2020. These were: 

•  Evaluation study on the ‘Opportunities and Challenges in Targeted Funding of Research and Innovation: Lessons learnt from the Horizon 2020 Focus Areas 

and implications for Horizon Europe Missions’, https://data.europa.eu/doi/10.2777/40351  
•  Evaluation study on the Open access policy study, https://data.europa.eu/doi/10.2777/268348 
•  Evaluation study on the proposal evaluation system under Horizon 2020, https://data.europa.eu/doi/10.2777/16211 
•  Evaluation study on the European Innovation Council (EIC) Pilot, https://data.europa.eu/doi/10.2777/261324 
•  Evaluation study on the external coherence & synergies of Horizon 2020 within European R&I support system, https://data.europa.eu/doi/10.2777/054469 
•  Evaluation study on the relevance & internal coherence of Horizon 2020 and its policy mix, https://data.europa.eu/doi/10.2777/058655  
•  Evaluation study on the implementation of cross-cutting issues in Horizon 2020, https://data.europa.eu/doi/10.2777/763665 

Five other studies covered specific policy objectives of the programme: 

•  Evaluation study on Excellent Science in the EU Framework Programmes for R&I, https://data.europa.eu/doi/10.2777/967813  
•  Evaluation  study  of  the  EU  Framework  Programmes  for  R&I  for  addressing  Global  Challenges  and  Industrial  Competitiveness  –  Focus  on  activities  for  a  more 

Resilient Europe, https://data.europa.eu/doi/10.2777/60819 

•  Evaluation study of the EU Framework Programmes for R&I for addressing Global Challenges and Industrial Competitiveness  – Focus on activities for the Digital 

and Industrial Transition, https://data.europa.eu/doi/10.2777/99438 

•  Evaluation study of the EU Framework Programmes for R&I for addressing Global Challenges and Industrial Competitiveness  – Focus on activities related to the 

Green transition, https://data.europa.eu/doi/10.2777/422725 

•  Evaluation study of the EU Framework Programmes for R&I for an Innovative Europe, https://data.europa.eu/doi/10.2777/144504 

•  Data from other EU institutions, such as the Conclusions on the Interim Evaluation of the Council, the work of the European Parliament’s ITRE 

• 

committee, relevant Court of Auditors’ reports8 and reports/evaluations of the European Economic and Social Committee. 
Input from the public consultation on the Horizon 2020 ex post evaluation. This consultation received input from 1 818 respondents and 21 position 
papers.  

Detailed descriptions of the models and methods used in the different information sources mentioned above are available in each respective external study 
and internal analysis report. Below is a short overview. 

7 https://ec.europa.eu/info/publications/horizon-2020-monitoring-flash_en  
8 For the section on leverage of JUs, the ECA Annual report on EU Joint Undertakings for the financial year 2021 was used and its underlying data. For S2R, the EU contribution includes 
administrative contributions of EUR 13.5 million. If these are disregarded, the expected leverage factor becomes EUR 1.22, instead of the EUR 1.18 reported. 

5 

 
 
 
Main methods used 

1.  Macroeconomic modelling 

Measuring the full impact of R&I, i.e. capturing indirect effects on top of direct ones, is an intricate question, compounded by the often relatively long time 
lags between policy initiatives and observed actual impacts. The European Commission uses complementary modelling platforms for both the ex ante and 
ex post evaluations of research and innovation policies.9 In this annex, macroeconomic modelling is used to quantify the economic impact of Horizon 2020 
in terms of GDP gain and job creation in the EU. While there is consensus that R&I is an important factor in increasing productivity, quantifying the impact 
of R&I policies at macroeconomic level requires modelling tools that accurately capture how R&I translates into economic gains.  

There are several models available to assess the dynamic transmission channels of R&I, each with specific features. This ex post evaluation uses results 
produced by three macroeconomic models: NEMESIS, QUEST and RHOMOLO. This is an  ex post assessment in the sense that the input data on the 
Horizon 2020 investments are up to date and reflect the actual disbursements during the programming period. However, the results should not be regarded 
as a way to exactly track and monitor the actual macroeconomic impact of the Horizon 2020 interventions. This is because they rely on assumptions both 
on the modelling setup and on the simulation strategy adopted to simulate the investments’ impact (i.e. the economic channels activated by them). 

Results from NEMESIS were produced by a team of external experts, while RHOMOLO and QUEST results were produced by European Commission 
departments (the Joint Research Centre for RHOMOLO and DG Economic and Financial Affairs for QUEST). The strength of these models lies in their 
distinct features. NEMESIS is considered one of the richest models covering different types of innovation.10 QUEST is the most appropriate for assessing 
the impact of R&I policies over time. By modelling regional economies, RHOMOLO is the most suitable model to address the geographical concentration 
of innovative activities.  

9 European Commission, Directorate-General for Research and Innovation, Benedetti Fasil, C., Martino, R., Ravet, J. (2020), Macroeconomic models for Research and Innovation policy: 
the present and the future, Publications Office https://data.europa.eu/doi/10.2777/339051. 
10 Di Comite and Kancs (2015). 

6 

 
 
 
 
1.1. NEMESIS 

Presentation of the model 
NEMESIS was developed by a European consortium11 in 2000 to analyse the macro-sectoral impacts of EU policies, based on R&D investments and related 
knowledge spillovers. The model became a reference tool for assessing EU and national R&I policies, and since 2004 has been used by the European 
Commission for several analyses. These include the assessments of: (i) the Lisbon Strategy target of 3% of EU GDP to be invested in R&D12; (ii) the RTD 
national action plan related to the Barcelona Objective13; and (iii) the impact of European R&I programmes (ex ante assessment of the 7th Framework 
Programme14, of Horizon 202015, and of Horizon Europe16).  

Structure of the model 

NEMESIS is a detailed sectoral macro-econometric model estimated for every country of the EU. It distinguishes between 30 sectors operating within five-
level  nested-CES  functions.17  The  model  covers  both  the  supply  and  demand  sides  of  the  economy  and  incorporates  endogenous  technical  change. 
Specifically, the representation of technical progress in NEMESIS is derived from the new growth theories, where innovations result from investment in 
R&D by private firms, and from R&D undertaken by the public sector. In the latest version of NEMESIS used for this ex post evaluation (as well as for the 
interim evaluation of the H2020 programme in 2017), innovations still arise from private and public investments in R&D, as well as investments in two 
other complementary innovation inputs: ICT and other intangibles (OI), including training and software. These enable improved accuracy in assessing R&I 
policies by considering the most up-to-date theoretical and empirical findings of economic literature (Le Mouël, 2019; Akcigit et al., 2022).  

Table 1: The innovation mechanism in NEMESIS 

Schematically, the innovation mechanisms of the model, at the level of a firm (or a sector), can be described as follows: 

➢  Firms determine their investments in the three innovation inputs (private R&D, ICT and OI), depending on their relative costs and their degree of complementarity. 
➢  The investment effort by firms increases their own knowledge (stock variable) as well as the knowledge of other firms, sectors and countries, through the knowledge spillover 
matrices (knowledge transfers). For each innovation input, the knowledge stock is modelled as a weighted sum of the stock of assets, R&D, ICT or OI, belonging to all sectors 

11 Lab. ERASME / Ecole Centrale Paris (now SEURECO), Federal Planning Bureau of Belgium, E3M3 lab. / ICCS /NTUA and Chambre d’Industrie et de Commerce de Paris. 
12 Brécard, D., Fougeyrollas, A., Le Mouël, P., Lemiale, L. and P. Zagamé (2006), ‘Macro-economic consequences of European Research Policy: Prospects of the NEMESIS model in 
the year 2030’, Research Policy, No 35(7), pp. 910-924. Doi:10.1016/j.respol.2006.03.001.  
13 Chevallier, C., Fougeyrollas, A., Le Mouël, P., and P. Zagamé (2006), ‘A time to sow, a time to reap for the European Countries: A macro-econometric glance at the RTD National 
Action Plans’, Revue de l’OFCE, 2006/5 (No 97 bis), pp. 235-257. Doi:10.3917/reof.073.0235 
14 Delanghe, H. and U. Muldur (2007), ‘Ex-ante impact assessment of research programmes: The experience of the European Union’s 7th Framework Programme’, Science and Public 
Policy, No 34(3), pp. 169-183, doi:10.3152/030234207X218125.  
15 European Commission (2012), The Grand Challenge – The design and societal impact of Horizon 2020, Directorate-General for Research and Innovation. Doi:10.2777/85874. 
16 European Commission, Directorate-General for Research and Innovation, (2018), A new horizon for Europe: impact assessment of the 9th EU framework programme for research and 
innovation, Publications Office https://data.europa.eu/doi/10.2777/194210. 
17 European Commission, Directorate-General for Research and Innovation, Benedetti Fasil, C., Martino, R., Ravet, J. (2020), Macroeconomic models for Research and Innovation policy: 
the present and the future, Publications Office. https://data.europa.eu/doi/10.2777/339051  

7 

 
 
and countries. The coefficients of the matrices used to build these stocks are calibrated based on patent citations between sectors and countries. These matrices combine the 
citations between patents allocated by technology classes and country with the OECD concordance table, in order to allocate these citations between sectors (Johnson, 2002). 
For R&D, the knowledge stock is also influenced by the public investments undertaken by the public sector. 

➢  The growth in knowledge stocks will generate innovations at a rate that is a positive function of the knowledge absorption capacity of the firm (measured by its investment 

➢ 

intensity in each innovation input). 
Innovations take two forms: product and process. Product innovations increase the intrinsic quality of the product sold by the firm, whereas process innovations improve the 
production process without changing the quality of the product (pure total factor productivity effect).  

➢  Product innovations have a direct positive impact on the internal and external demands addressed to the firm, while process innovations reduce its production cost, and, in a 

competitive market context, lowers its market price and increases its demand. 

➢  These dynamics at firm or sectoral level are brought together at the macro level by the input-output tables of the model, and the combination of the sectoral interdependencies 

(‘bottom-up’) with the ‘top-down’ macroeconomic forces finally drive the medium- and long-term dynamics of the model. 

Key assumptions for the ex post evaluation 

Key assumptions in NEMESIS for assessing the impact of the H2020 relate to the financing of the programme, the leverage effects of the  framework 
programme investments18, and the framework programme’s economic performance (EU added value)19. For the purpose of this evaluation, it is assumed 
that the H2020 programme was financed by a reduction of public investment equivalent, at Member State level, to the EC contribution. The importance of 
this cut in public investment in the individual Member States is proportional to their historical contribution to the EU budget. There are net contributors 
and net beneficiaries, depending on how successful each Member State was at benefitting from the framework programme and their relative contribution 
to the EU budget.20 Additionally, it is also assumed that the framework programme has no direct crowding-in and crowding-out effects on basic research. 
This assumption was retained from the ex post evaluation of FP7 by PPMI in 201721 and the survey of the dedicated literature realised for the ex ante 
assessment of Horizon Europe in 201822.  

The assumptions retained are summarised in Table 2. 

18 Amount of additional R&I expenditure leveraged by the initial R&I investment. Note that besides this  ‘direct’ crowding-in effect of the FP on the R&D investments made by its 
beneficiaries, there is an ‘indirect’ crowding-in effect that is the additional R&D investments engaged by a research entity, financed by the FP or not, as a response to the modification of 
the overall economic activity that the FP provokes (and not as the direct result of the EC financial support, as for the direct crowding-in effect). The total crowding-in is therefore the sum 
of the direct and the indirect crowding-in. 
19 i.e. how much the performance of the R&I investments provoked by the FP (in terms of R&I outcomes) is superior to those of the R&I support from other sources of funding (including 
national sources). 
20 Note that for the ex ante and ex post evaluations of FP7, and for the ex ante and interim evaluations of H2020, no financing of the programmes was considered, and the FP money was 
supposed to come from ‘nowhere’. 
21 PPMI, 2017. 
22 Boitier et al., 2018. 

8 

 
 
 
 
Table 2: Key assumptions of the NEMESIS model 

FINANCING 

Equivalent decrease of public 
investments 

DIRECT CROWDING-IN 
EFFECT 
Basic research: €0 
Appl. res.: +€0.15 
Average: +€0.087 

EU added value 

+15% 

The value of EUR 0.15 for the framework programme’s direct crowding-in effect on applied research was retrieved from the review of dedicated literature 
realised in 2018 for the ex ante assessment of HE23. The value of +0.15% for the EU added value of the programme was already used in the ex ante impact 
assessment of H2020 with NEMESIS in 201224, based on the evaluation of past FPs. Finally, Table 3 reports the assumptions underlying the precise outline 
of the framework programme budget, its annual layout, and the distribution of the Commission contribution between basic and applied research, Member 
States and the different economic activities. 

Table 3: Horizon 2020 budget and its repartition 

Programme 

H2020 

budget 
in 

(EC 
constant 

H2020 
contribution, 
billion € 2020) 
Horizon 2020 budget in constant 
prices 

average  duration  of  FP7 
projects (in month) 

Repartition  between  basic 
and applied research 

National allocation of funds 

Sectoral allocation of funds 

35 

Basic: 42% 

Applied: 58% 

Historical  (based  on  H2020 
dashboard) 

Historical  (based  on  H2020 
dashboard  and  Orbis 
for 
private for-profit entities) 

All the data used for this ex post evaluation of H2020 come from ‘observed’ information available in the H2020 dashboard. Out of a total budget of 68.4 
billion 2020 euro in the H2020 dashboard25, only 61.7 billion was retained in the study. This was done by subtracting the part of the EC contribution that 
benefits countries outside the EU-28. The split between basic and applied research was 42% for basic and 58% for applied26, and 40% and 60% respectively 
if we take into account the total cost of the projects financed. The national allocation of the funds was also based on the H2020 dashboard, and appears 
very stable compared to FP7 data and to the H2020 data up to August 2016 used in 2017 for the interim evaluation. The sectoral allocation of applied 
research was itself based on the H2020 historical data for private corporations and on the Orbis database, to make the sectoral mapping using the NEMESIS 
model. 

23 Boitier et al., 2018. 
24 EC, 2012. 
25 With this limitation that the H2020 data used for this ex post evaluation include only the figures encoded in E-Grants, related mainly to grants agreements signed in the H2020 dashboard. 
The overall H2020 covers also other actions (i.e. studies, management and administrative expenditure for the programme, subscriptions, financial instruments, etc.) that are not encoded 
into  E-grants.  As  such,  the  dashboard  does  not  contain  all  the  activities/actions/expenditure  financed  under  the  programme,  and  this  may  slightly  (about  5%)  under-evaluate  the 
programme’s impact on the main indicators presented in this report. 
26 For this repartition, the EC contribution that benefits public bodies and higher education institutions was considered as basic research, and the remaining as applied research. These 
shares of about 40% for basic research and of 60% for applied were also used in the interim evaluation of H2020 and correspond to the repartition targeted by the Commission when 
designing the H2020 programme. 

9 

 
 
 
 
Results 

Results from the NEMESIS model indicate that Horizon 2020 produces positive effects on GDP. Figure 1 displays the results from the simulation across 
three main phases: ‘Investment’, ‘Innovation’ and ‘Obsolescence’.27 Positive, although limited, GDP gains are already observed in the investment phase: 
about  +0.05 points in  average between 2014  and 2020. During  these first  years, the positive effects  of the programme on GDP come mainly  from  its 
crowding-in on the R&D investments by  programme beneficiaries. The programme is financed by an equivalent  cut in public investments in Member 
States, but the crowding-in effect induces a net positive impact on investment at the macroeconomic level. The high and direct content in labour of R&D 
investments, compared to other forms of investments, also raises households’ income and final consumption, from where the main GDP gains during this 
first phase originate. There are in return inflationary pressures that deteriorate the external balance during the first 4 years of simulation, but the situation 
begins to improve from 2018 with the arrival of the first innovations the programme contributed to financing. 

GDP gains increase significantly during the innovation phase, reaching a maximum of about +0.25 points in 2027 up to 2030, and an average of about 
+0.20 points on 2021-2030. After 2030, GDP gains start to gradually decline due to the gradual obsolescence of the new knowledge and the innovations 
the programme contributed to creating. 

Figure 1: The impact of H2020 on EU GDP and its components 

Source: PPMI - NEMESIS simulation. 

Positive impacts are observed also on the employment level. As reported in Figure 2, the investment phase (up to 2020) is characterised by a significant 
increase in the number of people employed in the research sector, with the creation of up to 100 000 jobs in research by 2019-2020, and an average rise of 
about 85 000 jobs compared to the situation in the reference scenario. In turn, high-qualified employment in production activities experience a reduction, 

27 For simplicity, the three phases follow each other chronologically in the figures, but in reality they overlap. 

10 

-0.05%0.00%0.05%0.10%0.15%0.20%0.25%0.30%2014201520162017201820192020202120222023202420252026202720282029203020312032203320342035203620372038203920402041204220432044204520462047204820492050GDP deviation and coontribution to GDP (%)Private ConsumptionGross Fixed Capital FormationTrade BalanceInvestment phaseInnovation phaseObsolecence phase 
 
 
partly due to a re-allocation of high-skilled workers from these sectors to research positions,  and partly due to the rise in  wages received by qualified 
workers, which in turn reduces the overall demand for this category of personnel. The same mechanism also induces a slight reduction in the number of 
low-qualified people employed in production. During the innovation phase, the market deployment of the innovations provoked by the H2020 programme 
translates into substantial job creation in every economic sector. On average, for 2021-2030 the gain in total employment is about +123 000 (with +15 000 
for research employment, and +23 000 and +85 000 respectively for high- and low-qualified employment in production). As with GDP, the employment 
gains decrease gradually during the obsolescence phase, from +229 000 in 2030, when they are at their maximum, down to +36 000 in 2050. 

Figure 2: The impact of Horizon 2020 on employment 

Source: NEMESIS simulation. 

Limitations of the model 

While NEMESIS’ strengths justify its relevance when measuring the impact of R&I policies, the model’s specific features also imply a number of limitations 
to be considered when interpreting the results. First, the model relies on the empirical observation of relationships and allows for flexibility in behavioural 
functions, which may generate inconsistencies between the most recent developments in macroeconomic theory. Furthermore, it uses adaptive expectations 
rather than forward-looking ones. NEMESIS also does not link the use of human capital with investments in the educational system. 

11 

-100-500501001502002502014201520162017201820192020202120222023202420252026202720282029203020312032203320342035203620372038203920402041204220432044204520462047204820492050Employment deviation (thousand)High Qualified (Production)Low Qualified (Production)ResearchTotal EmploymentInvestment phaseInnovation phaseObsolecencephase 
 
 
 
1.2. QUEST 

Presentation of the model 

The  QUEST  model  is  a  global  dynamic  general  equilibrium  model  developed  by  the  European  Commission’s  Directorate-General  for  Economic  and 
Financial Affairs.28 The different model variants have been extensively used for macroeconomic policy analysis and research, e.g. analysing the impact of 
fiscal  and  structural  reforms  and  assessing  the  impact  of  cohesion  policies29.  QUEST  is  a  fully  dynamic  structural  macro-model  with  rigorous 
microeconomic foundations derived from intertemporal utility and profit optimisation. The model also accounts for frictions in goods, labour and financial 
markets.30 

Structure of the model 

QUEST belongs to the class of micro-founded dynamic general equilibrium (DGE) models that are now widely used in economic policy institutions and 
are seen as the latest step in the development of macroeconomic modelling. These models are forward-looking and intertemporal, i.e. current decisions 
account for expectations about the future, subject to budgetary, technological and institutional constraints. The intertemporal forward-looking aspects are 
particularly relevant for the analysis of R&D promotion policies.  

This evaluation uses the semi-endogenous growth version of the European Commission’s QUEST model with an R&D production sector (QUEST R&D). 
The model economy is populated by households, final and intermediate goods-producing firms, a research industry, a monetary and a fiscal authority. In 
the final goods sector, firms produce differentiated goods which are imperfect substitutes for goods produced abroad. Final good producers use a composite 
of  intermediate  goods  and  three  types  of  labour:  low-,  medium-,  and  high-skilled.  The  model  has  liquidity  and  non-liquidity  constrained  households. 
Liquidity constrained households have no access to financial markets. They simply consume their current income at each period. Non-liquidity constrained 
households buy the patents of designs produced by the R&D sector and license them to the intermediate goods-producing firms. The intermediate sector is 
composed  of  monopolistically  competitive  firms,  which  produce  intermediate  products  from  rented  capital  input  using  the  designs  licensed  from  the 
household  and  by  making  an  initial  payment  to  overcome  administrative  entry  barriers.  The  production  of  new  designs  takes  place  in  research  labs, 
employing high-skilled labour and making use of the commonly available domestic and foreign stock of knowledge. Importantly, the model is a global 

28  For  the  different  QUEST  model  variants  and  their  applications,  see  https://economy-finance.ec.europa.eu/economic-research-and-databases/economic-research/macroeconomic-
models/quest-macroeconomic-model_en 
29 See Varga, J., Roeger W. and in 't Veld, J. ‘Growth effects of structural reforms in Southern Europe: the case of Greece, Italy, Spain and Portugal’, Empirica, 2014, vol. 41, issue 2, 
323-363., Varga, J. and in 't Veld, J. (2011). ‘A model-based analysis of the impact of Cohesion Policy expenditure 2000-06: Simulations with the QUEST III endogenous R&D model’, 
Economic  Modelling, 28 (1-2), 647-663. and https://ec.europa.eu/info/business-economy-euro/economic-and-fiscal-policy-coordination/economic-research/macroeconomic-models_en 
for other publications using the QUEST model. 
30 As European Commission (2020) points out, one of the main strengths of the QUEST R&D model is its theoretically consistent micro-founded approach based on the forward-looking 
optimising behaviour  of rational agents. As a general equilibrium model, it  makes it possible to disentangle the  dynamic transmission channels of R&I policies, and to account for 
reallocation effects and policy trade-offs. European Commission, Directorate-General for Research and Innovation, Annicchiarico, B., Licandro, O., Mohnen, P., et al., Moving the frontier 
of macroeconomic modelling of research and innovation policy, Publications Office, 2020, https://data.europa.eu/doi/10.2777/34199 

12 

 
 
 
multi-country model of the EU Member States and the rest of the world, in which individual country blocks are interlinked with international trade and 
knowledge spillovers. 

Key assumptions for the ex post evaluation 

The simulations assume that countries contributed to the Horizon 2020 budget according to their GDP shares. For this evaluation, results were produced 
based on three scenarios depending on the financing of the Horizon programme. The first and the second scenarios assume tax-based financing via (i) lump-
sum taxes and (ii) value added taxes respectively. In the third scenario (iii), financing is done at the expense of an equivalent reduction in national public 
investment. Table 3 lists the key modelling assumptions. 

Table 4: Key assumptions of the QUEST model 

Budget size 

Horizon 2020 budget in constant prices 

Budget allocation across years 

Across countries and years of the Horizon programme’s active operations from 2014 to 2021 

Key assumptions (Horizon 2014-21) 

Spillovers 

Direct leverage effect 

Economic performance 

Financing  

Results 

International trade and knowledge spillovers, based on trade statistics and elasticities in the relevant literature 

Identical leverage of EU funding and national funding 

Identical performance of EU funding and national funding 

Increase in lump-sum or value added taxes or reduction in public investment 

As  lump-sum  and  value  added  taxes  are  among  the  least  distortive  taxes,  financing  productivity-enhancing  R&D  investments  from  these  resources  is 
unambiguously beneficial at the EU level (Figure 3.a). By changing from lump-sum tax financing to public investment cuts (e.g. roads, buildings), Member 
States lose the potential productivity effects of these public investments and the GDP results are lower both in the short and long run. 

The GDP effects  build up gradually,  as it takes  time for the positive productivity effects  to  emerge in  the production process  via the accumulation of 
intangible  (R&D)  capital.  Member  States  also  benefit  from  the  simultaneous  support  of  R&D  via  knowledge  spillovers:  the  international  diffusion  of 
innovations fosters intangible capital formation in the medium term. Short-term GDP effects are limited due to crowding-out effects at the beginning of the 
intervention period: R&D subsidies stimulate innovation by helping R&D-intensive companies attract more high-skilled labour from traditional production 
into research with higher wages. Because of supply constraints for high-skilled workers, part of the fiscal stimulus is offset by higher wages. In the first 
two scenarios, GDP gains are up to 0.18% by 2021 relative to the baseline and gradually decrease below 0.1% after the programming period due to the 
depreciation of tangible and intangible capital. Note that in the simulations, both EU- and nationally funded R&I have the same leverage and performance 
effects. 

13 

 
 
 
Figure 3: EU-27 GDP and employment impact of Horizon 2020, QUEST R&D results 

a.  GDP 

b. Employment  

Horizon spending % GDP

GDP - Lump-sum fin.

Empl. - Lump-sum fin.

Empl.- VAT fin.

GDP - VAT fin.

GDP - Public inv. fin.

Empl. - Public inv. fin.

%

0,20

0,15

0,10

0,05

0,00

-0,05

%

0,04
0,03
0,03
0,02
0,02
0,01
0,01
0,00
-0,01
-0,01
-0,02
-0,02

Source: European Commission, DG ECFIN, QUEST R&D simulations, 2023. The effects are expressed in percent deviation from the baseline. 

In the third scenario, the expected GDP effects are much smaller compared to the tax-based financing scenarios. The output effects are even slightly negative 
in the short run and close to zero in the medium term. Similar to R&D investments, public investment is also productivity-enhancing. Therefore, this type 
of financing not only has a direct negative demand effect in the short run but also reduces the productivity and supply effects in the medium to long run.31 
This scenario illustrates that financing matters: even productivity-enhancing R&D programmes should not be implemented at the expense of reducing 
productive public expenditure. 

Given that the programmes are simulated as productivity-enhancing measures, the model results suggest only a slight short-run increase in employment 
during the demand boost, which disappears with rising real wages in the medium to long run after the end of the implementation period (Figure 3.b). 

Limitations of the model 

Although the model is well-suited to simulating the effect of public financed subsidies on private R&D, it does not distinguish between research undertaken 
in private or public R&I entities. All R&D activities are carried out by a single R&D sector. Being an aggregate macroeconomic model, QUEST also misses 
the extensive regional details present in RHOMOLO. 

31 The productivity effects of public capital in the private sector’s production process are captured via the output elasticity of public capital. This elasticity is calibrated based on empirical 
estimates in the literature. Bom, P., and Ligthart, J. (2014), ‘What Have We Learned From Three Decades Of Research On The Productivity Of Public Capital?’, Journal of Economic 
Surveys 28: 889-916. 

14 

 
                                                       
 
 
 
 
 
1.3. RHOMOLO 

Presentation of the model 

RHOMOLO32 is  the macroeconomic model of the European Commission  focusing on EU regions. It  has been developed and maintained by the Joint 
Research Centre, in cooperation with the Directorate-General for Regional and Urban Policy. It is used for policy impact assessment and provides sector-, 
region-  and  time-specific  simulations  on  investments  and  reforms  covering  a  wide  array  of  policies.  RHOMOLO  is  built  on  a  micro-founded  general 
equilibrium approach and is used to provide a breakdown of results by region and sector. 

Structure of the model 

RHOMOLO is a spatial dynamic computable general equilibrium (CGE) model with new economic geography features. The version of the model used for 
this evaluation includes 276 EU NUTS-2 regions. Each region contains 10 economic sectors operating under monopolistic competition (apart from the 
agricultural sector and the public services sectors, which operate under perfect competition). Regional goods are produced by combining labour and capital 
with domestic and imported intermediates. 

Final goods are consumed by households, the government and investors. Each region is inhabited by a representative household supplying labour of three 
skill types, consuming and saving a part of their income. The government levies taxes, purchases public consumption goods, invests in the economy, and 
transfers resources to the various agents in the economy. Goods and services can either be sold in the domestic economy or exported to other regions. Trade 
between regions is associated with a set of bilateral regional transportation costs33. The RHOMOLO model incorporates imperfect competition in the labour 
market,  allowing  for  unemployment.  Wage  formation  is  modelled  with  a  wage  curve  (Blanchflower  and  Oswald,  1995),  so  that  lower  levels  of 
unemployment increase workers’ bargaining power, thereby increasing real wages34. 

The RHOMOLO model contains two types of capital: sector-specific private capital, and public capital available to firms in all sectors within each region. 
Sector-specific private capital is accumulated by private investors. The investment-capital ratio is a function of the rate of return to capital and the user cost 
of capital, allowing the capital stock to reach its desired level in a smooth fashion over time. Public capital is accumulated by the government by means of 
public investment. Public capital services enter the production function as an unpaid factor of production, meaning that all firms in all sectors enjoy the 
same level of public capital at no cost. Public capital is subject to congestion35, so that its efficiency decreases as production increases. 

32 Christou, T., Crucitti, F., García Rodríguez, A., Lazarou, N.J., and Salotti, S. (2023), ‘The RHOMOLO ex-post impact assessment of the 2014-2020 European research and innovation 
funding programme (Horizon 2020)’, JRC Working Papers on Territorial Modelling and Analysis, JRC133690. 
33 The transportation costs are modelled as iceberg costs and come from the estimates illustrated by Persyn, D., Díaz-Lanchas, J., and Barbero, J. (2022), ‘Estimating distance and road 
transport costs between and within European Union regions’, Transport Policy, 124, 33-42. https://doi.org/10.1016/j.tranpol.2020.04.006.  
34 Blanchflower, D.G., and Oswald, A.J. (1995), ‘An introduction to the wage curve’, Journal of Economic Perspectives 9(3), 153-167. DOI: 10.1257/jep.9.3.153. 
35 Fisher, W.H., and Turnovsky, S.J. (1998), ‘Public investment, congestion, and private capital accumulation’, The Economic Journal, 108(447), 399-413. https://doi.org/10.1111/1468-
0297.00294. 

15 

 
 
Table 5: How RHOMOLO models innovation 

How RHOMOLO models innovation 

➢  Public  expenditure  in  support  for  applied  research  is  introduced  into  the  model  as  a  reduction  in  the  user  cost  of  capital,  which  in  turn generates  an  increase  in  private  investments.  R&I 
expenditure is modelled as private investments. Hence, R&I spending generates demand for capital goods, leading to a temporary increase in the private capital stock (which depreciates 
at a 15% yearly rate). In addition, R&I spending leads to accumulation of an intangible knowledge capital stock (the increase is subject to a 5% yearly decay rate) positively affecting total 
factor productivity (TFP).  

➢  The impact of R&I expenditure on TFP through the accumulated knowledge capital stock is captured by a set of regional elasticities which are positively related to regional R&D intensity. The 
intuition is that firms in regions that are already spending much on R&D signal their pre-existing capacity to generate value from innovation activities. The range of the R&D elasticities is 
between 0.01 and 0.04, in line with the existing literature on the subject.36 

➢  Public expenditure in support for basic research is introduced into the model as an increase in public investment, leading to a temporary increase in the public capital stock (which depreciates at 

a 5% yearly rate). This affects the productivity of firms, as public capital enters the production function as an unpaid factor of production. 

➢  Expectations are assumed to be myopic and the model is solved sequentially, with stocks being upgraded at the beginning of each period. For this particular exercise, capital mobility within the 

EU was assumed, but no labour mobility. 

Key assumptions for the ex post evaluation 

Besides the modelling setup, the analysis for the ex post impact assessment of Horizon 2020 is constructed following the NEMESIS analysis, which is 
based on historical H2020 administrative data. The key assumptions retained for the simulation of the results are summarised in Table 4. 

Table 6: Key assumptions for the RHOMOLO model 

Budget size and allocation 
Support to basic vs applied research 
Regional spillovers 
Direct leverage effect 
Economic performance 
Financing 

Key assumptions 

Horizon 2020 budget in constant prices. Allocation across regions (NUTS2 level) and years 
It is assumed that 40% of the funds support basic research and 60% support applied research 
Regional spillovers are conditional on R&D intensity within the regions 
Identical leverage of EU funding and national funding 
Identical performance of EU funding and national funding 
Lump sum 

In RHOMOLO, the basic research funds are simulated via an increase in public investment, therefore leading to a temporary increase in the public capital 
stock of regions. Due to the role of public capital in the production function, besides the demand-side effect of increased (public) investments, this increases 
the productivity of firms. The applied research funds are assumed to reduce the user cost of capital, leading to an increase in private investment. This entails 
a demand-side effect, which also leads to a temporary increase in the private capital stock. Also, it is assumed that these R&I investments lead to an increase 
in TFP, with an elasticity which depends on R&D intensity, as explained above.  

36 See, for instance, Männasoo, K., Hein, H., and Ruubel, R. (2018), ‘The contributions of human capital, R&D spending and convergence to total factor productivity growth’, Regional 
Studies, 52(12), 1598-1611. DOI: 10.1080/00343404.2018.1445848.  

16 

 
 
 
The investment is assumed to be financed via lump-sum transfers. To mimic how the EU budget is financed, the regional contributions are proportional to 
the GDP weight of each region over the EU GDP. In other words, a region does not necessarily have to finance the programme with a contribution which 
matches the amount of H2020 destined to the region itself: the contribution depends on the proportion of EU GDP generated in the region. 

Results 

The  GDP  impact  increases  steadily  during  the  implementation  period  to  reach  a  peak  of  +0.189%  in  2021.  Subsequently,  it  declines  gradually  as  the 
monetary injection related to the programme ends, the increased private and public capital stocks depreciate and the temporary increase in TFP decays. In 
2050,  the  programme  still  has  relatively  small  residual  effects,  as  the  GDP  is  0.040%  above  its  initial  level  (Figure  4).  The  investment  also  leads  to 
improvements in employment, whose impact reaches a peak of +0.095% in 2020, amounting to almost 220 000 people37. 

Figure 4: Horizon 2020 impact over time on GDP and employment 

0,20

0,15

%

0,10

0,05

0,00

Source: JRC - RHOMOLO simulations, 2023. 

Employment

GDP

Figure  5  presents  the  territorial  distribution  of  the  GDP  impact  of  H2020  investment,  expressed  as  percentage  deviations  from  the  baseline  (i.e.  a 
hypothetical scenario without H2020) in 2022, 2030, 2040 and 2050. The GDP impact in 2022 is stronger in the regions with more H2020 investment. 
Specifically, the programme’s macroeconomic effects are relatively larger in the Scandinavian regions, in Central Europe and in the Iberian Peninsula. 
Moreover, in  most countries, the capital  city regions benefit more than the other regions,  a phenomenon particularly  evident  in  countries like Poland, 
Czechia, Slovakia, Bulgaria and Romania. Over time, in countries like Spain, Italy, Greece, and Poland, the effects gradually spill over to regions receiving 
relatively less H2020 funds. Nevertheless, the spillover effects remain mostly concentrated in the richest regions. The magnitude of the impact decreases 
over time due to: (i) the H2020 investments being limited to the period 2014-2022; (ii) the depreciation rates of the temporarily increased private and public 
stocks of capital; and (iii) the decay rate of the TFP improvements. 

37 The total number of employed people in the model base year in the EU-28 is about 231 826 000. 

17 

 
 
 
Figure 5: Territorial distribution of Horizon 2020’s GDP impact, 2022-2050 

Source: JRC - RHOMOLO simulations. 

18 

 
 
 
 
 
 
 
 
The RHOMOLO model is calibrated with data organised over the following 10 NACE Rev. 2 sectors (see Tables 7, 8 and 9 below). Sectoral results 
should be read keeping in mind that the shocks applied to the model in order to simulate the Horizon 2020 interventions are not sector-specific. 
Therefore, the results mainly reflect the steady-state data on sectoral production and the input-output relationships across sectors. We report value added 
because GDP is not available at the sectoral level. 

Table 7: RHOMOLO economic sectors 

Code  NACE Rev.2  

A 

Agriculture, forestry and fishing 

B-E 

Industry (except construction) 

C 

F 

G-I 

Manufacturing 

Construction 

Wholesale and retail trade, transport, accommodation and food service 
activities 

J 

Information and communication 

K_L  Financial and insurance activities, real estate activities 

M_N 

Professional, scientific and technical activities; administrative and support 
service activities 

O-Q 

R-U 

Public administration, defence, education, human health and social work 
activities 

Arts, entertainment and recreation; other service activities; activities of 
household and extra-territorial organizations and bodies 

19 

 
 
 
 
 
Tables 8 and 9 present respectively the value added and employment sectoral results, for selected simulation years.  

Table 8: H2020 sectoral value added impact in selected years 

2014 
2015 
% change w.r.t. baseline 
-0.003  0.017 
A 
-0.002  0.030 
B-E 
-0.006  0.027 
C 
0.110 
0.152 
F 
-0.006  0.026 
G-I 
0.067 
0.023 
J 
0.042 
0.007 
K_L 
0.052 
0.012 
M_N 
0.073 
0.064 
O-Q 
0.028 
R-U 
0.003 
2015 
Change w.r.t. baseline in mn €  2014 
50 
A 
137 
B-E 
601 
C 
1 135 
F 
684 
G-I 
462 
J 
921 
K_L 
821 
M_N 
1 942 
O-Q 
R-U 
136 
Source: RHOMOLO simulations (and DG REGIO for the H2020 contribution). 

2018 
0.093 
0.132 
0.116 
0.260 
0.113 
0.180 
0.140 
0.156 
0.115 
0.099 
2018 
274 
598 
2 614 
1 944 
2 995 
1 243 
3 064 
2 470 
3 041 
485 

2016 
0.039 
0.061 
0.054 
0.205 
0.052 
0.107 
0.073 
0.087 
0.094 
0.050 
2016 
113 
277 
1 213 
1 530 
1 384 
736 
1 595 
1 385 
2 484 
246 

2017 
0.066 
0.098 
0.087 
0.233 
0.085 
0.147 
0.108 
0.124 
0.101 
0.076 
2017 
193 
444 
1 952 
1 742 
2 237 
1 015 
2 362 
1 972 
2 665 
373 

-7 
-10 
-128 
825 
-155 
159 
144 
188 
1 687 
16 

2019 
0.123 
0.166 
0.144 
0.299 
0.143 
0.216 
0.173 
0.189 
0.132 
0.124 
2019 
360 
751 
3 236 
2 231 
3 770 
1 492 
3 785 
3 000 
3 484 
608 

2020 
0.157 
0.205 
0.180 
0.298 
0.178 
0.245 
0.208 
0.220 
0.127 
0.151 
2020 
461 
930 
4 034 
2 224 
4 694 
1 693 
4 540 
3 500 
3 366 
738 

2021 
0.190 
0.240 
0.210 
0.200 
0.209 
0.247 
0.230 
0.233 
0.065 
0.170 
2021 
559 
1 086 
4 708 
1 496 
5 539 
1 705 
5 031 
3 693 
1 714 
832 

2022 
0.197 
0.237 
0.200 
0.155 
0.202 
0.223 
0.221 
0.214 
0.049 
0.166 
2022 
577 
1 076 
4 482 
1 156 
5 337 
1 538 
4 839 
3 389 
1 310 
814 

2030 
0.149 
0.159 
0.119 
0.081 
0.132 
0.123 
0.155 
0.122 
0.028 
0.121 
2030 
439 
721 
2 675 
608 
3 479 
848 
3 383 
1 944 
748 
591 

2040 
0.092 
0.095 
0.071 
0.050 
0.079 
0.073 
0.093 
0.073 
0.017 
0.073 
2040 
270 
432 
1 594 
370 
2 100 
502 
2 046 
1 161 
451 
359 

2050 
0.056 
0.058 
0.043 
0.030 
0.048 
0.044 
0.057 
0.044 
0.010 
0.045 
2050 
165 
261 
969 
225 
1 274 
304 
1 236 
704 
273 
218 

20 

 
 
 
Table 9: H2020 sectoral employment impact in selected years 

2017 
0.033 
0.040 
0.047 
0.183 
0.037 
0.063 
0.026 
0.062 
0.062 
0.016 
2017 

2014 
-0.024 
-0.026 
-0.019 
0.181 
-0.021 
0.014 
-0.022 
0.001 
0.073 
-0.011 
2014 

2016 
2015 
-0.002  0.011 
0.015 
0.001 
0.023 
0.008 
0.190 
0.157 
0.016 
0.002 
0.030 
0.047 
-0.006  0.006 
0.042 
0.024 
0.060 
0.070 
-0.005  0.003 
2016 
2015 

% change w.r.t. baseline 
A 
B-E 
C 
F 
G-I 
J 
K_L 
M_N 
O-Q 
R-U 
Change w.r.t. baseline in thousands 
of persons 
A 
B-E 
C 
F 
G-I 
J 
K_L 
M_N 
O-Q 
R-U 
Source: RHOMOLO simulations (and DG REGIO for the H2020 contribution). 

-0.694 
-1.458 
-8.019 
24.132 
-10.263 
1.428 
-2.600 
0.262 
45.263 
-0.988 

0.940 
2.260 

2018 
0.049 
0.058 
0.064 
0.204 
0.053 
0.073 
0.041 
0.076 
0.071 
0.028 
2018 

2019 
0.065 
0.075 
0.077 
0.240 
0.069 
0.089 
0.058 
0.093 
0.080 
0.043 
2019 

2020 
0.095 
0.106 
0.106 
0.200 
0.096 
0.099 
0.082 
0.111 
0.063 
0.060 
2020 

2021 
0.140 
0.152 
0.138 
0.021 
0.134 
0.095 
0.122 
0.119 
-0.018 
0.085 
2021 

2030 
2022 
0.090 
0.135 
0.090 
0.143 
0.070 
0.125 
0.042 
0.024 
0.081 
0.127 
0.047 
0.084 
0.089 
0.124 
0.106 
0.059 
-0.016  0.002 
0.073 
0.093 
2030 
2022 

2040 
0.056 
0.055 
0.042 
0.026 
0.049 
0.029 
0.054 
0.035 
0.002 
0.045 
2040 

2050 
0.034 
0.033 
0.026 
0.016 
0.030 
0.017 
0.033 
0.022 
0.001 
0.027 
2050 

2.723 
5.933 

1.852 
4.189 

1.390 
3.257 

-0.066  0.305 
3.994 
8.517 
0.861 
0.055 
3.561 
10.009  19.989  27.236  32.880  45.057  58.975 
20.949  25.285  24.398  27.116  31.912  26.668  2.800 
7.713 
18.542  26.190  34.246  47.426  66.126 
0.869 
9.338 
8.824 
6.173 
2.993 
4.596 
-0.655  0.696 
14.191 
6.759 
2.967 
6.140 
10.900  15.965  19.617  24.065  28.506  30.790 
37.058  43.201  38.375  43.814  49.518  38.954  -11.001 
2.423 
-0.440  0.256 

7.176 
4.754 

9.797 
9.548 

1.360 

3.669 

5.206 

7.298 

5.643 

3.508 

2.580 
5.038 

1.598 
3.080 

3.874 
0.980 
1.881 
8.053 
53.173  29.781  18.025  10.987 
3.131 
2.140 
63.003  40.200  24.472  14.901 
1.723 
8.263 
2.828 
4.675 
3.810 
14.466  10.325  6.269 
5.543 
27.337  15.186  9.125 
0.662 
1.039 
-9.652  1.444 
2.353 
3.859 
6.308 
8.024 

Limitations of the model 

While the spatial dimension of RHOMOLO is a key strength of the model, its extensive regional disaggregation requires that the dynamics are kept relatively 
simple.  This  implies  that  the  optimisation  problems  in  RHOMOLO  are  inherently  static  and  do  not  acknowledge  the  intertemporal  consequences  of 
innovation decisions that can change not only the level but also the rate of growth of regional economies. This is solved by recursive dynamics. Furthermore, 
RHOMOLO does not explicitly distinguish between private and public R&D investments or between types of endogenous innovation. 

2.  Analysis of monitoring data 

21 

 
Monitoring flashes on Horizon 2020, presenting internal analysis on specific topics of interest, were also used to feed into the evaluation report.  

3.  Documentary review / desk research 

Extensive desk research was conducted to ensure background information, as well as to provide evidence that was then triangulated with other sources of 
information to draft the answers to the evaluation questions. Documents reviewed included legal texts, strategic documents, previous evaluations and policy 
analyses. 

4.  Analysis of unstructured data 

Two main types of unstructured analyses were carried out: text mining and research topic analysis. Text mining was used to analyse the impact section of 
H2020 periodic reports and extract key sentences/claims of impact made by the beneficiaries. This work resulted in a dataset of the expected impacts of 
H2020 projects. This data was used to identify in-depth analysis/interviews to be conducted within case studies.  

In parallel, research topic analysis was used to establish how far the framework programme actions addressed new or fast-growing research topics. Subject 
to  availability of monitoring data, each H2020 project  was linked to  these topics and a score  calculated. This  ‘FET’ (future and emerging tech) score 
captures the extent to which different H2020 projects address new and growing R&I topics.  

5.  Interviews  

The primary purpose of the interviews was to  collect evidence from the different  actors concerned by the  framework programme. This would give an 
objective assessment of what has happened by taking into account the different points of view. This method was used  in particular in case studies and 
international  benchmarks.  Interviews  were  also  conducted  to  confirm  and  complement  the  data  collection,  with  a  view  to  drafting  the  findings  and 
conclusions. Some 1 403 interviews were conducted (including some with the same actors on different topics), gathering the perspectives of Commission 
staff, Member States, associated countries, and a large range of stakeholders (universities, companies, umbrella organisations, etc.)  

6.  Targeted surveys  

Different surveys were designed to collect data on the framework programme: 

a) Survey of successful and unsuccessful applicants participating in parts of the Excellent Science programme parts  

The survey programme in the Excellent Science evaluation study consisted of six online questionnaires designed to collect data on the opinions and 
perceptions of successful and unsuccessful applicants to Horizon 2020. In total, 5 417 complete and 449 partial responses were received.  

The following groups were surveyed:  

•  Horizon 2020 beneficiary organisations (including beneficiary organisations that participated in MSCA, SEWP, INFRA and SwafS)  
•  Horizon 2020 unsuccessful applicant organisations (organisations that unsuccessfully applied for MSCA, SEWP, INFRA and SwafS)  

22 

 
•  Horizon 2020 MSCA IF fellows  
•  Unsuccessful Horizon 2020 MSCA IF applicants  
•  Horizon 2020 ERC principal investigators 
•  Unsuccessful Horizon 2020 ERC applicants.  

b) For the evaluation study on the European Framework Programmes for Research and Innovation for addressing Global Challenges and Industrial 
Competitiveness – Focus on activities for the Digital and Industrial Transition: Survey of H2020 LEIT applicants/participants, successful applicants 

A single questionnaire was drawn up. It targeted all stakeholders, namely all partners in the projects funded (successful applicants) as well as all partners 
in proposals that were not funded even though the proposals reached scores above the threshold (‘high-quality’ proposals). A total of 1 342 valid 
responses to the survey were received, giving an overall response rate of 7%.  

c) For the evaluation study on the European Framework Programmes for Research and Innovation for addressing Global Challenges and Industrial 
Competitiveness – Focus on activities related to the Green Transition 

The survey of successful applicants was designed during the inception phase and further enriched and developed to take into account various factors. In 
total, the survey received 1 333 entries, with 771 of respondents reaching the end of the survey. The analysis (Annex 7 to the study) was organised by 
societal challenge for relevance, effectiveness and EU added value and for all societal challenges together for the evaluation criterion of efficiency. The 
analysis was also organised by topic within each evaluation criterion and included corresponding graphs.  

The studies on Resilient Europe and Innovative Europe did not have a survey in their methodology. 

7.  Network analysis 

Network analysis performed in the Excellent Science and Innovative Europe evaluation studies involved analysis of the structuring effect of framework 
programme funding, which was based on author networks. It also included analysis of funding concentration in Horizon 2020. 

8.  Patent analysis 

This analysis – in the Innovative Europe study – served to explore patent productivity under the H2020 programme. Preparation of the Horizon 2020 patents 
dataset involved the download of data from the Cordis database38. It contained 2 003 unique applications for 1 714 unique patent families. Applications 
that covered the same or similar inventions were grouped into families; the technical content of these families is considered (almost) identical, and was 
therefore treated as a unique patent family, a unit of analysis. Subsequently, Cordis patents were matched with patents on the PATSTAT39 database. A total 
of 36 patents in Cordis patents data were not found on the PATSTAT database, leaving 1 678 patent families. Only patents whose priority date was after 

38 https://data.europa.eu/data/datasets/cordish2020projects?locale=en  
39 https://www.epo.org/searching-for-patents/business/patstat.html 

23 

 
 
the start of the project were considered (referred to as ‘foreground’ patents). Removing patents with a priority date earlier than the start of the project left 
445 patent families. Therefore, the 45540 foreground patents constitute the basis for the analysis. 

The analysis looked at the following key metrics: 
•  patent productivity, i.e. number of foreground patents produced by the H2020 pillar and programme; 
•  patent quality metrics based on: (a) number of forward citations; (b) patent family size; (c) number of claims/CPC classes by foreground H2020 

patents; 

•  H2020 patents that are triadic, i.e. patents registered with the European, US and Japanese patent offices. 

9.  Bibliometric analysis 

This method was used in the Excellent Science study and in the Digital and Industrial transition study to assess the Excellent Science indicators in Horizon 
2020, using the key impact pathways (KIP) framework of Horizon Europe, in particular KIP No 1: ‘Creating High-Quality New Knowledge’, the aim being 
to be able to compare later on the evolution of these indicators over time. KIP1 proposes the use of several indicators that capture and benchmark the 
productivity and impact of the sponsored scientific outputs and in particular high-quality peer-reviewed publications influential in their field and worldwide. 

The first indicator is a quantity- and productivity-focused indicator, measured by the number of peer-reviewed scientific publications. Based on the EC 
administrative  and  monitoring  data  submitted  by  Horizon  participants  combined  with  Scopus  and  Web  of  Science  databases,  data  and  values  for  this 
indicator were disaggregated by: type of scientific publication; Horizon pillar, cluster and component; type of Horizon Europe action; programme year; 
and field of science. 

The second indicator is the impact and quality indicator. It uses the citations accrued by each of the above validated publications: field- or category-weighted 
(normalised) citation impact indicators are one of the most sophisticated indicators in the modern bibliometric toolkit41. A ‘field-normalised’ citation score 
(such as MNCS/CNCI/FWCI) calculates an indication of the citation impact of a publication. It is calculated by comparing the number of citations received 
by a publication with the number of citations expected for a publication of the same publication year and subject field. A citation impact score of more than 
1.00 indicates that the publications in the treatment sample have been cited more than would be expected based on the global average for similar publications 
(the control). Based on Web of Science and Scopus information, data and values for this indicator can be disaggregated by: type of scientific publication; 
Horizon pillar, cluster and component; type of Horizon Europe action; programme year; and field of science. 

The third indicator also looks at impact but tries to define world-class science. It does this by using the accrued citations to calculate the number and share 
of peer-reviewed publications resulting from the projects funded by the EC-funded programmes and that are core contributions to scientific fields, calculated 

40 Note that two patents were under two different projects, and these two projects were under two different pillars; this affects the sum of the totals in the tables. Patent IDs 68536640 and 
62002358. 
41 Normalised citation impact indicators account for differences in citation accrual over time, differences in citation rates for different document ages (e.g. older documents are expected 
to have accrued more citations than more recently published documents), document types (e.g. reviews typically attract more citations than research articles) and subject fields (e.g. 
publications in medicine accrue citations more quickly than publications in mathematics). 

24 

 
 
as a percentage of top cited publications (PPTop1%)42. This indicator is similar to the second indicator, except for the citation threshold introduced to 
differentiate between highly cited publications and the remaining less cited and un-cited publications. Based on Web of Science and Scopus information, 
data and values for this indicator can be disaggregated by: Horizon pillar, cluster and component; type of Horizon Europe action; programme year; and 
field of science. As the PPTop 1% and 10% indicator is generally accepted, this indicator was calculated using either the platform CNCI (Web of Science) 
or the FWCI (Scopus). 

Table 10: List of key metrics and dimensions 

KEY METRICS 

Number of publications 

Number of projects 

Number of peer-reviewed scientific publications (indexed by Web of Science) resulting from the programme 

Number of projects funded by the programme 

DESCRIPTIONS 

Publications with at least one citation 

The number of publications that have been cited at least once 

Top 1% most cited publications 

The top 1% highly cited publications 

Field-normalised citation impact 

Field-normalised citation score of peer-reviewed publications resulting from the programme 

Open access publications 

n/a 

Dimensions 

Descriptions 

Call closure year 

Project status 

Thematic priority 

H2020 PILLARS 

H2020 programme 

Scientific discipline 

Call closure year 

The status of the project, including ongoing, closed or cancelled project types 

EU.0 – EU.5 and Euratom 

The three pillars of H2020, i.e. Excellent Science, Industrial Leadership and Societal Challenges 

Funding programmes under each pillar 

250 Web of Science research areas are classified into 7 scientific disciplines, following the method proposed by CWTS43 

In addition, in the green transition evaluation study, bibliometrics strategies have been deployed to measure the effectiveness of research activities in 
this area, at scale across thousands of SC2- to SC5-funded projects.  

Descriptive findings, which provide bibliometric findings for selections of funded projects (in case studies) or partnerships measure absolute achievements 
by research area or thematic area relative to selected comparators. These analyses serve to uncover where H2020 support in green transition calls stands 
out relative to other EU research not funded by the framework programme. Any outstanding performance under H2020 could relate to the programme 

42  Bornmann,  L.,  L.  Leydesdorff  and  R.  Mutz  (2013),  ‘The  use  of  percentiles  and  percentile  rank  classes  in  the  analysis  of  bibliometric  data:  opportunities  and  limits’,  Journal  of 
Informetrics, 7(1), 158–165. 
43 https://www.cwts.nl/pdf/nowt_classification_sc.pdf.  

25 

 
 
 
having successfully selected for projects/awardees that stand out from the reference populations and/or to the programme having exerted a positive effect 
on the performance of awardees. To assess the relative contribution of the latter factor in observed difference, a counterfactual analysis was performed (see 
below). In this context, it should be kept in mind that descriptive findings are not able to tell us whether any strong performances recorded by supported 
researchers were induced by H2020 support itself, or whether the supported researchers would have attained these achievements otherwise, with other 
funding opportunities. Demonstrations of high-calibre work for any indicator may reflect H2020 support, the peer-reviewed selection process of researchers, 
or neither or both of these causes. 

In the study on the proposal evaluation system for the EU R&I framework programme, bibliometric analysis was used to analyse the predictive power 
of  funding  decisions  by  looking  at  whether  proposal  evaluation  processes  are  effective  in  selecting  the  ‘best’  applicants  in  terms  of  their  subsequent 
bibliometric performance. The analysis also addressed bias by analysing differences in bibliometric performance levels before and after the award, for 
success overall and for groups characterised by gender and country. To do this, the study identified the publication output of 3 815 applicants between 2009 
and 2019 in the Web of Science databases; output including publications from before and after applications were submitted from 2014 to 2016. Research 
performance before and after application was analysed in terms of output (number of publications) and impact (metrics based on the number of citations). 
The change in performance was compared with the outcome of the application (successful or unsuccessful). A further consideration was whether gender 
or country affected the outcome of applications. 

10. Counterfactual analysis 

Introduction 

A counterfactual analysis assesses the existence of a causal relationship between an observed outcome (e.g. the employment growth of a firm) and the 
intervention (for instance, Horizon 2020) and provides an estimate of the impacts compared to what would have happened in the absence of the intervention. 

Before providing more details on the use of the methods for the ex post evaluation of Horizon 2020, we provide some relevant definitions using Crato and 
Paruolo (2019)44.  

Causality 

This is the sufficient link from one factor or event, the cause, to another factor or event, the effect. In econometric methods, a plausible establishment of 
causality  requires  some  type  of  experiment  or  the  construction  or  identification  of  some  counterfactual  situation  (see  below  ‘Counterfactual  impact 
evaluation (CIE)’). This allows for a reasonable comparison of what happened in the presence of a given factor with what happened or can be reasonably 
accepted as likely to have happened in the absence of the same given factor.  

Control group 

This is a group adequate for comparison with the group of units that were subject to a given policy (or treatment group, in statistical terminology). Prior to 
the policy intervention, the control group should display average characteristics that were otherwise similar to those of the group of individuals subject to 

44 Crato, N., & Paruolo, P. (2019), Data-Driven Policy Impact Evaluation, Springer Nature. 

26 

 
 
the measures. The identification of a control group is critical for measuring the effect of a policy intervention, as it indicates what the situation would be 
for the group subject to the policy intervention had the intervention not been implemented. (See also ‘Counterfactual impact evaluation (CIE)’.)  

Counterfactual impact evaluation (CIE) 

This refers to statistical procedures for assessing the effect of a policy measure and gauging the degree to which it attained its intended consequences. In 
randomised control trials, one compares the outcomes of interest of those having benefited from a policy or programme (the ‘treated group’) with those of 
a group that are similar in all respects to the treatment group (the comparison or ‘control group’) except in that it has not been exposed to that policy or 
programme. The comparison group seeks to provide information on what would have happened to the members subject to the intervention had they not 
been exposed to it – the counterfactual case. The difference in the outcome of interest between the treated and control groups provides information about 
the effect of the policy.  

Randomisation 

This refers to the assignment of individuals to a group or groups (such as treated and control groups) at random.  

Differences in differences (DiD) 

This CIE technique estimates the average treatment effect by comparing the changes in the outcome variable for the treated group with those for the control 
group, possibly controlling for other observable determinants of the outcome variables. As this technique compares the changes and not the attained levels 
of the outcome variable, it is intended to eliminate the effect of the differences between the two populations that derive from potentially different starting 
points. Take, for example, an impact evaluation of the relative impacts of two different but simultaneous youth job-training programmes in two different 
cities. One should not look at the net unemployment rate at the end of the programmes, because the starting values for the unemployment rate in the two 
cities may have been different. A difference-in-differences (DiD) approach instead compares the magnitudes of the changes in the unemployment rate in 
the two cities. A basic assumption of DiD is the common trend assumption, namely that treated and control groups would show the same trends across time 
in the absence of policy intervention. Hence the change in the outcome variable for the control group can be used as an estimate of the counterfactual 
change in the outcome variable for the treated group.  

The difference-in-differences  (DiD) method  was  used to  measure SMEs’  performance in  terms  of employment, turnover and labour productivity. The 
control group of firms included non-funded framework programme applicants just below the funding threshold that never received H2020 funding due to 
budgetary constraints (i.e. their applications passed the minimum proposal quality thresholds). The Orbis database was used to retrieve longitudinal financial 
data for the panel analysis. In Orbis, longitudinal data can be retrieved for up to 10 years, which means that information on turnover, employment and 
labour productivity are available for the period 2013-2021. The analysis focused on private SMEs, i.e. firms that were flagged as private for-profit entities 
and SMEs in the CORDA database, which are validated via the Orbis database as being for-profit firms. In addition, the Orbis size classification was also 
applied to build the sample. Firms with inactive status variable in Orbis were removed (either because the firm has been dissolved or entered into bankruptcy 
or liquidation, or has been subject to a merger or takeover). 

27 

 
 
 
Regression discontinuity design (RDD) 

This CIE technique exploits situations in which eligibility for the programme depends on certain observable characteristics, such as a requirement to be 
above (or below) an age threshold such as 40 years of age. Individuals close to the threshold on either side are compared, and the jump of the expected 
outcome variable at the threshold serves as an estimate of the local average treatment effect. As an example, consider an EU regulation that applies to firms 
above a certain size; regression discontinuity design (RDD) can be used to compare the outcome of interest, such as the profit margin, of treated firms 
above but close to the firm-size threshold with the same figure for control firms below but also close to the firm-size threshold. Firms that lie around the 
cut-off level are supposed to be close enough to be considered similar except for treatment status. RDD requires policy participation assignment to be based 
on some observable control variable with a threshold. RDD is considered a robust and reliable CIE method, with the additional advantage of being easily 
presentable with the help of graphs. Since the observations that contribute to identifying the causal effect are mainly those around the threshold, RDD may 
require large sample sizes. 

Propensity score matching (PSM) 

This CIE technique compares the outcome variable for treated individuals with the outcome variable for matched individuals in a control group. Matching 
units are selected such that  their observed characteristics (controls) are similar to  those of treated units. The matching  is usually operationalised via a 
propensity score, which is defined as the probability of being treated given a set of observable variables. As an example, imagine that one needs to evaluate 
the impact of an EU-wide certification process for chemical firms on firms’ costs. This certification process is voluntary. Because the firms that applied for 
the certification are more likely to be innovative enterprises, one should compare the results for the treated firms with those for similar untreated firms. One 
possibility is to define the control group by matching on the level of R&D spending. PSM requires a (comparatively) large sample providing information 
on many variables, which are used to perform the matching. 

Counterfactual analysis in the ex post evaluation of Horizon 2020 

It is important to stress that for such analyses a longer time frame would be ideal, not only due to the nature of innovation policies, which take longer to 
materialise than other fields, but also due to data reporting delays. Hence, a positive observed result at this stage might be an underestimation of the actual 
result observed over a longer timeframe. For the same reason, the absence of positive results should not be alarming at this stage. 

Firms 

The overall causal impact of the Horizon 2020 on firm-level outcomes has been assessed by the mean of a difference-in-differences (DiD) approach45, 
comparing the changes in outcomes for successful applicants to the changes in outcomes for unsuccessful applicants.46 The study employs information on 
participants retrieved from CORDA and from Orbis balance sheets data from 2010 to 2022. To increase comparability between the treatment and control 
group, the analysis excludes unsuccessful applicants with low quality proposals, leading to a sample of about 41 000 unique private companies that applied 
for Horizon 2020 funding. The counterfactual results suggest that firms receiving H2020 grants increased on average their employment level by about 20% 
(compared to non-funded firms) and their total assets and revenues by about 30% in the years following the receipt of the first grant (Figure 6). Effects are 

45 As developed by Callaway, Brantly, & Pedro HC Sant’Anna (2021), ‘Difference-in-differences with multiple time periods’, Journal of Econometrics, 225(2), 200-230. Allowing for variation in treatment 

timing and heterogeneous treatment effects.  

46 Mitra, Alessio and Niakaros, Konstantinos (2023), The Horizon Effect: A Counterfactual Analysis of EU R&I Grants, https://data.europa.eu/doi/10.2777/584781.  

28 

 
 
present even after 2.5 years (the average duration of a project in our sample). Such  a result confirms the positive and causal impact of the  framework 
programme on beneficiary companies’ growth.  

Figure 6: DiD estimates on the effect of Horizon 2020 on firms’ outcomes 

Source: Mitra, Alessio, and Niakaros, Konstantinos (2023), The Horizon Effect: A Counterfactual Analysis of EU R&I Grants, https://data.europa.eu/doi/10.2777/584781  
Note: Confidence intervals at 99% level. Doubly robust DiD estimates incorporate individual and time-fixed effects, NACE and country-fixed effects, as well as the number of times 
companies have applied to Horizon 2020 calls. 
A counterfactual analysis47 covering Horizon 2020 for-profit SMEs found effects on the employment and turnover of Horizon 2020 beneficiaries. The 
study employed a difference-in-differences (DiD) method. The control group included those programme applicants that were above the threshold (so they 
could have received EU funding) but did not receive it due to budgetary constraints.  

A crucial assumption in a DiD setting is that in the absence of H2020 participation, the economic performance of the treatment group follows the same 
evolution of the control group (‘the parallel trend assumption’). It is important to stress that for analysis, using the DiD method a longer time lag would be 
needed to claim causality with certainty; however, the study team did not have access to earlier data. 

When looking at the whole programme, the employment level of the treated firms grew by more than 4% after 2016 compared to the control group, at an 
increasing rate until 2020. Similarly, the treatment group experienced a higher growth of turnover than the firms in the control group after 2014. After 
2017, the treated firms grew more than 10% compared to the control firms. When restricting the analysis to the industrial leadership, firms in the treatment 
group experienced a higher growth of employment compared to the firms in the control group after 2014. In 2017, the treated firms grew 5% more than the 
control firms. The results for turnover are in line with the results for employment: firms in the treatment group experienced a higher growth of turnover 
than the firms in the control group after 2014. In 2018, the treated firms grew 12% more than the control firms. The study found no effect on productivity. 

47 Innovative Europe study (2023), Annex 4, https://data.europa.eu/doi/10.2777/467162.  

29 

 
 
 
When restricting the analysis to societal challenges, firms in the treatment group experienced a higher growth of turnover than the firms in the control group 
after 2014. In 2015, the treated firms grew 6% more than the control firms. The study found no effect on employment or productivity. 

SME Instrument 
A paper48 focusing on the first 3 years of the SME instrument, phase II, studies various firm-level outcomes using an RDD approach, comparing successful 
applicants close to the threshold for funding.  

The findings suggest that these grants lead to an increase in subsequent firm investment, particularly in intangible assets. They also result in a boost in 
innovation output, as indicated by a rise of between 15% and 31% in citation-weighted patents, which occurs both through the intensive and extensive 
margins. In other words, R&D grants not only affect firms already involved in innovative activities but also lead to more firms engaging in patenting. 
Additionally, R&D grants are a catalyst for subsequent equity investments, with firms experiencing over a 100% increase in likelihood to receive private 
equity, leading to more significant funding rounds and a higher number of deals. Moreover, these grants have a positive impact on the rate of firm growth 
(28-56%) and reduce the probability of failure by over 100%. 

European Innovation Council Pilot 

An assessment of the European Innovation Council (EIC) Pilot used a comparative and counterfactual analysis to  compare beneficiaries’ performance 
before and after the support and against a comparator group. Matching difference-in-difference (M-DiD) was applied to accelerator calls that took place in 
2018.  

The analysis only considered projects approved in 2018 since company performance needs to be observed at least 2 years after the treatment to consider 
projects’ state of advancement. The analysis assessed company performance between 2015 and 2020. The years between 2015 and 2017 constitute the pre-
treatment  period,  2018  the  treatment  period,  and  the  years  between  2019  and  2020  are  the  post-treatment  period.  The  control  group  was  built  using 
propensity score matching (PSM)49.  

Companies that received EIC support performed better than other applicants when compared for some key performance parameters. Data show that on 
average, before accessing the EIC, beneficiary companies were smaller and on a growth path that strengthened after the EIC support. This result can be 
interpreted as a positive sign of the programme’s capacity to spot and select entrepreneurial and innovative talent. The counterfactual analysis results show 
a positive causal relationship between the EIC support and the companies’ capacity to generate additional jobs. 

However, it is important to stress that a more revealing analysis of economic impacts should be based on a longer time horizon than was possible for this 
evaluation. The short-term positive impact on beneficiary companies’ turnover and staff reveals the immediate effects of the grant and cannot be considered 
a sign of successful product commercialisation. 

Researchers: FP7 long-term impact 

48 Santoleri et al., The Causal Effects of R&D Grants: Evidence from a Regression Discontinuity. 
49 EIC Pilot study, technical annex, https://data.europa.eu/doi/10.2777/645064. 

30 

 
 
A recent paper50 estimates the causal impact of receiving a European Research Council (ERC) grant on researchers’ productivity, excellence, research 
networks and the ability to obtain additional funding up to 9 years later. The authors collected information on winning and non-winning ERC applicants 
between 2007-2013 as well as information on all their publications on Scopus until April 2021. 

When looking around the funding threshold (in a regression discontinuity design fashion), i.e. at researchers ranked last among those funded compared 
with those who ranked first among the non-funded, this study51 finds that obtaining an ERC grant does not improve researchers’ productivity (number of 
publications), excellence (h-index, publications in top 1% or top 10% ranked journals) or the research network. Nevertheless, those receiving an ERC 
advanced or starting grant acknowledge on average more EU funds (the ‘Matthew effect’) than the rejected applicants do, in particular in the fields of 
physical science and engineering and of social sciences and humanities. However, when the authors look at the total number of funds acknowledged (both 
EU and all others) there are no significant differences.  

Moreover, comparing the evolution of bibliometric outcomes before and after receiving the ERC grant between beneficiaries and non-beneficiaries (in a 
difference-in-difference  fashion)  the  study  finds  that  overall  ERC  grants  increased  research  productivity  in  the  long  term,  namely  up  to  9  years  after 
receiving the grant (i.e. on. H-Index and publications in top 1% and top 10% ranked journals). When looking at different academic fields, some areas, like 
universe  and  earth  sciences,  chemistry,  medicine,  human  mind  studies  and  institutions  and  behaviour,  showed  positive  effects  on  productivity  and 
excellence, depending on the measure used (i.e. number of papers, number of paper in top 10 % or top 1% ranked journals, H-index) and type of grant 
received (i.e. starting or advanced grant). The results of the DiD model also confirmed the evidence of a ‘Matthew effect’ across all fields with researchers 
who received an ERC grant. They were more likely to obtain other EU grants by themselves (or through their co-authors) even if the total number of funds 
they received was similar to that of non-beneficiaries (i.e. they compensate by seeking other types of funds and in the 9 years after applying to the ERC). 

11. Benchmarks 

Benchmarking activities provided evidence to inform evaluation questions set out in the Terms of Reference for the impact area evaluation studies. They 
identified  lessons  learnt  from  best  practices  worldwide  supporting  research  and  innovation  and  they  put  in  perspective  the  framework  programme’s 
performance in the area covered by the study. As a minimum, the benchmarks were based on data analysis and document review, completed by interviews 
with relevant stakeholders. The following benchmarks were used in the staff working document: 

Table 11: Benchmarks 

International benchmark  

Comparison with: 

Evidence from the benchmarking exercises demonstrated that the publication citation scores 
of Horizon 2020 were higher than all the other international funders in most of the analysed 
disciplines  –  both  in  terms  of  the  share  of  top  1%  most  cited  publications  and  average 
normalised citation score of its publications.  [Excellent Science study] 

NOW  (the  Netherlands),  the  French  National  Research  Agency  (ANR),  the 
Australian Research Council (ARC), the FCT (Portugal), the Austrian Science 
Fund  (FWF),  the  Natural  Sciences  and  Engineering  Research  Council  of 
Canada (NSERC) and the National Science Foundation (USA)  

Benchmarking related to the direct leverage was also conducted cautiously by DG RTD, to 
compare between same types of leverage (in this case direct leverage, and not investments 

National  Science  Foundation  (USA)  and  ERDF  funding  for  SME 
competitiveness (EU) 

50 Ghirelli C., Havari E., Meroni E. and Verzillo S. (2023) The Long-Term Causal Effects of Winning an ERC Grant, JRC133001. 
51 Ghirelli C., Havari E., Meroni E. and Verzillo S. (2023) The Long-Term Causal Effects of Winning an ERC Grant, JRC133001. 

31 

 
 
after the projects), and similar technology readiness levels (TRLs). Similar direct leverage 
factors were found. In particular,  

-  The U.S. National Science Foundation generally does not allow voluntary committed 
cost sharing in its proposals. This means a funding rate of 100%, and hence a direct 
leverage factor of 0, similar to Horizon 2020 for this type of research (accounting for a 
third of Horizon 2020 funding).  

-  ERDF funding for SME competitiveness has a leverage factor of 0.43. However, this 
funding  is  distributed  to  the  Member  States  and  regions,  rather  than  directly  to  the 
SMEs. For Horizon 2020, the leverage factor of funds going to SMEs is 0.34.  

12. Case studies 

Overall, 154 case studies were conducted, covering the different programme parts, as well as specific aspects of Horizon 2020 such as cross-cutting issues, 
the EIC Pilot and partnerships. The impact area evaluation studies had 15 case studies each, while the study on relevance and internal coherence had 20 
case studies, the study on external coherence – 23, the study on cross-cutting issues – 14, the EIC Pilot evaluation – 15, and the study on the proposal 
evaluation system for the R&I programme – 7.  

13. Policy workshops 

Some 20 policy workshops were conducted to support this evaluation. The workshops were implemented in the context of the independent external studies. 
They were used to consolidate and increase the robustness of the findings and conclusions arising from the data collection conducted through other methods, 
addressing evidence gaps whenever needed.  

14. Public consultation 

The public consultation on the ex post evaluation of Horizon 2020 was part of a larger joint consultation exercise looking at the past present and future of 
the R&I framework programmes (ex post evaluation Horizon 2020, interim evaluation Horizon Europe and the 2025-2027 strategic plan). In full compliance 
with the Better Regulation requirements, the online questionnaire was published, among other places, on the Have your Say portal52, also offering the 
possibility to submit position papers. It ran from 1 December 2022 until 23 February 2023. 

Excluding the respondent characteristics questions, the section on the ex post evaluation of Horizon 2020 included 20 questions. Among the 20 questions, 
there were 5 questions where respondents could give further insights in an open answer box. In total, the Commission received 2 788 responses to the 
questionnaire, with 1 818 replies received for the Horizon 2020 section. Statistical analysis of closed questions was performed to explore differences in 
opinion between types of respondents. The affiliations of respondents (e.g. SME, academia, public authority) are self-reported, and were not verified. The 
analysis of the open questions followed a qualitative method of approach. The questions were analysed either with the use of basic qualitative analysis (i.e. 
grouping respondents into broad groups and reading the responses to get an overview of the key themes) or with the use of a word cloud function (the 
function is used to indicate the most frequently used words in the qualitative responses and present the results graphically).  

52 https://ec.europa.eu/info/law/better-regulation/have-your-say_en  

32 

 
 
In addition to the online questionnaire, 229 position papers were received as part of the public consultation. After the compilation of all the position papers 
and the first screening and the removal of duplicates, campaigns and documents not addressing the Horizon 2020 programme directly (e.g. promotional 
material), 21 papers were analysed. The factual summary report and position papers have been published on Have your Say portal53. The analysis followed 
a qualitative method of approach. Based on a sample of position papers, a coding frame of broad themes was constructed. The final analysis per theme and 
any emerging sub-themes was conducted by type of respondent: academia, research organisations, public authorities, businesses, NGOs, citizens and others. 

Overall limitations of the Horizon 2020 ex post evaluation 

Methodological and data limitations were identified (listed below). Thanks to thorough checks ensuring that data is robust, these limitations did not affect 
the overall reliability of the analysis and the findings. Nevertheless, the evaluation faced the following challenges and limitations: 
a)  The ongoing implementation of 40% of projects represents a challenge for assessing impact (a challenge faced also by the ex post evaluation of FP 7 
when 50% of projects were still ongoing). This is compounded by the long period of time needed to contribute to impact in the field of R&I. However, 
the legal basis determined when the evaluation must be done - subsequent evaluations can follow up and assess the long-term impact of Horizon 2020. 
b)  Some parts of the programme, such as the EIC Pilot, were implemented at the end of the programme, and so only future analysis can provide more 

extensive information on its impacts.  

c)  As noted already in the interim evaluation, Horizon 2020 indicators refer only to parts of the programme’s intervention logic. There are limitations on 
data on the programme’s outcomes and impact (for example, most Horizon 2020 indicators focus on input/results but not on the uptake of the results 
or their impact). The set of official programme indicators did not cover the entire intervention logic equally across impact areas: economic effectiveness 
was thus easier to assess as official indicators focused on contribution to company turnover, employment and GDP. Similar indicators are not available 
for assessing contributions to the Innovation Union flagship or other EU policies.  

d)  Indicators are defined in a suboptimal way, as they are not systematically accompanied by baseline values (i.e. values before the programme) or by 
target values (i.e. expected values at the end of the programme). They are also sometimes ambiguous, particularly when presented in short form (e.g. 
KPIs 9, 10, 11). 

e)  Some cross-cutting issues did not have accompanying indicators, while others are so complex that the selected KPIs cannot really provide a complete 

picture of progress (i.e. ERA, Innovation Union, Widening participation).  

f)  Some indicators addressed only actions supported by a single DG, programme part or type of action, and their data could not be aggregated with data 
collected for other indicators (which addressed another target group, for example). This led to challenges in ‘telling the story’ of what the programme 
achieved as a whole, without delving into its parts. There were also challenges regarding: (i) the aggregation of datasets (for example, most indicators 
are collected for specific programme parts only and not for the whole programme, and monitoring data covering the entire programme come from 
various data sources, which are difficult to aggregate); and (ii) the disaggregation of datasets (for example, to see the results of partnerships alone).  
g)  Where possible, key performance indicators were to be calculated ad hoc for the evaluation, in most cases from Commission monitoring systems. For 
a few indicators (KPIs 1, 4, 9-10-11, 22-23), figures were provided by other Commission departments and implementing authorities. Occasionally, 
data from the 2022 programme statement for the framework programme are used, as publication of the Horizon 2020 data in the programme statement 
of 2023 was suspended. 

53 https://ec.europa.eu/info/law/better-regulation/have-your-say/initiatives/13444-Horizon-2020-programme-final-evaluation/public-consultation_en  

33 

 
 
h)  Monitoring data recorded in the Commission IT systems can minimise but not completely exclude cases of multiple counting. This is particularly true 
for  indicators  reported  in  aggregate  form,  such  as  innovations  and  dissemination  activities.  Granular  reporting  on  intellectual  property  rights  and 
publications  is  provided  voluntarily  by  beneficiaries.  This  may  result  in  inaccuracies,  such  as  under-reporting  (especially  for  patents)  and  poor 
formatting of data, which detract from the accuracy of counting and matching to external data sources. Both these points are examined more in depth 
below in this section. 

i)  Some specific weaknesses were identified for external impact evaluation studies: 

o  The steering committees agreed that the data collected were adequate for their intended use and robust, despite being somewhat limited. (i.e. not 
all data is available in eCorda or Innovation Radar). In some areas of the studies, the evaluators collected too little additional data/evidence, in 
addition to what was provided by the Commission and the implementing bodies.  

o  In evaluation studies, the analysis of data on IPR outputs was suboptimal as it did not systematically filter out obvious reporting errors. 
o  Use of external data sources: presentation of figures from Orbis, Crunchbase and Dealroom databases could have acknowledged the match rate 

and explained issues in more detail, e.g. in terms of geographical representation.  

o  Evaluation  studies  included several  attempts to  identify causal  effects  of participation  in  the  framework  programme. However,  expert review 
highlighted critical points weakening the relevance of their findings. Typical shortcomings include: no distinction made between mono-beneficiary 
and  multi-beneficiary  schemes;  short  time  series,  with  no  data  on  firm  performance  available  to  the  study  team  before  2013;  not  enough 
transparency on the characteristics of the control group and on limitations connected to matching to external data sources.  

o  The analysis of equity raised could have been more granular, with reference to the type of funding entities, with particular attention to other public 
funding,  and  to  the  origin  of  such  funding  and  to  the  type  of  entities  funded.  Limitations  of  matching  to  external  sources  should  also  be 
acknowledged more precisely. For the analysis on equity funding rounds, as no all-encompassing databases exist, contractors could have cross-
checked the validity of the results with other data sources such as Crunchbase and Zephyr.  

o  Occasional uneven thematic coverage in some areas of the programme: for example, findings and conclusions in the area of Societal Challenge 1 
were  too exclusively focused on research into pharmaceutical and medicinal results, with findings of research into demographic change and well-
being remaining extremely limited (e.g. research related to mental health); also, findings and conclusions in digital and industrial technologies 
were highly developed for the LEIT programme parts, but weaker for coverage of other programme such as Pillar I bottom-up schemes. 

A monitoring system based on self-reporting has inherent limitations. It is naturally prone to inaccuracies, especially when reporting questions may be 
interpreted differently by project  participants. This  limits the usability of figures reported in  aggregate form by project  beneficiaries, such as non-IPR 
innovation outputs, knowledge spillovers to other entities, and dissemination and exploitation activities.  

Systems allowing for granular reporting, such as the one in place for publications and IPR, are less affected by the issue above, as they allow for easier 
identification and quality control of unique outputs. However, they remain prone to discretionary under-reporting, which could be temporary (delayed 
reporting) but could also be permanent. For example, since there is no legal obligation to declare IPR outputs supported by EU funding, applications linked 
to framework programme projects may well never be reported. At the same time, the monitoring system suffers from a specific type of over-reporting: 

34 

 
participants indicate IPR applications filed before or around the start date of the project. These ‘background patents’ (see Glossary), which are an input 
of research rather than its output, are in fact a common occurrence, especially at the beginning of a new framework programme54.  

In this staff working document, we report figures on IPR from the monitoring system ‘as they are’ to ensure consistency with baselines and with other 
monitoring reports such as yearly programme statements55. Filtering out ‘background patents’ is an exercise subject to interpretation, as using the project 
date alone is not a guarantee that the IPR declared is a genuine project output56. 

Much of the analysis on innovation diffusion from the framework programme is based on analysis of patents and other IPR. However, IPR productivity 
is an indicator to monitor in the longer term. First, there is a considerable time lag between achieving research and innovation results and applying for 
protection. Evidence from earlier framework programmes shows that participants report them generally towards the end of the projects, and often well after 
project activities are finished57. This is even more important if the subject of interest  is awarded IPRs: as also remarked in the Horizon 2020 interim 
evaluation, the European Patent Office may take up to 5 years to grant a patent58. With such a long time lag, it is difficult at this stage to assess basic patent 
quality indicators, such as the number of citations or their economic value. 

Second, IPR figures obtained right after the end of a framework programme – even 2 years later, as for this evaluation – are far from final. As 
shown in Section 4.1.5 of the staff working document, the number of reported IPR applications in FP7 is currently much higher than known at time of the 
ex post evaluation for FP7. The difference with the ex post evaluation is not only composed of applications filed from 2015 onwards, but also of delayed 
project reporting. A sizeable number of patent applications (around 1 200) were indeed filed before 2015, but were reported by applicants later, and are 
hence not included in our FP7 baseline.  

As regards the public consultation, the views represented by the respondents are not representative of the EU population due to self-selection bias. 

To overcome/mitigate these limitations, the evaluation report is transparent in indicating its data sources. Evaluation results have not been based on one 
data source, but findings were triangulated using various data sources.  

54 European Commission, ‘Patents in the Framework Programme – From Horizon 2020 to Horizon Europe’, R&I monitoring and evaluation flash, August 2020, p. 5. 
55 European Commission, Programme Statement 2022, COM(2021) 300, June 2021, https://commission.europa.eu/system/files/2022-07/ps_db2023_he_h1_1.pdf 
56 The referenced Commission report on patents in the FP (‘Patents in the Framework Programme – From Horizon 2020 to Horizon Europe’, 2020) considers ‘background’ applications 
filed until one year after project start. 
57 European Commission, Directorate-General for Research and Innovation, Patents in the Framework Programme: from Horizon 2020 to Horizon Europe, Publications Office of the 
European Union, 2022, https://data.europa.eu/doi/10.2777/789674. 
58  European  Commission,  Interim  evaluation  of  Horizon  2020,  2017,  p.  133,  https://research-and-innovation.ec.europa.eu/knowledge-publications-tools-and-data/publications/all-
publications/interim-evaluation-horizon-2020-key-documents_en. 

35 

 
 
 
 
 
ANNEX III. EVALUATION MATRIX AND, WHERE RELEVANT, DETAILS ON ANSWERS TO THE EVALUATION QUESTIONS (BY CRITERION) 

Horizon 2020 official indicators are highlighted in blue in the matrix below. For more information, see the following publication: European Commission, Directorate-General for 
Research and Innovation, Horizon 2020 indicators: assessing the results and impact of Horizon, Publications Office, 2015, https://data.europa.eu/doi/10.2777/71098  

Indicators marked in orange come from Regulation (EU) No 1291/2013 of the European Parliament and of the Council of 11 Decemb er 2013 establishing Horizon 2020  – the 
Framework Programme for Research and Innovation (2014-2020), https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32013R1291  

Evaluation questions 

1.1.  How effective was 
Horizon 2020 
implementation? 

Judgement criteria:  
extent to which… 
Horizon 2020 
strengthened Europe’s 
science base  

1.  Effectiveness 

Indicators and where available – targets 

Main data sources 

Share of researchers in the total labour force  

% of publications from ERC-funded projects which are 
among the top 1% highly cited [KPI 1, no baseline – new 
under Horizon 2020, target: 1.8%, achieved value: 6.4%] 
% of publications published in the top 10% impact ranked 
journals at FET level [KPI 2, no baseline – new under 
Horizon 2020, target: 25 publications per EUR 10 million 
funding, achieved value: 25.4 publications per EUR 10 
million funding]  
Number of researchers undertaking (a) international mobility 
under the MSCA, and (b) mobility between academic and 
non-academic sectors [KPI 4, baseline: 50 000 researchers 
(2007-2013) – out of which 20% PhD, target: for the MSCA: 
65 000 researchers – out of which 25 000 PhD candidates, 
achieved value: 49 475 unique researchers – of which 25 676 
PhDs] 
KPI 5: Number of researchers who have physical or remote 
access to research (e)infrastructures  

•  Physical  access:  baseline:  22 000  researchers,  target: 
20 000 additional researchers, achieved value: 24 235 
•  Share  of  researchers  with  access  to  e-infrastructures:  no 

baseline or target, achieved value: 35.6% 

Number of joint public-private publications – LEIT [KPI 8, 
no baseline – new under Horizon 2020, no target, achieved 
value: 10 907] 

36 

Eurostat (RD_P_PERSLF), Professional position: 
Researchers, Percentage of population in labour force – 
numerator in FTE 

European Research Council Executive Agency, Annual 
Activity Report 2022, p. 6  

Commission monitoring system (CORDA), data on 
24 April 2023 

Excellent Science study, Section 4.4.3, Table 19. Figure 
based on CORDA data, updated on 19 January 2023 

Commission monitoring system (CORDA), data on 
24 April 2023 

Commission monitoring system (CORDA), data on 
24 April 2023  

 
 
 
 
 
 
Horizon 2020 has 
spread excellence and 
widened participation 
in the programme 

Horizon 2020 has 
boosted Europe’s 
industrial leadership 
and competitiveness 

Science with and for Society – SWAFS: Number of 
institutional change actions promoted by the programme 
[KPI 21, no baseline – new under Horizon 2020, target: 100, 
achieved value: 381 (total) – 348 (Member States only)] 
JRC’s annual number of peer-reviewed publications in high-
impact journals [KPI 23, baseline: 460, target: 500 in 2020, 
achieved value: 548] 
Publication citation scores of Horizon 2020 compared to 
other international funders (share of top 1% most cited 
publications and average normalised citation score of 
publications) 
Number of awards and prizes won by ERC grantees 
(including the Nobel Prize, Wolf Prize and Fields Medal) 
% of public consultation respondents who responded that 
Horizon 2020 contributed to the planned scientific effects 
‘somewhat’ or ‘to a great extent’ 
Spreading Excellence and Widening participation – SEWP: 
Evolution of publications in high-impact journals in the 
given research field [KPI 20, no baseline – new under 
Horizon 2020, no target: achieved value: 1 263 before EU 
funding, 3 098 after EU funding] 

EU research and development expenditure relative to GDP 
[target: 3%] 
The innovation output indicator in the context of the Europe 
2020 strategy  
% of FP contribution to GDP in the EU: average annual 
GDP gain and medium gain 

% of FP contribution to employment in the EU: average 
annual employment gain and medium gain 

Commission monitoring system (CORDA), data on 
24 April 2023 

Ex post evaluation of the activities of the Joint Research 
Centre under Horizon 2020 and Euratom 2014-2020 

Excellent Science study – benchmarking exercise (based 
on bibliometric analysis) 

Excellent Science study  

Replies to stakeholders’ consultation  

Commission monitoring system (CORDA), data at 
24 April 2023 

European Commission, Programme Statements 2022 

Science, Research and Innovation performance of the EU, 
2022 report 

Macromodelling analysis. Models: RHOMOLO (by DG 
Joint Research Centre), QUEST (by DG Economic and 
Financial Affairs), NEMESIS (by SEURECO ERASME 
in the Innovative Europe study) 

Macromodelling analysis. Models: RHOMOLO (by DG 
Joint Research Centre), QUEST (by DG Economic and 
Financial Affairs), NEMESIS (by SEURECO ERASME 
in the Innovative Europe study) 

Number of patent applications and awards in FET per EUR 
10 million of funding [KPI 3, no baseline – new under 
Horizon 2020, target: 1, achieved value: 0.84 applications 
and 0.55 patents awarded]  

Commission monitoring systems (CORDA), data on 
24 April 2023, Group of Independent Experts, Assessing 
the Influence of ERC-funded Research on Patented 
Inventions (2022) 

37 

 
Number of patent applications and patents awarded in the 
different enabling and industrial technologies per EUR 10 
million of funding – LEIT [KPI 6, no baseline – new under 
Horizon 2020, target: 3, achieved value: 0.56 applications 
and 0.38 patents awarded] 
Percentage of participating firms introducing innovations 
new to the company or to the market – LEIT [KPI 7, no 
baseline – new under Horizon 2020, target: 100% of 
participating firms, achieved value: 124.2%59] 
Total investments mobilised by Horizon 2020’s debt and 
equity facilities (InnovFin) [KPI 9, KPI 10: no baseline – 
new under Horizon 2020, target: EUR 25 billion investments 
mobilised; achieved value: EUR 77.5 billion as of April 
2022 (debt and equity combined)] 

Number of organisations funded and amount of private funds 
leveraged by Horizon 2020’s debt and equity facilities 
(InnovFin) [KPI 11, baseline: 300, target: 5 000 
organisations funded and EUR 35 billion of private funds 
leveraged, achieved value: 37 921 as of April 2022, 
EUR 43.6 billion in private funding] 
Number and percentage of participating SMEs introducing 
innovations new to the company or the market (covering the 
period of the project plus 3 years) [KPI 12, no baseline – 
new under Horizon 2020, target: 50%, achieved value: 
118%] 
Growth (turnover) and job creation in participating SMEs 
[KPI 13, no baseline – new under Horizon 2020, no target, 
achieved value: +4.2% turnover growth, +5.2% direct 
employment growth]60 
Number of organisations from universities, business and 
research integrated in the KICs [EIT KPI 1, baseline: 200 

Commission monitoring systems (CORDA), data on 
24 April 2023 

Commission monitoring systems (CORDA), data on 
24 April 2023 

EIB group estimate based on implementation data. 
Figures reported in European Commission, DG Research 
and Innovation annual activity report 2021, p. 47. 
Available at 
https://commission.europa.eu/system/files/2022-
05/annual-activity-report-2021-research-and-
innovation_en.pdf 
EIB group estimate based on implementation data, 
reported in annual activity report, p. 47. Available at 
https://commission.europa.eu/system/files/2022-
05/annual-activity-report-2021-research-and-
innovation_en.pdf  

Commission monitoring systems (CORDA), data on 
24 April 2023 

Commission monitoring systems (CORDA), data on 
24 April 2023 

European Commission, Programme Statements 2022 
(indicator 1 on p. 70), Innovative Europe study.  

59 The figure is reported as it appears in the monitoring system. Ambiguous reporting questions led beneficiaries to indicate more firms than unique participants in the programme. Since 
data are reported by beneficiaries in an aggregate way, no ex post controls on the figure are possible. 
60 Each SME is considered per each time it participated in a grant. The value reported should be intended as an average of all variations between turnover or employment at project start 
against  the  most  recent  value  reported by  the beneficiary  (which  may  vary from  project  to project depending  on  reporting  date).  Turnover  and  employment  values  are used  for  the 
calculation as they are self-reported by participants and may be inconsistent with official sources. 

38 

 
 
organisations in the 2010-2013 period, target: 1 200, 
achieved value: 2 153 in 2020] 
Number of new graduates from EIT-labelled PhD and 
Masters programmes [EIT KPI 2, achieved value: 3 845] 
Number of business ideas incubated [EIT KPI 3] 

Innovative Europe study (2023), Table 14, p. 45 

Innovative Europe study  

Sum of knowledge transfers/ adoptions [EIT KPI 4] 

Innovative Europe study  

Number of new or improved products/services/ processes 
launched onto the market as a direct output of a KIC activity 
[EIT KPI 5, achieved value: 1 501] 

European Commission, Programme Statements 2022 
(Indicator 2 on p. 71: collaboration inside the knowledge 
triangle leading to the development of innovative 
products, services and processes) 

Innovative Europe study 

Innovative Europe study  

European Commission, Programme Statements 2022 
(Indicator 2 on p. 71: collaboration inside the knowledge 
triangle leading to the development of innovative 
products, services and processes) 

Innovative Europe study 

Replies to stakeholders’ consultation  

Replies to stakeholders’ consultation  

Replies to stakeholders’ consultation  

Attractiveness of education programs: Ratio of the number 
of eligible applicants for EIT-labelled PhD and Masters 
programmes divided by the number of available places on 
these programmes [EIT KPI 6] 
Number of start-ups and spin-offs created as a direct output 
of a KIC activity [EIT KPI 7, baseline: 33, target: 66, 
achieved value: 305 start-ups and spin-offs across Climate 
KIC, EIT InnoEnergy, EIT Digital, EIT Health and EIT Raw 
Materials in 2010-2016; additional 36 start-ups created by 
students enrolled in and graduates of EIT-labelled MSc and 
PhD programmes in 2017-2020; 99 start-ups created as a 
result of innovation projects for the indicated KICs] 
% of public consultation respondents who responded that 
Horizon 2020 contributed to the planned economic effects 
‘to a great extent’ 
Communication: % of public consultation respondents who 
rated ‘communication activities on Horizon 2020 to attract 
applicants’ as ‘very good’ or ‘good’ 

Exploitation: % of respondents who reported the following 
developments since the end of their grant: (a) the project 
results have contributed to opening a new stream of research 
and innovation in the institution; (b) the project results have 
been/will be used in other (follow-up) projects; (c) the 
project results have been/will be commercialised (i.e. used to 
create new products and/or services); (d) the project results 
have been/will be used to improve available products and/or 
services 

39 

Dissemination, 
exploitation and 
communication 
measures have made it 
possible to reach these 
outcomes and impacts 

 
 
 
 
Dissemination: % of respondents who believe that various 
EU platforms and dashboards helped to disseminate, exploit 
and access research and innovation results 

Dissemination: Extent of compliance with the open access 
requirements of Horizon 2020, for both publications and 
research data 

Horizon 2020 
increased research and 
innovation’s 
contribution to key 
societal challenges  

Number of publications in peer-reviewed high-impact 
journals per EUR 10 million funding, for societal challenges 
[KPI 14, no baseline – new under Horizon 2020, target: 20 
for all societal challenges, achieved value: 7 per EUR 10 
million funding] 

Number of patent applications and patents awarded across 
the societal challenges, per EUR 10 million of funding [KPI 
15, baseline: to discuss, target: 2, achieved value: 0.35 
applications and 0.26 patents awarded, per EUR 10 million 
of funding] 
Number of prototypes and testing activities, [KPI 16, no 
baseline – new under Horizon 2020, no target, achieved 
values: prototypes 262 297, testing activities 386 07861] 
Number of joint public-private publications, for Societal 
Challenges [KPI 17, no baseline – new under Horizon 2020, 
no target, achieved value: 13 436] 
Number of new products, processes and methods launched 
into the market [KPI 18 – not legally compulsory. No 
baseline – new under Horizon 2020, no target, achieved 
values: Total number of projects introducing either a new 
methodology, a new process or a new product: 5 970; 
Total number of projects introducing a new methodology: 
2 955; 
Total number of projects introducing a new process: 2 941; 
Total number of projects introducing a new product: 5 30662] 
% of the overall energy societal challenge funds allocated to 
renewable energy, end user energy efficiency, smart grids 
and energy storage activities [KPI 19, no baseline – new 
approach under Horizon 2020, target: 85%, achieved value: 
69.6%] 

Replies to stakeholders’ consultation  

European Commission, Monitoring the open access 
policy of Horizon 2020 (2021) 
Replies to stakeholders’ consultation  
Commission monitoring systems (CORDA), data on 
24 April 2023 

Commission monitoring systems (CORDA), data on 
24 April 2023 

Commission monitoring systems (CORDA), data on 
24 April 2023 

Commission monitoring systems (CORDA), data on 
24 April 2023 

Commission monitoring systems (CORDA), data on 
24 April 2023 

Commission monitoring systems (CORDA), data on 24 
April 2023 

61 Data self-reported by beneficiaries in aggregate way: controls for e.g. duplication are not possible. The figures are reported as they appear in the monitoring system. 
62 Data self-reported by beneficiaries in aggregate way: controls for e.g. duplication are not possible. The figures are reported as they appear in the monitoring system. 

40 

 
 
 
 
international 
cooperation and, more 
specifically, 
association of third 
countries to the 
Horizon 2020 has 
made a difference in 
achieving its objectives 

use of different 
instruments, including 
partnerships, made it 
possible to achieve 
impact for science, 
economy and/or 
society 
Horizon 2020 
strengthened the 
European Research 
Area  

1.2.  To what extent has 
the FP contributed 
to achieving the EU 
policy priorities? 

Annual number of occurrences of tangible specific impacts 
on European policies resulting from technical and scientific 
support provided by the Joint Research Centre [KPI 22, 
baseline: 248, target 330, achieved value: 513] 
% of public consultation respondents who responded that 
Horizon 2020 contributed to the planned societal effects ‘to 
a great extent’ 
International cooperation [CCI KPI 7]: 
-  % of third-country participants in Horizon 2020 
-  % of budget of topics in the work programmes 
mentioning at least one third country or region 

Ex post evaluation of the activities of the JRC under 
Horizon 2020 and Euratom 2014-2020 (Table 263, p. 69) 

Replies to stakeholders’ consultation  

Commission monitoring data (R&I Dashboard, data on 
31 December 2022)  

Level of agreement among stakeholders that Horizon 2020 
enabled international collaborations that otherwise might not 
have come into existence  
Adequacy of the policy mix 

Replies to stakeholders’ consultation  
5 impact area evaluation studies (links in Annex 2) 

Evaluation study on the Relevance and Internal coherence 
of Horizon 2020 and its Policy Mix  

% of public consultation respondents who reported that 
partnerships have been ‘very much’ more effective than 
regular collaborative research 

Replies to stakeholders’ consultation 

5 impact area evaluation studies (links in Annex 2) 

Contribution to realising the European Research Area [CCI 
KPI 1]: 
-  Annual number of research positions advertised on 

EURAXESS Jobs 

-  Number of national research infrastructures networked  
-  Number and percentage of open access articles 

published in peer-reviewed journals 

-  Number of projects that make scientific data accessible 
and re-usable and number of scientific datasets made 
accessible and re-usable 

-  Number of multiannual implementation plans adopted 

by joint programming activities 

Level of agreement among stakeholders that Horizon 2020 
influenced the national policies 

Evaluation study on the implementation of cross-cutting 
issues in Horizon 2020  

Replies to stakeholders’ consultation  

63 Impact is defined as ‘the use of JRC results for policy preparation (e.g. impact assessments), monitoring (e.g. COM reports), implementation (e.g. methods, materials, guidance) and 

evaluation’. 

41 

 
 
 
 
 
enhanced participation 
of Horizon 2020 
widening countries in 
the programme and 
supported the 
expansion of a pan-
European R&I network 
cross-cutting issues 
were implemented 
within the FP projects 

1.3.  To what extent has the 
FP contributed to 
achieving the SDGs? 

projects funded by the 
FP contributed to EU 
policy priorities and 
SDG implementation 

Widening participation [CCI KPI 2]: Total number of 
participations by EU-28 Member States 

Evaluation study on the implementation of cross-cutting 
issues in Horizon 2020  

% of respondents who agreed or strongly agreed that 
‘Horizon 2020 spread excellence and widened participation 
in R&I’ 

SME participation [CCI KPI 3]: % of EU financial 
contribution committed through the SME instrument 

Social sciences and humanities: % of SSH partners in 
selected projects in all Horizon 2020 priorities [CCI KPI 4] 

Science & society: % of projects where citizens, civil society 
organisations and other societal actors contribute to the co-
creation of scientific agendas and scientific contents [CCI 
KPI 5] 
% of respondents who agreed or strongly agreed that ‘gender 
equality as a cross-cutting issue was effectively 
implemented’ 

Gender [CCI KPI 6]:  

-  % of women participants in Horizon 2020 projects 
-  % of women project coordinators in Horizon 2020  
-  % of women in EC advisory groups, expert groups, 

evaluation panels, individual experts, etc. 

-  % of projects taking into account the gender dimension 

in research and innovation content 

Sustainable development and climate change [CCI KPI 8]: 
-  % of EU financial contribution climate related in 
Horizon 2020 [target: 35%, achieved value: 32%] 
-  % of EU financial contribution in Horizon 2020 related 
to sustainable development [target: 60, achieved value: 
64.4%] 

Top three SDG projects contributed to, by impact area  

Replies to stakeholders’ consultation  

CORDA data 

Evaluation study on the implementation of cross-cutting 
issues in Horizon 2020 
Replies to stakeholders’ consultation 
Evaluation study on the implementation of cross-cutting 
issues in Horizon 2020  

Evaluation study on the implementation of cross-cutting 
issues in Horizon 2020  

Replies to stakeholders’ consultation  

European Commission, 'Study on the proposal evaluation 
system for the EU R&I framework programme' (2022)  

Evaluation study on the implementation of cross-cutting 
issues in Horizon 2020 

European Commission, 'Study on the proposal evaluation 
system for the EU R&I framework programme' (2022)  

Evaluation study on the implementation of cross-cutting 
issues in Horizon 2020 
Evaluation study on the relevance and internal coherence 
of Horizon 2020 and its policy mix (Annex C – eCorda 
data analysis) 
European Commission, 'Informing global climate action' 
(2022) 
5 impact area evaluation studies (links in Annex 2) 

42 

 
1.4.  Did the FP 

successfully reach a 
wide and balanced 
range of 
stakeholders?  

allocated funding is 
concentrated 
geographically 

allocated funding is 
concentrated to smaller 
or larger projects 

% of funding allocated by Member States 

% of funding allocated to EU-15, widening countries and 
associated countries 
Comparative likelihood of smaller and larger projects to get 
funded 

Evaluation study on the implementation of cross-cutting 
issues in Horizon 2020  

Evaluation study on the Relevance and Internal coherence 
of Horizon 2020 and its Policy Mix 

Replies to stakeholders’ consultation 

Questions 

2.1.  How efficient have 
the implementation 
processes of the FP 
been, and in 
particular the 
simplification 
measures taken in 
Horizon 2020?  

Judgement criteria:  
extent to which… 
administration and 
management have 
been efficient 

project application and 
selection processes 
have been efficient 

tangible benefits have 
resulted from 
simplified / 
streamlined rules, 
procedures or 
processes 

2.  Efficiency 

Indicators 

Main data sources 

European Commission administrative data 
5 impact area evaluation studies (links in Annex 2) 
Replies to stakeholders’ consultation 
Replies to stakeholders’ consultation  

Replies to stakeholders’ consultation  

Replies to stakeholders’ consultation  

5 impact area evaluation studies (links in Annex 2) 
European Commission administrative data 
Replies to stakeholders’ consultation  

Time-to-inform 
Time-to-sign 
Time-to-grant 
% of public consultation respondents who replied that 
cumbersome project implementation may have prevented 
participation in the framework programme 
% of public consultation respondents who replied that less or 
much less effort was required for the application process than 
expected  
% of public consultation respondents who replied that the effort to 
participate in Horizon 2020 was similar or lower than for the 
previous framework programme 
Tangible effects of specific key changes, e.g. reduced 
administrative burden, faster decision-making 
% of public consultation respondents who replied that the 
following measures largely or hugely reduced the administrative 
burden: 

•  alignment of funding rules applicable to all EU funds  
•  harmonisation of processes and guidance documents across 

the programme  

•  using an electronic-only grant management system  
•  further use of the two-stage application process (for some 

programme parts)  

•  using a new funding model with a single reimbursement rate 

and a single flat rate  

•  selected projects funded as proposed (no negotiation during 

grant preparation)  

43 

 
 
 
2.2.  How proportionate 

were the costs of 
application and 
participation borne 
by different 
stakeholders groups, 
taking into account 
the associated 
benefits?  

administrative costs 
borne by applicants 
and participants were 
lower compared to the 
previous FP 

barriers at application 
stage and during 
project 
implementation did 
not have negative 
consequences for the 
researchers and 
organisations involved 
drivers at application 
stage and during 
project 
implementation had 
positive consequences 
for the researchers and 
organisations involved 

Ranges of the average applicant’s cost of applying under Horizon 
2020 
Ranges of the average number of person days needed for proposal 
development under Horizon 2020 
Qualitative stakeholder feedback on application cost under 
Horizon 2020 compared to FP7 
Identification of barriers 
Level of agreement among applicants that these barriers did not 
have negative consequences 

Replies to stakeholders’ consultation  

Replies to stakeholders’ consultation  

5 impact area evaluation studies (links in Annex 2) 
Replies to stakeholders’ consultation 
5 impact area evaluation studies (links in Annex 2) 
Replies to stakeholders’ consultation  

Identification of drivers 
Level of agreement among applicants that these drivers had 
positive consequences 

5 impact area evaluation studies (links in Annex 2) 

Questions 

3.1.  How coherent has 

the R&I Framework 
Programme been 
between its 
programme parts, 
and with other EU 
programmes serving 
similar objectives 
and with national, 
regional and 
international 
initiatives? 

Judgement criteria:  
extent to which… 
implementation of 
Horizon 2020 was 
consistent between 
programme parts 

implementation of 
Horizon 2020 was 
consistent with the 
European Structural and 
Investment Funds 

3.  Coherence 

Indicators 

Main data sources 

Level of consistency/coordination of policy mix in terms of: (i) 
types of actions (e.g. grants, financial instruments, procurement 
support); (ii) forms of call implementation (e.g. calls for individual 
projects, use of focus areas, public-public partnerships, public-
private partnerships); (iii) technological readiness levels covered; 
(iv) targeted beneficiaries 

Evaluation study on the Relevance and Internal coherence 
of Horizon 2020 and its Policy Mix 

Innovative Europe study 

European Commission, 'Opportunities and challenges in 
targeted funding of Research and Innovation' (2021)  

Degree to which implementation of Horizon 2020 was consistent 
with Structural and Investment Funds’ European Research and 
Development Fund (including through the Seal of Excellence)  

Evaluation study on the external coherence and synergies 
of Horizon 2020 with the European research and 
innovation support system 
Special report 2022/23 of the European Court of Auditors: 
'Synergies between Horizon 2020 and European Structural 
and Investment Funds' 

Replies to stakeholders’ consultation 

44 

 
 
 
 
implementation of 
Horizon 2020 was 
consistent with relevant 
national, regional  
initiatives and joint 
programming 

Horizon 2020 has worked 
in synergy with other 
relevant EU programmes 

Degree to which implementation of Horizon 2020 was consistent 
with the objectives of national policies and programmes (especially 
partnerships and ERANETs) 

Evaluation study on the external coherence and synergies 
of Horizon 2020 with the European research and 
innovation support system 

Degree to which implementation of Horizon 2020 was consistent 
with other EU programmes such as (i) the Connecting Europe 
Facility; (ii) the European Fund for Strategic Investments; (iii) 
Erasmus+; (iv) Life 

Sections on ERANETs and joint programming (art. 185 
initiatives) in the impact area studies 

Evaluation study on the external coherence and synergies 
of Horizon 2020 with the European research and 
innovation support system 

Special report 2022/23 of the European Court of Auditors: 
'Synergies between Horizon 2020 and European Structural 
and Investment Funds' 

Replies to stakeholders’ consultation 

Questions 

4.1.  What was value 

resulting from the 
FP that is 
additional to the 
value that could 
result from 
interventions 
which would be 
carried out at 
regional or 
national level?  

Judgement criteria:  
extent to which… 
the stakeholders could 
not have implemented 
their research and 
innovation in another 
way (i.e. through other 
national or regional 
support) 
Horizon 2020 
minimised duplications 
via collaboration 

4.  EU added value 

Indicators 

Main data sources 

Level of stakeholder agreement that without Horizon 2020 projects 
would likely not have secured funding and would not have taken 
place at all 

-  Collaborative research 
-  Direct actions by JRC 

% of respondents who stated that participation in Horizon 2020 was 
beneficial compared to national and/or regional R&I programmes in 
EU Member States (for various reasons) 
Level of 
stakeholder 
agreement 
that Horizon 
2020 
minimised 
duplication 
through: 

-  H2020 widening actions  
- 

Innovation actions (demonstration and tests) 

-  EIT and its KICs 

5 impact area evaluation studies (links in Annex 2) 

Replies to stakeholders’ consultation 

Replies to stakeholders’ consultation  

Excellent Science study 
Ex post evaluation of the activities of the JRC under 
Horizon 2020 and Euratom 2014-2020 
European Commission, Programme Statements 2022 
Innovative Europe study 
Excellent science study 
Innovative Europe study  

Horizon 2020 leveraged 
additional resources for 
R&I 

Leverage factor of partnerships: additional resources mobilised by 
industry and Member States (absolute number in EUR)  

Partnerships Biennial Monitoring Report (2022, Section 
III) 

45 

 
 
 
 
Horizon 2020 created 
transnational 
cooperation 

Level of support for distinctive projects that are unlike those that the 
Member States and their regions fund  

Replies to stakeholders’ consultation 
Impact area studies (links in Annex 2)  

Horizon 2020 increased 
excellence 

Degree to which Horizon 2020 increased participants’ competitive 
advantage  

Degree to which Horizon 2020 spread excellence to lower R&I 
performing countries 
% of public consultation respondents who reported that stepping up 
support to breakthrough innovations through the introduction of the 
EIC Pilot strengthened the impact of Horizon 2020 

PPMI counterfactual analysis 
Mitra, Alessio, and Niakaros, Konstantinos (2023). The 
Horizon Effect: A Counterfactual Analysis of EU R&I 
Grants 
Excellent Science study 

Replies to stakeholders’ consultation  

Questions  

5.1.  How relevant has the 
support to innovation 
by the Framework 
Programme (FP), 
including 
partnerships, been 
given the 
stakeholders’ needs 
and considering the 
scientific, 
technological and/or 
socio-economic 
problems and issues 
identified at the time 
of its design and over 
time?  

Judgement criteria:  
extent to which… 
the design of the FP 
(including 
partnerships) was 
based on the analysis 
of needs of groups 
targeted for 
application/participat
ion 
the FP demonstrated 
that it was flexible in 
coping with changing 
circumstances in 
Europe and in the 
world 

Indicators 

Main data sources 

5.  Relevance 

Existence of needs falling outside the scope of the FP 

Degree to which the policy mix mobilised newcomers  

Evaluation study on the Relevance and Internal 
coherence of Horizon 2020 and its Policy Mix 

Evaluation study on the Relevance and Internal 
coherence of Horizon 2020 and its Policy Mix 

Newcomers monitoring flash 

% of respondents who reported that the FP reached its objectives  

Replies to stakeholders’ consultation  

Degree to which the programme integrated lessons learnt during 
implementation 

Evaluation study on the Relevance and Internal 
coherence of Horizon 2020 and its Policy Mix  

Degree to which the programme responded to unforeseen and 
emergency circumstances, such as the COVID-19 pandemic 

Evaluation study on the Relevance and Internal 
coherence of Horizon 2020 and its Policy Mix – case 
study no 21 on how Horizon 2020 and its policy mix 
came into play in reaction to the COVID-19 crisis  

Speed of response to COVID-19 compared to Zika and Ebola crises 

Replies to stakeholders’ consultation  

5 impact area evaluation studies (links in Annex 2) 

European Commission, 'Meeting the pandemic 
challenges - Contribution of EU R&I funding to COVID-
19 related research' (2022) 

46 

 
 
 
 
 
 
 
 
A more complete account of societal impacts of Horizon 2020 projects is provided below and in the underlying set of evaluative studies and materials, to 
complement Section 4.1.2 on Effectiveness: societal impacts. 

Additional details on societal effects of Horizon 2020 projects 

SC1 – Health, demographic change and well-being 

As a minimum, some of the SC1 project publications have been effective  in shaping EU public health policy development64. The effects range from 
project reports being cited in EU reports and the Eurosurveillance journal65, to influencing changes in WHO guidelines. The European Centre for Disease 
Prevention and Control (ECDC) cited the highest number (33) of SC1 publications, followed by the Directorate-General for Health and Food Safety (DG 
SANTE).  In  total,  53  (unique)  EU  policy  papers  were  found  to  have  cited  81  (unique)  SC1  project  publications.  The  policy  papers  include  types  of 
documents aimed at informing public health policy and providing guidelines66.  

Case studies on SC1 actions have been able to identify with some certainty an effect from some Horizon 2020-funded projects on the global health 
policy  discourse.  Examples  include  the  BEAT-AMR  project,  which  developed  clinical  recommendations  and  a  roadmap  on  antimicrobial  coatings  in 
healthcare settings, and members of the AMR project ‘DARWIN’ 67, who co-authored international recommendations for the WHO. Several partnership 
projects have equally contributed to shaping policies and global health advice. One such example was the AMBITION68-cm phase III clinical trial, whose 
results directly shaped an update of WHO guidelines69.  

Positive impacts  were  achieved by strengthening  international  collaboration in  research between European and African countries and institutions.  The 
public-public European & Developing Countries Clinical Trials Partnership (EDCTP)70 boosted the number of clinical studies and research activities 
by between 30 and 40% in South Africa, Uganda and Tanzania, and increased by over half the number of studies registered in other countries. It also helped 
secure a critical mass of resources and expertise, particularly in sub-Saharan Africa. This international collaboration in research was particularly relevant 
in these areas, where the existing activities had been fragmented.71 Globally, EDCTP272 was identified as the third-largest financial enabler of tuberculosis 
research globally and the second largest public funder. 

64 This finding is underpinned by analysis of SC1 projects cited in EU policy papers, studies supporting impact assessment, in combination with an assessment of the public health policy 
impacts in SC1. 
65 See for example the Eurosurveillance journal, https://www.eurosurveillance.org/. 
66 E.g. ‘Eurosurveillance: special edition: advanced diagnostics to inform public health policy. March 2019.’ and ‘Recommendations on FAIR metrics for EOSC.’ 
67 The DARWIN–AMR project co-authored international recommendations for the WHO aimed at promoting local national action plans and sociotechnical AMR mitigation options for 
countries around the world.   
68 The AMBITION-cm – EDCTP2 project led to the updated WHO guidelines on HIV treatment: ‘New guidelines developed by WHO strongly recommend a single high dose of liposomal 
amphotericin B as part of the preferred induction regimen for the treatment of cryptococcal meningitis in people living with HIV’.   
69 https://www.who.int/news/item/20-04-2022-rapid-advice-new-guidelines-for-simpler-safer-treatment-for-cryptococcal-disease-in-plhiv  
70 Resilient Europe study (2023), Annex1, Section 2.3.1 and case study 2. 
71 Evaluation study of the European FPs for Research and Innovation for a Resilient Europe, p. 66. 
72 https://www.treatmentactiongroup.org/wp-content/uploads/2021/12/tb_funding_2021.pdf. 

47 

 
 
ANNEX IV. OVERVIEW OF BENEFITS AND COSTS, TABLE ON SIMPLIFICATION AND BURDEN REDUCTION 

1.  Benefits of Horizon 2020 

Horizon  2020  gives  rise  to  a  range  of  benefits,  presented  in  Section  4.1  (Effectiveness)  in  the  context  of  the  framework  programme’s  structure  and 
Intervention Logic. In the framework of Better Regulation this means Horizon 2020 generates:  

1. Indirect long-term welfare benefits for EU society derived from scientific impact, and related benefits for participants 

The  R&I  framework  programmes,  including  Horizon  2020,  support  activities  that  generate  scientific  outputs  (e.g.  scientific  publications)  and 
outcomes/results (e.g. positive effects on researcher careers after participation), which are direct and indirect benefits for its participants (reported in 
section  4.1.1,  4.1.6,  and  also  4.1.2).  The  direct  outputs  of  Horizon  2020  and  some  indirect  long-term  outcomes  of  FP7  have  been  quantified  and  are 
summarised below. (Table 1, I. Benefits; column: Beneficiaries). 

In  the  long  run,  these  effects  on  beneficiaries  have  a  strong  potential  to  lead  to  indirect  welfare  benefits  for  EU  society,  i.e.  to  positive  social, 
environmental, or economic impacts from scientific progress (e.g. improvements in public health). These long-run benefits have not been systematically 
assessed and are only mentioned in Table 1 (column: Citizens/EU Society): Horizon 2020’s future long-term welfare benefits are difficult to predict and 
may arise in yet unknown areas. The links between past R&I support programmes and currently observable welfare improvements from scientific progress 
are not systematically monitored. In addition, many external factors contribute to any scientific breakthrough or advancement, which makes it difficult to 
establish a causal link with specific instances of R&I funding.  

2. Indirect wider economic benefits for EU society from diffusion of innovation, and related economic benefits for participants 

Horizon 2020 supported industrial research and innovation ‘from idea to market’, which had the aim to improve innovation diffusion in products, processes, 
and services. The related innovation outputs (e.g IPR applications) and resulting economic outcomes/results (e.g. improved economic performance of 
firm after participation) constitute direct and indirect economic benefits for private sector participants (reported in section 4.1.3, 4.1.6, and also 4.1.2). 
Innovation  outputs  of  Horizon  2020  and  some  of  the  indirect  (micro-)economic  benefits  for  participants  have  been  quantified  or  monetised  and  are 
summarised below. (Table 1, I. Benefits; column: Beneficiaries). 

In  the  long  run,  with  the  effects  of  innovations  in  processes  and  products  spreading  through  sectors  of  the  economy  and increasing  productivity  and 
competitiveness, Horizon 2020 has a strong potential to lead to indirect, wider economic benefits for the European economy as a whole (reported in 
section 4.1.3), ultimately raising the welfare of EU society. Macroeconomic impacts of Horizon 2020 have been estimated in terms of GDP gain and jobs 
created and are reported below. (Table 1, I. Benefits; column: EU society).   

Note on the effects of ‘Societal Challenges’ programme part: Horizon 2020 supports specific activities focused on tackling societal challenges by means 
of R&I, which have generated scientific outputs (included in benefit 1. above) and innovation outputs linked to economic benefits (included in benefit 2. 
above). Some activities on societal challenges are geared towards generating positive direct, non-market effects, such as informing the policy debate, 
48 

 
which are expected to contribute to indirect, non-monetary social and environmental benefits (e.g. positive impacts on gender equality, Europe’s security 
and on achieving SDGs), thus increasing the welfare of society in the long run. Although it is considered likely that at least some of these benefits have 
materialised, they have not been systematically assessed by the evaluation, due to the lack of a monitoring and evaluation framework.  

2.  Costs of Horizon 2020 

The framework programme, including the processes through which it was implemented, has given rise to four main types of costs incurred by different 
stakeholder groups, which also influence its efficiency: 

1. Horizon 2020’s operational expenditure - is the main direct cost of the framework programme. It is incurred by EU society and constitutes the input that 
enables the R&I activities and leads to the generation of benefits to society. The money is invested through calls and other activities (e.g. PPPs, events, 
studies) and comes out of the programme’s voted budget. (Reported below in Table 1, II. Costs) 

2. Administrative Cost of the European Public Sector – are funded separately through the Union’s budget as Horizon 2020’s Administrative Expenditure. 
This cost of administrating and running the framework programme is borne by the public sector at European level73 (and ultimately EU society) and is 
the second main cost factor that affects its overall efficiency. (Reported below in Table 1, II. Costs) 

Evidence collected on the cost of the proposal evaluation process (included in the total administrative cost) suggests that an average public sector time 
cost74 for the evaluation of one proposal to arrive at between 43.6 hours (multi-stage process) and 58.6 hours (single stage process) of evaluation expert 
time. An approximate monetised value of the evaluation of one proposal was estimated75 to about € 2 500 (multi-stage) and € 3 300 (single stage process), 
excluding administrative costs of organising the evaluation, based on the pay rate76 of an evaluation expert.  

3. Administrative costs of Horizon 2020 beneficiaries (non-additional, compensated for in operational expenditure) - The 41 575 participants77 of Horizon 
2020  spent  substantial  effort  on  the  actual  research  and  innovation  activities,  which  generated  the  benefits,  reported  under  effectiveness.  In  addition, 
participants  also  incurred  costs  (i.e.  spent  effort,  money  and  time)  to  fulfil  Horizon  2020’s  administrative  requirements  for  each  of  their  178  104 
participations78. These administrative costs of programme  beneficiaries, while to an extent operationally unavoidable, are not directly productive in 
terms of benefits. They have the potential to introduce substantial inefficiencies into R&I support. Reducing the administrative costs of beneficiaries was 
one of the aims of the simplification measures of Horizon 2020 (see section 4.2.2)  
In  the  public  consultation,  the  majority  of  respondents  stated  that  the  effort  needed  to  participate  in  Horizon  2020  was  ‘similar’  (39%;  692 
respondents), or even ‘greater’ (17%; 303) compared to that under FP7. Only a minority of respondents (12%; 219) spent less effort (with one-third 

73 Public sector administrative costs related to the FP at a national, regional, and local level are not covered in the evaluation. 
74 Study on the proposal evaluation system for the EU R&I framework programme. Annex (2022), p. 152.  
75 Horizon 2020’s Evaluation support study on Resilient Europe; Annex1 Section 5, Table 60, p. 122. 
76 The Horizon 2020 Model Contract for Experts indicates that the expert is entitled to a fee of EUR 450 for each full day worked in accordance with Article 3(2). Calculation assumes 8-
hour day. (https://ec.europa.eu/research/participants/data/ref/h2020/experts_manual/h2020-experts-mono-contract_v1.1_en.pdf ). 
77 Corda dashboard data: Unique participants in signed grants, as of 1/1/2023. 
78 Corda dashboard data: Participations in signed grants, as of 1/1 2023. 

49 

 
 
 
of the respondents not providing an opinion). Although exact percentages differed, the overall pattern of the feedback matched that of responding EU 
citizens (34% similar, 13% greater, 9% lower effort) and was influenced by responses from academic and research institutions (41% similar, 21% greater, 
12% lower effort), with over 190 respondents (over one-fifth) of this stakeholder group reporting a cost increase relative to FP7. Improvements seem to 
have been slightly more pronounced for the private sector, according to responses from business associations (44% similar, 16% greater, 28% lower effort) 
and companies/business organisations (40% similar, 12% greater, 14% lower effort). 

When comparing the participation of Horizon 2020 to ‘other research and innovation programmes’, a larger share considered the relative effort 
under Horizon 2020 ‘similar’ or ‘greater’, than ‘lower’. Responses varied substantially by sub-group. Whereas approx. 20% (26) of respondents from 
Associated Countries reported Horizon 2020 required comparatively lower effort, this was the case for only half that share of respondents from EU Member 
States (EU15: 12.3%, EU13: 10.1%). compared the programme favourably to alternative sources of funding. Effort to participate in Horizon 2020 vis-à-
vis other R&I programmes: among half of the respondents from both EU-associated countries as well as third countries was greater, compared to only 
33% among EU-13 respondents. Similarly, 50% of academic institutions reported the same, along with business associations (41%), companies and business 
organisations (37%) as well as EU citizens (40%). A lower share of respondents, notably from NGOs (33%), non-EU citizens (31%) and public authorities 
(29%), indicated that the effort was higher – leading to the assumption that for these respective groups, Horizon 2020 was more accessible.  

Around 300 respondents provided information on the time cost to manage participation in terms of the ‘average number of person-days79 spent during 
the entire project’, which allows for some very approximate quantification. As expected, projects of longer duration experienced higher total administrative 
time costs, however, particularly the 12-month (71 person-days) and 48-month (332 person-days) long projects showed high average administrative time 
costs of 6 to 7 person-days per month. Projects of 24 months (103 p.-d.) and 36 months (157 p.-d.) length spent just under 4.5 person-days per month 
on average.   

Expressing the two modes of average values, of 4.5 person-days/month and 7 person-days/month, as order of magnitude money values, would imply that 
the total beneficiaries’ administrative cost of Horizon 2020 amounted to between EUR 135 million and EUR 215 million80. The upper bound of the 
range could, however, be too low by an order of magnitude. A spot check81 against project reporting information suggests that projects of 60 months 
duration and more may have average monthly time costs (of tasks that constitute additional administrative burden) that far exceed 10 person-day per 
months. As the grouping of tasks into work packages and the numbering and labelling of these packages is different for each project, the reported cost 
information cannot be systematically used for an ex post assessment.  The time ranges for projects of the same duration, reported in the public consultation, 
were in some cases vast, likely reflecting the diversity of circumstances as well as potential question design issues: The 156 respondents with projects of 
36 months  duration, for instance,  reported costs from  2 person-days  per month  to a value of  1,500 person-days per  months (even correcting for 

79 i.e expressed in multiples of working days of one person. 
80 This monetisation uses the programme’s total number of project months (1 284 101 months) and is based on the wage tariffs of Better Regulation Tool 58 (footnote 836; One-In-One- 
Out calculator, using Eurostat and 2018 Labour Force Survey data), with an assumed 8 hour working day, European average values and the assumption that work is carried out by a 
person with profile ISCO 4. 
81 The average values have been spot checked for plausibility against a small number of randomly selected Horizon 2020 projects of different project durations, using reported effort by 
the coordinator (reported in person-months, that use 220 working days per year) The effort was taken from the work package that in each case closest matched tasks representing additional 
administrative burden.  

50 

 
 
outlier), the latter value suggesting that about 82 people were occupied with administrative requirements full-time. It cannot be excluded that responding 
beneficiaries may have interpreted the question in an unintended way, reporting too high values (including time for internal coordination or for project 
activities overall) or too low values (as not all time spent was perceived by them, due to the respondent’s role in the project).  

The uncertainty the respondents’ interpretation of the question, the non-representative nature of the public consultation as such, and the small sample of 
respondents overall (particularly for some project types) means that there is a very high level of uncertainty around the (time) cost values representing 
beneficiaries’ administrative costs of participating in Horizon 2020. The presented values are insufficiently robust to inform programme design. 

4. Costs of applicants - Successful and unsuccessful applicants to the framework programme incur a one-off cost: they invest effort, time, and money to 
prepare proposals. Application costs are mainly determined by the specific requirements of the programme part and rise with an increase in competition 
between applicants. For an evaluation, from the point of view of EU society, these application costs are relevant. They have the potential to introduce 
substantial  external  inefficiencies  into  the  R&I  support,  as  time,  effort,  and  money  spent  by  unsuccessful  applicants  can,  to  a  great  extent,  become  a 
deadweight loss to society unless captured suggests otherwise. High application costs also drive away potential applicants and so have the potential to 
distort the participation of applicants with respect to relevant characteristics, such as capital constraint, lack of experience, and the availability of outside 
options of funding to the applicant.  

How high were the costs of applicants?   
The evaluation has brought together around 40 individual instances of evidence of variable but generally low levels of robustness on the actual cost of 
application. The cost data received suggest that actual costs seem to vary widely by funding instrument. They are also influenced by the evaluation 
process  (multi/single  stage)  and  type  of  stakeholder  group  targeted.  The  information  available  is  of  insufficient  quality  to  generate  a  robust 
aggregated cost of applicants for the programme. It nevertheless illustrates the order of magnitude of the costs of preparing a proposal. 

In the context of several support studies, the evaluation collected some evidence from beneficiaries on costs of applications82 expressed in money terms. 
The average monetised cost of a single proposal varies substantially by funding instrument (and likely also by the potential grant size), with the lowest 
average value of  € 6 000 for a SME Instrument phase 1 proposal83 to over € 73 400 for an upper range INFRA proposal84. Past application cost evidence, 
provided by the European University Association (EUA) in the context of the Horizon 2020 interim-evaluation, can serve as a point of comparison, even 
though the reported range of € 10,000 to € 100 000 per proposal applies only to researchers85.   

82 Although the evidence predominantly stems from beneficiaries, success is not assumed to be generally correlated with application costs across the FP, although it likely has some 
influence at the margin. 
83 Evaluation study on Innovative Europe, the reported range for SME I proposals is approx. €6 000 to € 25 000. 
84 Evaluation study on Excellent Science, the reported range for INFRA proposals is approx. €44 900 to € 73 400. 
85  European  University  Association  (2016),  estimates  based  on  a  member  survey  with  contributions  from  more  than  150  universities  from  28  Europe  countries; 
https://eua.eu/resources/publications/346:eua-member-consultation-a-contribution-to-the-horizon-2020-mid-term-review.html 

51 

 
 
The application costs for some instruments appear to fall closer together than for others86 further reflecting the diversity of applications. The average cost 
of one proposal for Horizon 2020 was estimated87 to fall into the range of €18 000 to €37 000.  

This suggests that the application costs of beneficiaries alone, who submitted 33 80688 successful proposals, would amount to between €609 million and 
€1.25 billion. This value is rendered very uncertain by the lack of adequate, systematically collected evidence and should be read as a rough illustrative 
figure only.  

Unsuccessful proposals89  did  not  directly lead to productive outputs  under to  Horizon 2020  but also  cost  effort and money to  prepare.  Unsuccessful 
applicants are a stakeholder group the evaluation has not systematically consulted. Due to a severe lack of robust evidence the estimation of this cost has 
to rely on strong simplifying assumptions. Two available estimates calculated in the context of the evaluation are highly uncertain but provide an idea of 
magnitude: the evaluation support study on Resilient Europe arrived at totals between € 5 410 million (based on project size weights) and € 9 160 million 
(based on consortium size weights). The evaluation support study on Excellent Science generated a range between € 4 758 million and € 9 694 million90. 
The evaluation interprets these results as an indication that the total application cost embodied in the large number of unsuccessful proposals is likely 
very substantial and may well reach a value in the order of magnitude of € 5 billion to € 10 billion. 

Evaluations of earlier framework programmes did not report estimates that could be used as direct points of comparison. Quantitative evidence available 
from the interim evaluation also presented wide ranges: The EUA estimated that, in the first year of Horizon 2020 alone, its members spent between € 
268 million and € 2.68 billion on unsuccessful applications. Based on EUA’s figures, the interim evaluation estimated that around € 1.7 billion would be 
spent annually on writing unsuccessful proposals, of which € 643.0 million for non-funded high-quality proposals alone91.  

Public Consultation responses provide basic time cost ranges92 and show a pattern of diminishing proportions of applications as application time costs rise: 
Responses  suggests  that  56%  of  applications  take  less  than  50  person-days,  with  80%  of  applications  under  100  person-days.  The  study  on the 
proposal evaluation system93 finds that the average time spent by ‘applicants’ 94 on a single-stage application was 25 person-days and 47 person-days for 

86 For instance, relatively narrow ranges were found for ERC (approx. €19 600 to €28 700, evaluation study Excellent Science); as well as SC1 (€34 000 to €42 000, evaluation study on 

Resilient Europe. Wide range of application costs was reported by MSCA beneficiaries (around €11 800 to €35 600). 

87 Evaluation study on Resilient Europe estimated a range between €20 000 and €34 000 and the evaluation study Excellent Science €18 257 to €37 169 for one proposal. 
88 Dashboard, Horizon 2020, ‘Retained Proposals’ as of 01.01.2023 
89 Unsuccessful proposals here includes all proposals, which are effectively unsuccessful as they not lead to any funding under Horizon 2020, including those of high quality.  
90 Evaluation study Excellent Science, Annex I, https://data.europa.eu/doi/10.2777/353383.  
91 SWD on the interim evaluation of Horizon 2020, SWD(2017) 220 final,  Section 7.3.2 (Application and evaluation process), p. 60. Scaling the annual value up to a seven-year total for 
the framework programme is not possible, in the absence of evidence on the distribution of costs over time. An (unlikely) uniform distribution would imply € 11.9 billion, a similar order 
of magnitude as the higher value of the range found by this evaluation.  
92 Annex 5: Cost of proposal preparation, p. 94. 
93 Study on the Proposal Evaluation System for the EU R&I framework programme, 2022, Table p. 3, https://data.europa.eu/doi/10.2777/16211. 
94 No further breakdown was reported for the question, however, beneficiaries dominate responses. The time cost of the multi-stage application captures the higher time cost of successful 
applicants. The vast majority of (unsuccessful) applicants only incur costs for the simplified first application stage. The overall average may therefore be expected to be at least equal if 
not lower than for a single stage procedure. 

52 

 
 
 
 
a (successful) multi-stage application. For Societal Challenges 2 to 595 (588 responses) lower time costs are reported, with 47% of applications taking 
less than 10 person-days to prepare and over 80% taking under 20 person-days.  

Project size and the programme part may be two key determinants of the time cost but the evidence collected is insufficient to asses this overall. Time costs 
of reported LEIT proposals96were split by project size and suggest that costs for projects under €500 000 had a wide range from 15 to over 30 person-
days, whereas the majority of projects of €500 000 and above reported time costs over 30 person-days. At least half of all projects in all size ranges 
reported time costs of over 30 person-days, including 75% of coordinators of projects with a budget of at least €5 million, 70% of projects of € 2 million 
to €5 million and still over 50% of projects between €500 000 and €2 million. 

What does the oversubscription and low success rate mean for the efficiency of Horizon 2020? 

Horizon 2020 attracted an exceedingly high number of proposals, and even the number of proposals of sufficient quality to warrant funding was a multiple 
of those actually funded.  

In principle, oversubscription of an individual programme part primarily indicates where the resource constraint of the budget is particularly binding. The 
competition between applicants depends to some extent on the oversubscription of a programme, which in turn increases the likelihood that the eventually 
funded projects are of good quality and generate benefits and value-for-money.  

Very high oversubscription rates, however, have a negative effect on the overall efficiency as a very large proportion of the effort spent on preparing, 
and then evaluating, proposals does not lead to any benefits. Measures that ensure the costs of the application process are kept as low as possible 
and measures that allow successful yet unfunded proposals to be re-used in future applications at national or EU level therefore have the potential 
to address the situation and increase the programme’s efficiency from the point of view of society. 

Were application costs ‘proportionate’ for applicants, taking into account the potential benefits?  

The level of oversubscription of Horizon 2020 strongly suggests that up front, at the time the decision is made, the programme is relevant and attractive 
enough for a very large number of potential applicants to go ahead and invest effort.  

Potential applicants chose up front whether to face this application cost or not (in contrast to costs imposed on citizens by a regulation). They consider 
factors such as how likely they are to receive a grant, the size of the grant, the expected size of the application cost, their own risk profile, their capital 
constraint, and any alternative routes for funding that are open to them. They may also factor-in that they plan to apply again in the future, or that they can 
reuse parts of a previously unsuccessful proposal.  

95 Study on the Green Transition, 2023, https://data.europa.eu/doi/10.2777/422725.  
96 Study on the Digital and Industrial Transition, 2023, Annex VI, https://data.europa.eu/doi/10.2777/882919.  

53 

 
 
 
 
Once  proposals  have  been  evaluated  the  situation  changes.  The  vast  majority  of  applicants  will  not  receive  a  grant.  For  unsuccessful  applicants  the 
application costs are to a great extent lost and are unlikely proportionate. For successful applicants the proposal in the end leads to substantial benefits. 
Grants are typically vast relative to the costs of application. The fact that a beneficiary breaks even is, however, not automatically a sign that application 
costs had been proportionate, as the grant is meant to mainly cover R&I activities and administrative costs. The larger the grant, the higher the success rate 
of the call and the larger a consortium, the more likely application costs may be experienced as ‘proportionate’ by an applicant. In contrast, factors such as 
a smaller or financially weaker applicant (e.g. SME), stronger competition between applicants (low success rates) and a comparatively smaller grant size, 
increase the risk that the application costs were disproportionate, even for a successful applicant. 

Even applicants who directly compete, and therefore face similar application costs and the same level of competition, have different characteristics (e.g. 
risk aversion) and thus likely consider different rates of return acceptable and different levels of application costs ‘proportionate’. Quantitative evidence on 
what is seen as ‘proportionate’ by different stakeholder (sub-)groups of R&I funding is not available. Qualitative evidence from the public stakeholder 
consultation  confirmed  past  feedback  that  the  level  of  application  costs  (‘cumbersome  application  process’)  and  success  rates  (‘too  low  to  be  worth 
applying’) are seen as problematic for respondents97.  

At the level of the framework programme, in relation to the € 71 195 million of R&I support, applicants spent an indicative total of € 5 610 million 
(approx. 8%) to € 11 250 million (approx. 16%) of application costs. Given the insufficient level of confidence in the cost estimates it is not possible to 
base any firm conclusions on these values and it is a topic for further discussion, what percentage would still be acceptable and constitute ‘proportionate’ 
costs. Taken together, the available information is, however, sufficient to suggest that it cannot be ruled out that disproportionate application costs may 
have been an issue in Horizon 2020, particularly in some areas of the framework programme, possibly also at an aggregate level. The question of 
proportionality of application costs of R&I support therefore deserves continued attention. 

Participation of consultancies in Horizon 2020 consortia  

In the context of the evaluation, the question was discussed, whether the (perceived) common participation of ‘consultancies’ in consortia of Horizon 
2020 can be interpreted as an indirect indicator of the framework programme’s too high complexity and therefore a sign of its inefficiency. The 
participation of firms that could be considered consultancies is not monitored but has been assessed ex post. It was identified that 228 consultancy 
companies participated in 6.7% of all multi-beneficiary projects in Horizon 202098 overall. It also found that around half of the proposals submitted 
included a firm in the consortium (not necessarily as coordinator)  that was identified by the study to fall within this category99. Anecdotal evidence from 
National Contact Points suggests that consultants frequently attend information events for the framework programme. 

In of itself, the involvement of consultancies cannot be seen as an indicator for inefficiencies in the framework programme. Participants organise themselves 
using division of labour. Different skill sets are involved to carry out research and to prepare a proposal. The splitting of tasks may also just reflect the fact 

97 See, for instance, public consultation responses on ‘reasons preventing participation in Horizon 2020’, presented in Annex 5, (pp. 83 -85), Table 16,17 and 18, Figure 17.  
98 Study on Resilient Europe (2023, https://data.europa.eu/doi/10.2777/60819. These were consultancy firms ‘involved’ in any role in Horizon 2020 consortia. Overall, 382 firms were 
identified using keyword search in Technote database with cross-referencing participants. Website information used to narrow down to 228 consultancy companies. 
99 Study on Resilient Europe, 2023, p. 38, https://data.europa.eu/doi/10.2777/60819. Also Annex 1, p.140, https://data.europa.eu/doi/10.2777/57680. Their presence in the consortia was 
found to be correlated with a higher success rate but it is unclear whether this association is meaningful.  

54 

 
 
that consortia include individuals whose time is very valuable (e.g. top researchers, etc) and that there are costs associated with applying and fulfilling 
administrative obligations of projects (which is not an issue in itself, as long as these are proportionate). The involvement of consultancies in the application 
process would further point at the existence of economies of scale in proposal writing. Consultancies invest the hassle cost of finding relevant information 
and to get to know the structure of the framework programme, the rules and procedures, and may have practical experience on how to apply for it. In using 
this human capital, they add value to the consortium for which they get compensated. Having a niche for firms to specialise on certain tasks may just be 
the most efficient way for the applicants to organize themselves. 

When would the current situation constitute an efficiency problem for the framework programme? - This was the case if employing one of the consultancies 
was effectively a necessary condition for a successful application and consultancies therefore acted as gate keepers to the framework programme. This 
would  be  particularly  problematic  for  applicants  with  resource  constraints  (including  those  from  countries  to  benefit  from  ‘widening’  or  SMEs). 
Competition in the specialised consultancy sector (of a region/language) therefore matters. If the consultancy sector is not very competitive and substantial 
market  power  exists,  consultancies  can  extract  excess  profits.  Too  high  application  costs  and  administrative  costs  of  beneficiaries  (particularly  due  to 
unclear and cumbersome requirements) can generate barriers and favour such a situation, which also would negatively affect the efficiency of the framework 
programme. Costs of applicants and beneficiaries are therefore key areas to monitor and assess in any future evaluation of the framework programme, as 
is the share of applicants that make use of specialised consultancies in the different programme parts. 

55 

 
 
 
Annex IV Table 1 

Table 1. Overview of costs and benefits identified in the evaluation  

Citizens/  
EU Society  

EU Public  
Administration  

Horizon 2020  
Beneficiaries 

Horizon 2020 
Applicants 

Quantitative  

Comment 

Quantitative  

Comment 

Quantitative 

Comment  

Quantitative  Comment 

I. BENEFITS   

1. Indirect long-term 
welfare benefits 
 for EU society 
from 
scientific impact 
&  
related benefits to participants 
(Sections 4.1.1, 4.1.2, 4.1.6) 

one 
off 

-.- 
(No estimate 
available)  

 Horizon 2020 
strengthened frontier 
research, contributed 
to scientific 
breakthroughs and 
advancements. It 
increased the human 
capital of researchers 
through furthering EU-
wide career 
development and 
access to research 
infrastructures. 
In the long run, the 
supported activities are 
expected to lead to 
sizable and wide-
ranging welfare 
benefits to EU society 
(economic, social, and 
environmental benefits). 

1. Peer-reviewed 
publications 

H 2020: 276 784 
3.9% in top 1% 
most-cited 
publications  
26% on ‘future & 
emerging research& 
technology fields’ 

FP7+3yrs*: 219620  
(*no FP data on % 
top 1% most-cited 
and % in future and 
emerging fields) 

FP7 (ERC): 9 yrs  
persistent (+)long-
term effect on 
publication activity 
of researchers  

2. Researchers 
undertaking cross-
sector and cross-
country mobility, 
incl. PhD candidate 

Direct scientific 
output of Horizon 
2020 (benefit to 
researcher) 
linked to expected 
long-term welfare 
benefits from 
scientific impact 

(Number as of 
24/04/2023)  

Indirect benefit 
of FP7 to 
researcher 

Direct output  

56 

 
                        
 
 
 
 
 
 
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
H 2020: 49 475, of 
which PhDs: 25 676 
(52%) 
Target: 65 000, of 
which PhDs: 25 000 
(38%) 
FP7: 50 000, of 
which PhDs: 20% 

3. No. researchers 
who gained access 
to research  
(e-)infrastructures  

H 2020: 24 235* 
(incl. e-infrastr.:  
              162 810) 

Target: 20 000  

FP7: 22 000 (excl. 
e-infrastructures) 

4. long-term effect 
on visibility, 
productivity, and 
career of scientists 
FP7: 2 -5 years, 
after proposal, 
persistent (+) effect  

of Horizon 2020 
(benefit to 
researcher) 
linked to expected 
long-term welfare 
benefits  

(KPI4, Number 
shown as of 
01/01/2023; FP7 
figures for 2007-
2013.) 

Direct output  
of Horizon 2020 
(benefit to 
researcher) 
(KPI 5; *access 
gained from FP 
support; Horizon 
2020 figures as of 
01/01/2023.) 

Indirect benefit 
of FP7 to 
researcher 

5. Intellectual 
Property Rights 
applications 

H 2020:      
3 898 (0.57/€10m) 
of which     
patents 3 012 
(77.3%, 0.44/€10m) 

Innovation output 
under Horizon 2020, 
(benefit to 
participant) linked to 
expected long-term 
economic benefits.  
(IPR figures for H2020 
as of 24/04/2023, 
expected to increase; 
for FP7 as of 

2. Indirect wider 
economic benefits 
for the EU economy 
from 
diffusion of innovation 
& 
related benefits to participants  
(sections 4.1.3, 4.1.6, 4.1.2) 

one 
off  

Total impact 
on GDP  
(2020 prices) 
estimated: 
€ 429 bn 
(range:  
€421 bn to  
€ 798 bn) 
(2014-2040), 
of which  

Horizon 2020 supports 
industrial research and 
innovation “from idea 
to market”, with a view 
to improve innovation 
diffusion in products, 
processes and services, 
and thereby improving 
the competitiveness of 
industry participants 

57 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
  
 
 
 
 
 
 
 
309bn  
(€287 bn to 
€420 bn)  
over 17 years 
(2014 -2030) 

and ultimately boosting 
the productivity and 
competitiveness of the 
EU’s economy as a 
whole. 

GDP impact forecast: 
central scenario presented 
uses RHOMOLO model 
output; range/sensitivity 
uses QUEST, NEMESIS) 

--------- 
Employment impacts: 
central scenario presented 
uses RHOMOLO model 
output. 

Average 
GDP gain of 
around  
€15.9 bn 
(€15.6 bn to 
€28.5 bn) 
annually over 
2014 -2040 

FP7: Total 
impact on 
GDP 
estimated  
380bn 
over 17 year 
(2007-2023) 
---- 
Peak annual 
impact on 
employment 
in all sectors: 
220 000  
jobs created 
in 2020. 

(NEMESIS: 
229 000 jobs 
in 2030) 

Interim eval.: 
110 000 to 
179 000 jobs 
created 
(2014-2030) 

58 

01/12/2015, 2 years 
after end of FP). 

trademarks 499 
(12.8%, 0.44/€10m) 
H2020 excl. ERC: 
3 210 IPR appl. 
(0.58/€10m) 

FP7+2yrs   
excl. ERC*: 
2 266 (0.6/€10m) 
of which  
patents 1742 (77%) 

*Due to monitoring 
limitations at the time 
IPR applications 
linked to ERC could 
not be reconstructed 

FP7 +9yrs: 6 328 
of which 
patents 5545 
(95.4% awarded) 

6. Monetised value 
of one patent 
ranges between  
under €100 000 
and €1.1 million  
depending on sector 

Innovation output 
under Horizon 2020, 
(benefit to 
participant)  
(EIT Knowledge & 
Innovation 
Communities (KIC) 

Innovation output 
under Horizon 2020 
(benefit to 
participant)  

Innovation output 
under Horizon 
2020 (benefit to 
participant 

7.Start-ups 
created (EIT KICs) 
H2020: 440 
Target: 600 
Baseline 2012: 33 

8. Innovations 
generated 
H2020: 8 000 
tracked by 
Innovation Radar, 
of which 
31% new products  
28.6% significantly 
improved product  

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
(Micro-)economic 
benefit for SME  
(sect 4.1.3 access to 
risk capital) 

(Micro-)economic 
benefit for firm 

(Micro-)economic 
benefits for firm  

9. Access to 
additional private 
capital for SMEs 
Horizon 2020: 
€9.36 billion 
(LEIT) 

10. Growth of 
firm 
H 2020 (all private 
firms): 
20% increase in 
jobs compared to 
non-funded firms  
30% increase in 
total assets and 
revenue growth 
compared to non-
funded firms 
(persistent > 2.5yrs) 
-- 
(SME Instr.ph II) 
28-56% increase in 
growth of firm 

11. Increase in 
patenting (SME) 
Horizon 2020: 
15% -31% increase 
in cite-weighted 
patents  

59 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Citizens/  
EU Society  

II. COSTS 

EU Public  
Administration  

Horizon 2020  
Beneficiaries 

Horizon 2020 
Applicants 

Quantitative  

Comment 

Quantitative  

Comment 

Quantitative 

Comment  

Quantitative   Comment 

1. Direct economic cost 
of R&I funding 
to EU society 
(Operational Expenditure) 

one 
off 

€62 133.6 
million 

2. Administrative costs 
of implementing the R&I 
framework programme 
to EU Public Sector  
(Administrative Expenditure) 

one 
off 

Actual paid 
Operational 
Expenditure  
of Horizon 2020 
(as of 01/01/2023) 

Costs of administrating 
Horizon 2020 are 
incurred by the public 
sector at European level 
but are ultimately a cost 
on EU Society. 

€ 4 292. 3  
million 

In year 2020: 
3.37%  

Target for 2020:  
max 4.6%  

Actual paid 
Administrative 
Expenditure 
of Horizon 2020 
(as of 01/01/2023) 

Administrative 
expenditure as 
share of budget 
envelope  
(excl. JRC, EIT; 
committed as of 
01/01/2023) 

3. Beneficiaries’ 
administrative costs 
of participation 

(Not additional - already 
included in no.1 Operational 
Expenditure)  

one 
off 

Time cost: 
on average 
 4.5 - 7  
person-days  
per month of 
project duration 

Administrative 
costs per 
participation 
incurred to meet 
requirements (e.g. 
reporting). 

Monetised order-of-
magnitude estimate: 
Administrative 
costs of 
beneficiaries 

€ 135 million –  
€ 215 million 

Note: evidence not 
robust. Particularly 
upper bound value 
may be 
substantially higher 

60 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4. Costs of applications 
 Direct costs of preparing 
proposals 

of successful and 
unsuccessful applicants 

one 
off 

as projects of 60 
months and more 
exceed 10 person-
day per project 
months. 

€18 000 
to €37 000 
(evidence not 
robust) 

Successful: 
€609 million 
to €1.25 bn 

Unsuccessful: 

€ 5bn 
to 10 bn 

Cost per 
proposal 
(evidence 
not 
robust) 

Total 
cost of 
applicati
ons 
(evidence 
not 
robust) 

61 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
3.  Simplification measures - additional information on performance 

Expectations - simplification in Horizon 2020  

As presented under section 4.2.2 in the discussion of efficiency, ‘simplification’ was a central to Horizon 2020 and efforts to simplify the programme 
influenced its design, its rules, financial management, and its implementation.  

The  ambition  was  comprehensive,  and  expectations  were  clearly  expressed.  The  interim  evaluation  report  of  the  Seventh  Framework  Programme  had 
concluded that a more radical approach was needed to achieve a ‘quantum leap’ in simplification, and that the risk-trust balance needed to be redressed. 
The European Parliament100 and the European Council101 had called to  radically simplify access, to simplify to boost the attractiveness and lower the 
associated burden of EU research funding. The ‘radical overhaul’ of the administration of FP was seen as the highest priority to be tackled. 

The 2011 ‘Green Paper on a Common Strategic Framework for EU Research and Innovation Funding’ picked up on these findings and requests. It identified 
simplification as a ‘top priority’ in order to make EU research and innovation funding generate more impact and be more attractive to participants; it also 
prompted to simplifying participation by lowering administrative burden, reducing time to grant and time to payment, as well as achieving a better balance 
between cost and trust based approaches.102  

In consequence, Horizon 2020 substantially changed the set-up and management of R&I support. Previous expectations were repeated in the Regulation 
establishing  the  programme,  which  explicitly  introduced  ‘simplification’  as  a  central  aim103.  Simpler  funding  rules  were  expected  to  reduce  the 
administrative costs of participation and to contribute to the prevention and reduction of financial errors104.  

Horizon 2020 performance against time cost targets (TTI, TTS, TTG) 

The quantitative administrative target on time to grant shines a spotlight on one specific aspects of administrative efficiency. Horizon 2020’s binding TTG 
target of 8 months (245 days) per call was more stringent than FP7’s previous TTG target of 270 days. Even stricter TTG targets were set for the SME 
Instrument phase 1 calls (3 months or 92 days), SME I phase 2 calls and FTI calls (6 months or 183 days). An overview over targets and actual achieved 
values for Horizon 2020 and FP7 are summarised in Table 14. Table 15, and Figure 13 below. Variation across Horizon 2020 remained within a range of 
average actual TTGs of 154 to 218 days (FP7: 271-359 days) and of 74% to 98% of agreements signed on time (FP7: 9% - 60%).  

The performance was not uniform, with in particular the initial period of the EIC pilot generating delays for companies of up to up to 12 months 
and considerable uncertainty.  According to the EIC pilot evaluation (2022), ‘the EIC Fund was incorporated on 22 June 2020 and operations effectively 

100 ‘Simplifying the implementation of the research framework programmes’, European Parliament resolution of 11 November 2010 on simplifying the implementation of the Research 
Framework Programmes (2010/2079(INI)). 
101 Council conclusions on Europe 2020 flagship initiative: Innovation Union. 26.11.2010. 
102 GREEN PAPER From Challenges to Opportunities: Towards a Common Strategic Framework for EU Research and Innovation funding (COM(2011) 48 final).  
103 Ambition outlined in Recital 20 to Regulation No 1291/2013 of the European Parliament and of the Council of 11 December 2013  establishing Horizon 2020 - the FP for Research 
and Innovation (2014-2020), OJ L 347, 20.12.2013, p. 104. 
104 Ibid. 

62 

 
 
 
started on 1 September 2020. By that time, 140 companies had already been selected for investment support since the first call was launched in October 
2019. Due diligence only started in August 2020. During this process, some companies were surprised when they were eventually offered a convertible 
loan instead of a full equity investment. Others were taken aback when they understood the implication of the co-investment requirement and realised, they 
had to look for an additional investor by themselves’. 
Table 12: Actual Time-to-grant (TTG)  

Programme 

FP7  

H-2020 
(excl ERC) 

H-2020  
SMEI  
phase 1 

H- 2020  
FTI 

H-2020  
SMEI  
phase 2  

H-2020  
(excl. ERC, 
SMEI  
phases 
1&2,  
FTI) 

Target 
[avg. days] 

Actual TTG 
[avg. days  
(% < t)] 

270 days 

245 days 

245 days 

92 days 

183 days 

183 days 

313 (41%) 

187 (90%) 

208  

92  

194 

152 

Source: Monitoring dashboard data frozen 1.1.2023; FP7 TTG based on an overall number of 23’122 FP7 grants, excluding JTI, RSSF and  
Fusion grants.  

63 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
Figure 7: Actual average Time-to-Grant values of R&I support over time 

          Source: Monitoring dashboard data of FP7 and Horizon 20202. Average TTG values (blue bars) and number of signed R&I support grants of both FPs pooled. 

In addition to the Time-to-Grant target, covering the entire period from submission of proposal to signature of grant  agreement,  further (non-binding) 
expectations on sub segments of this process were formulated for Horizon 2020:  

•  Time-to-Inform (TTI) - target 5 months (153 days) - measures the time from the deadline for submission of proposals to the notification of the 

applicant on the evaluation outcome.  

•  Time-to-Sign (TTS) - target 3 months (92 days) - measures the time from the notification on evaluation outcome to the signature of the grant 

agreement.  

Horizon 2020 met expectations on time-to-inform and time-to-sign, with the actual average TTI (112 days105) beating the target by 41 days and the 
actual average TTS (76 days) by 16 days. This suggests that the TTS component of the time before grant signature was the more challenging period, 
which  is  also  reflected  that  two  programmes  (‘Societal  Challenge’  TTS:  99  days;  ‘Other  priorities’  TTS:  117  days)  on  average  did  not  meet  TTS 
expectations. In both the TTS period exceeded the TTI period, however, due to short average TTI periods (SC: 93; Other: 94) the overall TTG targets were 
met. Variation across the remaining programmes remained within a range of average actual TTIs of 79 to 139 days, and average actual TTSs of 61 to 80, 
meeting the expectations. A comparison to FP7 is not possible with respect to TTI and TTS as these periods were not monitored at the time.  

105 TTI and TTS monitoring dashboard data (frozen 1.1.2023). For 98.27% of the main listed proposals the TTI of Horizon 2020 stayed under 153 days. (Annual Activity report 2022) 
64 

 
 
 
 
 
Table 13: Actual average Time-to-Inform, Time-to-Sign and Time-to-Pay achieved under Horizon 2020 

Target 
[avg.days] 

Horizon 2020 

Excellent Science  

Industrial Leadership 

Societal Challenges 

Spreading Excellence and Widening Actions 

Other Priorities 

H- 2020 
TTI 

H-2020 
TTS 

H-2020 
TTG 

H-2020 
TTP  

153 days  92 days 

245days 

90 days; 30days if 
prefinanced 

112106 

139 

79 

93 

138 

94 

76 

61 

76 

99 

80 

187 (90%) 

68; 6.7(91.5%) 

199 (97%) 

154 (84%) 

191 (83%) 

218 (98%) 

117 

211 (74%) 

Source: Horizon 2020 Dashboard data, 1.1.2023; Time-to-Pay: (ABAC) accounting data 

4.  Potential for further simplification - additional information 

Adaptation costs linked to changes to programme design, rules and procedures, including simplification measures 

The evaluation found that the complexity of the framework programme overall has remained persistently high, resulting in a burden for applicants and 
beneficiaries. A potential for the reduction of this burden therefore does not only stem from the characteristics of any new design, rules or processes as 
such (i.e. costs of a new steady state) but also from the transition period during which the changes are designed, tested, announced, introduced and rolled 
out. The 2018 ECA report107 highlighted the (one off) adaptation costs for beneficiaries from any changes to the programme and the associated legal 
uncertainty, where changes occurred in too quick succession. Participants have to inform themselves about updates, interpret the changes, establish with 
certainty which rules apply to them at a given time and then adjust. The report therefore emphasised the importance of stable and well-designed rules to 
minimise participants’ administrative costs. This implies that, even where beneficiaries’ administrative costs or application costs would be lower once 

106 Calculated based on dashboard data grant numbers and pillar TTI TTS. Sum of Weighted averages/ total H2020 grant number (all excl ERC). 
107 Court of Auditors. Special Report. N.28 (2018) - Concretely, the report pointed at the introduction of simplified rules on personnel costs, which had to be adjusted again shortly after 
in response to negative side-effects that had emerged, leading to ‘confusion and legal uncertainty’. 

65 

 
 
 
 
 
 
 
 
 
simplification  measures  have  taken  hold,  the  transition  process  is  costly  and  can  cancel  out  at  least  some  of  these  positive  effects.  This  suggests  that 
infrequent, carefully designed and piloted step-wise adjustments, which pay close attention to the participants’ perspective and are accompanied by 
clear communication measures and ex post assessments can have a simplification potential in of themselves. 

Table 2 – Achieved simplification and burden reduction (see next page) 

As a European framework programme, Horizon 2020 strives to be as simple and efficient as possible. It falls under the Regulatory Fitness Programme 
(REFIT) of the European Commission, seeking opportunities to simplify and reduce administrative burden for people, businesses and administrations. As 
presented in section 4.2.2 of the evaluation report, Horizon 2020 introduced two main strands of simplification measures: 

•  Structural simplification and a general overhaul of implementation processes, that had primarily the objective to 1) lower the direct administrative 
cost  of  applicants  and  beneficiaries  that  are  associated  with  participating  in  Horizon  2020,  and  2)  increase  the  EU  public  sector’s  efficiency  of 
administering the framework programme through accelerating all processes relating to proposal and grant management, 

•  Simpler funding rules and a revised ‘control and risk strategy’. These measures primarily set out to optimise the balance between the administrative 

costs of beneficiaries and the benefits of reducing financial errors. 

Time to grant – any potential for further simplification? 

The average time-to-grant values reported outperform the targets across the programme (see Annex IV.3 above). At a first glance, a further tightening of 
the targets could hold further potential for simplification. However, like every target, the time to grant target is an imperfect proxy, in this case for the 
efficiency of activities related to the evaluation of grants and the preparation of the grant agreement. The setting (or tightening) of the target -without any 
accompanying measure that would make such a change plausible- is associated with the risk of generating (or increasing) negative unintended consequences, 
particularly when other connected administrative processes are changed at the same time. A shorter time span for grant preparation is associated with an 
increase in risk of errors. Considering the already material error rates of the programme, any such increase in risk would not be welcome. Furthermore, 
while an increased use of lump sum funding is expected to keep financial error rates in check, it will simultaneously shift some of the burden of financial 
checks from the reporting stage to the evaluation (of the proposed lump sums) of proposals. This will potentially reduce again the current scope to further 
tighten the time-to-grant target. Once a wider use of lump sums has been established, a new assessment of the performance against the target can be carried 
out to assess any room for manoeuvre. The evaluation therefore finds that it is not recommendable to incentivise a further shortening of the time-to-grant 
period until the effects of lump sum funding on the timing of the proposal evaluation and on the resulting error rate can be established. 

66 

 
 
 
 
 
TABLE 2:    Simplification achieved and further potential 
PART I: Simplification and burden reduction (savings already achieved)  
Simplification, burden reduction and cost savings achieved already by Horizon 2020, including points of comparison where available. 

Citizens/ EU Society 

EU Public Administration 

Horizon 2020 Beneficiaries 

Horizon 2020 Applicants 

Quantitative 

Comment 

Quantitative 

Comment 

Quantitative 

Comment 

Quantitative 

Comment 

 Administrative cost savings of applicants and beneficiaries (costs associated with participating in Horizon 2020) due to the structural simplification and general overhaul of 
implementation processes of Horizon 2020.  
Enforcement cost savings for beneficiaries, due the introduction of a revised ‘control and risk strategy’. 

One-off 

(change from FP7 to Horizon 
2020) 

 .   

Evidence on 
costs of 
applicants are not 
systematically 
collected. 
Savings due to 
measures have 
not been 
quantified. 

Approx. 2 500 
unique 
beneficiaries 
(5.74 percentage 
points reduction) 

Share of unique 
beneficiaries 
audited: 
H2020: 6.02 %   
FP7: 11.76%  
target: max. 7 % 

Number of 
beneficiaries 
who experienced 
enforcement 
cost saving 
(relative to FP7 
“control and risk 
strategy”.) 

Evidence on 
administrative 
costs of 
beneficiaries are 
not 
systematically 
collected. 
Savings due to 
measures have 
not been 
monetised. 

Administrative cost savings of the EU public service (costs associated with administering the framework programme) through accelerating all processes relating to proposal and grant 
management.  The evaluation found that the introduction of the electronic grant management workflow and the withdrawal of the negotiation stage were key drivers of the acceleration 

One-off 

 (change from FP7 to Horizon 
2020) 

Time-To-Grant (TTG) 
reduction 

Horizon 2020 saved over 
9 500 years of time in the 
EU public sector, relative to 

Time cost 
saving for 
public 
sector  
from 
accelerate
d 

Start dates of 
beneficiaries’ 
projects 
brought forward 
in total by over 
9 500 years of 
time. . 

Time cost saving 
for beneficiaries 
from accelerated 
administrative 
processes. 

67 

 
               
 
  
 
 
 
 
 
 
 
 
 
 
 
 
what it would have taken at 
the speed of FP7. 

H2020*: saving of 
126 days per grant on 
average vs. FP7. 

Expected saving was 
100 days per grant on 
average vs FP7. 

H2020 average TTG:  187 
days per grant. 
90% within target*of 245 
days per grant. 

FP7 average TTG:   
313 days per grant. 
41% within target of 
270 days per grant. 

administr
ative 
processes 
(from end 
of 
deadline 
for 
proposals 
to grant 
agreement 
signature) 

* 
excluding 
ERC 

Key 
drivers:  
- 
electronic 
grant 
manageme
nt 
workflow;  
- removal 
of 
negotiatio
n stage.  

68 

 
 
 
 
 
 
 
 
 
 
 
 
 
Identified further potential simplification and savings that could be achieved with a view to make the initiative more effective and efficient without prejudice to its policy objectives108. 

PART II: Potential simplification and burden reduction (savings)  

Citizens/ EU Society 

EU Public Administration 

Horizon 2020 Beneficiaries 

Horizon 2020 Applicants 

Quantitative  

Comment 

Quantitative  

Comment 

Quantitative 

Comment   Quant
itativ
e 

Comment 

Application of unsuccessful applicants are an area with a potential for efficiency savings for the framework programme.  The evidence base of the evaluation does not allow to specify any 
new simplification measures to the extent, that they could be assessed in terms of their expected costs savings’. Potential existing measures that could be extended include: a targeted, carefully 
tested and designed use of the two-stage evaluation processes; and any measures that prevent the loss of the value inherent in successful-unfunded proposals (proposals above the quality 
threshold but that remained unfunded due to the budget constraint) and allow it to be captured for alternative funding applications at EU or national level. This may include the Seal of 
Excellence measure, after a detailed ex ante assessment. 

One-off 

n/a 

Public sector 
administrative 
expenditure related 
to proposal 
evaluation costs  
are an area with a 
potential for 
efficiency savings, 
to the extent that a 
duplication of an 
evaluation can be 
avoided. 

n/a 

Application costs of 
unfunded proposals are 
an area with a potential 
for efficiency savings 
for the framework 
programme overall. 

Lump sum funding involves the paying out of pre-agreed lump sums (that were specified in the proposal by the grant beneficiary) after the completion of a work package. It renders obsolete 
the financial reporting (by beneficiary) and the checking of financial reports, as well as the reimbursement of detailed eligible costs by the EU public administration).The evaluation of the 
lump sum pilot suggests that a wider use of lump sum funding likely has some simplification potential to reduce beneficiaries’ administrative costs and address the persistence of frequent 
financial errors, highlighted by the European Court of Auditors. The net effect on costs depends on details of implementation. 

One-off 

n/a 

The use of lump sums 
has the potential to 
reduce financial errors 
by removing financial 
reporting and the 
reimbursement on the 
basis of eligible costs 
(both sources of 

n/a 

69 

Public sector 
administrative 
expenditure is 
expected to change 
due to multiple 
factors. The 
direction of the net 
effect on public 

n/a 

n/a 

The use of lump 
sums has 
potential to 
reduce the net 
administrative 
costs of 
beneficiaries, 
who no longer 

Application costs may 
increase, as proposals 
have to submit an 
additional budget table 
for the project, to 
justify the lump sums. 
The cost of generating 
the budget information 

 
 
 
 
 
 
financial errors in R&I 
funding). The extent to 
which a reduction of 
errors can be achieved, 
and a reduction of the 
error rate can be 
observed, depends on 
details of 
implementation, 
including that of ex post 
project reviews and any 
changes to the audit 
strategy. While the 
rationale of lump sum 
funding supports the 
assumption that financial 
errors will overall be 
reduced, the piloted 
projects have not yet 
generated any ex post 
evidence to allow for a 
validation of this 
assumption and an 
ex ante estimation of 
future simplification 
effects. 

sector costs 
depends on 
implementation 
details that 
determine the 
additional 
workload of 
proposal evaluators 
and possible 
adjustment costs 
for project officers. 
The net effect will 
also be affected by 
beneficiaries’ 
strategic behaviour 
(unintended 
effects) in response 
to the measure over 
the medium-term. 
The currently 
available evidence 
base is insufficient 
to assess the 
direction or 
magnitude of the 
net effect on public 
sector 
administrative 
costs. 

have to report 
on eligible costs 
and resources 
for 
reimbursements, 
but receive 
shares of the 
lump sum, once 
work packages 
have been 
completed. The 
net effect for 
beneficiaries 
depends on 
details of 
implementation 
and 
beneficiaries’ 
strategic 
behaviour 
(unintended 
effects) in the 
medium term.)  
The currently 
available 
evidence base is 
insufficient to 
assess the 
magnitude of 
the benefit to 
beneficiaries. 

is not fully additional 
but to a large extent 
part of the baseline: 
Project management 
best practice and 
existing requirements 
of the programme mean 
that applicants are 
assumed to calculate 
the project budget at 
proposal stage already. 
However, adapting the 
budget to the format, 
structure and level of 
detail requested in the 
proposal template and 
filling in the template 
gives rise to additional 
costs.  Any change will 
be affected by details of 
implementation, 
including the 
availability and user 
friendliness of guidance 
for applicants.  The 
currently available 
evidence base is 
qualitative and does not 
allow a quantification  
of the expected effect 
on applicants. 

108    This assessment is without prejudice to a possible future Impact Assessment. 

70 

 
 
 
 
5.  Value-for-money for EU Society  (Benefit cost ratio calculation) 

Table  14  below  shows  the  costs  and  benefit  figures  used  in  the  benefit  cost  ratio  calculation  presented  in  Section  4.2.1  of  the  evaluation  report.  The 
evaluation calculates a benefit cost ratio over the period 2014-2040. This period was chosen to allow time for the emergence of wider benefits of R&I 
investments. Projects will only have all ended by December 2028, eight years after the programme period came to an end in 2020. The 2040 cut-off date 
allows for a minimum of 13 years for all projects to have been completed and results to feed through to impacts that affect society. 

The benefit cost ratio uses GDP impacts, as the closest proxy for the overall welfare benefits for EU society. Macroeconomic impacts have been adjusted 
for inflation and are reported below in current prices. (GDP impact figures in section 4.1.3 are reported in 2020 prices.) In this way they could be used in a 
calculation with the expenditure figures. All the three model outputs have been used to reflect uncertainty. The results suggest that the benefit cost ratio 
(BCR) of Horizon 2020 is at least 5, which means that one euro of costs to society associated with the programme (programme costs and costs to 
applicants) is expected to bring about five euros of benefits for EU citizens (measured through GDP impact) in the period up to 2040. 

For comparison, a set  of benefit  cost  ratios over  the shorter period until  2030 has also been calculated, again  using the three model outputs to  reflect 
uncertainty. The three models describe different trajectories of how much and by when benefits grow and peak over time, which explains the difference in 
ranking (see Figure 9 in Section 4.1.3 of the evaluation report for a visualisation). During this shorter period, the total cost value does not change, as all 
costs are incurred at the start. As can be expected, by 2030 the BCR values are lower, with a central BCR still closer to 4. This is because in 2030, the 
last projects had only ended 2 years prior, and benefits had 10 years less to channel through to a marketable impact and for GDP gains to accumulate.  

Table 14: Benefit Cost Ratio calculation 

Costs of Horizon 2020 
(1) Operational Expenditure  
(budget, current prices)  

(2) Administrative Expenditure  
(budget, current prices) 

EUR 71.195 billion 

EUR 4.428  billion 

(3) Cost of application 
(Note: range of low to high average values; costs include successful 
and unsuccessful applicants.) 
Please note: evidence not robust/ low level of confidence in figures. 

EUR 5.61 billion 

EUR 11.25 billion 

Total Cost of Horizon 2020 (low, high; current prices) 

EUR 81.233 billion 

EUR 86.874 billion 

71 

 
 
 
 
 
 
 
 
 
 
 
 
 
Benefits of Horizon 2020 
(1) Total Benefits of Horizon 2020 - up to 2040 
(Note: GDP impact used in calculation as closest proxy to welfare 
impact; period 2014-2040; current prices) 

EUR 915.1 billion - estimated by NEMESIS model  

EUR 492.0 billion - estimated by RHOMOLO model  

EUR 482.3 billion - estimated by QUEST model  

Benefit Cost Ratio (BCR) – up to 2040 

NEMESIS 
RHOMOLO 
QUEST   

low 
10.5 
5.7 
5.6 

high  
11.3 
6.1 
5.9 

Benefit Cost Ratios using a shorter 2014 - 2030 period: 

Benefits of Horizon 2020 
(2) Total Benefits of Horizon 2020 - up to 2030 
(Note: GDP impact used in calculation as closest proxy to welfare 
impact; period 2014-2030; current prices) 

EUR 481.6 billion - estimated by NEMESIS model / 

EUR 354.7 billion - estimated by RHOMOLO model  

EUR 328.9 billion - estimated by QUEST model / 

Benefit Cost Ratio (BCR) – up to 2030 

NEMESIS 
RHOMOLO 
QUEST   

low 
5.5 
4.1 
3.8 

high  
5.9 
4.4 
4.0 

72 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
Figure 8: Comparison of approximate magnitudes of costs and wider economic benefit of Horizon 2020 

Source: EU Commission illustration. Fuzzy edge indicates uncertainty with regards to the size of impact.  

73 

 
 
 
ANNEX V. STAKEHOLDER CONSULTATION – SYNOPSIS REPORT  

In  support  of  this  evaluation,  a  broad  range  of  consultation  activities  were  conducted:  the  call  for 
evidence, the public consultation, interviews, surveys of participants and beneficiaries as well as targeted 
consultations.  

To  ensure  that  all  possible  views  are  well  reflected  and  to  ensure  transparency  and  accountability, 
consultations with various categories have been held in the frame of the ex post evaluation of Horizon 
2020.  The  consultation  process  did  not  start  from  zero,  as  the  Commission  based  its  work  on  the 
consultations that took place in 2016 for the interim evaluation of Horizon 2020109 which provided useful 
information on the mapping, priorities and views of all major interested parties.  

Stakeholder mapping 

Stakeholder  groups  that  are  concerned  by  Horizon  2020  as  a  whole  can  be  broken  down  into  the 
following categories: academia, businesses (including  small  and medium-sized enterprises), National 
Contact  Points110  and  public  authorities  as  well  as  non-governmental,  research  and  umbrella 
organisations.111   

Beyond that, the following Institutions have in the past contributed to the evaluation of the Framework 
Programme:  

• 
• 

• 

• 

the Council conclusions112 on the Interim Evaluation of Horizon 2020, adopted on 01/12/2017, 
the  European  Parliament,  which  reported  on  the  assessment  of  Horizon  2020113  and  the 
implementation in line with the interim evaluation,114   
the  European  Economic  and  Social  Committee  that  provided  recommendations  for  the  Interim 
Evaluation of Horizon 2020,115  
the Committee of the Regions and the European Research Area and Innovation Committee which 
is  a  policy  advisory  body  whose  main  mission  is  to  provide  strategic  input  on  any  research  and 
innovation issue relevant to the development of the European Research Area.116  

Other consultation activities conducted under the remit of the external evaluation studies 

Next  to  the  consultation  activities  that  were  accessible  via  the  ‘Have  your  say’  portal,  targeted 
consultations in the forms of workshops, interviews and surveys were conducted under the remit of the 
various external evaluation studies, specifically addressing applicants, participants national and regional 
authorities as well as business representatives.  

Interviews 

The main objective of conducting interviews was to gather evidence from different actors concerned by 
the  Framework  programme,  offering  the  possibility  to  give  an  objective  assessment  by  taking  into 
account the different views. Interviews were particularly used in case studies as well as international 
benchmarks.  Beyond  that,  interviews  were  conducted  to  confirm  and  complement  data  collection  to 

109 Results of the Horizon 2020 Stakeholder Consultation, 2018, https://op.europa.eu/s/yXBt  
110 National Contact Points (NCPs) are independent organisations of different nature (e.g. Ministries, Academies of Science, 
Research agencies) that act as information providers to applicants in their native language. They are based in all EU countries 
and Associated States as well as in some non-European countries. 
111 So-called ‘umbrella organisations’ are industry-specific associations of EU public interest. 
112 Council conclusions 15320/17 https://www.consilium.europa.eu/media/31888/st15320en17.pdf  
113 Briefing: Interim evaluation of Horizon 2020 
https://www.europarl.europa.eu/RegData/etudes/BRIE/2018/614771/EPRS_BRI(2018)614771_EN.pdf  
114 European Parliament Report on the assessment of Horizon 2020 implementation (A8-0209/2017) 
https://www.europarl.europa.eu/doceo/document/A-8-2017-0209_EN.pdf    
115  European  Economic  and  Social  Committee 
https://www.eesc.europa.eu/en/our-work/opinions-information-reports/information-reports/interim-evaluation-horizon-
2020  
116 European Research Area and Innovation Committee, 
https://www.eumonitor.eu/9353000/1/j9vvik7m1c3gyxp/vh7ej5swwyyh  

interim  evaluation  of  Horizon  2020 

recommendations: 

74 

 
 
 
 
support the drafting of findings and conclusions. In total, 1 403 interviews117 were conducted in support 
of  this  evaluation  –  these  interviews  do  include  same  actors  on  different  topics  by  gathering  large 
amounts  of  qualitative  data  among  Member  States’  and  associated  countries’  representatives, 
Commission staff and other stakeholders (as explored in the stakeholder mapping section).  

Surveys 

In the frame of the evaluation studies, different surveys were designed, disseminated and analysed to 
gather both quantitative and qualitative data on Horizon 2020.  
By  conducting  six  separate  online  questionnaires,  successful  and  unsuccessful  participants  were 
surveyed under the Excellent Science study – in total, 5 417 complete and 449 partial responses were 
received. The following groups were targeted by these surveys: 

•  Horizon 2020 beneficiary organisations (including beneficiary organisations that participated in 

MSCA, SEWP, INFRA and SwafS).  

•  Horizon 2020 unsuccessful applicant organisations (organisations that unsuccessfully applied for 

MSCA, SEWP, INFRA and SwafS).  

•  Horizon 2020 MSCA IF fellows.  
•  Unsuccessful Horizon 2020 MSCA IF applicants.  
•  Horizon 2020 ERC principal investigators. 
•  Unsuccessful Horizon 2020 ERC applicants.   

For the  Evaluation  Study  on the European  Framework Programmes for  Research and  Innovation for 
addressing Global Challenges and Industrial Competitiveness with a focus on activities for the activities 
related to the Digital and Industrial Transition, Horizon 2020 LEIT applicants and participants were 
targeted: A single questionnaire was developed that targeted all stakeholders, namely all partners in the 
projects funded (successful applicants) as well as all partners in proposals that were not funded even 
though the proposals reached scores above threshold (‘high-quality’ proposals).  A total of 1 342 valid 
responses were received on the survey, reaching an overall response rate of 7%.  

In  view  of  the  Green  Transition  evaluation  study,  a  survey  was  conducted  targeting  successful 
applicants  –  this  survey was  designed  during  the  inception  phase  and  was  later  further  enriched  and 
developed  with  gateways  to  take  into  account  various  factors.  In  total,  the  survey  received  1  333 
contributions  from  respondents  among  which  771  were  filled  out  in  its  entirety.    The  survey  and 
following analysis was ordered by societal challenges probing for relevance, effectiveness and EU added 
value whereas for efficiency and the overall green transition  process  information  was analysed at an 
aggregated  level.  Still,  the  analysis  was  further  organized  by  topic  within  each  evaluation  criterion 
illustrated with respective graphs. 

The studies on Resilient Europe and Innovative Europe did not conduct surveys within the frame of their 
methodologies. 

Policy workshops 

20 policy workshops were conducted in support of this evaluation. The workshops were implemented in 
the frame of the external evaluation studies. They aimed to consolidate and increase the robustness of 
findings and conclusions arising from the data collection conducted through other methods with the aim 
of addressing evidence gaps whenever needed. 

Call for evidence 

The  ‘call  for  evidence’  opened  on  1  July,  2022  and  closed  on  29  July  2022.  The  overall  number  of 
responses submitted was 35. This number includes two responses that were not considered in the analysis 
as  they  were  out  of  scope,  i.e.  not  concerning  the  Horizon  2020  programme.  Four  comments  were 

117 This included 223 interviews in the frame of the Resilient Europe study, 138 for the study on Relevance, 217 in view of 
Digital and Industrial Transition, 224 on Excellent Science, 195 on Innovative Europe, 131 on External Coherence and 85 in 
view of cross-cutting issues.  

75 

  
 
 
 
submitted from the same person and have been then considered as a single contribution. Moreover, 4 
organisations  and  one  individual  submitted  the  same  response  to  the  ‘call  for  evidence’  on  the  final 
evaluation of Horizon 2020 and the interim evaluation of Horizon Europe. Five position papers were 
received  and  all  of  them  were  considered  in  the  scope  of  the  consultation.  As  for  the  geographical 
distribution of the responses received, Germany is the country with the highest number of responses 
(11), accounting for more than one third of the responses, followed by France (6) and Belgium (4). In 
total,  replies  came  from  11  different  countries,  including  two  non-EU  Member  States  (the  United 
Kingdom and Switzerland). 

The findings from the feedback received during call for evidence were taken into consideration in the 
survey  design  for  the  public  consultation.  In  view  of  content  moderation,  no  feedback  had  to  be 
unpublished as all contributions were in line with the content moderation rules. For additional content-
related  information,  please  consult  the  ‘Supporting  Information:  Call  for  evidence’  section  further 
below. 

Public consultation: scope and objectives 

In line with the Better Regulation guidelines and toolbox118, the public consultation on Horizon 2020 
forms part of a combined consultation and evaluation exercise.119 It aimed to explore stakeholders’ views 
regarding  the  key  aspects  of  the  past  and  the  present  as  well  as  the  future  of  the  EU  Framework 
Programme for Research and  Innovation, notably for the  ex post evaluation of Horizon 2020 (2014-
2020),  the  interim  evaluation  of  Horizon  Europe  (2021-2023)  as  well  as  to  receive  inputs  from 
stakeholders to be used for the definition of strategic orientations for the Horizon Europe Strategic Plan 
(2025-2027).  

The  reason  for  conducting  a  joint  consultation  is  the  relatively  short  time  span  between  the  legal 
obligation  for  the  Horizon  2020  ex  post  evaluation  and  the  legal  obligation  for  the  Horizon  Europe 
interim evaluation. Additionally, another reason for conducting a joint consultation instead of reaching 
out to the broad public on three separate instances was to counter stakeholder fatigue, also bearing in 
mind that all three dimensions concern the same group of stakeholders. Nevertheless, it is important to 
note that this public consultation was geared towards anyone with an interest in the EU R&I Framework 
programmes, not only towards beneficiaries and the main stakeholder groups delineated in the section 
above but also unsuccessful applicants as well as independent experts.  

The combined public consultation was accessible in English, French and German on the Have Your Say 
web portal from 01/12/23 until 23/03/23. Respondents had the possibility to submit their replies  any 
official EU language resulting in in 2 788 responses and 265 position papers in total. For the section on 
Horizon 2020, 1 818  responses were submitted  along  with  21 position papers.  The factual summary 
report, along with all contributions to the three dimensions covered in this public consultation as well as 
position papers are accessible on the Have your Say portal. Findings in this consultation did not only 
feed into the analysis presented on the following pages as well as highlighted in the respective sections 
in the main Staff Working Document but also form basis for the development of the 10th Framework 
Programme for Research and Innovation.  

Methodology used for the analysis of the responses received through the public consultation 

Quantitative analysis 

Quantitative analysis was conducted by means of descriptive statistics, differentiating and comparing 
responses  of  different  groups  of  respondents.  Correct  representation  and  interpretation  of  results  are 
fundamental to drawing coherent conclusions which is why the number of respondents has been shown 
along with percentages. Linkages between answers and respondents’ characteristics such as participation 
in  the  programme,  country  affiliation  and  type  of  respondent  (e.g.  Member  State  and  business 
organisation representatives, researchers). When evident, correlations between answers given in closed 

118 Better Regulation Toolbox, notably Tool #52. 
119 Better Regulation Toolbox, Tool #50, p. 434. 

76 

  
 
 
 
questions have been explored. The summary statistics were bundled in .xml format which allowed for 
swift cross-comparison among the various dimensions covered in the public consultation survey.   

Qualitative analysis 

Key  messages  were  extracted  from  qualitative  contributions,  primarily  position  papers  and  open 
questions present in the public consultation survey. Same holds true for the analysis of the feedback 
contributions  received  for  the  call  for  evidence.  As  only  21  position  papers  exclusively  addressed 
Horizon 2020, instead of using tools such as Nvivo or Python, contributions were clustered by topics 
and  specific  aspects  raised  in  both  position  papers  and  open  questions  by  means  of  using  Excel, 
presenting findings in a contribution matrix.  

Content moderation according to Better Regulation Tool #54120 

In  view  of  content  moderation,  only  three  contributions  were  unpublished:  all  three  were  taken  into 
consideration content-wise, however in two cases GDPR-related concerns led to unpublishing on the 
Have your say portal. Another respondent reached out to the support team of the public consultation via 
the  indicated  functional  mailbox  asking  to  unpublish  the  contribution  as  a  wrong  attachment  was 
uploaded as a position paper – for the analysis, the newer position paper was taken into account.  

Identification of campaigns121 

Although there was some coordination between some of the respondents (e.g., those participating in the 
same network, cluster, or country), as testified by the uploading of the same position paper by multiple 
respondents, the analysis of the consultation results does not indicate any campaign affecting the overall 
results.122 

120 Better Regulation Tool #54, p. 478. 
121 Better Regulation Tool #54, p. 476. 
122 Overall, 23 campaigns (coordinated responses to the survey by more than one respondent and up to 8). The 23 campaigns 
include responses by 70 respondents, representing 3.9% of all responses. 

77 

  
 
 
 
 
 
Public consultation: Participants 

In  total,  1 818  chose  to  complete  the  section  of  the  consultation  on  Horizon  2020  programme. 
Contributions were received from a wide range of actors. 51% (934) of the respondents are part of 
academic or research institutions, 18% (317) are companies or business organisations, and 16% (284) 
are  citizens  (EU  and  not  EU).  The  remaining  16%  (283)  of  respondents  cover  different  types  of 
stakeholders (e.g., public authorities, NGOs, business associations, environmental organisations, trade 
unions).  Among  the  92  public  authorities  that  responded,  38  represent  the  national  level,  21  the 
international level, 19 the regional level and 14 the local level.  

Figure 9: I am giving my contribution as… (N=1 818) 

61; 3%

36; 2%

3; 0%

1; 0%

68; 4%

83; 5%

92; 5%

Academic/research institution

Company/business organisation

EU citizen

Public authority

Other

223; 12%

317; 18%

934; 51%

Non-governmental organisation (NGO)

Non-EU citizen

Business association

Environmental organisation

Trade union

Two-thirds (1 191) of the respondents provided personal views, whereas one-third (586) contributed as 
a representative of an organisation or institution. Two-thirds (963) of organisations that responded were 
large,  while  one-third  of  respondents  comprised  micro  (179),  small  (147)  or  medium  (245)  size 
organisations. 

Figure 10: What is the size of your organisation? (N=1 534) 

179; 12%

147; 9%

245; 16%

963; 63%

Large (250 or more)

Medium (50 to 249
employees)

Small (10 to 49 employees)

Micro (1 to 9 employees)

Geographical coverage 
The consultation gathered responses from 74 different countries, including all 27 EU Member States 
+ UK123. 89% (1 620) of the contributions came from respondents based in EU28 Member States, 8% 
(139)  from  Horizon  2020  Associated  Countries,  and  3%  (59)  from  Third  Countries124.  The  largest 
number of contributions came from Italy (13%; 244), followed by Spain (12%; 212), Germany (12%; 

123 An EU Member State when Horizon 2020 was implemented.  
124 The responses received from Third Countries are from: Argentina (3), Australia (2), Bangladesh (1), Belarus (1), Brazil 
(6), Canada (3), Cape Verde (1), China (4), Colombia (4), Congo (1), Egypt (2), El Salvador (1), Ethiopia (1), Guatemala 
(1), India (5), Japan (1), Jordan (1), Kenya (2), Kosovo (1), Lebanon (1), Mexico (1), Mozambique (1), New Zealand (1), 
Pakistan (1), Palestine (1), Philippines (1), Russia (1), South Africa (1), Taiwan (1), Uganda (1), United States (5), Venezuela 
(1), Yemen (1). 

78 

  
 
 
 
 
 
211) and France (10%; 183). Looking at non-EU countries, the largest number of contributions came 
from Switzerland (2.3%; 42), Norway (1.8%; 32) and Turkey (1.3%; 24). 

Figure 11: What is your country of origin? – EU 28 Member States (N=1 620) 

244

212

211

183

117

103

66 66 62

300

250

200

150

100

50

0

48 46 43 40 36 28 22 22

12 9

8

7

7

7

6

5

5

3

2

y
l
a
t
I

i

n
a
p
S

e
c
n
a
r
F

y
n
a
m
r
e
G

s
d
n
a
l
r
e
h
t
e
N

i

m
u
g
e
B

l

a
i
r
t
s
u
A

e
c
e
e
r
G

l

a
g
u
t
r
o
P

n
e
d
e
w
S

k
r
a
m
n
e
D

d
n
a
n
i
F

l

i

a
h
c
e
z
C

d
n
a
e
r
I

l

i

a
n
a
m
o
R

y
r
a
g
n
u
H

i

a
n
e
v
o
l
S

a
i
t
a
o
r
C

a
i
r
a
g
u
B

l

s
u
r
p
y
C

a
i
k
a
v
o
l
S

i

a
n
o
t
s
E

a
t
l
a
M

a
i
v
t
a
L

i

a
n
a
u
h
t
i
L

g
r
u
o
b
m
e
x
u
L

d
n
a
o
P

l

i

m
o
d
g
n
K
d
e
t
i
n
U

Figure 12: What is your country of origin? – Horizon 2020 Associated Country (N=139) 

N. of responses by Associated country (N=139)

42

32

24

45
40
35
30
25
20
15
10
5
0

9

8

5

4

4

4

3

2

1

1

0

0

0

79 

  
 
 
 
 
 
 
 
Experience with the framework programmes 
The respondents were asked to select one or more options describing their experience with Horizon 2020 
and Horizon Europe.125 91% of respondents (1 648) participated in Horizon 2020126 and 64% (1 163) 
are Horizon Europe beneficiaries.127 39% (703) of respondents stated that they ‘proposed project(s) to 
receive  funding  from  Horizon  2020  but  were  unsuccessful’.  However,  considering  that  the  same 
respondent  could  select  multiple  options,  only  2.5%  (45)  of  respondents  applied  for  Horizon  2020 
funding  and  were  never  successful.128  Respondents  also  include  organisations  supporting  other 
entities  that  apply  for  or  participate  in  the  EU  R&I  framework  programmes  (26%;  478)  and 
organisations that have never applied for funding but are interested in R&I (4%; 76).  

Figure 13: Please select the option(s) that best describe(s) your experience with the European Research 
and Innovation programmes (N=1 818; multiple answers possible) 

I/my organisation has participated/is currently
participating in at least one project funded by Horizon
2020 (2014-2020)

I/my organisation has participated/is currently
participating in at least one project funded by Horizon
Europe (2021-2027)

I/my organisation has proposed project(s) to receive
funding from Horizon 2020 (2014-2020) but was
unsuccessful

I/my organisation has proposed project(s) to receive
funding from Horizon Europe (2021-2027) but was
unsuccessful

91%

64%

39%

33%

I/my organisation supports other entities that apply for /
participate in the EU R&I Framework Programmes

26%

I/my organisation is interested in research and innovation
but has not applied so far for EU funding under the
European Research and Innovation Programme

4%

None of the above

1%

0

200

400

600

800 1000 1200 1400 1600 1800

The majority of respondents (73%; 1321) were mainly active or interested in the part of Horizon 2020 
concerning  societal  challenges  (Pillar  III)  and  more  than  one-third  of  them  participated  or  were 
interested in Excellent Science actions (Pillar I), namely the Marie Skłodowska-Curie Actions (MSCA) 

125  ‘Please  select  the  option(s)  that  best  describe(s)  your  experience  with  the  European  Research  and  Innovation 
programmes)’. The question allowed multiple answers. Therefore, the same organisation could be, for instance, a beneficiary 
of both Horizon 2020 and Horizon Europe, or an unsuccessful applicant of Horizon 2020 but a beneficiary of Horizon Europe, 
or  both  an  unsuccessful  applicant  and  a  beneficiary  of  Horizon  2020,  if  it  submitted  multiple  proposals  with  different 
outcomes. 
126  They  selected  the  response  option  ‘I/my  organisation has participated/is  currently  participating  in  at  least  one  project 
funded by Horizon 2020 (2014 – 2020)’. 
127  They  selected  the  response  option  ‘I/my  organisation has participated/is  currently  participating  in  at  least  one  project 
funded by Horizon Europe (2021 – 2027)’. 
128 They selected the response option ‘I/my organisation has proposed project(s) to receive funding from Horizon 2020 (2014 
–  2020)  but  was  unsuccessful’  alone  or  with  other  response  options,  but  they  did  not  select  ‘I/my  organisation  has 
participated/is currently participating in at least one project funded by Horizon 2020 (2014 – 2020)’. 

80 

  
 
 
 
 
(46%; 843), the European Research Council (ERC) (35%; 638) and Future and Emerging Technologies 
/ Pathfinder (34%; 310). 

Figure 14: In which of the following areas of Horizon 2020 are you or your organisation mainly active  
and or interested in? (N=1 818; multiple answers possible) 

73%

46%

35%

34%

R&I to tackle societal challenges

Marie Skłodowska-Curie Actions

European Research Council

Future and Emerging Technologies / Pathfinder

Leadership in Enabling and Industrial Technologies

European Research Infrastructures

Cooperation between science and society

Spreading Excellence and Widening Participation

SME Instrument / European Innovation Council Pilot

Fast Track to Innovation

26%

26%

24%

20%

18%

17%

European Institute for Innovation and Technology

15%

All of the above

Access to Risk Finance

4%

4%

None of the above

1%

0

200

400

600

800

1000

1200

1400

Overview of position papers 
21 position papers uploaded in response of this consultation included content relevant to the part of the 
consultation on Horizon 2020. Among the 21 position papers, 8 were written by academic or research 
institutions, 6 by non-governmental organisations and 3 by business associations. The largest number 
of position papers came from Belgium (5) and France (3). 

Results of the consultation 

The respondents’ experience with Horizon 2020 

The benefits of participating in Horizon 2020 

The  majority  of  respondents  (74%;  1 324)  agreed  that  participating  in  Horizon  2020  ‘improved 
cooperation  with  partners  from  other  countries  (within  the  EU  and  beyond)’,  39%  (697)  agreed  that 
Horizon  2020  ‘improved  excellence  in  research  and  innovation’  compared  to  other  programmes 
available  in  EU  Member  States  or  Associated  Countries,  and  34%  (602)  agreed  that  Horizon  2020 
brought the ‘possibility to finance projects which otherwise could not be supported at national and/or 
regional level’. Less than 1% (0.08%; 15) of respondents stated that there was ‘no additional benefit’ in 
participating in Horizon 2020 compared to other national and/or regional R&I programmes. 

81 

  
 
 
 
 
 
Figure 15: According to you, what are the main benefits of participating in Horizon 2020 compared to 
national and/or regional R&I programmes in EU Member States or Associated countries? Select 
maximum 3 answers (N=1 790) 

1324

697

602

596

433

386

306

302

Improved cooperation with partners from other countries (within
the EU and beyond)

Improved excellence in research and innovation (e.g., more high
impact publications and patents)

Possibility to finance projects which otherwise could not be
supported at national and/or regional level

Improved international visibility

Strengthened interdisciplinary cooperation

Strengthened critical mass to address pan-European challenges

Horizon 2020 provided financial means at a scale not provided in
national and regional schemes

Strengthened cooperation between academia and the private
sector

Reimbursement of costs at a higher level than in national and/or
regional research and innovation programmes

Horizon 2020 covered a topic not covered by national and regional
R&I support

Horizon 2020 supported the development of emerging
technologies

Horizon 2020 helped to bring innovations to the market

Additional risk capital provided to companies

There are no additional benefits compared to national / regional
support

204

155

109

76

22

15

0

200

400

600

800

1000

1200

1400

The analysis by country group shows that, although the ranking of response options is aligned, certain 
benefits are considered particularly relevant for some country groups: 

▪  76.6%  of  respondents  from  EU  Associated  Countries  selected  ‘improved  cooperation  with 
partners from other countries’ compared to 74% of respondents from EU15, 73.3% from EU13 
and 61% from Third Countries. 

▪  49.4% of respondents from EU13 and 45.3% from EU Associated Countries selected ‘improved 
excellence in research and innovation’ compared to 42.4% of respondents from Third Countries 
and 36.8% from EU15. 

▪  45.8% of respondents from Third Countries and 40% from EU13 selected ‘improved international 
visibility’  compared  to  38.7%  of  respondents  from  EU  Associated  Countries  and  31.4%  from 
EU15. 

▪  23.6% of respondents from EU15 selected ‘strengthened critical mass to address pan-European 
challenges’ compared to 19% of respondents from EU Associated Countries, 12.8% from EU13 
and 6.8% from Third Countries. 

82 

  
 
 
 
 
 
 
Table 15: According to you, what are the main benefits of participating in Horizon 2020 compared to 
national  and/or  regional  R&I  programmes  in  EU  Member  States  or  Associated  countries?  Select 
maximum 3 answers (EU15 N= 1 414; EU13 N= 180; EU Associated Countries 

RESPONSE OPTION 

EU15  EU13 

EU 
ASSOCIATED 
COUNTRIES 

THIRD 
COUNTRIES 

Improved cooperation with partners from other 
countries (within the EU and beyond) 

Improved excellence in research and innovation 
(e.g., more high impact publications and patents) 

74.0%  73.3% 

76.6% 

61.0% 

36.8%  49.4% 

45.3% 

42.4% 

Possibility to finance projects which otherwise could 
not be supported at national and/or regional level 

35.3%  28.3% 

25.5% 

28.8% 

Improved international visibility 

Strengthened interdisciplinary cooperation 

Strengthened critical mass to address pan-European 
challenges  

Horizon 2020 provided financial means at a scale 
not provided in national and regional schemes 

Strengthened cooperation between academia and the 
private sector 

Reimbursement of costs at a higher level than in 
national and/or regional research and innovation 
programmes 

Horizon 2020 covered a topic not covered by 
national and regional R&I support 

Horizon 2020 supported the development of 
emerging technologies  

31.4%  40.0% 

24.3%  22.8% 

23.6%  12.8% 

38.7% 

23.4% 

19.0% 

45.8% 

28.8% 

6.8% 

17.3%  19.4% 

11.7% 

18.6% 

16.7%  15.0% 

19.7% 

20.3% 

11.9%  9.4% 

10.2% 

8.5% 

8.3%  10.0% 

11.7% 

5.1% 

6.2%  6.7% 

4.4% 

6.8% 

5.1% 

0.0% 

1.7% 

Horizon 2020 helped to bring innovations to the 
market 

4.2%  2.8% 

5.8% 

Additional risk capital provided to companies 

1.3%  1.7% 

There are no additional benefits compared to 
national / regional support 

1.0%  0.0% 

0.7% 

0.0% 

83 

  
 
 
 
 
 
The reasons preventing participation in Horizon 2020 

The  main  reasons  that  held  back  potential  beneficiaries  from  Horizon  2020  were  all  linked  to 
application costs, namely the low success rates of applicants, which both successful and unsuccessful 
applicants agree on (57%; 924 and 69%; 31 respectively), the cumbersome application process  (42%; 
681  among  successful  and  53%;  24  among  unsuccessful  applicants,  and  50%;  67  respondents  from 
associated  countries),  as  well  as  the  lack  of  resources:  Interestingly,  a  larger  fraction  of  successful 
applicants (41%; 670) than unsuccessful candidates (27%; 12) deemed the potential applicant’s lack of 
resources  to  prepare  a  proposal  as  a  reason  negatively  affecting  participation.  Compared  to  EU-13 
respondents,  respondents  from  associated  countries  are  10  percentage  points  less  likely  to  identify 
limited resources as a deterring factor for participation. 

Low success rates were also considered a further deterring factor to participation by 59% (830) of EU-
15  respondents,  64%  (115)  of  EU-13  respondents  and  40%  (54)  of  respondents  from  associated 
countries. Other deterring reasons mentioned by the respondents129 are: difficulties in finding a project 
coordinator,  difficulties  in  involving  industrial  partners,  concerns  about  sharing  valuable  knowledge 
with partners (especially for businesses), lack of expertise to prepare a proposal and high costs for hiring 
external consultants for proposal preparation.  

Figure 16: In your view, what are the main reasons that may have prevented potential beneficiaries from 
participating in Horizon 2020? Select maximum 3 answers. (N=1 781) 

Success rates in Horizon 2020 are too low to be worth applying

1019

The Horizon 2020 application process is cumbersome

Limited financial/human resources to prepare a proposal

Inadequate knowledge of the EU research and innovation
framework programme

The Horizon 2020 project implementation rules are cumbersome

Difficulties in finding project partners

Other

Lack of a relevant area/topic for my research interests

756

728

698

417

396

284

273

Lack of expertise

226

Limited attractiveness compared to other national or regional
programmes

Concerns about sharing valuable knowledge with partners

153

109

Limited attractiveness compared to other European or
international programmes

35

0

200

400

600

800

1000

1200

129 Question: “if other, please specify”. 

84 

  
 
 
 
 
The top four reasons selected by successful and unsuccessful applicants are the same, albeit with some 
variation:  

Table  16:  Top  4  for  answers  for  ‘In  your  view,  what  are  the  main  reasons  that  may  have  prevented 
potential  beneficiaries  from  participating  in  Horizon  2020?  Select  maximum  3  answers’  (Successful 
applicants N= 1 622; Unsuccessful applicants N= 45) 

RESPONSE OPTION 

HORIZON 2020 
SUCCESSFUL 
APPLICANTS 

HORIZON 2020 
UNSUCCESSFUL 
APPLICANTS 

Success rates in Horizon 2020 are too low to be worth 
applying 

The Horizon 2020 application process is cumbersome 

Limited financial/human resources to prepare a 
proposal 

Inadequate knowledge of the EU research and 
innovation framework programme 

57.0% 

41.9% 

41.3% 

39.3% 

68.9% 

53.3% 

26.7% 

33.3% 

Looking at disaggregation by country and stakeholder group for these top four reasons: 

Table 17: Top 4 answers for ‘In your view, what are the main reasons that may have prevented potential 
beneficiaries from participating in Horizon 2020? Select maximum 3 answers’ (EU15 N= 1 407; EU13 
N= 179; EU Associated Countries N= 136; Third Countries N= 59) 

RESPONSE OPTION 

EU15 

EU13 

Success rates in Horizon 2020 are 
too low to be worth applying 

The Horizon 2020 application 
process is cumbersome 

Limited financial/human resources 
to prepare a proposal 

Inadequate knowledge of the EU 
research and innovation framework 
programme 

59.1% 

44.2% 

40.2% 

38.0% 

64.2% 

25.7% 

50.8% 

36.3% 

EU ASSO. 
COUNTRIES 

THIRD 
COUNTRIES 

40.4% 

28.8% 

50.0% 

33.9% 

39.0% 

32.2% 

49.3% 

54.2% 

85 

  
 
 
Figure 17: Stakeholder breakdown of top four reasons preventing participation in Horizon 2020  
up to three responses allowed, environmental organisations and trade unions not represented due to low 
response rate (3 or fewer) 

70%

60%

50%

40%

30%

20%

10%

0%

Inadequate knowledge of
the EU research and
innovation framework
programme

Limited financial/human
resources to prepare a
proposal

The Horizon 2020
application process is
cumbersome

Success rates in Horizon
2020 are too low to be
worth applying

Academia

Business associations

Companies

EU citizens Non-EU citizens NGOs

Public authorities

Priorities and objectives 

Assessing the achievement of the Horizon 2020 objectives 

Respondents  generally  expressed  positive  opinions  concerning  the  achievement  of  the  Horizon  2020 
objectives.  Most respondents  agreed that  ‘Horizon 2020 supported the development  of the European 
Research  Area’  (83%;  1 483),  ‘fostered  excellence  science’  (80%;  1 432),  ‘spread  excellence  and 
widened participation in R&I’ (74%; 1 305), and ‘helped develop and implement EU policies’ (70%; 
1 257). The most controversial aspect was the ability of Horizon 2020 to ‘support cooperation between 
science and society’, with 6% (99) of respondents maintaining that the programme did not do enough to 
support such cooperation. 

Figure  18:  To  what  extent  do  you  agree  with  the  following  statements  concerning  the  objectives  of 
Horizon 2020? 

Horizon 2020 supported the development of the European
Research Area (N=1785)

Horizon 2020 fostered excellent science (N=1780)

694

702

789

14

127

138

23

24

730

192

77

Horizon 2020 spread excellence and widened participation in R&I
(N=1773)

537

768

281

Horizon 2020 helped develop and implement EU policies (such as
the "Europe 2020" strategy) (N=1791)

Horizon 2020 supported cooperation between science and
society (N=1780)

372

426

885

744

264

211

42

371

20

140

79

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

55

20
99

68

17

Strongly agree

Agree

Neither agree nor disagree

Disagree

Strongly disagree

I do not know / No opinion

86 

  
 
 
 
 
 
 
Development of the European Research Area 

Similarly,  EU  citizens  and  respondents  from  academic  and  research  organisations  (both  respectively 
84%; 775, 184) either agreed or strongly agreed that Horizon 2020 supported the development of the 
European Research Area. Same as shown with regard to fostering excellent science, 88% (52) of non-
EU respondents indicated that Horizon 2020 supported the development of the European Research Area, 
showing that the views regarding the scientific dimension of Horizon 2020 are coherent. 

Excellent Science 

Overall, 80% (1 432) of respondents to the consultation support the claim that Horizon 2020 encouraged 
excellent  science:  this  view  is  held  by  82%  (760)  of  respondents  from  academic  and  research 
organisations and 65% (20) from business associations, 84% (175) of EU citizens and 88% (53) of non-
EU  citizens.  Among  all  other  stakeholder  groups  (including,  among  others,  companies,  public 
authorities, trade unions, NGOs and environmental organisations, 77% (424) agreed or strongly agreed 
with this view. 

Spread excellence and widen participation 

In total, 74% (1 305) of respondents agreed or strongly agreed in the stakeholder consultation with the 
notion that Horizon 2020 spread excellence and widened participation in R&I. This view was held by 
81% (677) of respondents from business associations, 75% (26) of respondents replying on behalf of 
companies  and  businesses,  74%  (677)  of  respondents  from  academia  and  73%  (48)  of  respondents 
replying on behalf of NGOs. At the same time, 73% of non-EU citizens agreed or strongly agreed (43), 
compared to 68% of EU citizens (148). Breaking down the responses of all respondents by countries, it 
becomes clear that 77% (138) of respondents from EU-13 countries, 74% (1 027) of EU-15 countries, 
73% (99) of associated countries and 70% (41) of third countries agree or strongly agree with the notion 
that Horizon 2020 spread excellence and widened participation in R&I. 

Develop and implement EU policies 

Overall, 70% of respondents (1 483) in the stakeholder consultation conducted for this evaluation agreed 
or  strongly  agreed  that  ‘Horizon  2020  helped  develop  and  implement  EU  policies’  (such  as  the 
‘Europe  2020’  strategy).  The  strongest  support  for  this  statement  has  been  shown  among  public 
authorities (78%; 72), followed by business associations (75%; 24), companies (73%; 229) and academia 
(70%;  644).  Only  NGOs  (66%;  44),  EU  citizens  (66%;  145)  and  non-EU  citizens  (60%;  36)  have 
indicated a lesser agreement with the statement, either agreeing or strongly agreeing that Horizon 2020 
helped to develop and implement EU policies. 

Overall,  few  respondents  (2%;  37)  across  all  stakeholder  groups  expressed  unfavourable  opinions 
regarding the capacity of Horizon 2020 to help develop and implement EU policies indicating that across 
all stakeholder groups are indeed overwhelmingly positive about the development and implementation 
of EU policies by means of Horizon 2020. 

Building R&I capacity in EU countries lagging behind 

Only  62%  (1  097)  of  respondents  believed  that  Horizon  2020  helped  building  R&I  capacity  in  EU 
countries lagging behind: this view was primarily shared by non-EU citizens (70%; 41), environmental 
organisations (67%; 2), companies and businesses (64%; 199), academia / research organisations (62%; 
566) followed by EU citizens (61%; 132). The views of business associations were less favourable, only 
having 55% (17) supporting the claim that the programme helped building R&I capacity in EU countries 
lagging behind, whereas 23% (199) of respondents replying on behalf of businesses indicated that the 
effect of Horizon 2020 in this endeavour was neutral. Nevertheless, other stakeholder groups held more 
favourable views, with less respondents deeming the effect neutral, e.g. 11% of academia (200) and 10% 
of companies (31). Overall, only a small fraction of respondents indicated that it had no effect at all: 3% 
of businesses (8) and companies and 1.5% of academia (14).  

87 

  
 
 
 
Supporting cooperation between science and society 

‘Horizon 2020 supported cooperation between science and society’: 6% of respondents (99) maintained 
that the programme did in fact not do enough to support said cooperation. Nevertheless, this constitutes 
only  a  small  fraction  of  respondents  within  each  stakeholder  group.  It  is  important  to  note  that  the 
majority  of  responses  were  rather  favourable,  suggesting  an  overall  positive  sentiment  towards  the 
support of cooperation between science and society.  

Among the various stakeholder groups, favourable views were shared the most on behalf of NGOs 71% 
(47), whereas business associations (59%; 19) were the least favourable compared to other stakeholder 
groups  indicating  that  they  either  agree  or  strongly  agree  with  the  fact  that  Horizon  2020  supported 
cooperation between science and society. 

Flexibility to respond to unforeseen events 

The majority of participants (70%; 1 248) stated that Horizon 2020 was flexible enough to respond to 
unforeseen emergencies such as the COVID-19 pandemic or Zika. Only 3% (55) expressed the opposite 
opinion. One position paper also appreciated the flexibility of Horizon 2020 in delivering a response to 
the COVID-19 pandemic. 

Figure 19: To what extent was Horizon 2020 flexible enough to respond to unforeseen emergencies such 
as the COVID-19 pandemic, Zika, etc. (N=1 793) 

551

697

55

204

286

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

To a great extent

Somewhat

Little

Not at all

I do not know / No opinion

Overall,  70%  (1  248)  of  respondents  stated  that  ‘Horizon  2020  is  flexible  enough  to  respond  to 
unforeseen emergencies such as the COVID-19 pandemic, Zika and others’. Respondents that shared 
views on behalf of an institution or organisation agreed to a greater extent (76%; 435) with this statement 
compared to respondents that shared their views in an individual capacity (67%; 788).  

Among the various stakeholder groups, respondents generally perceive Horizon 2020 as being flexible 
enough to  respond  to  emergencies like the COVID-19 pandemic, Zika and others. Most respondents 
were  positive  about  the  flexibility  of  the  programme  with  32%  of  academia  (296),  31%  of  non-EU 
citizens (18), 30% of public authorities (28), 29% of companies (92), 30% of public authorities (28) and 
27% of EU citizens  (60) strongly agreeing. The  percentages of respondents sharing a more sceptical 
view on the matter were relatively low. Still, the ‘I don’t know / no opinion’ option was chosen by 12% 
(public authorities, 11) to 18% (business associations, 6) of respondents. 

88 

  
 
 
 
Figure 20: Stakeholder breakdown - flexibility to respond to unforeseen emergencies (N= 1 793) 

Academic/research institution

Business association

Company/business organisation

Environmental organisation

EU citizen

Non-EU citizen

Non-governmental organisation (NGO)

Public authority

Trade union

Other

0%

10%

20%

30%

40%

50%

60%

70%

80%

90% 100%

To a great extent

Somewhat

Little

Not at all

I do not know / No opinion

Implementation and administrative procedures 

Design and implementation of the calls for proposals 
Most respondents agreed that  ‘the descriptions of Horizon 2020 call for proposals were clear’ (67%; 
1 170)  and  frequent  enough  (63%;  1 117),  that  ‘the  priority  setting  via  the  work  programmes  was 
adequate’ (61%; 1 091) and that ‘the communication activities to attract applicants were adequate’ (58%; 
1 029).  However,  27%  (478)  of  respondents  maintained  that  finding  the  right  call  for  proposals  was 
difficult.   

Figure 21: To what extent do you agree with the following statements concerning the calls for proposals 
under Horizon 2020? 

The descriptions of the calls for proposals were clear (N=1779)

The calls for proposals were published frequently enough
(N=1780)
Priority setting (via the work programmes) was adequate
(N=1780)

The communication activities to attract applicants were adequate
(N=1780)

Gender equality as a cross-cutting issue was effectively
implemented (N=1779)

288

251

181

196

225

882

866

910

833

309

34

55

211

20

344

166

133

353

164

24

148

438

16

95

202

19

624

478

185

248

Finding the right call for my proposal was easy (N=1785)

191

568

450

386

92

98

Calls for proposals sufficiently took Social Sciences and
Humanities into account (N=1773)

147

441

There was a good balance between small and large projects
(N=1777)

94

449

406

487

70

215

50

296

494

401

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Strongly agree

Agree

Neither agree nor disagree

Disagree

Strongly disagree

I do not know / No opinion

Among  the  different  types  of  stakeholders,  31%  (286)  of  research  institutions  as  well  as  business 
associations  found  it  hard  to  find  the  right  call,  whereas  companies  and  business  organisations 
indicated  that  they  had  slightly  less  difficulties  (26%;  81).  Likewise,  42%  (384)  among  research 
institutions,  50%  (16)  among  business  associations  and  47%  among  companies  and  business 
organisations found it easy to find the right call for their proposals, presenting a relatively even split 
between the views of whether calls were designed in a clear way.  

89 

  
 
 
 
 
Figure 22: Stakeholder breakdown - priority setting via the work programmes was adequate (N= 1 780) 

Academic/research institution

Business association

Company/business organisation

Environmental organisation

EU citizen

Non-EU citizen

Non-governmental organisation (NGO)

Public authority

Trade union

Other

0%

10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Strongly agree

Agree

Neither agree nor disagree

Disagree

Strongly disagree

I do not know / No opinion

Figure 23: Stakeholder breakdown - finding the right call for my proposal was easy (N= 1 785) 

Academic/research institution

Business association

Company/business organisation

Environmental organisation

EU citizen

Non-EU citizen

Non-governmental organisation (NGO)

Public authority

Trade union

Other

0%

10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Strongly agree

Agree

Neither agree nor disagree

Disagree

Strongly disagree

I do not know / No opinion

Social Sciences and Humanities 

Overall, 37% (656) of respondents indicated in the public consultation that they either agree or strongly 
agree  with  the  notion  that  calls  for  proposals  sufficiently  took  Social  Sciences  and  Humanities  into 
account. It is important to mention that 29% (527) of respondents did not answer this survey question or 
indicated that they do not know or have no opinion. This indicates that a significant share of respondents 
has rather limited knowledge of the integration of Social Sciences and Humanities in the programme. 
Bearing this caveat in mind, 35% of respondents representing companies and businesses, 33% (110), 
33% (306) of respondents from academia, 32% (29) of respondents from public authorities deemed that 
the calls for proposals did indeed sufficiently take Social Sciences and Humanities into account. Among 
both EU and non-EU citizens, 30% believe Social Sciences and Humanities were sufficiently taken into 
account  (68  and  18  respectively).  Still,  respondents  that  neither  agree  nor  disagree  with  the 
abovementioned statement include representatives from NGOs (26%; 17), research institutions (24%; 
218), companies and business associations (both 22% 7 and 69 respectively). Notably NGOs (31%; 20) 

90 

  
 
 
 
 
and public authorities (26%; 24) either disagree or strongly disagree with the notion that Social Sciences 
and Humanities were sufficiently taken into account. 

Figure 24: Stakeholder breakdown - calls for proposals sufficiently took Social Sciences and 
Humanities into account (N= 1 773) 

Academic/research institution

Business association

Company/business organisation

Environmental organisation

EU citizen

Non-EU citizen

Non-governmental organisation (NGO)

Public authority

Trade union

Other

0%

10%

20%

30%

40%

50%

60%

70%

80%

90% 100%

Strongly agree

Agree

Neither agree nor disagree

Disagree

Strongly disagree

I do not know / No opinion

Gender equality 

48% of respondents (849) either agreed or strongly agreed that gender equality as a cross-cutting issue 
has been effectively implemented. 27% (478) neither agreed nor disagreed, while 11% (204) disagreed 
or disagreed strongly. 14% (248) did not express an opinion.  

The stakeholder groups that were most positive about the implementation of gender equality as a cross-
cutting issue were academia (50%; 455), followed by public authorities (49%; 45), companies (47%; 
143) and non-EU citizens (47%; 28). EU citizens (45%; 98), NGOs (44%; 29) and business associations 
(31%;  11)  on  the  other  hand  were  less  positive:  the  difference  between  academia  and  business 
associations showed a 19 percentage point difference which might be rooted in the smaller variation 
within  the  stakeholder  group  –  only  32  business  associations  replied  to  the  related  question  overall 
whereas the figure was significantly higher for respondents from academia (919).    

91 

  
 
 
 
Figure 25: Stakeholder breakdown - gender equality has been effectively implemented as a cross-cutting 
issue (N= 1 779) 

Academic/research institution
Business association
Company/business organisation
Environmental organisation
EU citizen
Non-EU citizen
Non-governmental organisation (NGO)
Public authority
Trade union
Other

0%

10%

20%

30%

40%

50%

60%

70%

80%

90% 100%

Strongly agree

Agree

Neither agree nor disagree

Disagree

Strongly disagree

I do not know / No opinion

Evaluation of proposals 

Most respondents agreed or strongly agreed that the ‘time to evaluate the proposals’ (66%; 1 175) and 
the ‘time to sign the grant agreement’ (69%; 1 230) was ‘adequate’. Whilst 50% (884) of respondents 
think  that  the  feedback  received  from  the  evaluation  was  ‘clear  and  informative’,  24%  (426)  of 
respondents disagreed or strongly disagreed with this statement.   

Figure 26. To what extent do you agree with the following statements concerning proposal evaluation 
under Horizon 2020? 

The time taken to sign the grant agreement was adequate
(N=1776)

312

The time taken to evaluate proposals was adequate (N=1780)

225

918

950

29

226

120

171

256

191

36
122

82

The feedback received from the evaluation was clear and
informative (N=1779)

197

687

363

344

106

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Strongly agree

Agree

Neither agree nor disagree

Disagree

Strongly disagree

I do not know / No opinion

Project implementation 
Most respondents agreed or strongly agreed that their ‘organisation’s usual accounting practices were 
accepted’ (69%: 1 227), that ‘the mechanisms for project monitoring and reporting were adequate’ (69%; 
1 214),  that  ‘the  cost  calculation  rules  were  clear’  (66%;  1 177).  The  majority  of  respondents  were 
‘satisfied with the support received by the EC services (including agencies) during grant preparation and 
implementation’ (58%; 1 022) and agreed or strongly agreed that ‘the balance between checking and 
trust in beneficiaries was adequate’ (57%; 1 012). 

92 

  
 
 
 
 
 
 
 
 
 
Figure  27.  To  what  extent  do  you  agree  with  the  following  statements  concerning  the  project 
implementation under Horizon 2020? 

Our organisation’s usual accounting practices were accepted 
(N=1786)

The mechanisms for project monitoring and reporting were
adequate (N=1769)

346

310

881

904

34

202

99

224

258

152

35

110

28

The cost calculation rules were clear (N=1782)

285

892

265

186

126

I am satisfied with the support received by the EC services
(including agencies) during grant preparation and…

The balance between checking and trust in beneficiaries was
adequate (N=1767)

351

276

671

736

35

374

112

230

313

171

219

52

Strongly agree

Agree

Neither agree nor disagree

Disagree

Strongly disagree

I do not know / No opinion

0%

10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

A  large  majority  of  respondents  ‘agreed’  or  ‘strongly  agreed’  that  their  ‘organisation’s  usual 
accounting practices were accepted’ (69%; 1 227). The agreement among respondents from NGOs and 
companies was even higher with both at 74% (49; 651) respectively. EU citizens (61%; 135) and non-
EU citizens (51%; 30) agreed to a lesser extent. Only a small fraction of respondents found that their 
usual  accounting  practices  were  not  accepted,  namely  public  authorities  (2%;  2),  NGOs  (8%;  3), 
companies (7%; 23), business associations (6%; 2) and academia (8%; 77).  

Beyond that, stakeholders agreed that ‘the mechanisms for project monitoring and reporting were 
adequate’  (69%;  1  214),  showing  the  highest  level  of  agreement  among  companies  (74%;  226), 
followed by respondents from academia (70%; 634), business associations (68%; 21), EU citizens (66%; 
143), non-EU citizens (60%; 35) and NGOs (58%; 38) and business associations (68%; 21). At the same 
time,  another  fraction  of  respondents  from  academia  were  of  the  opinion  that  the  monitoring  and 
reporting mechanisms were not adequate (10%; 93), followed by NGO-associated respondents (17%; 
11), respondents on behalf of companies (9%; 28) and business associations (3%; 1).   

Overall, respondents agreed with the statement that ‘the cost calculation rules were clear’ (66%; 1 
177).  Nevertheless,  there  was  some  variation  between  the  different  stakeholder  groups:  business 
associations  agreed  to  the  greatest  extent  (74%;  23),  followed  by  companies  (72%;  224),  academia 
(68%; 620), NGOs (62%; 41), EU citizens (61%; 120), public authorities (57%; 52) and non-EU citizens 
(54%; 37) respectively. At the same time, the level of dissatisfaction with the clarity of cost calculation 
rules varies to a smaller extent among the different types of stakeholders, ranging from 9% of companies 
(27), up to 13% for academia (118), public authorities (12) and business associations (4) respectively.    

More  than  half  of  respondents  were  ‘satisfied  with  the  support  received  by  the  EC  services 
(including  agencies)  during  grant  preparation  and  implementation’  (58%;  1  022):  business 
associations are beyond the average of all respondents satisfied with the support received by the EC 
services (67%; 21), along with companies (64%; 199). At the same time, respondents from academia 
(57%; 524) and NGOs (48%; 32) were less satisfied. Interestingly, the level of satisfaction between EU 
citizens and non-EU citizens differs: non-EU citizens (64%; 27) are less satisfied401 with the support 
received by the EC services, compared to EU citizens (55%; 122).   

The effects of simplification measures 

Most respondents stated that ‘using an electronic-only management system’ (82%; 1 403) and having 
‘harmonised  processes  and  guidance  documents  across  the  framework  programme’  (72%;  1 220) 
reduced their administrative burden – somewhat or to a great extent. Likewise, most respondents think 
that removing the negotiation stage during grant preparation (67%; 1 133) and ‘using a funding model 
with  a  single  reimbursement  rate  and  a  single  flat  rate’  (59%;  1 033)  are  effective  simplification 
93 

  
 
 
 
 
measures.  Although  positive  responses  (771)  outnumber  negative  ones  (255),  the  simplification 
measures  that  received  the  highest  number  of  negative  responses  is  ‘further  use  of  the  two-stage 
application process’.  

Figure  28:  Based  on  your  experience,  to  what  extent  did  the  following  simplification  measures, 
introduced in Horizon 2020, help reduce your administrative burden? 

Using an electronic-only grant management system (N=1711)

1042

Harmonisation of processes and guidance documents across the
programme (N=1688)

Selected projects are funded as proposed (no negotiation during
grant preparation) (N=1691)

Using a new funding model with a single reimbursement rate and
a single flat rate (N=1700)

663

643

361

26
78

14
190

15

24
138

291

36

557

490

142

65

315

533

470

156

49

460

32

34

Alignment of funding rules applicable to other EU funds (N=1699)

330

493

200

84

558

Further use of the two-stage application process (for some
programme parts) (N=1708)

278

493

240

153

102

442

To a great extent

Somewhat

Administrative burden remained the same

Very little

Not at all

I do not know / No opinion

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Specifically regarding the implementation processes of the calls, stakeholders indicated that ‘Using an 
electronic-only management system’ (82%; 1 403) and having ‘harmonised processes and guidance 
documents across the framework programme’ (72%; 1 220) were understood to have reduced at least 
‘somewhat’ the administrative burden for respondents. Likewise, most respondents think that effective 
simplification measures included: removing the negotiation stage during grant preparation (67%; 1 
133) and ‘using a funding model with a single reimbursement rate and a single flat rate’ (59%; 1 
033).  

The simplification measure with the highest number of negative views was the ‘further use of the two-
stage application process’. Although positive responses (42%; 711) still outnumber the negative ones 
(15%;  255),  the  majority  of  respondents  agrees  with  the  notion  of  further  expanding  the  two-stage 
application process for some programme parts. Notably respondents from academia (56%; 433), non-
EU citizens (52%; 27), public authorities (49%; 36), business associations (48%; 11) and EU citizens 
(48%; 87) were in favour of furthering the use of the two-stage application process for some programme 
parts.    

The effort to participate in Horizon 2020 

Views on whether the effort needed to participate in Horizon 2020 was lower compared to FP7 were 
not  uniform:  39%  (692)  of  respondents  think  that  the  effort  was  similar  (the  largest  share  across  all 
country and stakeholder groups), 17% (303) that it was greater and 12% (219) that it was lower. One-
third of respondents (565) do not have an opinion.  

On  the  other  hand,  relative  to  other  research  and  innovation  funding  programmes,  the  effort  to 
participate  in  Horizon  2020  is  deemed  greater  (43%;  771,  particularly  academic  and  research 
institutions)  or  similar  (39%;  692)  by  most,  across  all  stakeholder  groups.  Only  a  small  minority  of 
respondents (7%; 219) consider it lower.  

94 

  
 
 
 
 
 
Figure 29: The effort needed to participate in Horizon 2020 compared to the 7th Framework Programme 
and other R&I programmes  

The effort needed to participate in Horizon 2020 compared to the
7th Framework Programme (FP7) was...  (N=1779)

219

692

303

565

The effort needed to participate in Horizon 2020 compared to that
of other research and innovation programmes was... (N=1783)

123

544

771

345

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Lower

Similar

Greater

I do not know / No opinion

Costs of proposal preparation 

Most (56%; 939) of the consultation respondents declared that their ‘proposal preparation for Horizon 
2020’ took overall less than 50 days (across all stakeholder groups, especially public authorities – 70% 
and  companies  –  65%).  This  is  followed  by  30%  of  respondents  (505)  who  stated  that  the  proposal 
preparation took ‘more than 50 but less than 100 days’ and 13% (219) who said ‘more than 100 days’. 
Considering the different types of respondents, academic, research institutions and NGOs reported to 
spend more time preparing proposals than companies, business organisations or public authorities.130  

Figure 30. Approximately, how much time did the proposal preparation for Horizon 2020 take overall? 
Please indicate the total number of person-days. (N= 1 663) 

56%

1000

800

600

400

200

0

30%

13%

Less than 50
person-days

Between 50 and
99 person-days

More than 100
person-days

130 The share of respondents spending less of 50 days for proposal preparation is 50% in the first group, whereas it is more 
than 65% for the second group.  

95 

  
 
 
 
 
 
Figure 31: Stakeholder breakdown – How much time did the proposal preparation for Horizon 2020 
take overall? (N =1 663) 

Academic/research institution

Business association

Company/business organisation

Environmental organisation

EU citizen

Non-EU citizen

Non-governmental organisation (NGO)

Public authority

Trade union

Other

0%

10%

20%

30%

40%

50%

60%

70%

80%

90% 100%

Less than 10 person-days

More than 10 but below 20 person-days

More than 20 but below 30 person-days More than 30 but below 40 person-days

More than 40 but below 50 person-days More than 50 but below 60 person-days

More than 60 but below 100 person

More than 100 person-days

The consultation also asked participants about the ‘time spent managing participation’. The analysis of 
responses  cannot  be  conclusive  because  there  is  great  variability  in  the  responses  in  terms  of  who 
provided information or not, which programme parts the respondents were linked to and their specific 
role in the projects.131 The chart below shows the information that was provided on ‘average number of 
person-days spent during the entire project’, by project duration. 

Figure 32. Approximately, how much time does your project spend on managing participation in Horizon 
2020?  Total  number  of  person-days  spent  overall  on  managing  participation.  Average  estimate  of 
person-days by project duration (N= 301) 

332

71

103

157

350

300

250

200

150

100

50

0

Duration: 12 months
(N=29)

Duration: 24 months
(N=29)

Duration: 36 months
(N=156)

Duration: 48 months
(N=87)

The results to the open question ‘Approximately, how much time does your project spend on managing 
participation in Horizon 2020?’ can only be interpreted with caution because of the following:  

▪  332 responses could not be processed because respondents did not provide the required information 

(could not provide an estimate or when they did it was not in the required format).  

▪  Comparability is difficult because respondents interpreted the question in different ways (e.g., some 
respondents  included  in  the  estimate  the  time  spent  for  the  internal  project  coordination,  others 
provided an estimate of the time that it took for the whole project activities). 

131 For example, among projects with a duration of 36 months, the estimates range from 2 person-days to 1,500 person-days 
(after the outliers were removed). 

96 

  
 
 
 
 
 
▪  The resources spent on managing participation largely depended on the type of action (i.e., Research 
& Innovation Action or Coordination and Support Action), the role of the respondent in the project 
(i.e., coordinator or partner), and the project size. 

Effectiveness of the novelties introduced for the period 2018 – 2020 

The  consultation  results  show  varying  degrees  of  familiarity  with  the  novelties  introduced  by  the 
programme. 70% (1 231) of the respondents agreed or strongly agreed that reinforcing the ‘support for 
open sharing of data and open access to publications’ increased the framework programme’s impact. 
Similarly, the majority of respondents see positive impacts from ‘increasing the research and innovation 
investments for sustainable development and climate action’ (54%; 948) and from ‘increasing the focus 
on digitalisation’ (50%; 884).  

On the other hand, more than one-third of the respondents did not express an opinion on novelties such 
as  the  introduction  of  the  European  Innovation  Council  (EIC),  the  launch  of  international  flagship 
initiatives,  the  introduction  of  the  lump  sum  funding  and  the  investments  to  fight  COVID-19.  The 
analysis by type of respondent does not highlight significant differences among the stakeholders’ groups 
as the distribution of responses among the different options is similar. 

Figure 33. For the period 2018-2020, a number of novelties were introduced to increase the 
programme's impact. To what extent do you agree with the following statements? 

The support for open sharing of data and open access to
publications was reinforced (N=1762)

456

775

211

68

221

31

The increase of research and innovation investments for
sustainable development and climate action (e.g., the Green
Deal call for proposals) improved the impact of the…

322

626

259

108

422

28

Increased focus on Digitalisation improved the impact of the
programme (N=1750)

283

601

369

86

384

27

Work programmes were formulated more clearly than in the
past (N=1762)

93

648

514

124

369

14

The research and innovation investment to fight the COVID-19
pandemic (the COVID-19 call for proposals) improved the impact
of the programme (N=1754)

192

477

326

91

625

43

The identification of a limited number of Focus Areas
rationalised the innovation funding landscape (N=1747)

98

539

386

152

536

36

The pilot on lump sum funding simplified participation in the
programme (N=1757)

170

423

279

171

633

81

The launch of international flagship initiatives on dedicated
topics (e.g., targeted actions with Africa on food and sustainable
agriculture) boosted international cooperation (N=1758)

130

427

293

86

787

35

The introduction of the European Innovation Council Pilot
stepped up the support for breakthrough innovations (N=1751)

111

339

312

92

873

24

Strongly Agree

Agree

Neither agree nor disagree

Disagree

Strongly disagree

I do not know / No opinion

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Across all stakeholder groups, the support for the lump sum pilot is held the most with respondents 
associated with public authorities (38%; 35), which is followed by respondents from companies (36%; 
110). Respondents from companies favour the introduction of the lump sum pilot to a greater extent 
compared  to  respondents  from  academia  (33%;  299)  however  not  significantly  (only  3  percentage 

97 

  
 
 
 
 
points). Similarly, 16% of respondents from academia (144) are negative about the introduction of the 
lump sum pilot, compared to 13% of respondents from companies (37) once again showing a difference 
of 3 percentage points. 

Figure 34: Stakeholder breakdown - "The pilot on lump-sum funding simplified participation (N= 1 757) 

Academic/research institution

Business association

Company/business organisation

Environmental organisation

EU citizen

Non-EU citizen

Non-governmental organisation (NGO)

Public authority

Trade union

Other

0%

10%

20%

30%

40%

50%

60%

70%

80%

90% 100%

Strongly Agree

Agree

Neither agree nor disagree

Disagree

Strongly disagree

I do not know / No opinion

According to 35% (32) of public authorities, 34% of academia (311), 31% of NGOs (20) and 23% of 
companies  (71)  deem  that  the  introduction  of  international  flagship  initiatives  have  boosted 
international cooperation. However, around half of the respondents said that they do not know or do 
not have an opinion on this matter. 

Figure  35:  Stakeholder  breakdown  -  launch  of  flagship  initiatives  boosted  international  cooperation  
(N= 1 758) 

Academic/research institution

Business association

Company/business organisation

Environmental organisation

EU citizen

Non-EU citizen

Non-governmental organisation (NGO)

Public authority

Trade union

Other

0%

10%

20%

30%

40%

50%

60%

70%

80%

90% 100%

Strongly Agree

Agree

Neither agree nor disagree

Disagree

Strongly disagree

I do not know / No opinion

98 

  
 
 
 
 
 
Effectiveness of the Horizon 2020 partnerships 

44%  (780)  of  respondents  stated  that  the  public-public  or  public-private  partnerships  were  ‘more 
effective  than  regular  collaborative  research  in  achieving  impact  for  science,  the  economy  and/or 
society’. Only 16% (285) of them selected that partnerships were not more effective, or they were more 
effective only ‘to a limited extent’. However, more than 26% (461) of respondents did not express any 
opinion. 

Figure 36: In your opinion, to what extent have public-private and public-public Partnerships supported 
by Horizon 2020 been more effective, compared to regular collaborative research in achieving impact 
for science, the economy and/or society? (N=1 769) 

320

460

243

218

67

461

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

To a great extent

Somewhat

Neutral

To a limited extent

Not at all

I do not know / No opinion

Among the different types of respondents, businesses and business organisations have a more positive 
opinion  than  academic  /  research  institutions.  42.2%  (386)  of  academic  and  research  institutions 
answered ‘to a great extent’ or ‘somewhat’ compared to 63.6% (21) of business association respondents 
and 53.5% (167) of companies and business organisations. 

Scientific effects 

More  than  60%  of  the  respondents  agree  that  Horizon  2020  produced  the  expected  scientific  effects 
listed in the consultation, for instance, by improving the skills of Europe’s researchers and by facilitating 
the emergence of new researchers (85%; 1 500) or by making Europe more attractive for world-class 
researchers from abroad (70%; 1 240).  

Figure 37. In your view, to what extent did Horizon 2020 produce the following scientific effects? 

Improving the skills of Europe’s researchers and facilitating the 
emergence of new researchers (N=1770)

Facilitating cross-sector and cross-border mobility of researchers
(N=1767)

Scientific breakthroughs, higher risk research and research in
emerging areas of science and technology (N=1775)

Making Europe more attractive for world class researchers from
abroad (N=1764)

949

913

643

632

551

54

28

85

36

103

16

566

82

154

35

696

106

96

199

50

608

162

92

220

14

Reinforced physical and knowledge infrastructures and facilities
across the EU (N=1765)

534

702

172

65

278

34

Building R&I capacity in EU countries lagging behind (N=1769)

434

666

194

99

342

To a great extent

Somewhat

Neutral

A little

Not at all

I do not know / No opinion

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

78% (711) of respondents from academic and research institutions, 75% (24) of business associations 
and 73% (226) of respondents from companies, Horizon 2020 fosters scientific breakthroughs, higher 

99 

  
 
 
 
 
risk research and research in emerging areas of science and technology. This claim was supported 
by EU citizens (76%; 163) and an even greater share of non-EU citizens (86%; 59).    

More  respondents  agreed  or  strongly  agreed  that  Horizon  2020  improved  the  skills  of  Europe’s 
researchers and facilitated the emergence of new researchers (85%; 1 500): 92% of non-EU citizens 
(54),  88%  of  respondents  from  academia  (803),  82%  of  respondents  from  NGOs  (61)  and  81%  of 
respondents from business associations (26) supported this claim.  

In terms of facilitating cross-sector and cross-border mobility of researchers, 88% of respondents 
from academia (804), 76% of respondents from companies (235) as well as 71% of business associations 
(22) agreed or strongly agreed that Horizon 2020 had a positive effect. Similarly, 73% of respondents 
from academia (666), 67% of respondents from business associations (20), as well as 60% of companies 
(90) deemed that the programme is making Europe more attractive for world class researchers from 
abroad.  Still,  this  claim  is  only  supported  by  66%  of  non-EU  citizens  (52),  contrasting  88%  of  EU 
citizens responding (164).       

Among  academia  and  research  institutions,  30%  (913)  agreed  to  a  great  extent  that  Horizon  2020 
reinforced physical and knowledge infrastructures and facilities across the EU which is similar to 
views of public authorities (32%; 91). Figures were lower for NGOs (28%; 65) and companies (27%; 
309).  

Only 62% (1 097) of respondents believed that Horizon 2020 helped building R&I capacity in EU 
countries  lagging behind:  this view was primarily  shared by environmental organisations (67%; 2), 
companies and businesses (64%; 199) and academia / research organisations (62%; 566). The views of 
business  associations  were  less  favourable,  only  having  55%  (17)  supporting  the  claim  that  the 
programme  helped  building  R&I  capacity  in  EU  countries  lagging  behind,  whereas  23%  (199)  of 
respondents replying on behalf of businesses indicated that the effect of Horizon 2020 in this endeavour 
was neutral. Nevertheless, other stakeholder groups held more favourable views, with less respondents 
deeming the effect neutral (e.g. academia 11%; 200, companies 10%; 31). Overall, only a small fraction 
of respondents indicated that it had no effect at all (3%; 8 businesses and companies and 1.5%; 14 of 
academia).    

Economic effects 

Around  one-third  of  respondents  did  not  express  any  opinion  on  the  economic  effects  produced  by 
Horizon 2020. Nonetheless, most of the respondents stated that the programme fostered market uptake 
of European innovations (51%; 886), digitalised the industry and the economy (54%; 945), opened up 
new markets for participants (56%; 983) and supported the internationalisation of SMEs participating in 
Horizon 2020 (63%; 1 094). As for the other economic effects, the respondents with a positive opinion 
outnumber  those  with  a  negative  opinion.  Only  518  respondents  expressed  an  opinion  on  whether 
Horizon 2020 increased the availability of debt financing. 

100 

  
 
 
 
 
 
 
 
 
 
Figure 38. In your view, to what extent did Horizon 2020 produce the following economic effects? 

Supporting the internationalisation of participating SMEs
(N=1749)

462

632

129

77

Opening up new markets for H2020 participants (N=1758)

280

703

186

109

Digitalising the industry and the economy (N=1743)

373

572

178

72

Fostering market uptake of European innovations (N=1746)

253

633

211

92

33

50

Producing new and better product-service offerings addressing
customer needs (N=1741)

Fostering sustainability across the entire product-service lifecycle
(N=1745)

257

263

565

558

Creating smart global value chains that enable value capture in
Europe (N=1746)

220

524

219

Increased availability of equity finance for R&D and innovation-
driven companies (N=1744)

262

459

161

77

44

228

78

50

224

87

68

58

657

746

39

53

23

42

426

438

515

507

569

563

Improved time-to-market for European manufacturers and
service providers (N=1746)

194

502

246

92

659

55

48

Increasing availability of debt (N=1735)

52

193

218

1169

To a great extent

Somewhat

Neutral

A little

Not at all

I do not know / No opinion

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

The following graphs provide a stakeholder breakdown for the top three effects identified by 
respondents: 

Figure 39: Stakeholder breakdown – ‘Horizon 2020 supported the internationalisation of participating 
SMEs’ (N= 1 749) 

Academic/research institution

Company/business organisation

EU citizen

Public authority

Other

Non-governmental organisation (NGO)

Non-EU citizen

Business association

Environmental organisation

Trade union

0%

10%

20%

30%

40%

50%

60%

70%

80%

90% 100%

To a great extent

Somewhat

A little

Neutral

Not at all

I do not know / No opinion

101 

  
 
 
 
 
Figure 40: Stakeholder breakdown – ‘Horizon 2020 opened up new markets for programme 
participants’ (N = 1 758) 

Academic/research institution

Company/business organisation

EU citizen

Public authority

Other

Non-governmental organisation (NGO)

Non-EU citizen

Business association

Environmental organisation

Trade union

0%

10%

20%

30%

40%

50%

60%

70%

80%

90% 100%

To a great extent

Somewhat

A little

Neutral

Not at all

I do not know / No opinion

Figure 41: Stakeholder breakdown – ‘Horizon 2020 digitalised the industry and the economy’  
(N = 1 743) 

Academic/research institution

Company/business organisation

EU citizen

Public authority

Other

Non-governmental organisation (NGO)

Non-EU citizen

Business association

Environmental organisation

Trade union

0%

10%

20%

30%

40%

50%

60%

70%

80%

90% 100%

To a great extent

Somewhat

A little

Neutral

Not at all

I do not know / No opinion

102 

  
 
 
 
 
 
 
Societal effects 

Respondents  generally  reported  that  Horizon  2020  produced  positive  societal  effects.  These  effects 
included ‘improving lifelong health and well-being of all’ (52%; 916), ‘supporting the transition to a 
reliable,  affordable,  publicly  accepted,  sustainable  and  competitive  energy  system’  (53%;  918),  
‘fostering  smart,  green  and  integrated  transport’  (53%;  925),  ‘fostering  a  greater  understanding  of 
Europe’ (57%; 997), and ‘engaging with wider society to build effective cooperation between science 
and society’ (63%; 1,106). More than 40% of respondents (720) did not express an opinion on whether 
Horizon 2020 contributed to ‘securing sufficient supplies of safe, healthy and high-quality food’. 

Figure 42: In your view, to what extent did Horizon 2020 produce the following societal effects? 

Engaging with wider society to build effective cooperation between
science and society (N=1744)

Fostering a greater understanding of Europe, providing solutions and
supporting inclusive, innovative and reflective European societies…

Fostering smart, green and integrated transport (N=1738)

Supporting the transition to a reliable, affordable, publicly accepted,
sustainable and competitive energy system (N=1743)

416

401

300

300

41

690

143

118

336

596

154

93

456

44

625

618

136

89

142

92

45

62

48

543

529

Improving the lifelong health and well-being of all (N=1753)

247

669

169

104

516

Fostering secure and inclusive European societies while contributing
to upholding fundamental rights (N=1750)

283

556

193

78

582

Securing sufficient supplies of safe, healthy and high-quality food
(N=1743)

178

514

198

92

720

41

58

To a great extent

Somewhat

Neutral

A little

Not at all

I do not know / No answer

0%

10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

66%  (1  170)  of  respondents  in  the  stakeholder  consultation  conducted  for  this  evaluation  agreed  or 
strongly  agreed  that  ‘Horizon  2020  supported  cooperation  between  science  and  society’.  6%  of 
respondents  (99)  maintained  that  the  programme  did  not  do  enough  to  support  said  cooperation. 
Nevertheless, this constitutes only a small fraction of respondents within each stakeholder group. It is 
important to note that the majority of responses were rather favourable, suggesting an overall positive 
sentiment  towards  the  support  of  cooperation  between  science  and  society.  Among  the  various 
stakeholder  groups,  favourable  views  were  shared  the  most  on  behalf  of  NGOs  71%  (47),  whereas 
business associations (59%; 19) were the least favourable compared to other stakeholder groups. Non-
EU citizens found that Horizon 2020 supported cooperation between science and society to a greater 
extent (67%; 39) compared to EU citizens (60%; 129).  

103 

  
 
 
 
 
 
 
 
 
 
 
 
The  following  graphs  provide  a  stakeholder  breakdown  for  the  top  three  effects  identified  by 
respondents: 

Figure  43:  Stakeholder  breakdown  –  ‘Horizon  2020  engaged  with  wider  society  to  build  effective 
cooperation between science and society’ (N = 1744) 

0%

10%

20%

30%

40%

50%

60%

70%

80%

90% 100%

Academic/research institution

Business association

Company/business organisation

Environmental organisation

EU citizen

Non-EU citizen

Non-governmental organisation (NGO)

Other

Public authority

Trade union

To a great extent

Somewhat

A little

Neutral

Not at all

I do not know / No answer

Figure  44:  Stakeholder  breakdown  –  ‘Horizon  2020  fostered  a  greater  understanding  of  Europe, 
providing solutions and supporting inclusive, innovative and reflective societies’ (N= 1 744) 

Academic/research institution

Company/business organisation

EU citizen

Public authority

Other

Non-governmental organisation (NGO)

Non-EU citizen

Business association

Environmental organisation

Trade union

0%

10%

20%

30%

40%

50%

60%

70%

80%

90% 100%

To a great extent

Somewhat

A little

Neutral

Not at all

I do not know / No answer

104 

  
 
 
 
 
Figure  45:  Stakeholder  breakdown  –  ‘Horizon  2020  fostered  smart,  green  and  integrated  transport’  
(N = 1 738) 

Academic/research institution

Company/business organisation

EU citizen

Public authority

Other

Non-governmental organisation (NGO)

Non-EU citizen

Business association

Environmental organisation

Trade union

0%

10%

20%

30%

40%

50%

60%

70%

80%

90% 100%

To a great extent

Somewhat

A little

Neutral

Not at all

I do not know / No answer

Synergies with other EU programmes 

54%  of  respondents  expressed  an  opinion  on  the  synergies  between  Horizon  2020  and  Erasmus+. 
Between 22% and 39% of all the respondents expressed an opinion on the synergies between Horizon 
2020 and other EU programmes. The majority of respondents selected that ‘synergies were exploited’ 
or  ‘fully  exploited’  with  Erasmus+  (61%;  566  out  of  928)  and  the  Programme  for  Environment  and 
Climate Action (LIFE) (54%; 364 out of 676). Conversely, the majority of respondents selected that 
‘few’ or ‘no synergies were exploited’ with the Programme for Competitiveness of Enterprises and Small 
and  Medium-sized  Enterprises,  COSME  (52%;  258  out  of  495),  Connecting  Europe  Facility  (CEF) 
(53%;  241  out  of  454),  Common  Agricultural  Policy  (CAP)  (59%;  248  out  of  421),  the  European 
Structural and Investment Funds (ESIF) (63%; 331 out of 529)132, and the European Fund for Strategic 
Investments (EFSI) (66%; 253 out of 385).  

Figure 46: How did the following EU programmes work in synergy (complement and reinforce) Horizon 
2020? 

Erasmus+ (N=1729)

171

395

264

98

801

281

214

98

1048

Programme for Environment and Climate Action (LIFE) (N=1724)

Programme for Competitiveness of Enterprises and small and
medium-sized enterprises (COSME) (N=1713)

83
44

44

193

156

102

Connecting Europe Facility (CEF) (N=1718)

169

140

101

European Structural and Investment Funds (ESIF) (N=1732)

Common Agricultural Policy (CAP) (N=1731)

42

35

27

156

206

125

138

112

136

European Fund for Strategic Investments (EFSI) (N=1722)

105

127

126

1218

1264

1203

1310

1337

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Synergies fully exploited

No synergies exploited

Several synergies exploited

I do not know / No opinion

Few synergies exploited

132 Respondents from EU13 countries have a more positive opinion on the synergies with the ESIF compared to respondents 
from EU15 countries.  

105 

  
 
 
 
 
 
As the Erasmus+ programme as well as LIFE stand out positively across all respondents, further analysis 
revealed  that  non-EU  citizens  (48%;  27)  and  respondents  from  the  field  of  academia  indicated  that 
synergies with Erasmus+ were either fully or partly exploited (36%; 326), whereas the figure is lower 
among EU citizens (32%; 67), companies (21%; 65) and business associations (16%; 5). This leads to 
the assumption that the field of academia leaves greater room for synergies. This should not come as a 
surprise considering that the two programmes in view of academia have a strong link.  

In view of LIFE, a similar trend can be seen: 22% of non-EU citizens (12), 21% of EU citizens (43) as 
well  as  21%  of  respondents  associated  with  academia  (191),  18%  of  respondents  associated  with  a 
company or business (56) and 13% of respondents associated with business associations (4) indicate that 
synergies were either fully or partly exploited. At the same time, 20% of respondents from academia 
(178), 22% of respondents  from  business  associations (7) and 13% of respondents  associated  with  a 
company or business (41) indicated that either few or no synergies were exploited.  

Figure 47: Stakeholder breakdown - Synergies between Horizon 2020 and Erasmus+ (N= 1 729) 

Academic/research institution

Business association

Company/business organisation

Environmental organisation

EU citizen

Non-EU citizen

Non-governmental organisation (NGO)

Public authority

Trade union

Other

0%

10%

20%

30%

40%

50%

60%

70%

80%

90% 100%

Synergies fully exploited

Several synergies exploited

Few synergies exploited

No synergies exploited

I do not know / No opinion

106 

  
 
 
 
Figure 48: Stakeholder breakdown - Synergies between Horizon 2020 and LIFE (N = 1724) 

Academic/research institution

Business association

Company/business organisation

Environmental organisation

EU citizen

Non-EU citizen

Non-governmental organisation (NGO)

Public authority

Trade union

Other

0%

10%

20%

30%

40%

50%

60%

70%

80%

90% 100%

Synergies fully exploited

Several synergies exploited

Few synergies exploited

No synergies exploited

I do not know / No opinion

Exploitation and dissemination of results 

Respondents indicated scientific publication(s), workshops or other events, project website and social 
media  (especially  LinkedIn)  as  the  initiatives  of  Horizon  2020  that  mostly  helped  dissemination, 
exploitation and access to research and innovation results. In particular, 69% (1 213) and 62% (1 083) 
of respondents respectively stated that scientific publication(s) and workshops or other events133 ‘helped 
disseminate, exploit and access research and innovation results’ to a great extent. 

In  the  open  comments  to  the  question,  participants  mentioned  ‘personal  contacts’  and  ‘direct 
networking’ as useful instruments to disseminate, exploit and access research and innovation results. 
Other comments highlighted the beneficial role of creating consortia under Horizon 2020 projects and 
the multi-actor approach that supports interaction between research providers and users within consortia. 
According to the comments, this approach helped find European innovation partners with similar long-
term strategic goals, which was beneficial for cooperation beyond the duration of the supported project. 
Traditional media (TV, radio) and non-scientific publications (such as press releases, whitepapers, and 
books) were mentioned as other initiatives that strongly helped the dissemination, exploitation of R&I 
results and access to them.   

Regarding the helpfulness of EU-wide dissemination and exploitation support services initiated by the 
Commission, a significant share of respondents did not have an opinion or did not know: over 50% for 
the Innovation Radar and IPR Helpdesk, 30-40% for the Horizon Dashboard, Horizon Results Platform 
and Horizon Results Booster, and 22% for CORDIS. For publications, the project website, social media 
and workshops, this share is 6-11%. 

133 Such as scientific conferences and congresses, events dedicated to sharing results such as the Road Transport Research 
Days and the Transport Research Arena, specific events organised by national contact points or agencies. 

107 

  
 
 
 
 
 
 
 
Figure 49: To what extent have the following initiatives helped disseminate, exploit and access research 
and innovation results? 

Scientific publication(s) (N=1758)

Workshops, events, etc. (N=1754)

1213

1083

Project website (N=1758)

Social media (N=1746)

CORDIS (N=1742)

Patent filing (N=1746)

802

711

490

409

357

467

15

78

95

14

106

84
38

594

606

178

146

53

192

184

584

216

66

386

522

208

72

535

Horizon Dashboard (N=1740)

Horizon Results Platform (N=1736)

Horizon Results Booster (N=1739)

IPR Helpdesk (N=1724)

235

250

193

212

448

433

270

143

252

127

380

349

238

139

186

111

644

674

789

866

Innovation Radar (N=1724)

166

294

10

17

180

102

982

Other (N=994)

92

39

836

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

To a great extent

Somewhat

A little

Not at all

I do not know / No opinion

Overall,  this  indicates  that  particularly  in  view  of  the  Innovation  Radar  and  the  IPR  Helpdesk, 
stakeholders are not sufficiently convinced of these tools’ usefulness for dissemination and exploitation. 
Nevertheless, while EU citizens (15%; 31), non-EU citizens (14%; 8) and respondents from academia 
(12%  111)  favour  the  IPR  Helpdesk  over  the  Innovation  Radar  (7%;  15,  11%;  6  and  9%;  80 
respectively), only 16% of business associations (5) and 12% of companies (36) hold the belief that the 
IPR Helpdesk fosters dissemination and exploitation of results to a great extent. 

108 

  
 
 
 
Figure 50: Stakeholder breakdown - Extent to which the IPR Helpdesk helped to disseminate, exploit 
and access research and innovation results (N = 1 724) 

Academic/research institution

Business association

Company/business organisation

Environmental organisation

EU citizen

Non-EU citizen

Non-governmental organisation (NGO)

Public authority

Trade union

Other

0%

10%

20%

30%

40%

50%

60%

70%

80%

90% 100%

To a great extent

Somewhat

A little

Not at all

I do not know / No opinion

Figure  51:  Stakeholder  breakdown  -  Extent  to  which  the  Innovation  Radar  helped  to  disseminate, 
exploit and access research and innovation results (N = 1 724) 

Academic/research institution

Business association

Company/business organisation

Environmental organisation

EU citizen

Non-EU citizen

Non-governmental organisation (NGO)

Public authority

Trade union

Other

0%

10%

20%

30%

40%

50%

60%

70%

80%

90% 100%

To a great extent

Somewhat

A little

Not at all

I do not know / No opinion

109 

  
 
 
 
 
 
 
Among all stakeholder categories, CORDIS is deemed the most relevant EU-wide exploitation support 
service:  32%  of  non-EU  citizens  (18),  29%  of  EU  citizens  (61),  28%  (255)  of  respondents  from 
academia,  28%  (85)  from  companies  and  24%  (22)  from  public  authorities  indicated  that  CORDIS 
helped disseminate and exploit results to a great extent.   

Figure 52: Stakeholder breakdown - Extent to which CORDIS helped to disseminate, exploit and access 
research and innovation results (N = 1 724) 

Academic/research institution

Business association

Company/business organisation

Environmental organisation

EU citizen

Non-EU citizen

Non-governmental organisation (NGO)

Public authority

Trade union

Other

0%

10%

20%

30%

40%

50%

60%

70%

80%

90% 100%

To a great extent

Somewhat

A little

Not at all

I do not know / No opinion

Following  CORDIS,  the  Horizon  Dashboard  is  most  used  among  business  associations  (16%;  5), 
companies (16%; 48), non-EU citizens (16%; 9), EU citizens (15%; 33) and public authorities (15%; 
14), similar to  the  Horizon Results Booster which was assessed by business  associations  (16%;  5), 
NGOs (14%; 9), non-EU citizens (14%; 8), companies (13%; 38) and EU citizens (12%; 25) as helpful 
to a great extent. 

Figure 53: Stakeholder breakdown - Extent to which the Horizon Dashboard helped to disseminate, 
exploit and access research and innovation results (N = 1 740) 

Academic/research institution

Business association

Company/business organisation

Environmental organisation

EU citizen

Non-EU citizen

Non-governmental organisation (NGO)

Public authority

Trade union

Other

0%

10%

20%

30%

40%

50%

60%

70%

80%

90% 100%

To a great extent

Somewhat

A little

Not at all

I do not know / No opinion

110 

  
 
 
 
 
Although  respondents  from  research  or  academia  favoured  the  effectiveness  of  the  Horizon  Results 
Platform over the Horizon Results Booster (14%; 122) by 4 percentage points, which is a similar trend 
also pronounced in the responses of companies (16%; 48) by 3 percentage points. 

Figure  54:  Stakeholder  breakdown  -  Extent  to  which  the  Horizon  Results  Platform  helped  to 
disseminate, exploit and access research and innovation results (N = 1 736) 

Academic/research institution

Business association

Company/business organisation

Environmental organisation

EU citizen

Non-EU citizen

Non-governmental organisation (NGO)

Public authority

Trade union

Other

0%

10%

20%

30%

40%

50%

60%

70%

80%

90% 100%

To a great extent

Somewhat

A little

Not at all

I do not know / No opinion

Figure  55:  Stakeholder  breakdown  -  Extent  to  which  the  Horizon  Results  Booster  helped  to 
disseminate, exploit and access research and innovation results (N = 1 739) 

Academic/research institution

Business association

Company/business organisation

Environmental organisation

EU citizen

Non-EU citizen

Non-governmental organisation (NGO)

Public authority

Trade union

Other

0%

10%

20%

30%

40%

50%

60%

70%

80%

90% 100%

To a great extent

Somewhat

A little

Not at all

I do not know / No opinion

111 

  
 
 
 
 
 
 
SUPPORTING INFORMATION: FEEDBACK FROM CALL FOR EVIDENCE 

Design of Horizon 2020 

General  comments:  Several  comments  from  different  stakeholder  groups  acknowledge  that  Horizon  2020 
successfully contributed to stimulating research and innovation in the EU and provided the framework to enhance 
collaboration  in  R&I  projects.  However,  some  respondents  also  raised  concerns  about  the  changes  to  the 
Framework  Programme’s  budget  due  to  the  annual  inter-institutional  negotiations  as  well  as  about  the 
redistribution of the budget among different priorities during the programme’s implementation. Another concern 
regarded the role of Horizon 2020 in facilitating complementarities and synergies among EU, Member States and 
international initiatives. 

Type of funded R&I actions and Technology Readiness Levels (TRL) coverage: Respondents from NGOs, 
academia and research institutions perceived that Horizon 2020’s project portfolio was not balanced in supporting 
projects at different TRLs, whereas focusing more on projects on technology deployment (TRL 7-9), rather than 
on projects on observations of basic principles, formulation of technology concepts, and experimental Proof of 
Concept (POC) (TRL 1-3). According to some respondents, this tendency was linked to the increasingly impact-
oriented approach of Horizon 2020 that negatively affected the balance between Research and Innovation Actions 
(RIA) and Innovation Actions (IA). The Horizon 2020 Green Deal call was mentioned as an example of additional 
funding targeting projects with a high TRL, rather than basic research – although the issue should be addressed at 
all levels of research. At the same time, bigger and continuous support was suggested for technology validation 
and demonstration in relevant environments (TRL 4-6) to tackle the ‘innovation valley of death’ that sometimes 
threatens innovation uptake. Finally, one respondent from academia questioned the choice of funding incremental 
research in addition to breakthrough research through Horizon 2020. 

Approach for topics / calls for proposals:   Respondents from academia appreciated the bottom-up approach 
applied by the European Research Council, the Marie Skłodowska-Curie Action and the European Innovation 
Council Pathfinder. One respondent pointed out that narrow and restrictive calls limit the chances of developing 
research across different fields of science, reduce research creativity, and lead to limited innovation. Likewise, 
one SME stakeholder observed that targeted calls of proposals with a very narrow scope, limit the possibility for 
SMEs to participate as, compared to large companies, it is harder for SMEs to adjust their R&I activities to fit the 
specific topic. On the other hand, some stakeholders asked for more targeted calls on specific topics (e.g. research 
on paediatric cancer, Lyme disease). 

Some stakeholders appreciated that the same or similar topics are addressed in several calls as it ensured coherence 
and strengthened the long-term vision of the work programme. 

Horizon  2020  Key  Performance  Indicators:  Some  stakeholders  are  not  convinced  by  the  KPIs  chosen  to 
evaluate the impact of Horizon 2020 projects and consider them too focused on short-term, quantitative outputs 
(e.g.,  number  of  publications,  number  of  patents,  number  of  dissemination  activities)  and  not  on  the  broader 
societal impact or on the further exploitation of projects’ results. 

Horizon 2020 Implementation  

Publication of the calls for proposal: There were concerns about the timing of the calls for proposals. According 
to  various  stakeholders,  a  large  number  of  calls  were  announced  too  close  to  the  deadline,  which  made  it 
challenging to form a consortium and write a high-quality proposal. It was recommended to extend the call for 
proposal deadlines and to arrange them at regular intervals to facilitate long-term application planning, and to 
launch open calls for continuous submission over one year.  

Proposal  evaluation:  Several  respondents recommended  to  improve the  scoring  bandwidth  in  order  to better 
reflect differences in the quality of the proposals. With the current system, respondents noted that there is very 
little variation in the final score between successful and unsuccessful, high-quality proposals. Some respondents 
expressed their frustration about proposals scoring very high but not being selected for funding. This could be 
tackled  by  increasing  synergies  with  other  EU-funded  programmes  (i.e.  financing  excellent  but  unsuccessful 
proposals through other programmes). Another recommendation was to offer lump sums to proposals reaching a 
certain threshold, instead of asking to improve and submit the proposal under another call. The review process of 
some programmes (e.g. the ERC) is appreciated more than that of others (e.g. SME instrument) – with respect to 
the different composition of evaluation panels. 

112 

  
 
 
Dissemination of projects’ results: The dissemination of the projects’ results was pointed out as a major area for 
improvement  -  some  project  beneficiaries  required  more  support  from  the  European  Commission  services  in 
communicating  the  projects’  results  to  attract  additional  funding  for  further  development  and,  ultimately,  for 
market uptake. Results and data sharing were considered important to advance research, especially in the scientific 
(e.g. medical) field. 

Administrative burden for applicants and participants 

Application  process:  Respondents  appreciated  the  simplifications  operated  under  Horizon  2020  for  the 
preparation  and  submission  of  proposals  compared  to  the  Seventh  Framework  Programme  (FP7).  However, 
feedback  suggests that respecting  the  page limit  of  the  proposal  template created  some  difficulties,  especially 
when  the  projects  did  not focus  on  a  single technology  or  product  but  were  more  complex  and  encompassed 
several aspects (e.g., for R&I infrastructures). Applicants found it difficult and time-consuming to describe in a 
short section the projects’ activities and expected outcomes. According to one SME, the application process is 
particularly  demanding  for  SMEs,  which  have  limited  resources  for  proposal  writing  and  can  leverage  fewer 
connections to be part of large consortia. Respondents pointed out that the Funding and Tenders Portal required 
previous knowledge and experience with EU funding.  

Grant  agreement  and  project  reporting:  The  preparation  of  Grant  Agreement  amendments  was  raised  as 
particularly tedious for participants. Some respondents complained about an excessive, time-consuming request 
for administrative documents in this phase. There are contrasting views on whether the documentation effort for 
project reporting was reasonable. Finally, in some instances, project officers were reportedly too slow to respond 
to requests concerning project report amendment and validation, thus generating additional workload for project 
coordinators. 

Audit procedures: Various respondents consider the period to receive feedback from auditors as too long, while 
the time allowed to comply with their demands was too short. A prompter publication of audit reports would 
contribute  to  improving  the  follow-up and  would allow  beneficiaries to  quickly  take  auditors’ comments  into 
consideration.  According  to  some  stakeholders,  financial  reporting  requirements  were  too  complex.  Overlaps 
between the different types of audits under Horizon 2020 were reported, pointing to a lack of coordination between 
the  Commission  and  the  contracted  audit  firms.  According  to  one  contribution,  the  rules  and  guidelines  for 
Horizon  2020  financial  reporting  were  interpreted  in  different  ways  by  the  auditing  institutions,  leading  to 
inconsistencies and duplications. More consistency and better coordination between the different types of audits 
and the institutions involved (the European Commission, the European Court of Auditors, the European Anti-
Fraud Office and external audit firms) could have avoided overlaps. 

Widening participation 

Access and participation from associated and non-associated third countries: It was recommended to increase 
access  and  participation  from  associated  and  non-associated  third  countries.  This  would  lead  to  a  wider 
dissemination of Horizon 2020 project results in these countries. 

International  collaboration:  One  respondent  expressed  disappointment  about  the  lower  level  of  support  for 
international collaboration under Horizon 2020, in particular for Low and Medium Income Countries (LMICs), 
where  funding  decreased  compared  to  the  FP7  programming  period.  Participants  perceived  Horizon  2020  as 
primarily  focused  on  strengthening  the  EU’s  competitiveness  rather  than  tackling  global  challenges  through 
international cooperation.  

113 

  
 
 
 
 
 
ANNEX VI: ADDITIONAL DATA ON HORIZON 2020 STATE OF PLAY 

Applications to the programme 
(Source: CORDA data – cut-off date: 1 January 2023) 

Over 1 million applications were submitted through over 285 000 eligible proposals. Organisations 
located in EU28 Member States account for 88% of all applications, followed by associated countries 
(8.3%) and non-associated third countries (3.7%).  

More detail is available on the tables that follow, on applications and success rates by country by 
organisation type and by pillar. 

Table 18: Applications and success rates by type of organisation 

Type of Organisation 

Applicatio
ns in 
eligible 
proposals 

% of all  
applications 

Applications in 
retained 
Proposals 

Success 
rate of 
application 

% 

EU 
contribution 
requested in 
retained 
proposals (in 
million EUR) 

Higher Education Institutions 
(HES) 

357 364 

35.6% 

50 330 

14.1% 

24 950  38.6% 

Research Organisations (REC) 

171 199 

Private for-profit entities (PRC)  

404 260 

Public bodies (PUB) 

Other (OTH) 

 Total 

17.0% 

40.2% 

3.3% 

3.8% 

31 987 

55 684 

8 234 

7 392 

18.7% 

13.8% 

24.8% 

19.2% 

16 405  25.4% 

19 118  29.5% 

2 222 

3.4% 

2 007 

3.1% 

33 232 

38 544 

1 004 599 

100% 

153 627 

15.3% 

64 702 

100% 

114 

  
 
 
 
 
 
 
 
 
 
Table 19: Applications by country and programme part in Horizon 2020 (Source: CORDA data – cut-off date: 1 January 2023) 

115 

EU-28 Member StateCountry codeCountry groupHorizon 2020 applications in eligible proposals over 2014-2020Excellent ScienceIndustrial LeadershipSocietal challengesSpreading excellence and widening participationOtherExcellent ScienceIndustrial LeadershipSocietal challengesSpreading excellence and widening participationOtherExcellent ScienceIndustrial LeadershipSocietal challengesSpreading excellence and widening participationOtherAustriaATEU-1525,6638,5775,7709,7953001,22133%22%38%1%5%2%3%3%3%3%BelgiumBEEU-1540,30713,7607,45917,0013061,78134%19%42%1%4%4%3%5%3%4%BulgariaBGEU-136,6959631,6443,45216647014%25%52%2%7%0%1%1%2%1%CyprusCYEU-136,6541,4741,4323,02032240622%22%45%5%6%0%1%1%3%1%CzechiaCZEU-1310,4413,5152,3993,58830263734%23%34%3%6%1%1%1%3%2%GermanyDEEU-15108,12744,21424,89233,6831,2144,12441%23%31%1%4%12%11%9%12%10%DenmarkDKEU-1523,93410,9124,0647,84819891246%17%33%1%4%3%2%2%2%2%EstoniaEEEU-135,7541,1921,4182,63619831021%25%46%3%5%0%1%1%2%1%GreeceELEU-1534,5707,0379,39216,2592071,67520%27%47%1%5%2%4%5%2%4%SpainESEU-15110,66134,92228,70341,9534964,58732%26%38%0%4%9%13%12%5%11%FinlandFIEU-1522,0557,0396,1187,90120079732%28%36%1%4%2%3%2%2%2%FranceFREU-1577,95034,59816,71623,5005272,60944%21%30%1%3%9%8%7%5%6%CroatiaHREU-134,8481,0847812,52319626422%16%52%4%5%0%0%1%2%1%HungaryHUEU-1310,5802,6202,5714,64315459225%24%44%1%6%1%1%1%1%1%IrelandIEEU-1517,9066,3964,3646,12513488736%24%34%1%5%2%2%2%1%2%ItalyITEU-15109,62334,59427,26742,7117834,26832%25%39%1%4%9%12%12%8%11%LithuaniaLTEU-134,1619238991,81914937122%22%44%4%9%0%0%1%1%1%LuxembourgLUEU-153,2717971,0111,2874712924%31%39%1%4%0%0%0%0%0%LatviaLVEU-133,4807078741,57312919720%25%45%4%6%0%0%0%1%0%MaltaMTEU-131,6263322758386511620%17%52%4%7%0%0%0%1%0%NetherlandsNLEU-1555,95424,0549,71919,5954572,12943%17%35%1%4%6%4%5%4%5%PolandPLEU-1318,2515,2034,2597,53141284629%23%41%2%5%1%2%2%4%2%PortugalPTEU-1525,3828,2795,6919,8045611,04733%22%39%2%4%2%3%3%5%3%RomaniaROEU-1310,6401,8522,4225,38931166617%23%51%3%6%0%1%2%3%2%SwedenSEEU-1529,54212,1696,2919,7373251,02041%21%33%1%3%3%3%3%3%3%SloveniaSIEU-1310,4582,5132,5964,48522863624%25%43%2%6%1%1%1%2%2%SlovakiaSKEU-134,4599301,0941,96617529421%25%44%4%7%0%0%1%2%1%United KingdomUKEU-15100,60753,56016,17527,1986842,99053%16%27%1%3%14%7%8%7%7%Total EU-28883,599324,216196,296317,8609,24635,98137%22%36%1%4%86%90%88%89%89%Total EU-1398,04723,30822,66443,4632,8075,80524%23%44%3%6%6%10%12%27%14%Total EU-15785,552300,908173,632274,3976,43930,17638%22%35%1%4%80%79%76%62%75%Associated countries83,37730,94819,91727,8921,0483,57237%24%33%1%4%8%9%8%10%9%Thrid countries37,62320,7742,59213,4744873555%7%36%0%2%6%1%4%0%2%Total Horizon 20201,004,599375,938218,805359,22610,34240,28837%22%36%1%4%100%100%100%100%100%% of all applications per pillar% of applications of the country by pillarNumber of applications 
 
Table 20: Applications by country and type of participating organisation in Horizon 2020 ((Source: CORDA data – cut-off date: 1 January 2023) 

116 

EU-28 Member StateCountry codeCountry groupHorizon 2020 applications in eligible proposals over 2014-2020HESPRCPUBRECOTHHESPRCPUBRECOTHHESPRCPUBRECOTHAustriaATEU-1525,6638,68610,0005135,3481,11634%39%2%21%4%2%2%2%3%3%BelgiumBEEU-1540,30712,41713,8009156,5546,62131%34%2%16%16%3%3%3%4%17%BulgariaBGEU-136,6951,0773,2394161,27868516%48%6%19%10%0%1%1%1%2%CyprusCYEU-136,6542,1533,47436431734632%52%5%5%5%1%1%1%0%1%CzechiaCZEU-1310,4414,1013,9402901,65645439%38%3%16%4%1%1%1%1%1%GermanyDEEU-15108,12736,49742,3131,86424,5532,90034%39%2%23%3%10%10%6%14%8%DenmarkDKEU-1523,93411,9328,7771,1311,47761750%37%5%6%3%3%2%3%1%2%EstoniaEEEU-135,7542,0322,66529933242635%46%5%6%7%1%1%1%0%1%GreeceELEU-1534,5709,47313,6761,2629,0121,14727%40%4%26%3%3%3%4%5%3%SpainESEU-15110,66126,16447,7044,93227,9293,93224%43%4%25%4%7%12%15%16%10%FinlandFIEU-1522,0558,9408,4426003,53453941%38%3%16%2%3%2%2%2%1%FranceFREU-1577,95016,46832,7381,81623,8223,10621%42%2%31%4%5%8%5%14%8%CroatiaHREU-134,8481,5711,81143282520932%37%9%17%4%0%0%1%0%1%HungaryHUEU-1310,5802,5255,6794971,44743224%54%5%14%4%1%1%1%1%1%IrelandIEEU-1517,9068,3627,90052559152847%44%3%3%3%2%2%2%0%1%ItalyITEU-15109,62334,12650,0453,54718,3813,52431%46%3%17%3%10%12%11%11%9%LithuaniaLTEU-134,1611,3491,67138251824132%40%9%12%6%0%0%1%0%1%LuxembourgLUEU-153,2716411,8388253517520%56%3%16%5%0%0%0%0%0%LatviaLVEU-133,4809841,47629550222328%42%8%14%6%0%0%1%0%1%MaltaMTEU-131,626589764149725236%47%9%4%3%0%0%0%0%0%NetherlandsNLEU-1555,95423,27422,2961,3506,8912,14342%40%2%12%4%7%6%4%4%6%PolandPLEU-1318,2516,0677,4727743,25268633%41%4%18%4%2%2%2%2%2%PortugalPTEU-1525,3826,6319,8981,2486,4771,12826%39%5%26%4%2%2%4%4%3%RomaniaROEU-1310,6402,7134,6269351,62773925%43%9%15%7%1%1%3%1%2%SwedenSEEU-1529,54214,60611,1771,1812,04053849%38%4%7%2%4%3%4%1%1%SloveniaSIEU-1310,4582,3084,6745462,45247822%45%5%23%5%1%1%2%1%1%SlovakiaSKEU-134,4591,3382,13524048925730%48%5%11%6%0%1%1%0%1%United KingdomUKEU-15100,60758,91032,0452,2465,1962,21059%32%2%5%2%16%8%7%3%6%Total EU-28883,599305,934356,27528,831157,10735,45234.6%40.3%3.3%17.8%4.0%86%88%87%92%92%Total EU-1398,04728,80743,6265,61914,7675,22829.4%44.5%5.7%15.1%5.3%8%11%17%9%14%Total EU-15785,552277,127312,64923,212142,34030,22435.3%39.8%3.0%18.1%3.8%78%77%70%83%78%Associated countries83,37732,01736,4623,1729,6782,04838.4%43.7%3.8%11.6%2.5%9%9%10%6%5%Thrid countries37,62319,41311,5141,2294,4141,04451.6%30.6%3.3%11.7%2.8%5%3%4%3%3%Total Horizon 20201,004,599357,364404,25133,232171,19938,54435.6%40.2%3.3%17.0%3.8%100%100%100%100%100%Legend:HES - Higher Education InstritutionsREC- Research OrganisationsPRC - Private-for-profit entitiesPUB - Public bodiesOTH - Other% of all applications per organisation type% of all applications per countryNumber of applications 
 
Table 21: Applications by country and success rates of application in Horizon 2020 (Source: CORDA data – cut-off date: 1 January 2023) 

117 

EU-28 Member StateCountry codeCountry groupHorizon 2020 applications in eligible proposals over 2014-2020% of totalHorizon 2020 applications in eligible proposals per yearHorizon 2020 applications in high quality proposals% of high quality applicationsRetained ApplicationsSuccess rate of application in Horizon 2020'000 of scientists and engineers per country*Share of scientists and engineers in EU-28Applications per '000 of scientists and engineers in the populationEuropean Innovation Scoreboard Country Group (2014)European Innovation Scoreboard Country Group (2020)AustriaATEU-1525,6632.6%3,66615,44060.2%4,44917.3%329.221.9%78FOLLOWERSTRONGBelgiumBEEU-1540,3074.0%5,75826,09764.7%7,62318.9%466.962.7%86FOLLOWERSTRONGBulgariaBGEU-136,6950.7%9562,67339.9%84812.7%191.51.1%35MODESTMODESTCyprusCYEU-136,6540.7%9513,60854.2%88113.2%26.180.2%254FOLLOWERMODERATECzechiaCZEU-1310,4411.0%1,4925,70454.6%1,62015.5%331.221.9%32MODERATEMODERATEGermanyDEEU-15108,12710.8%15,44766,66161.7%18,21616.8%3090.618.0%35LEADERSTRONGDenmarkDKEU-1523,9342.4%3,41915,06963.0%3,61515.1%282.941.6%85LEADERLEADEREstoniaEEEU-135,7540.6%8222,88750.2%79113.7%44.440.3%129FOLLOWERSTRONGGreeceELEU-1534,5703.4%4,93919,26755.7%4,81713.9%250.761.5%138MODERATEMODERATESpainESEU-15110,66111.0%15,80961,97056.0%15,82914.3%1380.328.0%80MODERATEMODERATEFinlandFIEU-1522,0552.2%3,15111,92254.1%3,12214.2%277.421.6%80LEADERLEADERFranceFREU-1577,9507.8%11,13648,92662.8%13,62217.5%1739.6210.1%45FOLLOWERSTRONGCroatiaHREU-134,8480.5%6932,28247.1%65713.6%96.660.6%50MODERATEMODERATEHungaryHUEU-1310,5801.1%1,5115,13248.5%1,35312.8%260.31.5%41MODERATEMODERATEIrelandIEEU-1517,9061.8%2,55810,54658.9%2,65714.8%231.441.3%77FOLLOWERSTRONGItalyITEU-15109,62310.9%15,66056,46951.5%14,24513.0%1043.386.1%105MODERATEMODERATELithuaniaLTEU-134,1610.4%5941,97647.5%54113.0%97.280.6%43MODERATEMODERATELuxembourgLUEU-153,2710.3%4671,95559.8%54116.5%26.940.2%121FOLLOWERLEADERLatviaLVEU-133,4800.3%4971,56745.0%47713.7%47.540.3%73MODESTMODERATEMaltaMTEU-131,6260.2%23281249.9%22714.0%16.020.1%101MODERATEMODERATENetherlandsNLEU-1555,9545.6%7,99335,71763.8%9,66617.3%834.524.9%67FOLLOWERLEADERPolandPLEU-1318,2511.8%2,6078,95949.1%2,48313.6%1169.986.8%16MODERATEMODERATEPortugalPTEU-1525,3822.5%3,62613,92454.9%3,29613.0%371.822.2%68MODERATESTRONGRomaniaROEU-1310,6401.1%1,5204,86845.8%1,38913.1%534.363.1%20MODESTMODESTSwedenSEEU-1529,5422.9%4,22017,61959.6%4,53815.4%581.63.4%51LEADERLEADERSloveniaSIEU-1310,4581.0%1,4944,99647.8%1,24811.9%78.360.5%133FOLLOWERMODERATESlovakiaSKEU-134,4590.4%6371,95143.8%59313.3%104.440.6%43MODERATEMODERATEUnited KingdomUKEU-15100,60710.0%14,37261,53461.2%15,38215.3%3280.0519.1%31FOLLOWERSTRONGTotal EU-28883,59988.0%126,228510,53157.8%134,72615.2%17185.87100.0%51FOLLOWERSTRONGTotal EU-1398,0479.8%14,00747,41548.4%13,10813.4%2998.2817.4%33Total EU-15785,55278.2%112,222463,11659.0%121,61815.5%14187.5982.6%55Associated countries83,3778.3%11,91145,87655.0%12,13914.6%Thrid countries37,6233.7%5,37525,37667.4%6,76218.0%Total Horizon 20201,004,599100.0%143,514581,78357.9%153,62715.3%* Source = Eurostat  
 
 
Funding allocation and participants in signed grants (Source: CORDA data – cut-off date: 1 January 2023) 

EUR 68.3 billion were allocated through 35 426 grants. In total, 41 575 different organisations 
benefited from Horizon 2020 funding, some participating in several Horizon 2020 projects.  

Figure 56: Funding per type of organisation 

Figure 57: Funding by Member State (in EUR million) 

118 

 
 
 
 
 
Figure 58: Funds received by Member States (by EUR million GERD) 

119 

 
 
 
 
 
 
Figure 59: Funding allocation per pillar 

Figure 60: Participation by country group 

178 104 participations were registered in Horizon 2020, a same organisation participating sometimes in several projects.  

(Source: CORDA data – cut-off date: 1 January 2023) 

More  detail  is  available  in  the  tables  that  follow  on  Horizon  2020  investment  (a)  by  pillar,  (b)  by 
organisation type, and (c) by country. 

120 

 
 
 
 
 
 
 
 
 
Table 22: Horizon 2020 investments by country and by pillar of Horizon 2020 (Source: CORDA data – cut-off date: 1 January 2023) 

121 

EU-28 Member StateCountry codeCountry groupHorizon 2020 investments in signed grants (EUR million) Excellent ScienceIndustrial LeadershipSocietal challengesSpreading excellence and widening participationOtherExcellent ScienceIndustrial LeadershipSocietal challengesSpreading excellence and widening participationOtherExcellent ScienceIndustrial LeadershipSocietal challengesSpreading excellence and widening participationOtherAustriaATEU-151 956706448742124836%23%38%1%2%3%3%3%1%2%BelgiumBEEU-153 3959136861 40926712027%20%42%8%4%4%5%5%26%6%BulgariaBGEU-1316222257630914%15%47%19%6%0%0%0%3%0%CyprusCYEU-13318605510788819%17%34%28%3%0%0%0%9%0%CzechiaCZEU-1351316291177503232%18%35%10%6%1%1%1%5%2%GermanyDEEU-1510 1193 9422 1783 5584439639%22%35%0%4%16%16%13%4%19%DenmarkDKEU-151 76473527670154842%16%40%0%3%3%2%3%0%2%EstoniaEEEU-13274384912853614%18%47%19%2%0%0%0%5%0%GreeceELEU-151 71728956082963417%33%48%0%2%1%4%3%1%2%SpainESEU-156 3791 7671 6252 797918128%25%44%0%3%7%12%11%1%9%FinlandFIEU-151 537457413601155230%27%39%1%3%2%3%2%1%2%FranceFREU-157 4442 7631 5692 8561623937%21%38%0%3%11%11%11%2%11%CroatiaHREU-1313829177510621%12%55%7%5%0%0%0%1%0%HungaryHUEU-1337011670139252031%19%38%7%5%0%1%1%2%1%IrelandIEEU-151 20341630143165035%25%36%0%4%2%2%2%1%2%ItalyITEU-155 6991 7131 2952 4591521830%23%43%0%4%7%9%9%1%10%LithuaniaLTEU-139514264051015%27%42%6%10%0%0%0%1%0%LuxembourgLUEU-152014664845223%32%42%2%1%0%0%0%0%0%LatviaLVEU-1311712215128410%18%44%24%4%0%0%0%3%0%MaltaMTEU-133775204119%14%52%11%4%0%0%0%0%0%NetherlandsNLEU-155 3812 1588922 2181210140%17%41%0%2%9%6%8%1%5%PolandPLEU-13746197217244602826%29%33%8%4%1%2%1%6%1%PortugalPTEU-151 1523162454541023427%21%39%9%3%1%2%2%10%2%RomaniaROEU-133014649183111215%16%61%3%4%0%0%1%1%1%SwedenSEEU-152 317887412945136038%18%41%1%3%4%3%4%1%3%SloveniaSIEU-133806984181261918%22%48%7%5%0%1%1%3%1%SlovakiaSKEU-1313726206418919%15%47%13%6%0%0%0%2%0%United KingdomUKEU-157 8494 1069652 5423520152%12%32%0%3%16%7%10%3%10%Total EU-2861 70122 01312 65724 1149691 94836%21%39%2%3%88%92%91%95%93%Total EU-133 5898007291 48640916522%20%41%11%5%3%5%6%40%8%Total EU-1558 11221 21311 92822 6275601 78337%21%39%1%3%85%86%86%55%85%Associated countries6 0942908110618934813948%18%31%1%2%12%8%7%5%7%Thrid countries52810436382.505.520%7%72%0%1%0%0%1%0%0%Total Horizon 202068 32325 02513 79926 3891 0172 09237%20%39%1%3%100%100%100%100%100%Investment by pillar% of all investments in the same pillar% of H2020 investments by country and by pillar 
 
 
Table 23: Horizon 2020 investments by type of organisation (Source: CORDA data – cut-off date: 1 January 2023) 

122 

EU-28 Member StateCountry codeCountry groupHorizon 2020 investments in signed grants (EUR million) HESPRCPUBRECOTHHESPRCPUBRECOTHHESPRCPUBRECOTHAustriaATEU-151 9567766232442810440%32%1%22%5%3%3%1%3%4%BelgiumBEEU-153 3951 0977366979070432%22%2%23%21%4%4%3%5%24%BulgariaBGEU-13162415211471025%32%7%29%6%0%0%1%0%0%CyprusCYEU-1331813312910351242%41%3%11%4%0%1%0%0%0%CzechiaCZEU-1351322714916992244%29%3%19%4%1%1%1%1%1%GermanyDEEU-1510 1193 6222 7371383 33728636%27%1%33%3%13%14%7%20%10%DenmarkDKEU-151 7641 0484441031066259%25%6%6%4%4%2%5%1%2%EstoniaEEEU-13274133921872548%34%6%2%9%0%0%1%0%1%GreeceELEU-151 717473511356653428%30%2%39%2%2%3%2%4%1%SpainESEU-156 3791 3942 1712972 29122622%34%5%36%4%5%11%14%13%8%FinlandFIEU-151 537652400413905542%26%3%25%4%2%2%2%2%2%FranceFREU-157 4441 1282 5962003 25426615%35%3%44%4%4%13%9%19%9%CroatiaHREU-1313845391335732%28%9%25%5%0%0%1%0%0%HungaryHUEU-133709812218834927%33%5%22%13%0%1%1%0%2%IrelandIEEU-151 20368941646322157%35%4%3%2%3%2%2%0%1%ItalyITEU-155 6991 9012 1081691 36315733%37%3%24%3%7%11%8%8%5%LithuaniaLTEU-1395283211131229%33%11%14%12%0%0%1%0%0%LuxembourgLUEU-152016386736931%43%3%18%5%0%0%0%0%0%LatviaLVEU-1311747331217740%28%11%14%6%0%0%1%0%0%MaltaMTEU-133718972148%25%19%6%3%0%0%0%0%0%NetherlandsNLEU-155 3812 6661 3131111 00029150%24%2%19%5%10%7%5%6%10%PolandPLEU-13746206190492119028%25%7%28%12%1%1%2%1%3%PortugalPTEU-151 152301322574373526%28%5%38%3%1%2%3%3%1%RomaniaROEU-133016311120792921%37%7%26%9%0%1%1%0%1%SwedenSEEU-152 3171 3246001651923457%26%7%8%1%5%3%8%1%1%SloveniaSIEU-133807213525139919%36%7%37%2%0%1%1%1%0%SlovakiaSKEU-1313748547191035%39%5%14%7%0%0%0%0%0%United KingdomUKEU-157 8495 4201 46417863215669%19%2%8%2%20%8%8%4%5%Total EU-2861 70123 71317 6741 85415 7382 72238%29%3%26%4%88%92%88%93%93%Total EU-133 5891 1581 14721778528232%32%6%22%8%4%6%10%5%10%Total EU-1558 11222 55516 5271 63814 9532 44039%28%3%26%4%84%86%77%88%83%Associated countries6 0943 0301 5582221 13415050%26%4%19%2%11%8%10%7%5%Thrid countries52822168411277042%13%8%24%13%1%0%2%1%2%Total Horizon 202068 32326 96419 3002 11716 9992 94239%28%3%25%4%100%100%100%100%100%Investment by type of organisation% of investments to an org. type, by country% of investments in a country by org. type  
         Table 24: Participation in Horizon 2020 by country (Source: CORDA data – cut-off date: 1 January 2023) 

123 

EU-28 Member StateCountry codeCountry groupHorizon 2020 participations in signed grants over 2014-2020% of totalHorizon 2020 participations in signed grants per year000 scientists and EngineersShare of scientists and engineers in EU28Participations  per '000 scientists and engineers in the populationNumber of Horizon 2020 projects with at least 1 participant from the country% of totalAustriaATEU-155,0922.9%727329.221.9%15.53,2329.1%BelgiumBEEU-158,4524.7%1,207466.962.7%18.15,05614.3%BulgariaBGEU-139980.6%143191.51.1%5.26661.9%CyprusCYEU-139860.6%14126.180.2%37.77362.1%CzechiaCZEU-131,8831.1%269331.221.9%5.71,3983.9%GermanyDEEU-1520,78711.7%2,9703090.618.0%6.79,93728.1%DenmarkDKEU-154,0022.2%572282.941.6%14.12,9108.2%EstoniaEEEU-138990.5%12844.440.3%20.27012.0%GreeceELEU-155,5023.1%786250.761.5%21.92,9018.2%SpainESEU-1518,88510.6%2,6981380.328.0%13.78,80824.9%FinlandFIEU-153,5122.0%502277.421.6%12.72,2436.3%FranceFREU-1517,1559.6%2,4511739.6210.1%9.98,00522.6%CroatiaHREU-138200.5%11796.660.6%8.55841.6%HungaryHUEU-131,5550.9%222260.31.5%6.01,1443.2%IrelandIEEU-152,9661.7%424231.441.3%12.82,1616.1%ItalyITEU-1517,1769.6%2,4541043.386.1%16.57,89322.3%LithuaniaLTEU-136190.3%8897.280.6%6.45041.4%LuxembourgLUEU-156290.4%9026.940.2%23.35391.5%LatviaLVEU-135510.3%7947.540.3%11.64371.2%MaltaMTEU-132630.1%3816.020.1%16.41920.5%NetherlandsNLEU-1511,1316.2%1,590834.524.9%13.36,17817.4%PolandPLEU-132,8551.6%4081169.986.8%2.41,9595.5%PortugalPTEU-153,9612.2%566371.822.2%10.72,4476.9%RomaniaROEU-131,6190.9%231534.363.1%3.01,0563.0%SwedenSEEU-155,2142.9%745581.63.4%9.03,4089.6%SloveniaSIEU-131,4800.8%21178.360.5%18.91,0162.9%SlovakiaSKEU-136970.4%100104.440.6%6.75161.5%United KingdomUKEU-1517,4589.8%2,4943280.0519.1%5.310,51729.7%Total EU-28157,14788.2%22,45017185.87100.0%9.132,55291.9%Total EU-1315,2258.5%2,1752998.2817.4%5.16,37418.0%Total EU-15141,92279.7%20,27514187.5982.6%10.031,36488.5%Associated countries13,6697.7%1,9537,95822.5%Thrid countries7,2884.1%1,0413,2219.1%Total Horizon 2020178,104100.0%25,44335,426100.0% 
Table 25: Horizon 2020 investments by country (Source: CORDA data – cut-off date: 1January 2023) 

124 

EU-28 Member StateCountry codeCountry groupHorizon 2020 investments in signed grants (EUR million) over 2014-2020% of total H2020 investment% of H2020 investment in EU-28Horizon 2020 investments in signed grants (EUR million) per yearGERD (in EUR million*Horizon 2020 investment in EUR per EUR million of GERDEuropean Innovation Scoreboard Country Group (2014)European Innovation Scoreboard Country Group (2020)AustriaATEU-151 9562.9%3.2%27912 57522 215FOLLOWERSTRONGBelgiumBEEU-153 3955.0%5.5%48515 78930 718FOLLOWERSTRONGBulgariaBGEU-131620.2%0.3%2353643 100MODESTMODESTCyprusCYEU-133180.5%0.5%45196231 351FOLLOWERMODERATECzechiaCZEU-135130.8%0.8%734 52116 204MODERATEMODERATEGermanyDEEU-1510 11914.8%16.4%1 446109 71613 175LEADERSTRONGDenmarkDKEU-151 7642.6%2.9%2529 36326 909LEADERLEADEREstoniaEEEU-132740.4%0.4%3951675 872FOLLOWERSTRONGGreeceELEU-151 7172.5%2.8%2452 56595 660MODERATEMODERATESpainESEU-156 3799.3%10.3%91116 50955 198MODERATEMODERATEFinlandFIEU-151 5372.2%2.5%2207 21230 446LEADERLEADERFranceFREU-157 44410.9%12.1%1 06354 27319 594FOLLOWERSTRONGCroatiaHREU-131380.2%0.2%2067629 168MODERATEMODERATEHungaryHUEU-133700.5%0.6%532 36422 364MODERATEMODERATEIrelandIEEU-151 2031.8%2.0%1724 54837 795FOLLOWERSTRONGItalyITEU-155 6998.3%9.2%81425 77331 591MODERATEMODERATELithuaniaLTEU-13950.1%0.2%1460022 710MODERATEMODERATELuxembourgLUEU-152010.3%0.3%2971440 275FOLLOWERLEADERLatviaLVEU-131170.2%0.2%1722075 581MODESTMODERATEMaltaMTEU-13370.1%0.1%59158 863MODERATEMODERATENetherlandsNLEU-155 3817.9%8.7%76918 90540 660FOLLOWERLEADERPolandPLEU-137461.1%1.2%1077 77313 714MODERATEMODERATEPortugalPTEU-151 1521.7%1.9%1653 40148 407MODERATESTRONGRomaniaROEU-133010.4%0.5%431 08439 721MODESTMODESTSwedenSEEU-152 3173.4%3.8%33117 40119 018LEADERLEADERSloveniaSIEU-133800.6%0.6%541 06251 112FOLLOWERMODERATESlovakiaSKEU-131370.2%0.2%2087922 266MODERATEMODERATEUnited KingdomUKEU-157 84911.5%12.7%1 12143 13425 996FOLLOWERSTRONGTotal EU-2861 70190.3%100.0%8 814362 39524 323Total EU-133 5895.3%51320 51724 985Total EU-1558 11285.1%8 302341 87824 283Associated countries6 0948.9%Thrid countries5280.8%Total Horizon 202068 323100.0%* Source: Eurostat     
 
Funding rates and leverage 

Table 26: Horizon 2020 funding rates and direct leverage factors, by type of action 

Funding rate, in %134 

Direct leverage factor 
(PRC), in EUR 

Horizon 2020 

Horizon 2020 excluding fundamental 
research (ERC, MSCA) and CSA 
RIA 

IA 

SME Instrument 

JTIs 

ERA-NET Cofund 

EJP Cofund 

KICs 

Art. 185 

varies 

varies 

100 

70 (100 for non-profit) 

- 

varies 

33 

70 

- 

varies 

0.23 (0.57) 

0.35 (0.63) 

0.05 (0.08) 

0.24 (0.62) 

0.43 

1.08 

2.2 

0.92 

0.23 

0.52 

Notes: Definitions provided in glossary. Data is retrieved from  e-CORDA (cut-off date 07/02/2023). For Art. 185  TFEU 
initiatives only 40% of the cost data is available. 

Table 27: Horizon 2020 direct leverage factors for Innovation Actions (IA), by programme part 

Programme part  

Direct leverage 
factor, in EUR 

Secure, clean and efficient energy 

Smart, green and integrated transport 
Secure  societies  -  protecting  the  freedom  and  security  of  Europe  and  its 
citizens 
Leadership in enabling and industrial technologies (LEIT) 
Climate action, environment, resource efficiency and raw materials 

0.44 

0.26 
0.21 

0.2 
0.15 

Notes: Definitions provided in the glossary. Data from ECORDA (cut-off date 07/02/2023). Only the programme parts with 
the highest leverage (first five) are presented. 

134 Actual funding rate can be lower as applicants can request less funding than the maximum amount defined by the funding 
rate. The reasons some applicants request less funding than the maximum vary, but one is that they may receive funding from 
another source. Lower requested amount (lower funding rate) translates into higher direct leverage factor (see glossary for 
definitions and formal relationship among the two). 

125 

 
 
 
 
 
 
 
 
 
 
 
Level of TRLs 

Figure 61: Selected features of Horizon 2020 support across the TRL scale 

Source:  European  Commission,  Directorate-General  for  Research  and  Innovation,  Evaluation  study  on  the  European 
Innovation  Council 
the  European  Union,  2022,  p. 60, 
https://data.europa.eu/doi/10.2777/261324 - updated by PPMI for the Innovative Europe evaluation study (2023) and adapted 
by DG RTD in May 2023. 

report,  Publications  Office  of 

(EIC)  pilot: 

final 

126 

Type of instrument           Grants                                                                                 Equity                     Advisory And networking support                                                                        TRL 1 2 3 4 5 6 7 8 9 Innovation maturity Initial  idea   Technology validated at  a lab  Prototype  demonstration  EIC Pilot SME Instrument EIT Innovation support projects / KICs Other Horizon 2020 Innovation Actions under LEIT  MSCA and ERC EIC Pilot Accelerator blended finance InnovFin Equity EIC Pilot Business Acceleration Services InnovFin Advisory EIT Business Creation and Support activities IP Helpdesk Societal Challenges Fast Track to Innovation SME Instrument business support services and coaching EIC Pathfinder