Study on the proposal evaluation system for the EU R&I framework programme : final report : annexes.
Date: 2022-01-13
Institutions: Directorate-General for Research and Innovation
Subjects: EU research policy, European official, Framework Programme for Research and Development, document retrieval, financing, information analysis, innovation, organisation of research, research programme, research staff

Study on the proposal 
evaluation system for 
the EU R&I framework 
programme 
Final report: Annexes 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Study on the proposal evaluation system for the EU R&I framework programme 

European Commission 
Directorate-General for Research and Innovation 
Directorate G — Common Policy Center 
Unit G2 — Common Programme Analysis & Regulatory Reform  
Contact   Diletta Zonta  
Email  

diletta.zonta@ec.europa.eu 
RTD-PUBLICATIONS@ec.europa.eu 

European Commission 
B-1049 Brussels 

Manuscript completed in October 2021. 

This document has been prepared for the European Commission, however it reflects the views only of the authors, and the 
European Commission shall not be liable for any consequence stemming from the reuse. 

More information on the European Union is available on the internet (http://europa.eu). 

PDF  

ISBN 978-92-76-42546-5 

doi: 10.2777/246820  KI-01-21-370-EN-N 

Luxembourg: Publications Office of the European Union, 2022 

© European Union, 2022 

a 

under 

Creative 

authorised 

The  reuse  policy  of  European  Commission  documents  is  implemented  by  Commission  Decision  2011/833/EU  of  12  December 
2011 on the reuse of Commission documents (OJ L 330, 14.12.2011, p. 39). Unless otherwise noted, the reuse of this document 
is 
licence 
(https://creativecommons.org/licenses/by/4.0/). This means that reuse is allowed provided appropriate credit is given and any  
changes are indicated. 
For any use or reproduction of elements that are not owned by the European Union, permission may need to be sought directly 
from the respective rightholders. 
Image credits: 
Cover  page:  ©  Lonely  #46246900,  ag  visuell  #16440826,  Sean  Gladwell  #6018533,  LwRedStorm  #3348265,  2011;  kras99 
#43746830, 2012. Source: Fotolia.com. 

International 

Attribution 

Commons 

(CC-BY 

4.0) 

4.0 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
EUROPEAN COMMISSION 

Study on the proposal 
evaluation system for the EU 
R&I framework programme 

Final report: Annexes 

Daniela Rodriguez-Rincon, Carolina Feijao, Cagla Stevenson, Hamish 
Evans, Alexandra Sinclair, Simon Thomson and Susan Guthrie 

2022 

Directorate-General for Research and Innovation 

EN 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
PREFACE 

RAND  Europe  was  commissioned  by  Directorate-General  for  Research  and  Innovation 
(known by the acronym DG RTD) to conduct a comprehensive review of proposal evaluation 
processes  used  in  Horizon  2020  (H2020),  and  to  conduct  international  benchmarking  of 
these processes against accepted practice and the wider knowledge base in this area. This 
document  contains  the  details  of  the  methods  used,  the  data  used  and  produced  for 
reference and the specific analysis used to develop the report containing the main findings.   

This study incorporates bibliometric analysis conducted by Clarivate and expert advice from 
Prof. Jonathan Grant, Prof. Liv Langfeldt, Prof. Ana Marušić and Prof. Ulf Sandström. 

2 

 
 
 
CONTENTS 

FIGURES .......................................................................................................... 5 

TABLES ............................................................................................................ 9 

BOXES ............................................................................................................. 13 

ABBREVIATIONS AND ACRONYMS ....................................................................... 14 

CHAPTER 1: INTRODUCTION............................................................................... 17 

CHAPTER 2: APPROACH AND METHOD ................................................................. 18 

1  Approach to the study ................................................................................................. 18 
2  Approach and method by task ...................................................................................... 22 
3  Limitations of the approach .......................................................................................... 43 

CHAPTER 3: MAPPING OF THE PROCESSES OF PROPOSAL EVALUATION IN H2020 .... 46 

1  H2020 proposal evaluation process ............................................................................... 48 

CHAPTER 4: LITERATURE REVIEW ON THE CHARACTERISTICS OF GRANT PROPOSAL 
EVALUATION ..................................................................................................... 63 

1  The  grant  funding  process  is  burdensome  on  applicants,  reviewers  and  coordinating 
institutions ..................................................................................................................... 63 
2  The burden of the funding process is exacerbated by a low success rate ............................ 64 
3  R&I  funders  have  developed  strategies  in  attempts  to  reduce  the  burden  of  the  funding 
process .......................................................................................................................... 64 
4  Reducing the burden of the funding process may lead to more timely funding decisions ....... 65 
5  Increased training and support in  grant writing can lead to better outcomes and decrease 
the perception of burden .................................................................................................. 65 
6  Despite the important role of feedback as a means to improve the quality of research, there 
is little evidence on what constitutes ‘proper’ feedback in grant peer review ........................... 66 
7  Bias in the evaluation process occurs in a range of forms across the funding process, from 
programme design to funding allocation ............................................................................. 71 
8  Funders have developed initiatives aimed at increasing fairness and equality of opportunity 
in the proposal evaluation process ..................................................................................... 71 
9  Lack  of  clarity  in  the  definition  of  evaluation  criteria  decreases  reliability  of  the  funding 
process .......................................................................................................................... 72 
10 The use of group discussion can increase reliability of the peer review process ................... 73 
11 Peer review is described as ‘anti-innovation’ ................................................................... 74 
12 Evaluating multi- and interdisciplinary research and innovation presents specific challenges 
and requires adaptation of the process to ensure evaluation is adequate ................................ 75 
13 The ability of the peer review process to predict successful research projects (i.e. predictive 
power) is limited by the reviewers’ understanding of the evaluation criteria ............................ 77 
14 Transparency may be related to the design of the funding process, but this relationship has 
not been measured extensively ......................................................................................... 77 

CHAPTER 5: ANALYSIS OF THE H2020 PROPOSAL EVALUATION SYSTEM .................. 79 

1  Proposal scoring data .................................................................................................. 79 
2  Evaluator survey data .................................................................................................. 117 
2  Analysis of feedback letters provided to H2020 applicants ................................................ 167 
3  Transparency analysis ................................................................................................. 175 
4  Bibliometric analysis of predictive power and bias ........................................................... 181 

CHAPTER 6: CASE STUDIES ................................................................................ 194 

1  Canadian Foundation for Innovation – Canada ................................................................ 195 
2  Deutsche Forschungsgemeinschaft (German Research Foundation) ................................... 204 

3 

 
 
 
3  National Health and Medical Research Council – Australia ................................................. 211 
4  National Research Foundation – Singapore ..................................................................... 217 
5  Narodowe Centrum Nauki (National Science Centre) – Poland .......................................... 226 
6  Swiss National Science Foundation23337United Kingdom Research and Innovation ............. 241 

REFERENCES .................................................................................................... 248 

ANNEX A. INTERVIEW PROTOCOL WITH EC STAFF ................................................. 258 

ANNEX B. SURVEYS TO ASSESS BURDEN OF THE PROCESS .................................... 260 

1  EU Proposal Evaluation Survey – Applicants ................................................................... 260 
2  EU Proposal Evaluation Survey – Evaluators ................................................................... 272 

ANNEX C. INTERVIEW PROTOCOL FOR CASE STUDY DEVELOPMENT ........................ 288 

ANNEX D. INTERVIEWEE CODES ......................................................................... 291 

4 

 
 
 
 
FIGURES 

Figure 1: Conceptual framework .............................................................................19 

Figure 2: Prisma flow diagram for the studies included in the review ...........................25 

Figure 3: Diagram of the evaluation process for MSCA Individual Fellowships. Source: 

European Commission, 2017 ...........................................................................50 

Figure 4: Diagram of the ERC’s two-step evaluation process. Source: RAND Europe .....60 

Figure  5:  Diagram  of  the  SME  instrument’s  two-step  evaluation  process.  Source: 

European Commission, n.d. .............................................................................62 

Figure 6: Summary of reviewer characteristics .........................................................80 

Figure 7: Latent class analysis of reviewer data ........................................................81 

Figure 8: Summary of applicant characteristics ........................................................82 

Figure 9: Latent class analysis of applicant data .......................................................84 

Figure 10: Median and range of scores by action and evaluation criterion ....................99 

Figure 11: Distribution of median scores per evaluation criterion and action .............. 108 

Figure 12: Number of reviewers per proposal across the H2020 programme .............. 111 

Figure 13: Distribution of the coefficient of variation across reviewer categories for the 

actions analysed .......................................................................................... 112 

Figure  14:  Demographic  characteristics  of  reviewers  responding  to  the  survey 

assessing 2016/2017 H2020 calls .................................................................. 118 

Figure 15: Distribution of nationalities of reviewers responding to the survey assessing 

2016/2017 H2020 calls. ................................................................................ 119 

Figure 16: Demographics (gender) for feedback provided as free text in the survey .... 120 

Figure 17: Demographics (age) for feedback provided as free text in the survey ........ 121 

Figure  18:  Demographics  (organisation)  for  feedback  provided  as  free  text  in  the 

survey ........................................................................................................ 123 

Figure 19: Demographics (previous experience) for feedback provided as free text in 

the survey .................................................................................................. 124 

Figure  20:  Latent  class  analysis  of  reviewers  responding  to  the  survey  assessing 

2016/2017 H2020 calls. ................................................................................ 125 

Figure 20: Survey answers on different aspects of the proposal evaluation process..... 126 

Figure  21:  Survey  answers  on  the  level  of  satisfaction  with  the  proposal  evaluation 

process overall ............................................................................................ 128 

Figure 22: NVIVO analysis of the number of mentions of issues related to the coded 

themes at each stage of the proposal evaluation process .................................. 129 

Figure 23: Survey answers on the level of satisfaction with the briefing sessions ........ 130 

5 

 
 
Figure 24: Survey answers on the level of satisfaction with consensus meetings ........ 131 

Figure 26: Survey answers on the level of satisfaction with hearings meetings ........... 132 

Figure 27: Survey answers on the level of satisfaction with panel meetings ............... 133 

Figure 28: Survey answers on the level of satisfaction with peer review .................... 134 

Figure 29: Survey answers on the level of satisfaction with issues related to conflict of 

interest ...................................................................................................... 135 

Figure 30: Survey answers on the level of satisfaction with logistics of the evaluation 

for evaluation carried out centrally in Brussels ................................................. 136 

Figure 31: NVIVO analysis of the number of mentions of issues related to the coded 

themes at each stage of the proposal evaluation process by reviewer gender ...... 137 

Figure 32: NVIVO analysis of the number of mentions of issues related to the coded 

themes at each stage of the proposal evaluation process by reviewer age ........... 138 

Figure 33: NVIVO analysis of the number of mentions of issues related to the coded 

themes at each stage of the proposal evaluation process by reviewer affiliation ... 139 

Figure 34: NVIVO analysis of the number of mentions of issues related to the coded 

themes at each stage of the proposal evaluation process by reviewer experience . 139 

Figure 35: Number of survey participants across the two stakeholder groups ............. 140 

Figure  36:  Number  of  survey  participants  across  the  two  stakeholder  groups,  by 

gender ....................................................................................................... 140 

Figure 37: Applicant survey participants by current country of work.......................... 141 

Figure 38: Evaluator survey participants by current country of work ......................... 142 

Figure 39: Burden across the application process ................................................... 143 

Figure 40: Median time spent on written proposal................................................... 144 

Figure 41: Median time spent on written proposal,  by gender, country of work, and 

previous experience of H2020. MS = Member State ......................................... 146 

Figure 42: Time spent on interview ....................................................................... 147 

Figure 43: Median time spent on interview, by gender, country of work, and previous 

H2020 experience. MS = Member State .......................................................... 149 

Figure 44: Applicant perception of burden ............................................................. 150 

Figure 45: Applicant perception of the proposal submission tool ............................... 151 

Figure 46: Applicant perception of the usefulness of feedback .................................. 152 

Figure 47: Burden across the proposal submission process, for evaluators ................. 153 

Figure 48: Median time spent on written proposal evaluation ................................... 154 

Figure 49: Median time spent on evaluation of written proposal, by evaluator gender, 

country of work, and prior experience of H2020. MS = Member State................. 156 

6 

 
 
Figure 50: Median time spent on consensus group discussion................................... 157 

Figure 51: Median time spent on consensus group discussion, by gender, by country 

of work, by prior H2020 experience. MS = Member State ................................. 159 

Figure 52: Median time spent on panel review ....................................................... 160 

Figure 53: Median time spent on panel review, by gender, by country, by prior H2020 
experience. MS = Member State ........................................................................... 162 

Figure 54: Median time spent on interview ............................................................ 163 

Figure 55: Median time spent on interview, by gender, by country of work, by H2020 
experience. MS = Member State ........................................................................... 165 

Figure 56: Evaluator perception of burden across tasks ........................................... 166 

Figure 57: H2020 Transparency Framework analysis on access to information ........... 177 

Figure 58: H2020 Transparency Framework analysis on timing to information ............ 177 

Figure 59: Transparency of H2020 compared with other funding agencies with regard 
to access to information ...................................................................................... 179 

Figure 60: Transparency of H2020 compared with other funding agencies with regard 
to timing of the information ................................................................................. 180 

Figure  61:  Number  of  publications  per  year  pre-  and  post-award  for  successful  and 
unsuccessful applicants ....................................................................................... 182 

Figure  62:  Category-normalised  citation  impact  pre-  and  post-award  for  successful 
and unsuccessful applicants ................................................................................. 183 

Figure 63: Percentage of highly cited papers pre- and post-award for successful and 
unsuccessful applicants ....................................................................................... 184 

Figure 64: Relationship between applicant’s category-normalised citation impact after 
application and the score given to their application ................................................. 185 

Figure 65: Average number of papers per year by gender, prior to application ........... 186 

Figure 66: Average category-normalised citation impact by gender, prior to application
 ........................................................................................................................ 187 

Figure  67:  Average  percentage  of  highly  cited  publications  by  gender,  prior  to 
application ......................................................................................................... 187 

Figure 68: Average number of papers per year for successful applicants by gender .... 187 

Figure 69: Average number of papers per year for unsuccessful applicants by gender . 188 

Figure 70: Average category-normalised citation impact for successful applicants by 
gender .............................................................................................................. 188 

Figure 71: Average category-normalised citation impact for unsuccessful applicants by 
gender .............................................................................................................. 189 

Figure 72: Average% of highly cited publications for successful applicants by gender .. 190 

7 

 
 
Figure 73: Average% of highly cited publications for unsuccessful applicants by gender
 ........................................................................................................................ 190 

Figure 74: CFI proposal review process ................................................................. 196 

Figure 75: Overview of the merit review process for the CFI Innovation Fund ............ 197 

Figure 76: CFI rating scale for expert committees ................................................... 198 

Figure 77: CFI Innovation Fund objectives and assessment criteria ........................... 200 

Figure 78: Call timeline for the CFI Innovation Fund 2020 ....................................... 201 

Figure 79: DFG funding portfolio .......................................................................... 204 

Figure 80: DFG research funding by programme for 2019 (in million EUR and%) ........ 205 

Figure  81:  Funding  rates  in  the  individual  DFG  grant  programmes  by  scientific 
discipline, 2017–2019 ......................................................................................... 205 

Figure 82: DFG decision-making process ............................................................... 206 

Figure 83: Importance of selection criteria for DFG reviewers ................................... 207 

Figure 84: Framework for track record assessment. Source: Kelso, 2019 .................. 215 

Figure 85: Proposal evaluation process for the Central Gap Fund .............................. 222 

Figure 86: Overview of the first-stage proposal evaluation process of the NCN ........... 228 

Figure 87: Overview of the second-stage proposal evaluation process of the NCN ....... 229 

Figure 88: Decision making in the Postdoc.Mobility pilot .......................................... 237 

Figure 89: UKRI Councils’ Investment by Foundation in 2019/2020 .......................... 242 

8 

 
 
TABLES 

Table 1: Topics addressed by the research tasks ......................................................20 

Table 2: Search strategy for the rapid evidence assessment ......................................23 

Table 3: Inclusion and exclusion criteria for the literature review for the rapid evidence 
assessment .........................................................................................................24 

Table 4: Sub-tasks for Task 2 ................................................................................27 

Table 5: Overview of scoping interviews ..................................................................28 

Table  6:  Framework  for  assessing  transparency  of  funders  (developed  by  David 
Gurwitz et al., 2014) .............................................................................................29 

Table 7: Actions and respective pillars from the available data ...................................32 

Table 8: Indicative metrics used in analysis of H2020 proposal data ...........................33 

Table 9: Data fields used in the analysis ..................................................................34 

Table 10: Analytical approaches used for the quantitative analysis of evaluator survey 
data ....................................................................................................................35 

Table  11:  Applications  identified  from  the  H2020  data  set  for  inclusion  in  the 
bibliometric analysis .............................................................................................37 

Table 12: Proportion of sample of  selected awards that was  matched  in the Web  of 
Science database ..................................................................................................39 

Table 13: Funding agencies selected for case studies ................................................41 

Table 14: Number of interviews conducted for the different case studies .....................42 

Table 15: Main actions of H2020 (adapted from the interim evaluation of H2020) ........47 

Table  16:  Award  criteria  for  the  different  types  of  action  under  H2020  (work 
programme 2018–2020) .......................................................................................51 

Table 17: Summary of the evaluation guidance for reviewers in H2020 .......................58 

Table 18: Guidance for a good peer review ..............................................................68 

Table 19: Class membership and item response probabilities for reviewers ..................81 

Table 20: Summary of applicant characteristics ........................................................83 

Table 21: Class membership and item response probabilities for applicants .................84 

Table 22: Success rate by action ............................................................................85 

Table 23: Success rate by applicant gender (mono-beneficiary actions only) ................86 

Table 24: Success rate by applicant nationality (mono-beneficiary grants only) ............87 

Table 25: Success rate by applicant affiliation (mono-beneficiary only) .......................88 

Table 26: Outcome of mono-beneficiary grants by applicant and reviewer gender ........89 

9 

 
 
Table 27: Outcome of mono-beneficiary grants by applicant and reviewer nationality ...90 

Table 28: Outcome of mono-beneficiary grants by applicant and reviewer organisation
 ..........................................................................................................................91 

Table 29: Median scores and IQR by evaluation criteria and by applicant gender ..........92 

Table 30: Median scores and IQR by evaluation criteria and applicant nationality .........93 

Table 31: Median scores and IQR by evaluation criteria and applicant affiliation ...........95 

Table  32:  Significance  testing  of  differences  between  median  scores  for  applicant 
gender, nationality and organisation, paired by evaluation criteria ..............................97 

Table 33: Median scores across H2020 ....................................................................98 

Table 34: Comparison of demographic characteristics of applicants in proposals with 
low,  medium  and  high  coefficient  of  variation  (percentages  calculated  across 
demographics) ................................................................................................... 101 

Table  35:  Significance  of  differences  in  proportions  for  applicant  demographic 
characteristics with a high, medium and low coefficient of variation .......................... 103 

Table 36: Comparison of demographic characteristics of reviewers in proposals with 
low,  medium  and  high  coefficient  of  variation  (percentages  calculated  across 
demographics) ................................................................................................... 104 

Table  37:  Significance  of  differences  in  proportions  for  reviewer  demographic 
characteristics, between proposals with a high, medium and low coefficient of variation
 ........................................................................................................................ 105 

Table  38:  Comparison  of  characteristics  of  proposals  with  low,  medium  and  high 
coefficient  of  variation  (percentages  for  actions  calculated  across  coefficients  of 
variability) ......................................................................................................... 106 

Table  39:  Significance  of  differences  in  proportions  of  proposal  characteristics, 
between proposals with a high, medium and low coefficient of variation .................... 106 

Table 40: Characteristics of generous reviewers (scoring mean equal to or above 4) 
and less generous reviewers (scoring mean equal to or below 1) .............................. 110 

Table  41:  Significance  of  differences  in  proportions  for  generous  and  less  generous 
reviewers .......................................................................................................... 111 

Table 42: Simple logistic regression of evaluation outcome as a function of number of 
reviewers per proposal and coefficient of variation between reviewer mean scores per 
proposal for ERC actions, ..................................................................................... 115 

Table 43: Simple logistic regression of evaluation outcome as a function of number of 
reviewers per proposal and coefficient of variation between reviewer mean scores per 
proposal for SME actions ..................................................................................... 115 

Table 44: Simple logistic regression of evaluation outcome as a function of number of 
reviewers per proposal and coefficient of variation between reviewer mean scores per 
proposal for CSA actions ...................................................................................... 116 

Table 45: Simple logistic regression of evaluation outcome as a function of number of 
reviewers per proposal and coefficient of variation between reviewer mean scores per 
proposal for MSCA actions ................................................................................... 116 

10 

 
 
Table 46: Simple logistic regression of evaluation outcome as a function of number of 
reviewers per proposal and coefficient of variation between reviewer mean scores per 
proposal for RIA actions ...................................................................................... 117 

Table 47: Simple logistic regression of evaluation outcome as a function of number of 
reviewers per proposal and coefficient of variation between reviewer mean scores per 
proposal for IA actions ........................................................................................ 117 

Table  48:  Number  of  evaluation  summary  reports  reviewed  per  programme  and 
outcome ............................................................................................................ 167 

Table 49: Number of reviewers ............................................................................ 167 

Table 50: Page length of the evaluation summary reports ....................................... 168 

Table 51: Whether at least one reviewer summarises the proposal ........................... 169 

Table 52: RAG rating for alignment of scores and written assessment ....................... 169 

Table 53: RAG rating for structure of review around assessment criteria ................... 170 

Table 54: Whether the feedback includes critical appraisal of the proposal ................. 171 

Table 55: RAG rating for provision of examples by reviewers to illustrate feedback ..... 172 

Table 56: RAG rating for comment on grammar/language by reviewer ...................... 173 

Table 57: Positivity of language used by reviewer ................................................... 173 

Table 58: Constructiveness of reviews................................................................... 174 

Table 59: Themes related to identified weaknesses and areas for improvement.......... 174 

Table 60: Overall performance of successful and unsuccessful applicants post-award .. 181 

Table 61: Number of publications per year before and after application ..................... 182 

Table 62: Category-normalised citation impact before and after application ............... 183 

Table 63: Percentage of highly cited publications before and after application ............ 184 

Table  64:  Correlation  and  significance  of  the  relationship  between  applicants’ 
category-normalised  citation  impact  after  application  and  the  score  give  to  their 
application ......................................................................................................... 185 

Table 65: Summary of performance indicators pre-application by gender .................. 186 

Table 66: Change in number of papers published per year after application ............... 187 

Table 67: Mean change in average category-normalised citation impact post-award by 
gender .............................................................................................................. 188 

Table 68: Mean change in average percentage of highly cited publications post-award 
by gender .......................................................................................................... 190 

Table 69: Summary of performance metrics pre-application by country group ............ 191 

Table  70:  Mean  change  in  average  number  of  papers  published  per  year  post-
application by country group ................................................................................ 192 

11 

 
 
Table  71:  Mean  change  in  average  category-normalised  citation  impact  post-
application by country group ................................................................................ 192 

Table 72: Mean change in average percentage of highly cited papers post-application 
by country group ................................................................................................ 193 

Table 73: Funding agencies selected for case studies .............................................. 194 

Table 74: Assessment criteria for the CFI Innovation Fund ...................................... 199 

Table 75: Overview of the NHMRC’s restructured grant programme .......................... 212 

Table 76: Overview of THE NHMRC proposal evaluation processes by grant type ........ 213 

Table 77: Funding and grants available from the NRF .............................................. 218 

Table 78: Examples of impact metrics and tracking indicators provided by the NRF .... 220 

Table 79: Evaluation criteria for Central Gap Fund proposals .................................... 223 

Table 80: NCN domestic funding schemes ............................................................. 227 

Table 81: Assessment criteria and weighting for three domestic NCN funding calls ..... 230 

Table 82: Overview of proposal evaluation processes by SNSF grant type .................. 235 

Table 83: Interviewee codes ................................................................................ 291 

12 

 
 
 
 
 
 
 
 
 
 
BOXES 

Box 1: Data extraction criteria ...............................................................................26 

Box 2: Exceptions to the standard H2020 proposal evaluation process ........................61 

Box  3:  Key  findings  from  the  literature  review  on  international  practice  in  proposal 

evaluation .....................................................................................................63 

Box 4: Good practice in the evaluation of MIDRI (adapted from McLeish and Strang, 

2016) ...........................................................................................................76 

13 

 
 
 
 
ABBREVIATIONS AND ACRONYMS 

CFI 

CI 

CIHR   

CNCI   

CR 

CRC 

CRP 

CSA 

Canadian Foundation for Innovation 

confidence interval 

Canadian Institutes of Health Research 

category-normalised citation impact 

consensus report 

Collaborative Research Centres 

Competitive Research Programme 

Coordination and Support Actions 

DG RTD 

Directorate-General for Research and Innovation 

DFG 

Deutsche Forschungsgemeinschaft (German Research Foundation) 

EASME  

Executive Agency for SMEs 

EIC 

EJP 

ERA 

European Innovation Council 

European Joint Programme 

European Research Executive Agency 

ERA-NET 

European Research Area Net 

ERC 

European Research Council 

ERCEA  

European Research Council Executive Agency 

ESR 

evaluation summary report 

ESRC   

Economic and Social Research Council 

EU 

European Union 

FET Open 

Future and Emerging Technologies Open 

FP 

FPA  

FTI 

framework programme 

Framework Partnership Agreement 

Fast Track to Innovation 

FWF 
                               Science Fund) 

Fonds zur Förderung der wissenschaftlichen Forschung (Austrian     

GDPR   

H2020  

General Data Protection Regulation 

Horizon 2020 

H2020-EEN-SGA 

Horizon 2020 Enterprise Europe Network- Specific Grant Agreement  

14 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
I&E 

IA  

ICO 

ID 

IDR 

IEO 

IER 

Innovative and Enterprising  

Innovation Actions 

Information Commissioner’s Office 

identifier 

interdisciplinary research  

Innovation and Enterprise Office 

individual evaluation report 

INEA 

Innovation and Networks Executive Agency 

IP 

IPR 

IQR 

intellectual property 

intellectual property rights 

interquartile range 

ISCF 

Industrial Strategy Challenge Fund 

IT 

LCA 

LEIT 

MAC 

information technology 

latent class analysis 

leadership in enabling and industrial technologies 

Multidisciplinary Assessment Committee 

MIDRI  

multi- and interdisciplinary research and innovation 

MREA   

Medical Research Endowment Account 

MSCA   

Marie Skłodowska-Curie Actions 

MSCA IF  

Marie Skłodowska-Curie Actions Individual Fellowships 

NCCR   

National Centres of Competence in Research 

NCN 

Narodowe Centrum Nauki (National Science Centre) 

NHMRC 

National Health and Medical Research Council 

NIH 

NRF 

NRP 

National Institutes of Health 

National Research Foundation [Singapore] 

National Research Programme 

NSFC   

National Natural Science Foundation of China 

OR 

PCP  

odds ratio 

Pre-commercial Procurement 

15 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
PI  

POC 

PPI 

R&I 

RAG 

RIA  

principal investigator 

Proof of Concept 

Public Procurement of Innovative Solutions 

research and innovation 

red, amber, green 

Research and Innovation Actions 

SME  

small to medium-sized enterprise 

SNSF    

Swiss National Science Foundation (in German: Schweizerischer  
Nationalfonds zur Förderung der wissenschaftlichen Forschung) 

TRL 

technology readiness level 

UKRI 

United Kingdom Research and Innovation 

16 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
CHAPTER 1: INTRODUCTION 

The  purpose  of  this  study  is  to  provide  a  comprehensive  review  of  proposal  evaluation 
processes in European Union (EU) Research & Innovation (R&I) framework programmes to 
inform the development and, where required, adaptation of the approach taken to proposal 
evaluation  in  Horizon  Europe.  This  work  is  timely  because  Horizon  Europe  is  due  to  be 
launched in 2021. We have been commissioned by DG RTD to conduct this work, which 
consists of a comprehensive review of the processes used in Horizon 2020 (H2020), and 
international  benchmarking  of  these  processes  against  accepted  practice  and  the  wider 
knowledge base in this area. The research questions for this work are as follows: 

1.  How relevant is the current proposal evaluation system to achieve the objectives of 

Horizon 2020? 

2.  How efficient is the current proposal evaluation system? 

3.  How effective is the current proposal evaluation system? 

4.  What are the lessons learnt and areas for improvement for the future? What needs 

to be adapted to accommodate the novel features of Horizon Europe? 

The main findings of our analysis are provided in the accompanying final report document. 
This document serves as an annex to that report and sets out in detail the different tasks 
conducted  which  underpin  that  analysis.  with  the  following  chapters  set  out  our  overall 
approach and methods across tasks (Chapter 2), and then summarise the findings of each 
task as follows: 

  mapping the processes of proposal evaluation in H2020 (Chapter 3), 

 

literature review (Chapter 4), 

  analysis of the Horizon 2020 proposal evaluation system (Chapter 5), including the 

following: 

o  analysis of proposal scoring data 

o  analysis of evaluator survey data 

o 

reviewers and applicants surveys on burden 

o 

transparency analysis 

o  bibliometric analysis 

o  analysis of feedback letters 

 

 

case studies, 

references. 

Finally, some additional materials are provided in annexes A–C: the interview protocol used 
in  interviews  with  European  Commission  staff,  the  full  text  of  the  burden  survey,  and 
proposal scoring detailed data tables. 

17 

 
 
CHAPTER 2: APPROACH AND METHOD 

1 

Approach to the study 

1.1  Aims and conceptualisation 

The  purpose  of  this  study  is  to  provide  a  comprehensive  review  of  proposal  evaluation 
processes in EU R&I framework programmes to inform the development and adaptation, 
where  required,  of  the  approach  taken  to  proposal  evaluation  in  Horizon  Europe.  The 
research questions for this work are the following: 

1.  How relevant1 is the current proposal evaluation system to achieve the objectives 

of H2020? 

2.  How efficient2 is the current proposal evaluation system? 

3.  How effective3 is the current proposal evaluation system? 

4.  What are the lessons learnt and areas for improvement for the future? What needs 

to be adapted to accommodate the novel features of Horizon Europe? 

Combining the research questions with our existing knowledge of the evidence around the 
effectiveness  and  efficiency  of  proposal  evaluation  systems,  we  developed  an  initial 
conceptual framework for the study (Figure 1). This was intended to evolve over the course 
of the study, and we provide an updated version of this framework, reflecting our work so 
far, in Chapter 6. 

1 We consider relevance of the proposal evaluation system to be the extent to which it is efficient 
and  effective  and  thus  allows  Horizon  2020  to  deliver  on  its  objectives  –  i.e.  is  it  an  appropriate 
system for the needs of a modern R&I funder of the scale and scope of the framework programmes? 

2 We consider efficiency of the proposal evaluation system as the burden of the process on different 
individuals involved in submitting and evaluating proposals. 

3 We consider the effectiveness of the proposal evaluation system as the extent to which it delivers 
a good outcome, taking into account some of the known critiques of peer review–based processes – 
does it deliver a fair, consistent, appropriate assessment of the quality of applications? 

18 

 
 
 
                                                 
Figure 1: Conceptual framework 

To implement the study, we operationalised our conceptual framework across four primary 
tasks.  Our  overall  approach  delivers  cross-cutting  insights  spanning  the  key  study 
questions  as  set  out  in  Table  1,  which  characterises  the  key  aims  and  outputs  of  the 
different  tasks  and  sub-tasks,  and  their  alignment  with  different  study  questions  and 
requirements.

19 

 
 
 
 
in 

Setting 
context 
Horizon Europe 

the 
of 

Table 1: Topics addressed by the research tasks 

Task and method 

Key questions addressed 

Task  1:  Literature 
review 

What is the existing evidence regarding the characteristics of 
proposal evaluation systems that influence their effectiveness 
and  burden?  What  novel  approaches  have  been  used  to 
address these issues, and do they work? 

Task  2:  Review  of 
H2020  process  – 
Analysis  of  scoring 
data 

Is the proposal evaluation process fair and replicable? How do 
scores  and  outcomes  vary  by  personal  and  institutional 
characteristics  of  applicants  (if  at  all)?  How  do  scores  and 
outcomes vary depending on the reviewer (if at all)? How do 
these intersect?  

Task  2:  Review  of 
H2020  process  – 
of 
Analysis 
reviewers 
survey 
data 

Task  2:  Review  of 
H2020  process  – 
Bibliometric analysis 

What are the perceptions of evaluators on the effectiveness 
of  different  elements  of  the  proposal  evaluation  processes? 
What are areas of concern, strengths and weaknesses? 

Is  the  proposal  evaluation  process  effective  in  predicting 
outcomes? 

Task 2: Survey of a 
sample of applicants 
and reviewers 

What is the burden of the proposal evaluation process,  and 
how is it distributed across groups and across stages of the 
process and different actions? Is the burden proportionate to 
the outcomes? 

Relevance of H2020 proposal evaluation 
process 

in 
international 

Benchmarking 
the 
context 

Efficiency 

Effectiveness 

X 

X 

X 

X 

X 

X 

20 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Task  2:  Review  of 
H2020  process  – 
Analysis 
of 
documentation 

How  transparent  is  the  proposal  evaluation  process  across 
schemes?  How  timely  is  the  proposal  evaluation  process 
across schemes? 

Task  2:  Interviews 
with 
European 
Commission staff 

What is the rationale for the design of the H2020 evaluation 
process, and how does it align with the aims of H2020 and 
specific actions? 

Task 3: Case studies  How  do  other  funders  structure  their  proposal  evaluation 
processes? What is the rationale for their approach, and what 
measures,  if  any,  have  they  taken  to  address  effectiveness 
and  burden?  What  evidence,  if  any,  is  there  of  the 
effectiveness of these approaches? How transparent are their 
evaluation systems? 

X 

X 

X 

X 

Task 4: Analysis and 
lessons learnt 

Across tasks, what are the lessons learnt about the proposal 
evaluation  process,  and  how  can  these  inform  Horizon 
Europe? 

X 

X 

X 

X 

X 

X 

X 

21 

 
 
 
 
 
 
 
 
 
 
 
 
 
2 

Approach and method by task 

This section sets out the details of the methodology used for this study. 

2.1  Task 1: Literature review 

We undertook a rapid evidence assessment building on previous work from three studies 
conducted  by  RAND  Europe  that  focused  on  grant  peer  review  in  the  health  sciences 
(Guthrie,  Ghiga,  and  Wooding,  2018;  Guthrie,  Rodriguez-Rincon,  et  al.,  2019a)  and  an 
ongoing  study  on  peer  review  for  Wellcome  (Guthrie,  forthcoming).  A  rapid  evidence 
assessment is a systematic review of the available literature on a topic. However, unlike a 
full systematic review, the scope and coverage of the review is restricted through search 
criteria, which are refined throughout the process to ensure that the review focuses on the 
most relevant literature and that the amount of literature to review is manageable within 
the scope of the work (Varker et al., 2015). 

As the EU’s R&I framework programme encompasses multiple disciplines, not just health 
science, the rapid evidence assessment conducted in this study updated our previous work 
and extended it to cover all research areas, focusing on the efficiency and effectiveness of 
proposal evaluation systems. 

The rapid evidence assessment used a structured approach based around three stages: 1) 
conducting literature searches; 2) screening results; and 3) conducting data extraction and 
quality review. 

Stage 1: Literature searches 

Together  with  RAND’s  specialist  in-house  librarians,  we  developed  and  piloted  a 
comprehensive  search  strategy,  focusing  on  proposal  evaluation  systems  in  all 
geographical locations. As we have previously undertaken two robust literature reviews on 
this topic, we did not wish to duplicate that work and have therefore expanded the scope 
from  a  health  science  focus  to  R&I  more  broadly,  filling  any  gaps  that  may  exist  and 
updating with literature that has appeared since the previous rapid evidence assessment 
was  conducted.  We  conducted  the  search  shown  in  Table  2  using  Scopus  and  removed 
duplicates  from  the  articles  already  reviewed  for  our  previous  work,  reducing  the  scope 
and extent of literature to a feasible overall number of studies to be reviewed.

22 

 
 
Table 2: Search strategy for the rapid evidence assessment 

Group 

Search items 

1  

2 

3 

4 

5 

6 

7 

8 

TITLE-ABS-KEY  (“grant  review”  OR  “peer  review”  OR  “grant  program”  OR  “grant 
scheme”  OR  “funding  scheme”  OR  “funding  application”  OR  “grant  application”  OR 
“grant award” OR “fellowship funding” OR “fellowship award” OR “fellowship scheme” 
OR “proposal evaluation” OR “proposal assessment” OR “proposal review") 

(TITLE-ABS-KEY (innovat* OR creativ* OR novel* OR interdisciplin* OR inter-disciplin* 
OR conservative OR multi-disciplin* OR multidisciplin*) 

TITLE-ABS-KEY (bias OR equit* OR equality OR “equal opportunit*” OR fair*) 

TITLE-ABS-KEY (burden OR workload OR efficien*) 

TITLE-ABS-KEY (“career trajector*” OR “early career” OR “career stage” OR “post*doc*” 
OR post-doc*) 

TITLE-ABS-KEY (transparen* OR acceptab*) 

TITLE-ABS-KEY (consist* OR reliab* OR agree* OR replica* OR reproduc*) 

TITLE-ABS-KEY (timel* OR lag OR delay*)) 

Search strategy: GROUP 1 AND (GROUP 2 OR GROUP 3 OR GROUP 4 OR GROUP 5 OR GROUP 6 OR GROUP 
7 OR GROUP 8) 

Results on 20 October 2020: 6 471 

Results after removing duplicates: 5 294 

Stage 2: Screening 

In this stage, we screened articles for relevance against predefined inclusion and exclusion 
criteria  (Table  3)  and  selected  articles  for  inclusion  in  the  review  based  on  titles  and 
abstracts of identified studies. Two researchers conducted the screening, in three stages. 
First, criteria were applied on the titles. Those articles appearing to fit the criteria or where 
there was uncertainty were included. The abstracts of these titles were then read, and the 
inclusion  criteria  were  applied  a  second  time.  Again,  those  articles  fitting  the  criteria  or 
those where there was uncertainty were included. Third, reviewers retrieved full reports of 
studies that had passed the first round.

23 

 
 
 
 
 
Table 3: Inclusion and exclusion criteria for the literature review for the rapid evidence 
assessment 

Criterion 

Include 

Exclude 

Topic relevance 

Studies  on 
research  proposal 
evaluation  systems,  addressing 
efficiency, 
of 
elements 
effectiveness 
relevance. 
and 
Studies  including  the  strengths 
and  weaknesses  of  the  current 
processes, methods for measuring 
the  magnitude  of  these,  potential 
ways to mitigate weaknesses 

that 

Articles 
focus  on  wider 
proposal  evaluation  (i.e.  not  for 
for 
competitive  grant 
evaluate 
research) 
research  activity 
than 
proposals 

funding 

rather 

that 

or 

Geographical location 

No restriction 

Year of publication 

2010 onwards 

2009 or earlier  

Study characteristics 

empirical 

Academic  publications,  focusing 
or 
on 
syntheses. 
research 
articles or reviews 

evidence 

Primary 

or 

theoretical/conceptual 
Purely 
pieces. 
work 
Conference  papers  or 
letters. 
Comment  pieces  or  unstructured 
reviews. Grey literature 

opinion 

Language  

English 

Other languages 

Stage 3: Data extraction and analysis 

Following  screening,  we  obtained  106  studies  that  were  included  in  our  qualitative 
synthesis  (Figure  2).  The  final  selection  of  sources  for  extraction  was  based  on  a  gap 
analysis performed on our previous work. We mapped the existing analysis conducted for 
evidence on bias, burden, conservatism, inter- and multidisciplinary research, predictive 
power,  reliability,  timeliness  and  transparency  in  three  previous  RAND  Europe  studies 
(Guthrie,  Ghiga,  and  Wooding,  2018;  Guthrie,  Rodriguez-Rincon,  et  al.,  2019;  ongoing 
study on peer review led by Guthrie) with regard to characteristics, methods used for their 
measurement and approaches taken to address these factors in proposal evaluation. Our 
gap  analysis  revealed  that  more  information  was  needed  on  inter-  and  multidisciplinary 
research, innovation and feedback to applicants. 

We extracted information from each included source using a data-extraction template in 
Excel – a pre-determined coding framework based on the study aims to record data about 
each selected paper that fits within the inclusion criteria in a standard template. 

24 

 
 
 
 
 
Figure 2: Prisma flow diagram for the studies included in the review 

25 

 
 
 
 
 
 
 
 
 
 
 
 
Box 1: Data extraction criteria 

Researcher extracting data (initials) 
Source (author, date) 
Title 
Year of publication 

- 
- 
- 
- 
-  Objectives (aims of the work) 
-  Methods 
- 
- 
- 

Field of peer review (e.g. grant, journal, conference) 
Stage of the evaluation 
Evidence on factors relating to effectiveness and efficiency of the process: 
o  Burden (on researchers, institutions, funding bodies) 
o  Bias (gender, career stage, research type, institution, ethnicity) 
o  Transparency (which can be broken down into transparency of the application process 
itself and transparency of the review process (i.e. applicant feedback), trust in the 
process from the research community) 

Predictive power (the ability of the process to predict successful research) 

o  Conservatism (innovation) 
o  Reliability (consistency among reviewers) 
o 
o  Timeliness 
o  Multi- and interdisciplinary research and innovation (often shortened to MIDRI) 
o 
Changes and adaptations – ways in which proposal evaluation processes have been 
adapted to improve their relevance, efficiency and effectiveness – and any evaluation of 
these changes 
Additional notes and comments 

Feedback to participants 

- 

- 

2.2  Task 2: Analysis of Horizon 2020 proposal evaluation system 

Understanding the existing Horizon 2020 proposal evaluation system and its strengths and 
weaknesses  in  terms  of  effectiveness  and  efficiency  is  crucial  to  considering  any 
refinements or changes for future evaluation processes. Task 2 is broken down into four 
components, each of which adds an important facet to the evidence collected to inform the 
analysis  of  the  H2020  proposal  evaluation  system.  These  components  and  their  main 
anticipated contributions are summarised in Table 4. 

26 

 
 
 
 
Table 4: Sub-tasks for Task 2 

Sub-task 

Methods 

Main purpose 

A  Mapping  of 
proposal 
different funding instruments 

the  processes  of 
across 

evaluation 

  Document review 
  Interviews  with  European 

Commission staff 

To  collect  evidence  on  such 
aspects  as  the  processes  for 
proposal  evaluation,  analysis 
of  transparency  of  criteria, 
the  timeliness  of  processes, 
and  the  content and  focus of 
feedback 
evaluators 
and 
reports 

B 

Assessment of the effectiveness of 
the proposal evaluation system  

  Analysis  of  transparency 

and timeliness; 

  Analysis  of  evaluation 

summary reports; 

  Analysis  of  application 

data; 

  Analysis  of  the  evaluator 

survey data; 

  Interviews  with  European 

Commission staff; 
  Bibliometric analysis  

C 

Assessment  of  the  burden  of  the 
proposal evaluation system 

  Survey  of  applicants  and 
reviewers  on  the  burden 
of the process; 

  Interviews  with  European 

Commission staff  

system 

To 
the 
understand 
effectiveness  of  the  proposal 
evaluation 
with 
burden, 
regard 
fairness, 
transparency, 
timeliness, 
replicability, 
predictive 
and 
relevance 

power 

to 

with 

establish, 

To 
through 
bibliometric analysis, whether 
is 
the  application  process 
better 
associated 
research  performance 
for 
successful  applicants  relative 
to  unsuccessful  applicants, 
and thereby to gain an insight 
into  whether  the  process  is 
effective 
predicting 
in 
outcomes4. 

To  gather 
information  on 
reviewer 
and 
applicant 
perceptions  and  experiences 
of the burden of the proposal 
evaluation  process,  including 
quantifying  the  burden  on 
applicants  and  reviewers  and 
whether 
is 
proportionate 

burden 

the 

D 

Synthesis  and  compilation  of 
analysis report 

  Internal workshop 

and 

To  gather  the  main  evidence 
sources 
synthesise 
findings,  drawing  preliminary 
conclusions 
for  exploration 
based on the analysis, and to 
highlight  any  problems  or 
difficulties 

4 We acknowledge that receiving funding in itself may be a driver of better outcomes. However, as 
mentioned  in  Chapter  4,  evidence  from  the  literature  looking  at  other  funding  schemes  does  not 
indicate that this is the case. 

27 

 
 
 
                                                 
 
 
A:  Mapping  the  processes  of  proposal  evaluation  across  different  funding 
instruments 

In  order  to  map  the  proposal  evaluation  process  in  H2020,  we  conducted  a  review  of 
publicly available documentation and reports on the proposal evaluation process, as well 
as scoping interviews with Commission officials to better understand the nuances in the 
proposal evaluation system across the different actions under H2020. The interviews were 
used  to  collect  information  and  views  from  European  Commission  staff  about  their 
perceptions of the proposal evaluation systems in place for H2020, the rationale for the 
design of the process, and the burden involved from their perspective. Table 5 provides an 
overview of the interviews conducted in this mapping phase. The protocol used for these 
interviews can be found in Annex A. Findings from these interviews will also be used for 
the  assessment  of  the  effectiveness  of  the  system  (sub-task  B)  and  the  burden  of  the 
process (sub-task C). 

Table 5: Overview of scoping interviews 

Unit/agency 

Number of interviews 

Directorate-General for Research and Innovation (DG RTD) Unit B3 

Research Executive Agency 

European Research Council Executive Agency (ERCEA) 

Executive Agency for SMEs (small to medium-sized enterprises) (EASME) 

Innovation and Networks Executive Agency (INEA) Agency 

2 

2 

2 

1 

1 

In  order  to  draw  conclusions  on  potential  changes  for  Horizon  Europe  with  regard  to 
implementation of proposal evaluations from H2020, we will review documentation around 
Horizon  Europe,  with  a  particular  focus  on  its  structure,  aims  and  objectives  and  with 
reference to changes and adaptations in comparison to H2020. We will also rely on findings 
from the scoping interviews to better understand how the proposal evaluation system from 
H2020 can be improved for Horizon Europe. 

B: Assessment of the effectiveness of the proposal evaluation system 

Document review 

First,  we  conducted  an  analysis  of  transparency  by  reviewing  publicly  available 
documentation  on  H2020  funding  processes.  We  used  the  framework  developed  by 
Gurwitz,  Milanesi,  and  Koenig  (2014)  to  assess  the  transparency  of  a  range  of  funders. 
This framework characterises a range of ‘knowledge items’, specified in Table 6 across four 
‘knowledge  categories’.  Through  analysis  of  websites  and  documentation,  these  items 
categorise the following: 

 

 

information on the level of access: open, if publicly available; restricted, if available 
only to a defined group of people; and closed, if not available at all. 

information  on  the  timing  of  that  access:  providing  access  to  a  knowledge  item 
either  before  the  evaluation  procedure,  during  the  procedure,  or  after  the 
procedure. 

This framework has been used to assess the transparency of 27 major international public 
funders, which will serve as a comparison point for H2020 funding schemes regarding the 
level of transparency. 

28 

 
 
 
Table 6: Framework for assessing transparency of funders (developed by David 
Gurwitz et al., 2014) 

Knowledge item 

Knowledge category 

Description of evaluation procedure 

Evaluation details (questions to the reviewers) 

Names of applicants 

Names of funded principal investigators (PIs) 

Proposals submitted to call 

Proposals/Lay Abstracts of funded projects 

Rules 

Rules 

Content 

Content 

Content 

Content 

Names of reviewers (at minimum, in aggregated form) 

Expertise 

Review summaries of all proposals 

Review summaries of funded proposals 

Funding budget granted to projects 

Call success rate 

Ranking of proposals 

Expertise 

Expertise 

Produce 

Produce 

Produce 

Final report (at minimum, the final report abstract) 

Produce 

We also reviewed and compared templates of  evaluators’ reports in order to get insight 
into the consistency of evaluators’ understanding and use of key assessment criteria and 
into the practical operation of panel meetings and consensus meetings. The templates were 
provided by DG RTD. We assessed the guidance provided to reviewers, how this guidance 
differs by action and whether there was the potential to harmonise templates across the 
framework. 

Review of evaluation summary reports 

In order to assess the extent to which reviewers of H2020 proposals provide feedback that 
can be useful to applicants, both successful and unsuccessful, we reviewed 246 evaluation 
summary reports (ESRs) across six H2020 programmes – Research and Innovation Actions 
(RIA);  Innovation  Actions  (IA);  Marie  Skłodowska-Curie  Actions  (MSCA);  European 
Research Council (ERC) Proof of Concept; Coordination and  Support Actions (CSA); and 
H2020-EEN-SGA (Horizon 2020 Enterprise  Europe Network- Specific Grant Agreement)– 

29 

 
 
 
comprising 234 applications5. We aimed to obtain a sample of 50 ESRs per programme, 
with an equal mix of successful and unsuccessful proposals. 

To conduct the analysis, we developed an extraction template in Excel with the following 
entries. 

  Researcher: This entry was used to assign the different ESRs to the different research 
team member. Three research team members (HE, LH, MV) carried out the extraction 
independently. Consistency in extraction was checked by LH and confirmed by a fourth 
researcher (DRR). 

 

ID: This entry corresponded to the proposal identifier (ID) provided by DG RTD. Each 
application has its unique ID. 

  Programme: This entry captured the programme to which the application was made. 

  First  or  second  stage:  This  entry  noted  whether  the  ESR  corresponded  to  a  first-  or 

second-stage review, if relevant. 

  Above or below threshold: This entry was used to determine whether the score received 

was above or below the funding threshold. 

  Length of the review: This entry captured the length of the review in pages. 

  Score and assessment criteria: This entry captured the alignment between the score(s) 
provided by the reviewer and the written feedback, using a red, amber, green (RAG) 
rating system. Green indicates the score was well aligned with the written feedback, 
amber suggests some inconsistencies, while red indicates a large difference between 
the score and the feedback. For example, if a proposal received 4/5 and the reviewer’s 
feedback focused mainly  on the negative  aspects  of the proposal, the ESR would be 
designated as red in this field. Researchers were asked to provide a comments column 
for the basis and justification for their rating, if relevant. 

  Review structured around criteria: This entry captured the extent to which reviewers 
had followed the published criteria in writing their review. We again used a RAG rating, 
where  green  indicates  the  assessment  addresses  all  assessment  criteria,  amber 
suggests there were some inconsistencies in addressing the criteria, and red indicates 
that the review did not address the criteria in significant ways. For example, if reviewers 
were asked to comment on ‘clarity and pertinence of the objectives’, ‘soundness of the 
concept,  and  credibility  of  the  proposed  methodology’,  and  ‘quality  of  the  proposed 
coordination and/or  support measures’ and the reviewer gave feedback on all three, 
the  ESR  would  be  designated  green  in  this  field.  Researchers  were  able  to  provide 
comments to detail the rationale for their rating. 

  Do they summarise the proposal? This entry captured whether reviewers summarised 
the  proposal  in  their  review  to  show  an  understanding  of  the  proposed  research. 
Researchers assigned a ‘yes’ or ‘no’ response. 

  Critical appraisal: This entry captured a researchers’ subjective assessment on whether 
reviewers  provided  a  critical  appraisal  of  the  proposal  or  whether  the  review  was 
relatively generic and uninformative. Researchers assigned a ‘yes’ or ‘no’ response. 

  Review  provides  examples  that  illustrate  the  different  points:  Using  a  RAG  rating 
system,  in  this  field  researchers  captured  the  use  of  examples  by  the  reviewer  to 

5  The  number  of  applications  and  the  number  of  ESRs  reviewed  are  not  the  same  because  some 
programmes will have a two-stage evaluation process and therefore one application may have two 
ESRs.  

30 

 
 
                                                 
illustrate their feedback. Green indicates the reviewer used many useful examples to 
illustrate their point, amber suggests there was some (albeit limited) use of examples 
and  red  indicates  that  no  (or  a  very  small  number  of)  examples  were  provided. 
Researchers were able to provide comments to detail the rationale for their rating. 

  Grammar  or  language  usage:  Using  a  RAG  rating  system,  researchers  assessed 
whether the reviewer commented on the grammar or language used in the proposal. 
Green indicates the reviewer did not comment on the applicant’s grammar/language at 
all,  amber  suggests  there  were  some  references  to  language/grammar  and  red 
indicates that language/grammar was  commented on to a great extent. Researchers 
were able to provide comments to detail the rationale for their rating. 

  Narrative review or bullet points: This entry captured whether the review was written 

in bullet points or in narrative format. 

  Weaknesses of the review: This column provided a space for researchers to comment 

on their assessment of the weaknesses of the review as a whole. 

  Potential  improvements:  This  column  provided  a  space  for  researchers  to  suggest 

improvements for the review as a whole. 

  Constructive,  unconstructive,  neutral:  This  entry  captured  whether  the  review  was 
constructive, neutral or unconstructive. For example, if a reviewer provided concrete 
examples  of  how  to  improve  the  weaknesses  identified,  this  would  be  considered 
‘constructive’,  and,  conversely,  if  a  reviewer  criticised  a  proposal  in  vague,  general 
terms with no examples, this might be considered ‘unconstructive’. 

  Language: Here, researchers assessed whether the reviewer used positive, negative or 

neutral language in their review. 

Analysis of scoring data 

Data  were  provided  by  DG  RTD  on  applications  submitted  across  several  actions  within 
H2020, in the time period 2014–2020. The data contained information on applicant and 
reviewer characteristics, as well as the outcomes of proposals and scores by action – ERC, 
Marie Skłodowska-Curie Actions (MSCA), IA, RIA, CSA, SME instrument, Pre-commercial 
Procurement  (PCP),  Public  Procurement  of  Innovative  Solutions  (PPI),  EJP  Cofund 
(European  Joint  Programme  Cofund).  Prior  to  the  analysis,  we  conducted  a  data  quality 
assessment to determine the utility of each variable based on sample variability and case-
level and variable-level missing data. We found differences in the proposal numbers in the 
data held for the overall programme applicants and the proposals assessed by actions. In 
these  instances,  it  was  not  possible  to  reconcile  proposals  across  the  data  sets.  To 
streamline  the  analysis,  the  actions  were  grouped  according  to  the  categories  shown  in 
Table  7.  Analysis  was  performed  with  the  R  statistical  software  (R  Core  Team,  2020), 
making use of its large-scale data processing capacity and extensive range of data analysis 
and visualisation tools. 

31 

 
 
 
 
 
Table 7: Actions and respective pillars from the available data6 

Action category 

Actions included 

ERC 

MSCA 

IA 

RIA 

CSA 

ERC-POC,  ERC-LVG,  ERC-COG,  ERC-ADG,  ERC-STG, 
ERC-SyG, ERC-POC-LS 

MSCA 
COFUND, MSCA IF 

ITN,  MSCA-RISE,  MSCA-NIGHT,  MSCA-

IA, IA-LS, Shift2Rail-IA-LS, EuroHPC-IA 

RIA, RIA-LS, Shift2Rail-RIA-LS, EuroHPC-RIA 

CSA, CSA-LSP, EuroHPC-CSA 

SME instrument 

SME-1, SME-2, SME-2b 

Other 

PCP, PPI, EJP Cofund 

6 ERC-POC: European Research Council-Proof of Concept; ERC-LVG, European Research Council-Low 
Value  Grant;  ERC-COG:  European  Research  Council-Consolidator  Grants;  ERC-ADG:  European 
Research  Council-Advanced  Grants;  ERC-STG:  European  Research  Council-Starting  Grants;  ERC-
SyG: European Research Council-Synergy Grants; ERC-POC-LS: European Research Council-Proof of 
Concept Lump Sum; MSCA ITN: Marie Skłodowska-Curie Innovative Training Network; MSCA-RISE: 
Marie Skłodowska-Curie-Research and Innovation Staff Exchange Evaluations; MSCA-NIGHT: Marie 
Skłodowska-Curie-European  Researchers'  Night;  MSCA-COFUND:  Marie  Skłodowska-Curie-Co-
funding of regional, national and international programmes; MSCA IF Marie Curie: Marie Skłodowska-
Curie  Individual  Fellowships;  IA:  Innovation  actions;  IA-LS:  Innovation  Lump  Sum  actions; 
Shift2Rail-IA-LS:  Innovation  Action  Lump-Sum;  EuroHPC-IA:  European  High  Performance 
Computing-Innovation  Actions;  RIA:  Research  and  Innovation  Actions;  RIA-LS:  Research  and 
Innovation  Lump  Sum  Actions;  Shift2Rail-RIA-LS:  Research  and  Innovation  Lump  Sum  Actions; 
EuroHPC-RIA:  European  High  Performance  Computing-  Research  and  Innovation  Actions;  CSA: 
Coordination and Support Actions; CSA-LSP: Coordination and Support Actions- Lump Sum Pilots; 
EuroHPC-CSA: European High Performance Computing- Coordination and Support Actions; SME-1: 
Small  and  Medium-sized  Enterprises-1;  SME-2:  Small  and  Medium-sized  Enterprises-2;  SME-2b: 
Small  and  Medium-sized  Enterprises-2b;  PCP:  Pre-Commercial  Procurement;  PPI:  Public 
Procurement of Innovative solutions; EJP Cofund: European Joint Programme Cofund. 

32 

 
 
                                                 
 
Table 8 indicates the metrics we calculated based on these data. We analysed each metric 
across the data set as a whole, and by H2020 pillar. 

Table 8: Indicative metrics used in analysis of H2020 proposal data 

Characteristic 

Fairness 

Replicability 

Metrics 

 
 
 

Success rate by characteristic 
Scores by assessment criterion by characteristic 
Proportion  of  reviewers  and  applicants  by 
characteristic 

  Outcome for different applicant characteristics by 

reviewer characteristics 

  Range of reviewer median and average scores 
  Characteristics  of  more  and  less  ‘generous’ 
reviewers,  i.e.  those  who  typically  give  higher 
scores (by institution type, age and gender) 
  Relationship  between  score  by  criterion  and 

funding decision 

Conservatism and interdisciplinarity 

  Median  and  range  of  difference  in  score  of 

applications between criteria 

We conducted an in-depth statistical analysis of the scoring data in order to analyse the 
fairness, replicability and conservatism of funding processes in H2020 – as a whole, and 
by action. Specifically, variables were processed to derive the data fields outlined in Table 
9.  We  analysed  the  metrics  described  in  Table  8  at  univariable,  bivariable,  and 
multivariable levels as appropriate. Descriptive univariable analyses included applicant and 
reviewer characteristics, range of reviewer scores, and difference in score of applications 
between  criteria.  Bivariable  analyses  included  outcomes  by  applicant  and  reviewer 
characteristics.  We  also  used  multivariable  approaches  to  provide  a  more  in-depth 
investigation of the data set in terms of the simultaneous impact of multiple factors at the 
applicant,  application,  and  reviewer  levels  on  funding  outcomes.  Latent  class  analysis 
(LCA),  a  statistical  technique  for  identifying  subgroups  based  on  a  set  of  participant 
characteristics (Lanza and Rhoades, 2013; Lanza, Coffman, and Xu, 2013), was also used 
to describe different profiles of applicants and reviewers. 

33 

 
 
 
 
Table 9: Data fields used in the analysis 

Data fields for each applicant/team 

Proposal number 

 
  Gender 
  Nationality 
 

Institution type 

Data fields for each application 

Proposal number 
Action 
Score on each evaluation criterion 
Total score 

 
 
 
 
  Reviewer ID 
 

Funded or not funded 

Data fields for each reviewer 

Proposal number 

 
  Gender 
  Country of origin 
 
 

Age 
Institution type 

Reviewer survey 

The reviewers survey provides feedback from individuals involved in the H2020 proposal 
evaluation  process  on  their  views  on  the  effectiveness  of  different  elements  of  the 
evaluation process. This includes both quantitative and qualitative information from free-
text questions, which were assessed using the approaches outlined below. 

DG  RTD  provided  data  on  the  feedback  from  reviewers  involved  in  the  H2020  proposal 
evaluation process regarding their views on the effectiveness of different elements of the 
process. The data consisted of responses from two surveys addressing H2020 calls from 
2014/2015  and  2016/2017.  The  2014/2015  survey  contains  the  answers  of  44 
respondents, whereas the 2016/2017 survey contains the answers of 3 602 respondents. 
To ensure an appropriate sample size, we focused our analysis on the 2016/2017 survey. 

For the quantitative analysis, we performed the data analysis with the R statistical software 
(R Core Team, 2020). For the qualitative analysis, we used NVIVO software to conduct the 
iterative categorisation of the qualitative textual data. 

Quantitative analysis 

We analysed the quantitative data in the first instance (using the R statistical software) to 
assess the demographics of the reviewers responding to the survey and identify areas of 
dissatisfaction in the process. 

The  survey  provided  information  on  the  H2020  pillars/priorities  (Excellent  Science, 
Industrial  Leadership,  Societal  Challenges),  cross-cutting  work  programmes  and  focus 
areas.  To  streamline  the  analysis,  we  grouped  the  data  according  to  this  segmentation. 
Additionally, given the substantial number of issues covered in the survey, we grouped the 

34 

 
 
 
 
 
responses into categories that reflect the key components of the H2020 proposal evaluation 
process.  These  categories  are  a)  general  aspects  of  proposal  evaluation  processes;  b) 
briefing session; c) consensus meetings; d) hearings meetings; e) panel meetings; f) peer 
review;  and  g)  logistics.  To  obtain  an  overview  of  the  quantitative  data  and  identify 
potential differences between H2020 pillars, we used the approaches outlined in Table 10. 
The  methods  proposed  allowed  for  the  identification  of  areas  of  dissatisfaction  in  the 
process,  which  were  used  for  further  qualitative  analysis  (described  in  the  section  titled 
‘Qualitative analysis’, below). For ordinal data, the Likert-scale categories were coded into 
numerical values to facilitate the visualisation and qualitative analysis. However, we note 
that the resulting numerical categories do not correspond to a true numerical scale, as the 
interval ranges are not clearly defined. 

Additionally, we explored the application of LCA to enable incorporation of evaluator profile 
information  into  the  analyses.  As  described  in  Section  2.2.2,  LCA  allows  for  the 
identification of ’profiles’ (latent classes) from a set of individual characteristics (recorded 
as categorical data). Each latent class is composed of individuals with a similar pattern of 
association  between  their  characteristics,  and  the  resulting  set  of  profiles  can  be 
incorporated in the qualitative analysis proposed in section below. 

The overall data analysis and cleaning process was conducted in the R statistical software. 

Table 10: Analytical approaches used for the quantitative analysis of evaluator survey 
data 

Category 

Type of data 

Approach/display 

Demographic  characteristics 
of evaluators 

Nominal and ordinal   Descriptive statistics 

Bar graphs to visualise individual demographics 

Latent class analysis 
Graphs  and  tables  depicting  the  latent  classes 
identified and probabilities of class membership 

Nominal  

Descriptive statistics 
Bar graphs to visualise data 

Ordinal (scale) 

Descriptive statistics 
Box plots to visualise the distribution of the data 

General  aspects  of  proposal 
evaluation processes 
Briefing session 
Consensus meetings 
Hearings meetings 
Panel meetings 
Peer review 
Logistics 

Qualitative analysis 

For  areas  of  the  proposal  evaluation  process  with  higher  levels  of  dissatisfaction,  as 
indicated by the previous quantitative analysis, we used a maximum variation sampling 
approach (Palinkas et al., 2015) to identify the relevant qualitative responses and analyse 
a sample of those dissatisfied – and those more satisfied – to explore what the emerging 
issues and the causes of any dissatisfaction might be. Our sample consisted of the free-
text answers associated with the 25% lowest- and the 25% highest-scoring of quantitative 
answers.  Our  analysis  identified  that  the  vast  majority  of  the  free-text  answers  were 
associated  with  the  lowest-scoring  quantitative  answers.  For  this  reason,  our  analysis 
focused on the negative feedback provided by reviewers. 

The free-text answers were analysed in NVIVO using an  Iterative Categorization (Neale, 
2016) approach, whereby codes were developed to  represent the key themes emerging 
from the textual answers. In a first stage, this methodology consisted of developing codes 

35 

 
 
 
 
 
 
 
according to the key themes emerging from the textual data. The codes provided a means 
of ordering and summarising the key points in the data. In a second stage of the analysis, 
we used the coded data to identify patterns in the evaluators’ feedback and explore the 
meaning of any potential findings within the context of the broader evaluation. 

This analysis gave us an overall impression of the perceptions of the evaluators regarding 
the strengths and weaknesses of the  existing processes and areas  of concern, including 
detailed  insights  into  these  at  a  specific  action  level  and  also  depending  on  the 
characteristics and experiences of the evaluator. 

Bibliometric analysis 

The aim of this task was to analyse the predictive power of funding decisions by looking at 
whether  proposal  evaluation  processes  are  effective  in  selecting  the  ‘best’  applicants  in 
terms  of  their  subsequent  bibliometric  performance.  The  analysis  also  addresses  bias, 
analysing  differences  in  bibliometric  performance  levels  before  award  for  success  and 
analysing bibliometric performance post-award for different groups by gender and country. 

To do this, we identified a sample of 5 004 successful and unsuccessful funding applications 
from the Marie-Skłodowska-Curie Actions and ERC made between 2014 and 2016 from the 
data set provided by DG RTD. These awards were selected because the analysis required 
us to focus on mono-beneficiary funding schemes. We selected awards from 2014–2016 
to allow some time for post-award publications to occur and for these to be cited (although 
this  window  is  still  relatively  short).  As  noted,  each  application  was  made  by  a  single 
researcher, and the data set included the following:  

 

 

researcher  information:  applicant  title,  first  name,  last  name,  gender  (when 
available) and email address, 

location  information:  country,  host  institution,  department  and  job  title  at  that 
institution, 

  grant  information:  unique  proposal  number,  year  application  was  made, 

programme applied to and if application was successful or unsuccessful. 

Table 11 provides a summary of the sample of applications analysed. The data set contains 
5 004 applications, with slightly more successful applications (51.4%) than unsuccessful 
applications  (48.6%).  The  large  majority  of  applications  were  to  the  Marie-Skłodowska-
Curie  Action  (82.4%,  4 122  applications),  half  of  which  were  successful.  A  larger 
percentage  of  the  882  ERC  applications  were  unsuccessful  (57.9%).  Around  80%  of 
applicants  have  gender  data  associated  with  them,  of  which  a  quarter  are  female. 
Applications by male researchers are slightly more successful, representing just over half 
(51.4%) of accepted applications, compared with 46.4% accepted applications by female 
researchers

36 

 
 
 
Table 11: Applications identified from the H2020 data set for inclusion in the 
bibliometric analysis 

All applications 

Successful applications  Unsuccessful 

applications 

5 004 

- 

2 432 

48.6% 

2 572 

51.4% 

4 122 

82.4% 

2 061 

50.0% 

2 061 

50.0% 

882 

17.6% 

371 

42.1% 

511 

57.9% 

All 
applications  

Funding 
Stream 

Marie-
Skłodowska-
Curie Actions 

European 
Research 
Council 

Gender 

Male 

2 989 

59.7% 

1 536 

51.4% 

1 453 

48.6% 

Female 

1 024 

20.5% 

476 

46.5% 

548 

53.5% 

Unknown 

991 

19.8% 

420 

42.4% 

571 

57.6% 

We looked to match these awards in the Web of Science database in order to conduct our 
analysis. Identifying collections of publications by the same author is non-trivial; different 
authors  may  share  the  same  name,  and  different  publications  may  present  the  same 
author’s name in different formats. To mitigate this, we use a machine learning algorithm 
that groups publications that are likely to be by the same author into clusters. We created 
a data set of applicant publications by identifying clusters of publications authored by the 
5 004 applicants between 2010 and 2019. This relies on applicant information provided by 
DG RTD. 

Three  different  approaches  were  used  sequentially  to  match  awards  in  the  database;  if 
applicant publications could not be identified by the first approach, the second approach 
was tried and then the third approach. Approaches 1 and 2 were least likely to return false 
positives (clusters that did not belong to the applicant, which were manually removed from 
the data set). 

1.  Approach  1:  Clusters  containing  publications  that  listed  the  applicant’s  email 
address. Email addresses tend to be more unique than applicants’ names and are 
linked to institutions as well as researcher. 

2.  Approach 2: Clusters containing publications that list the applicant’s given name, 
family name, and country exactly as they appear in the data set provided by RAND. 
We could not filter by institution, as institution names in the RAND-provided data 
were inconsistent with how institutions names appear in the Web of Science. 

3.  Approach 3:  Clusters determined by a Web of Science  author search  using  given 
name  and  family  name.  This  search  is  more  flexible,  allowing  for  name  variants, 
but  can  find  too  many  clusters  by  multiple  researchers  with  the  same  or  similar 
name. Manual review was required to ensure that only the appropriate clusters were 
selected. 

Clusters were manually reviewed to ensure that the research areas and author affiliations 
were consistent with the applicant’s department, host institution and country. If analysts 

37 

 
 
 
 
 
  
were  still  unsure,  applicant  profiles  on  LinkedIn  and  institution  websites  were  checked. 
Ideally one cluster would capture the entire scholarly output of a single researcher, but in 
this study, a researcher’s publications appeared in an average of 1.8 clusters. 

Analysts identified clusters of publications (published in all years) for 3 911 out of 5 004 
applicants – that is, for 78% of the applicants. We could not find publications for around 
one in five applicants, for one of the following reasons: 

  An applicant’s name was too common to identify a cluster. 

  Clusters  that  contained  an  applicant’s  publications  also  contained  publications  by 

different researchers with the same or similar name. 

  Applicants published in research areas (e.g. some humanities subjects) or produced 

research outputs (e.g. books) with limited Web of Science coverage. 

  Applications  were  made  in  the  name  of  an  administrator  rather  than  an  active 

researcher. 

The initial data set of applicants’ publications includes one or more publication clusters for 
3 911 out of 5 004 applicants. This contains publications of all document types published 
in any year. For the analysis, we applied filters and only included publications that met the 
following criteria: 

  Published between 2010 and 2019, 

  With  document  types  classified  as  articles  or  reviews  (collectively  referred  to  as 

papers; see the section titled ‘Definitions’ for more information). 

Applying filters resulted in a data set of papers authored by 3 815 applicants; fewer than 
100 (2%) applicants were removed. The final statistics are presented in Table 12. We found 
papers for a slightly higher percentage of successful applications (79.8%) compared with 
unsuccessful applications (72.9%). This was also the case when comparing successful with 
unsuccessful applications from male and female applicants. Coverage is weakest for ERC 
applications, with papers found for just over one fifth of applicants. Papers were found for 
a similar percentage of male and female applicants.

38 

 
 
Table 12: Proportion of sample of selected awards that was matched in the Web of 
Science database 

% of applications with 
papers 

% of successful 
applications with 
papers 

% of unsuccessful 
applications with 
papers 

All applications 

Marie-Skłodowska-
Curie Actions 

76.2% 

88.0% 

79.8% 

90.5% 

72.9% 

85.5% 

Research 

21.1% 

19.9% 

21.9% 

European 
Council 

Male 

Female 

Unknown 

EU Member State 

Other Member State 

EU-associated 
country7 

Widening  participation 
group Member State 

88.6% 

85.4% 

29.6% 

76.0% 

76.3% 

79.1% 

73.2% 

91.1% 

88.0% 

29.0% 

79.8% 

80.1% 

85.9% 

83.2% 

29.9% 

72.3% 

72.2% 

79.1% 

79.1% 

74.1% 

72.9% 

For each applicant, we assessed research publications between 2000 and 2019 in the years 
before and after applications were made, using the following indicators. 

  Publication output is indicated by the average number of papers published per year. 

  Citation impact is indicated by two citation-based indicators: 

  Category-normalised  citation  impact  (CNCI).  This  is  the  number  of  citations 
received by an applicant’s papers normalised to the average citation count for all 
papers published globally in the same year and Journal Subject Categories. As CNCI 
is normalised to global averages as the world average therefore always equals 1.00, 
the performance of papers in different fields can be directly compared. 

  Percentage or proportion of an applicant’s papers that are classified as highly cited. 
In this report, a paper is considered highly cited if it is among the top 10% most 
highly cited papers published in the same year and Journal Subject Categories. 

7 EU-associated countries for H2020 are Iceland, Norway, Albania, Bosnia and Herzegovina, North 
Macedonia,  Montenegro,  Serbia,  Turkey,  Israel,  Moldova,  Switzerland,  Faroe  Islands,  Ukraine, 
Tunisia, Georgia and Armenia. 

39 

 
 
 
 
 
                                                 
 
Where  statistical  significance  has  been  reported,  this  has  been  assessed  at  the  95% 
confidence level. 

C: Assessment of the burden of the proposal evaluation process 

Survey of a sample of applicants and reviewers 

In order to assess the burden of the evaluation process, we conducted a survey of a sample 
of applicants and reviewers. The aim of this analysis was to understand the level of burden 
placed on different stakeholders by the current proposal evaluation process, and the extent 
to  which  different  elements  of  the  process  are  contributing  to  that  burden  for  different 
funding instruments within the Horizon 2020 programme. 

The  survey  was  initially  piloted  internally  with  researchers  at  RAND  and  externally  with 
members from DG RTD, and the survey was revised after feedback. The survey was then 
sent  to  applicants  and  reviewers  across  the  varying  funding  schemes  under  H2020  to 
ensure a representative sample across the different schemes. The survey was targeted at 
applicants and reviewers, aiming to achieve 1 000 respondents for each stakeholder group 
and each action. As we know that different actions have a different number of applications 
(e.g. MSCA receive more than 10 000 applications), we targeted applicants and reviewers 
from either the past year (in cases where applications exceed 3 000) or the past 2 years 
(in cases where applications are below 3 000). In order to reduce the impact of people not 
being able to recall time spent on past activities, we limited the sample to people who had 
participated in the process within the past 2 years. 

The survey was conducted using the online platform SmartSurvey. The survey was open 
from  5  November  2020  to  3  December  2020.  Between  the  launch  and  the  close  of  the 
survey, we sent three follow-up emails to survey respondents who had not yet completed 
the  survey,  to  encourage  participation.  Full  details  on  the  surveys,  including  their 
development, conduct and a copy of the survey instruments, can be found in Annex B. 

2.3  Task 3: Case studies 

We  conducted  seven  in-depth  international  case  studies  on  proposal  evaluation  for  R&I 
funding. The aim of the case studies was to compare H2020 with other proposal submission 
and  evaluation  processes,  to  identify  best  practice  internationally  and  to  assist  in 
identifying lessons learnt that may be applicable for Horizon Europe. The different tasks 
conducted to develop the case studies are described below. 

Case study selection 

In  consultation  with  DG  RTD,  we  selected  seven  case  studies  that  highlighted  different 
aspects of proposal evaluation in different contexts, to include funders varying in terms of 
amount  of  annual  available  funding  and  focus  discipline  (for  example,  biomedical, 
innovation, engineering or physical sciences) (Table 13). 

40 

 
 
Table 13: Funding agencies selected for case studies 

Funding agency (country) 

Annual budget (EUR) 

Discipline 

Canadian Foundation for Innovation 
(CFI) (Canada) 

250 million  

Scientific R&I 

Deutsche  Forschungsgemeinschaft  
(German 
Foundation 
(DFG)) (Germany) 

Research 

3.2 billion  

National  Health 
Research 
(Australia)  

Council 

and  Medical 
(NHMRC) 

540 million  

National Research Foundation (NRF) 
(Singapore) 

3.2 billion 

Narodowe  Centrum  Nauki  (National 
Science Centre (NCN)) (Poland)  

- 

Science 
across 
research 
disciplines  (humanities  and  social 
sciences,  life  sciences,  natural 
sciences and engineering)  

Fundamental 
clinical 
science,  health  services,  public 
health.  

science, 

Scientific research 

Basic research  

Swiss  National  Science  Foundation 
(SNSF) (Switzerland) 

988.4 million 

All disciplines 

United  Kingdom  Research  and 
Innovation (UKRI) (UK) 

8.5 billion 

Research  disciplines 
arts, 
biosciences, physical sciences) 

(including 
economics, 

humanities, 

Preliminary  desk  research  and  document  review  of  publicly  available  material  on  the 
proposal submission and evaluation process 

We  began  by  reviewing  publicly  available  documentation  and  reports  on  the  selected 
funding bodies to obtain a good understanding of the purpose and objectives, and we then 
set out each body’s proposal evaluation process. For each case study, our aim was collect 
sufficient information to map the evaluation process. 

Our initial approach to identifying relevant publicly available documentation was to start 
from the organisation’s own website, followed by targeted searching using keywords and 
a  ‘snowballing’  strategy  to  identify  additional  articles  from  reference  lists  of  selected 
articles.  We  looked  at  documentation  around  the  proposal  evaluation  processes  for  that 
funder, as well as for any evaluations or assessments of their processes. We also identified 
information  on  the  rationale  for  the  structure  of  the  evaluation  process  in  place,  any 
revisions made to processes and the rationale for these, and any performance measures 
collected around their proposal evaluation processes (e.g. on success rate, diversity and 
inclusion). 

Interviews 

We conducted semi-structured interviews that allowed us to tailor our questions according 
to our preliminary research into the main characteristics of the proposal evaluation system 
and the depth of knowledge we have been able to obtain from publicly available documents. 
We  aimed  to  conduct  at  least  one  interview  per  case  study,  with  the  exception  of  the 

41 

 
 
 
 
National  Health  and  Medical  Research  Council  (NHMRC)8.  However,  we  were  unable  to 
secure an interview with a representative from either the NRF or the DFG. We sent up to 
three follow-up emails to the organisations that did not respond. We conducted a total of 
six  interviews  (Table  14),  with  each  interview  being  around  1  hour  in  length  and  being 
conducted  by  telephone  or  videoconference.  The  anonymised  interview  codes  for 
interviews can be found in Annex D.  

Table 14: Number of interviews conducted for the different case studies 

Funding agency (country) 

Number of interviews  

Canadian Foundation for Innovation 

German Research Foundation  

National Health and Medical Research Council (Australia) 

National Research Foundation (Singapore) 

National Science Centre (Poland) 

Swiss National Science Foundation  

United Kingdom Research and Innovation  

3 

0 

0 

0 

1 

1 

1 

The interviews focused on the following themes: 

  Ensuring an understanding of the funding organisation, including different funding 
programmes within the organisation; the size, focus and number of funding rounds; 
and any constraints in its activities, 

  Exploration  of  the  detailed  processes  for  proposal  evaluation,  or,  where  full 
information  on  this  was  already  available  from  documentation,  confirmation  that 
our process mapping is correct, 

  An understanding of the rationale behind the chosen system, and the interviewee’s 

perspective of the strengths and weaknesses, 

8 Although no interviews were conducted for this case study, we have recently conducted an in-depth 
study  developing  a  framework  for  NHMRC  to  assess  the  effectiveness  of  their  updated  grant 
processes, with a particular focus on proposal evaluation processes (Guthrie, Rodriguez-Rincon, et 
al. 2019). Therefore, we were able to draw on a wealth of material from that study, including a series 
of workshops and consultations with their staff. This work was completed in 2019 and reflects the 
latest developments at NHMRC. 

42 

 
 
 
                                                 
  How efficiency and effectiveness are defined by the organisations, and other criteria 

important in designing the system, 

  Key indicators of the effectiveness of the proposal evaluation system (for example, 
How does the funding organisation assure itself that the funding decisions are fair 
and  replicable?  How  transparent  are  the  evaluation  criteria  and  processes?  Does 
the  organisation  collect  any  information  on  the  extent  to  which  the  process  is 
effective in predicting the outcomes of the research?), 

  Key indicators relating to the burden of the system, including any evidence collected 
on  burden  for  applicants,  for  the  funding  organisation  and  for  other  researchers 
involved, if peer review was used, 

  Review of processes and metrics used to assess the effectiveness and burden of the 

proposal evaluation system and its application, 

 

Information  on  any  recent  changes  to  the  proposal  evaluation  system  and  any 
assessment of their effect. 

The interview protocol was developed by the study team and agreed upon with DG RTD 
prior to the interviews. The interview protocol for case studies can be found in Annex C. 

2.4  Analysis of the case studies 

We conducted a comparative analysis of the different case studies with the aim of drawing 
out  international  good  practice  in  different  aspects  of  proposal  evaluation  and  to 
benchmark different features of the H2020 proposal evaluation process against the case 
study  examples,  thereby  indicating  some  possible  lessons  learnt  for  the  future  Horizon 
Europe  programme.  We  used  a  framework  analysis  approach  based  around  the  study 
conceptual framework (Figure 1). Evidence and examples of novel and promising practice 
from  each  case  study  were  mapped  against  the  conceptual  framework  and  the  overall 
evidence collected in each area was synthesised to provide an overview of key observations 
and learning with respect to each aspect of the framework. These observations were then 
reviewed in the context of H2020 and Horizon Europe to draw out potential lessons for the 
framework programmes. 

3 

Limitations of the approach 

We  identify  a  number  of  potential  limitations  to  the  study  approach  in  relation  to  the 
analyses conducted so far. 

Literature  review:  We  conducted  a  rapid  evidence  assessment.  This  approach  limits 
search terms and inclusion criteria to ensure the most relevant literature is identified and 
to enable the review to be conducted in an appropriate timeframe. This means that, relative 
to a systematic review, some relevant literature may have been missed. However, because 
we  were  building  on  several  prior  reviews  and  a  comprehensive  evidence  base,  we  are 
fairly confident that most of the state of the art in the field has been captured. Searches 
were  conducted  in  one  database,  Scopus.  This  is  a  comprehensive,  cross-disciplinary 
database.  However,  all  databases  have  some  limitations.  Scopus  is  largely  focused  on 
academic  literature,  so  there  may  be  some  grey  literature  that  has  been  missed  in  the 
search. Literature was subject to narrative synthesis by one member of the study team. 
With all narrative syntheses, there is a risk that the author may bring unconscious biases 
into  the  analysis  and  synthesis  of  literature.  This  bias  is  minimised  by  having  the  work 
conducted by an experienced reviewer. 

Burden survey: Collecting suitable retrospective data on the time burden associated with 
tasks is challenging because survey respondents may not recall the specific amount of time 
spent on detailed tasks. By pooling responses and analysing the median response across 
task and stakeholder group, we aimed to minimise effects of respondent’s inaccurate recall. 

43 

 
 
We  should  also  note  that  we  were  engaged  in  data  collecting  during  the  COVID-19 
pandemic. Our survey asked participants to reflect on activities covering the past 2 years, 
and this time is likely to be impacted by changes due to restrictions from COVID-19. This 
likely had a greater impact on some aspects of the proposal evaluation process than others, 
for  example  impacting  travel  times  to  interviews,  which  no  longer  occurred  in  person. 
Feedback  from  survey  respondents  also  highlighted  that  they  were  sometimes  unsure 
whether they had attended particular stages (for example, consensus group discussion), 
and  therefore  there  maybe  estimates  associated  with  some  tasks  that  are  not  wholly 
accurate. 

Analysis of  application data: Analysis is limited by the  extent and nature of the data 
available. For example, we are unable to look at fairness by ethnicity since these data are 
not available. For parts of the analysis, we have used LCA. This is a useful technique as it 
allows  researchers  to  collapse  multiple  variables  to  facilitate  statistical  analysis  across 
clusters  of  individuals  with  similar  characteristics,  identifying  key  relationships  between 
variables  within  a  data  set  rather  than  looking  at  all  possible  variable  combinations. 
However, a limitation is that it can be difficult to understand which aspects are driving any 
trends and relationships seen. Therefore, the analysis is supplemented by analysis at the 
individual variable level. Data quality was good and completeness of data was high, so this 
did not significantly limit the analysis. Sizes of some groups within the population did affect 
analysis – for example, the number of applicants from some types of institution was low 
(e.g. only 84 applications from public bodies within the data set). For some parts of the 
analysis, groups were merged either to facilitate statistical testing where binary variables 
were needed or to create larger group sizes. Analysis is based on application data from 
2015. Processes may have changed since then, and some aspects of the analysis may in 
part be reflecting historic rather than present systems. 

Analysis of evaluator survey data: Analysis is based on an existing data set provided 
from  a  survey  conducted  in  relation  to  evaluators  participating  in  H2020  in  2016/2017. 
These data are a few years out of date, and processes may have changed in the intervening 
period.  The  survey  provides  a  picture  of  the  views,  perceptions  and  experiences  of 
evaluators.  There  may  therefore  be  differences  of  opinion  and  biases,  and  the  results 
should be considered as the views and perceptions of this group rather than the absolute 
truth. Views of others – for example, applicants – are not reflected in this data set. In our 
analysis, as well as analysing the quantitative aspects of the survey, we have coded the 
free-text responses for the most and least satisfied respondents. We note that very few 
text comments were left by the most satisfied respondents, so this analysis largely reflects 
the views of the 25% of the respondent population least satisfied with the process overall. 
Therefore,  although  this  qualitative  analysis  provides  useful  information  on  potential 
weaknesses and areas for improvement, it provides little information on what is working 
well and should be retained, and it also does not necessarily represent the views of the 
average evaluator. 

Transparency analysis: The analysis conducted is based on publicly available information 
from the H2020 website as well as interviews conducted with European Commission staff. 
We have also generalised across different actions to provide evidence at a broader level. 
It  is  possible  that  there  are  differences  between  some  actions;  however,  we  find  that 
transparency  standards  are  broadly  consistent  across  H2020  as  a  whole,  so  there  are 
unlikely to be differences for a significant number of cases. 

Bibliometric  analysis:  A  key  limitation  of  the  bibliometric  analysis  is  that  it  was  only 
feasible to conduct it for mono-beneficiary programmes, due to the need to be able to track 
the bibliometric output of the named applicant. Additionally, it would not be appropriate 
for actions that do not target academic publications as a key outcome of the funding, since 
it would be very limited as an outcomes measure in these cases. Therefore, it was only 
possible to conduct the bibliometric analysis for ERC and MSCA applicants. As such, the 
analysis  provides  a  partial  picture  of  the  predictive  power  of  H2020  proposal  evaluation 
processes.  Even  within  those  schemes,  it  should  be  noted  that  academic  output  is  an 
important  aspect  of  the  aims  of  the  funding,  but  it  is  not  the  only  one,  so  even  there 

44 

 
 
academic  output  does  not  provide  a  complete  view  of  the  output  or  achievements  of 
successful and unsuccessful applicants. Finally, it is worth noting the wider limitations of 
bibliometric analyses. For example, coverage of bibliometric databases is more complete 
for some disciplines than others (although analysis is normalised by field of research, which 
should  ensure  that  those  publications  available  are  treated  equitably),  and  coverage  is 
higher for English-language publications. 

ESR review: Although the analysis aimed to capture descriptive information regarding the 
ESR,  inevitably  some  value  judgements  had  to  be  made.  This  was  particularly  true  in 
regard to the ‘potential improvements’ and ‘weaknesses of the review’, where researchers 
had to make a mostly subjective judgement. In addition to this, as different researchers 
carried out the ESR analysis, different judgements could have been made. The subjectivity 
of  this  analysis  was  limited  by  consistent  briefing  of  researchers  beforehand,  a  high 
proportion  of  columns  containing  drop-down  options,  and  an  independent  researcher  to 
confirm  consistency  in  extraction  across  all  three  extraction  researchers.  In  addition, 
although we sampled almost 234 ESRs, we only covered proposals submitted to H2020-
EEN-SGA,  no  Framework  Partnership  Agreements  (FPA)  or  EJP  Cofund  proposals. 
Therefore, the ESR analysis is not fully representative of the entire H2020 programme. 

Case studies: We aimed to select a range of organisations for the case studies that would 
cover  a  variety  of  geographies  as  well  as  disciplines  and  that  would  draw  out  learnings 
relevant to different aspects of the framework programme. We also aimed to identify some 
examples  where  novel  processes  are  in  place  or  where  changes  have  been  made  in 
response  to  challenges  in  the  proposal  evaluation  systems.  However,  this  selection  will 
have  limitations,  and  there  may  be  useful  and  novel  examples  we  have  not  identified. 
Equally, a different set of case studies might have identified a different set of observations 
and learning for Horizon Europe. We also note the challenge in identifying true comparators 
for  the  framework  programmes,  in  terms  of  both  scale  and  diversity.  Therefore,  these 
comparators may each be more relevant to specific aspects of H2020 and Horizon Europe. 
Despite multiple attempts to secure interviews, we were unable to get interviews for two 
of  the  case  studies.  In  addition,  with  the  exception  of  the  Canadian  Institutes  of  Health 
Research (CIHR), the remaining case studies were informed by a single interviewee. This 
means  that  much  of  the  evidence  draws  on  the  documentation  that  is  available  on  the 
processes.  Although  we  have  found  that  in  most  cases  documentation  is  relatively 
comprehensive  and  has  given  us  a  good  picture  of  existing  processes  and  how  they 
emerged, we also note that some of the lessons around challenges in implementing those 
processes or the wider contextual issues to be taken into account may be absent where 
interviews  could  not  be  conducted.  Additionally,  where  only  one  interview  has  been 
conducted,  there  may  be  limitations  in  the  perspectives  offered.  We  also  note  that 
interviewees are from within the organisations in question and therefore provide only one 
perspective on the effectiveness of the proposal evaluation processes in place. To address 
this  lack  of  diversity  in  perspectives,  we  have  also  looked  for  wider  commentary  and 
evaluative evidence on the quality of the processes and the challenges in the literature and 
included this evidence where available to give an alternative perspective. Finally, the set 
of case studies highlight a range of ideas and approaches to address potential challenges 
in proposal evaluation processes – as well as highlighting areas of challenge that have been 
identified and are still a work in progress. Although the case studies highlight examples of 
practice  and  their  rationale,  in  many  cases  the  approaches  implemented  have  not 
necessarily  been  evaluated  and  found  successful.  Therefore,  it  is  difficult  to  conclude 
whether  these  actions  constitute  ‘best’  practice.  Rather,  they  provide  some  useful 
reflections  on  the  ways  in  which  other  funding  organisations  have  addressed  common 
challenges and their experiences of that process. 

45 

 
 
CHAPTER 3: MAPPING OF THE PROCESSES OF PROPOSAL 

EVALUATION IN H2020 

Horizon 2020 is the eighth framework programme of the EU and is the main mechanism 
for  funding  of  research  and  innovation  in  Europe  from  2014  to  2020.  With  a  budget  of 
nearly EUR 77 billion, H2020 is the largest  EU research and innovation programme yet. 
The  primary  goals  of  H2020  are  to  ensure  that  Europe  produces  world-class  science,  to 
remove barriers to innovation, and to make it easier for the public and private sectors to 
work together. Along with these goals, H2020 also focuses on a number of different areas 
in terms of how it functions as a funding mechanism. These areas include the increased 
involvement  of SMEs, increased involvement and integration of  socio-economic sciences 
and humanities, widening participation such that excellent science is funded wherever it 
is, and increasing international cooperation.  One notable difference between  H2020 and 
previous EU framework programmes is that H2020 is meant to be simpler to access than 
previous funding programmes. 

Funding  through  H2020  is  set  out  in  a  number  of  multi-annual  work  programmes, 
organised under three  pillars, or priorities: Excellent Science,  Industrial Leadership, and 
Societal Challenges. 

  The  Excellent  Science  pillar  aims  to  reinforce  and  extend  the  excellence  of  the  EU’s 
science  base  and  to  consolidate  the  European  Research  Area  to  increase  the 
competitiveness of research and innovation in Europe. It includes funding for the ERC, 
which  funds  frontier  research  by  the  best  individual  teams  and  receives  the  largest 
portion of the funding for Excellent Science. This pillar also includes funding for Future 
and  Emerging  Technologies,  which  supports  collaborative  research  and  innovation; 
Marie Skłodowska-Curie Actions (MSCA), which support opportunities for training and 
career  development;  and  support 
including  e-
infrastructures. 

infrastructures, 

for  research 

  The Industrial Leadership pillar aims to speed up the development of technologies and 
innovations that will support businesses in Europe, and in particular SMEs. This pillar 
includes funding for Leadership in Enabling and Industrial Technologies (LEITs), which 
receives  the  biggest  portion  of  the  funding  under  the  Industrial  Leadership  pillar,  as 
well as access to risk finance and an SME-specific fund to support innovation in SMEs. 

  The Societal Challenges pillar aims to address major concerns in Europe and elsewhere, 
and  reflects  the  policy  priorities  of  the  European  2020  strategy.  Funding  under  this 
pillar is organised in challenges focused on Health, demographic change and well-being; 
Food  security,  sustainable  agriculture  and  forestry,  marine  and  maritime  and  inland 
water research, and the bioeconomy; Secure, clean and efficient energy; Smart, green 
and  integrated  transport;  Climate  action,  environment,  resource  efficiency  and  raw 
materials; Europe in a changing world; and Secure societies. 

Along with the work programmes set out under these three pillars, there are also cross-
cutting work programmes in H2020, including Science with and for society and Spreading 
excellence  and  widening  participation.  In  addition,  there  are  also  programmes  that  fall 
within H2020 but are outside of the scope of the three pillars: the European Institute for 
Innovation  and  Technology;  Euratom  (a  complementary  funding  programme  to  support 
nuclear research and training); and a pilot of the European Innovation Council (EIC), which 
will be rolled out on a wider scale during the next European framework programme. There 
are  also  a  number  of  focus  areas  with  increased  funding  through  2018–2020,  including 
Building  a  low-carbon,  climate  resilient  future;  Connecting  economic  and  environmental 
gains  through  the  Circular  Economy;  Digitising  and  transforming  European  industry  and 
services; and Boosting the effectiveness of the Security Union. 

Calls  under  H2020  represent  different  types  of  actions,  or  different  types  of  funding 
schemes. Depending on the type of action, the call will differ in terms of the scope of what 

46 

 
 
is funded, the reimbursement rate and the standard forms that are used, such as the lump 
sums provided for each cost. Additionally, the type of action also determines the specific 
criteria that will be used to evaluate proposals submitted under each call. The main types 
of  actions  are  Research  and  innovation  actions  (RIAs),  which  support  basic  and  applied 
research  to  establish  new  knowledge;  IAs,  which  support  plans  for  new,  altered  or 
improved  products;  and  CSAs,  which  assist  in  processes  supplemental  to  research  and 
innovation  activities,  such  as  standardisation  efforts,  awareness  raising,  dissemination, 
communication, networking and coordination. Table 15 shows the different actions under 
H2020, as well as a brief description of their objective. 

Table 15: Main actions of H2020 (adapted from the interim evaluation of H2020) 

Type of action 

Objective 

Research and Innovation Actions (RIAs) 

Innovation Actions (IAs) 

Fast Track to Innovation (IA) 

European Joint Programme Cofund (EJP Cofund) 

Comprise  activities  that  aim  to  establish  new 
knowledge and/or to explore the feasibility of a new 
or improved technology, product, process, service 
or solution 

Comprise activities that aim to produce plans and 
arrangements  or  designs  for  new,  altered  or 
improved products, processes or services. IAs are 
used  to  turn  existing  scientific  and  technological 
insights into applications 

Continuously open action calls aimed at addressing 
through 
technological  or  societal  challenges 
innovation 

Provide  support  to  coordinated  national  R&I 
programmes in implementing a joint programme of 
activities,  including  training,  dissemination  and 
financial support to third parties 

ERA-NET Cofund (Executive Research Agency Net 
Cofund) 

Provide support to public–public partnerships in the 
design,  implementation  and  coordination  of  joint 
activities, as well as establishing networks 

Pre-commercial Procurements (PCPs) 

Encourage  public  procurement  of 
research, 
development  and  validation  of  new  solutions  that 
can bring high-quality and  efficient improvements 
in areas of public interest 

Public Procurement of Innovative solutions (PPI) 

Enable  groups  of  procurers  to  share  the  risks  of 
acting as early adopters of innovative solutions 

Coordination and Support Actions (CSA) 

Marie Skłodowska-Curie Actions (MSCA) 

MSCA Individual Fellowships (MSCA IF) 

and 

Support standardisation, dissemination, awareness 
raising 
networking, 
coordination, support services,  learning exercises, 
dialogues  that  enable  the  development  of  new 
infrastructure 

communication, 

Provide grants for researchers at all career stages 
–  from  doctoral  candidates  to  highly  experienced 
researchers  –  and  encourage 
transnational, 
intersectoral and interdisciplinary mobility 

Individual  fellowships  that  support  mobility  of 
researchers within and beyond Europe,  as well as 
help attract the best foreign researchers to work in 
the EU  

47 

 
 
MSCA Innovative Training Networks (ITN) 

MSCA Research and Innovation Staff Exchanges 
(RISE) 

MSCA-COFUND 

ERC Frontier Research 

SME instrument Phase 1 (IA) 

Support  doctoral  programmes,  implemented  by 
partnerships  of  universities,  businesses  and  other 
research  performing  organisations  across  Europe 
and  broader;  partnerships  can  be  in  the  form  of 
European  Training  Networks,  European  Industrial 
Doctorates, or European Joint Doctorates 

international 

Support 
intersectoral 
collaboration  through  R&I  staff  exchanges,  and 
knowledge  sharing  from  research  to  market  and 
vice versa 

and 

to 

stimulate 

Aim 
and 
international doctoral or fellowship programmes to 
foster  excellence  in  researcher  training,  mobility 
and career development 

regional,  national 

Fund projects based on scientific excellence in any 
field of research, conducted by a research team led 
by a principal investigator 

study 

verifying 

feasibility 

idea/concept;  SME 

Cover 
the 
technological/practical and economic viability of an 
innovative 
in 
general supports all types of innovative SMEs with 
a  strong  ambition  to  grow  internationally  and 
provides staged support that covers the entirety of 
the innovation cycle in three phases 

instrument 

SME instrument Phase 2 (IA) 

Aimed at innovation projects that address a specific 
challenge and demonstrate high potential in terms 
of 
growth 
and 
competitiveness 
underpinned by a strategic business plan 

company 

1 

H2020 proposal evaluation process 

H2020 has issued more than 3 850 calls during its implementation9. Each call receives on 
average more than 1 500 proposals. 

The following two main elements shape the awarding process of H2020 funding. 

  Proposal  submission:  Applicants  submit  a  proposal  once  (single-stage)  or  twice 

(two-stage). 

  Proposal evaluation: Once submitted, proposals are evaluated in a single step or in 

multiple steps. 

The majority of  H2020  actions apply either a  single- or a two-stage  submission process 
and a single-step evaluation process. Two parts of the programme apply tailored, multi-
step  evaluation,  namely  the  ERC  and  the  EIC  pilot.  Each  programme  has  nuances 
associated with the proposal evaluation process, and in this study we seek to explore the 
similarities and differences of the different processes. 

9 According to data available on the Single Electronic Data Interchange Area for funding and tender 
opportunities for H2020, on 12 October 2020. 

48 

 
 
 
                                                 
Four  executive  agencies  are  responsible  of  the  framework  programme’s  proposal 
evaluation system. 

  The Research Executive Agency manages the logistics of the evaluation process 
and the evaluation experts for all services, in addition to the proposal evaluation of 
the  majority  of  H2020  actions  –  Marie  Skłodowska-Curie  Actions  (MSCA);  future 
and emerging technologies; space research; food security, sustainable agriculture 
and  forestry,  marine,  maritime  and  inland  water  research  and  the  bioeconomy; 
Innovative,  inclusive  and  reflective  societies;  Secure  societies;  Spreading 
excellence and widening participation; and Science with and for society). It is also 
in charge of the entire life cycle of the different programmes (not only the evaluation 
process). 

  The  European  Research  Council  Executive  Agency  manages  the  proposal 

evaluation and experts for the ERC part of the programme. 

  The Executive Agency for SMEs manages parts of Industrial Leadership, the EIC 
Accelerator (formerly the SME instrument), and Fast Track to Innovation (FTI), as 
well  as  Societal  Challenge  5  (Climate,  environment,  natural  resources  and  raw 
materials), and the energy-efficiency part of Societal Challenge 3 (Energy). 

  The Innovation and Networks Executive Agency manages proposal evaluation 
and experts for Societal Challenge 4 (Smart, green and integrated transport) and 
part of Societal Challenge 3 (Secure, clean and efficient energy). In addition, some 
calls are evaluated by the Commission services, with the help of external experts. 

1.1  Proposal application 

Applicants submit proposals to H2020 calls in two parts, Part A: Administrative form10 and 
Part B: Project proposal11. Part A includes such information as participant name, type of 
organisation, registration number, contact details, budget, and other administrative data. 
Part  B  consists  of  a  workplan  divided  into  five  sections,  namely  excellence,  impact, 
implementation, participants profile, ethics and security. 

Depending on the call description in the H2020 work programmes, applicants are invited 
to submit their proposals either in one stage (single-stage call) or in two stages (two-stage 
calls). Proposals are submitted according to the relevant proposal template. In the one-
stage  submission,  applicants  submit  the  full  proposal,  which  will  go  through  the  full 
evaluation process. In a two-stage evaluation, there are two deadlines per submission. In 
the first stage, applicants submit a short proposal for evaluation, consisting of two of the 
five sections of a full proposal, namely excellence and impact. Stage-one submissions are 
then  assessed  by  external  experts,  and  some  participants  are  invited  to  submit  a  full 
proposal for stage 2. The number of proposals admitted to stage 2 should be three times 
the  available  budget,  and  in  any  case,  not  less  than  two  and  a  half  times  the  available 
budget. Because of potential changes in the proposal between stages, experts once again 
review the proposal in terms of the criteria ‘Excellence’ and ‘Impact’, in addition to the new 
criterion of ‘Quality and efficiency of implementation’. The targeted number of proposals 
invited to the second stage is close to 3 but no less than 2.5 times the available budget 
(identified in the work programme), depending on the overall quality of the proposals.

10 Part A: Administrative form includes general information on applicants and contacts, budget, ethics 
table and call-specific questions.  

11 Part B: Project proposal includes a summary list of applicants, objectives of the proposals and its 
relevance  to  the  Horizon  2020  work  programme,  concept  and  methodology,  ambition,  expected 
impact, management structure, milestones and procedures, description of consortium and resources 
committed. The template for Part B can be tailored for each individual call.  

49 

 
 
                                                 
1.2  Admissibility check 

A designated project officer at the European Commission or staff in operation units (i.e. 
FET Open (Future and Emerging Technologies Open)) performs admissibility checks (e.g. 
proposal  is  readable,  accessible  and  printable  and  requested  forms  are  complete)  of  all 
submitted proposals. 

1.3  Eligibility check 

A designated project officer at the European Commission or staff in operation units (i.e. 
FET Open) performs eligibility checks (e.g. minimum number of partners as set out in the 
call conditions, relevance to the call/topic, compliance with any other topic-specific criteria) 
of  all  the  submitted  proposals.  The  project  officer  or  external  experts  (i.e.  vice-chairs) 
allocate proposals to independent experts for evaluation. An eligibility review committee 
decides on unclear cases. 

1.4  Proposal evaluation 

In general, full proposals (whether they are the only stage of the one-stage or the second 
stage  of  the  two-stage  process)  are  evaluated  as  described  below.  A  diagram  of  the 
evaluation process for MSCA is provided in Figure 3 as an illustration of the process. 

Figure 3: Diagram of the evaluation process for MSCA Individual Fellowships. Source: European Commission, 2017 

1.5 

Individual evaluation 

All  proposals  submitted  under  H2020  are  evaluated  by  independent  experts.  More  than 
22 000  different  independent  experts,  from  114  countries,  have  evaluated  H2020 
proposals so far. In total, more than 850 000 evaluations have been conducted. Experts 
are recruited through a continuously open call for expression of interest published in the 
Funding & Tenders Portal. Registration is open to experts of any nationality, including those 
outside of the EU and EU-associated countries. Experts are compensated for their time. 

A  minimum  of  three  external  experts  evaluate  each  proposal  individually  and  remotely 
according to three common criteria:  ‘Excellence’, ‘Impact’, and ‘Quality and efficiency of 
the  implementation’.  Different  actions  will  have  different  description  of  these  criteria, 
depending  on  the  objectives  of  the  programme.  Table  16  provides  an  overview  of  the 
descriptions of the award criteria for the different types of actions. 

50 

 
 
 
 
Table 16: Award criteria for the different types of action under H2020 (work programme 2018–2020)12 

Type of action 

Excellence 

Impact 

All types of action 

Clarity and pertinence of the objectives; 
Soundness  of  the  concept  and  credibility 
of the proposed approach 

The  extent  to  which  the  outputs  of  the 
project  would  contribute  to  each  of  the 
expected impacts mentioned in the work 
programme under the relevant topic 

Quality and efficiency of the 
implementation 

line  with 

Quality  and  effectiveness  of  the  work 
plan,  including  extent  to  which  the 
resources assigned to work packages are 
in 
their  objectives  and 
deliverables; 
Appropriateness  of 
the  management 
structures and procedures, including risk 
and innovation management; 
Complementarity  of  the  participants  and 
extent to which the consortium as a whole 
brings together the necessary expertise; 
Appropriateness of the allocation of tasks, 
ensuring that all participants have a valid 
role and adequate resources in the project 
to fulfil that role 

Research  and  Innovation  Actions 
(RIA); Innovation Actions (IA); 
Coordination  and  Support  Actions 
(CSA) 

(e.g. 

potential 

Extent that the proposed work is beyond 
the  state  of  the  art  and  demonstrates 
innovation 
ground-
breaking  objectives,  novel  concepts  and 
approaches,  new  products,  services  or 
business and organisational models); 
Appropriate 
of 
interdisciplinary  approaches  and,  where 
relevant,  use  of  stakeholder  knowledge 
and  gender  dimension  in  research  and 
innovation content 

consideration 

Any substantial impacts not mentioned in 
the work programme that would enhance 
innovation  capacity,  create  new  market 
strengthen 
opportunities, 
competitiveness 
of 
companies,  address  issues  related  to 
climate  change  or  the  environment,  or 
bring other important benefits for society; 
Quality of the proposed measures to: 
-  Exploit  and  disseminate  the  project 
of 
results 

(including  management 

growth 

and 

12 This table is not exhaustive of H2020 actions. It presents the award criteria for collaboration-based grants and mono-beneficiary grants but does not provide 
information on non-grant actions.  

51 

 
 
                                                 
 
 
ERA-NET Cofund actions 

Level of ambition in the collaboration and 
commitment  of  the  participants  in  the 
proposed ERA-NET action. Namely to pool 
national  resources  in  terms  of  budget, 
number  of  partners  and  participating 
their 
to 
countries,  and 
national/regional research programmes 

coordinate 

intellectual property rights (IPR)), and to 
manage research data where relevant 
-  Communicate  the  project  activities  to 
different target audiences 

to 

the 

partners 

establishing 

to  better  alignment  of 

Contribution 
national activities and policies; 
Contribution 
and 
strengthening  a  durable  cooperation 
between 
their 
national/regional research programmes; 
Quality of the proposed measures to: 
-  Exploit  and  disseminate  the  project 
results  (including  management  of  IPR), 
and  to  manage  research  data  where 
relevant 
- Communicate the project, to activities to 
different target audiences 

and 

Pre-commercial 
(PCP)/Public 
Innovative Solutions (PPI) actions 

Procurement 
of 

Procurement 

Progress beyond the state of the art in 
terms  of  the  degree  of  innovation 
needed to satisfy the procurement need 

Strengthening  the  competitiveness  and 
growth  of  companies  by  developing 
innovations meeting the needs of European 
and global procurement markets; 
Quality of the proposed measures to: 
-  Exploit  and  disseminate  the  project 
results (including management of IPR) and 
to manage research data where relevant 
-  Communicate  the  project  activities  to 
different target audiences; 
concerted 
forward-looking 
More 
procurement  approaches 
reduce 
fragmentation  of  demand  for  innovative 
solutions 

that 

EJP Cofund 

Level  of  ambition  in  the  collaboration 
and  commitment  of  the  participants  in 
the  proposed  ERA-NET  action  to  pool 
national 
resources  and  coordinate 
national/regional research programmes 

Contribution to better alignment of national 
activities and policies; 
Effectiveness of the proposed measures to 
exploit  and  disseminate  the  programme's 
results and to communicate the programme 

52 

 
 
 
 
 
Framework  Partnership  Agreements 
(FPA) 

Clarity and pertinence of the objectives 

Marie  Skłodowska-Curie  Actions 
(MSCA) 

aspects 

innovative 

MSCA ITN: 
Quality, 
and 
credibility  of  the  research  programme; 
Quality  and  innovative  aspects  of  the 
training  programme;  Quality  of  the 
supervision; Quality of the proposed 
Interaction 

the 

and 

and 

credibility 

MSCA IF: 
of 
Quality 
research/innovation project (level 
of novelty, appropriate consideration of 
inter-/multidisciplinary 
gender 
aspects);  Quality  and  appropriateness 
of  the  training  and  of  the  two-way 
transfer  of  knowledge  between  the 
researcher and the host; Quality of the 
supervision and of the integration in the 
team/institution;  Potential  of 
the 
to  reach  or  re-enforce 
researcher 
professional 
maturity/independence 
during the fellowship 

of 

and 

credibility 

RISE: 
the 
Quality 
research/innovation  project;  Level  of 
novelty and appropriate consideration of 
inter-/multidisciplinary, 
intersectoral 
and  gender  aspects;  Quality  and 
appropriateness  of  knowledge  sharing 
among the participating organisations in 
light  of  the  research  and  innovation 
objectives;  Quality  of  the  proposed 
interaction  between  the  participating 
organisations 

COFUND: 

The extent to which the action plan of the 
FPA  would  contribute  to  each  of  the 
expected  impacts  mentioned  in  the  work 
programme under the relevant topic 

MSCA ITN: 
Enhancing the career prospects and 
employability 
and 
researchers 
of 
contribution  to  their  skills  development; 
Contribution  to  structuring  doctoral/early-
stage  research  training  at  the  European 
to  strengthening  European 
level  and 
innovation 
the 
capacity;  Quality  of 
proposed  measures 
to  exploit  and 
disseminate  the  project  results;  Quality  of 
the proposed measures to 
communicate the project activities to 
different target audiences 

Complementarity  of  the  partners  and 
balance of expertise; 
Potential 
among the partners 

cooperation 

long-term 

for 

tasks  and 

MSCA ITN: 
Coherence and effectiveness of the work 
plan,  including  appropriateness  of  the 
resources; 
allocation  of 
the  management 
Appropriateness  of 
including 
structures  and  procedures, 
quality 
risk 
management;  Appropriateness  of  the 
infrastructure 
participating 
organisations;  Competences,  experience 
and complementarity of the participating 
organisations  and  their  commitment  to 
the programme 

management 

and 

the 

of 

MSCA IF: 
Enhancing  the  future  career  prospects  of 
the researcher after the fellowship; Quality 
of  the  proposed  measures  to  exploit  and 
disseminate  the  project  results;  Quality  of 
the proposed measures to communicate the 
project  activities 
target 
audiences 

to  different 

MSCA IF: 
Coherence and effectiveness of the work 
plan,  including  appropriateness  of  the 
resources; 
allocation  of 
Appropriateness  of 
the  management 
structure  and  procedures,  including  risk 
management;  Appropriateness  of  the 
institutional environment (infrastructure) 

tasks  and 

RISE: 
Enhancing  the  potential  and  future  career 
prospects of the staff members; Developing 
new  and  lasting  research  collaborations, 
achieving  transfer  of  knowledge  between 
participating organisations and contribution 
innovation 
to 
potential at the European and global levels; 
Quality of the proposed measures to exploit 
and disseminate the project results; Quality 
of the proposed measures to communicate 
the  project  activities  to  different  target 
audiences 

improving  research  and 

tasks  and 

RISE: 
Coherence and effectiveness of the work 
plan,  including  appropriateness  of  the 
resources; 
allocation  of 
the  management 
Appropriateness  of 
structures  and  procedures, 
including 
risk 
quality 
management;  Appropriateness  of  the 
institutional 
(hosting 
infrastructure); 
arrangements, 
Competences, 
and 
complementarity of the 
participating  organisations  and 
commitment to the project 

management 

environment 

experience 

their 

and 

53 

 
 
 
 
 
 
 
 
 
 
the 
composition 

Quality  of  the  selection/recruitment 
researchers 
process 
for 
(transparency, 
and 
organisation  of  selection  committees, 
evaluation criteria, equal opportunities); 
Quality  of  the  research  options  offered 
by the programme in terms of science, 
interdisciplinarity,  intersectorality  and 
level  of  transnational  mobility;  Quality 
of  career  guidance  and 
training, 
including  supervision  arrangements, 
training in 
transferable skills 

COFUND: 
Enhancing  the  potential  and  future  career 
prospects  of  researchers;  Strengthening 
human  resources  on  regional,  national  or 
international  level;  Aligning  practices  of 
the 
participating 
principles  set  out  by  the  EU  for  human 
resources  development  in  research  and 
innovation;  Quality  of 
the  proposed 
measures  to  exploit  and  disseminate  the 
results;  Quality  of  the  proposed  measures 
to  communicate  the  results  to  different 
target audiences. 

organisations  with 

effectiveness 

COFUND: 
and 
Coherence, 
appropriateness  of 
the  work  plan; 
Appointment  conditions  of  researchers; 
to 
Competence  of 
implement the programme 

the  participant 

European Research Council 

SME instrument 

Starting  Grant,  Consolidator  Grant  and 
Advanced Grant: 
Research  project:  Ground-breaking 
nature  and  potential  impact  of  the 
research project; Scientific approach 
investigator: 
principal 
capacity and creativity; Commitment 

Intellectual 

Proof  of  Concept  Grant:  Does  the 
proposed 
activity 
proof-of-concept 
the  output  of 
greatly  help  move 
research  towards  the  initial  steps  of  a 
process  leading  to  a  commercial  or 
social innovation? 

Soundness  of  the  concept,  including 
transdisciplinary  considerations,  where 
relevant; 

Extent that proposed work is ambitious, 
has innovation potential, and is beyond 
the  state  of  the  art  (e.g.  ground-
breaking objectives, novel concepts and 
approaches) 

Proof of Concept Grant: Is the project to be 
taken  to  proof  of  concept  expected  to 
generate  any  effect  or  benefit  to  the 
economy,  society,  culture, public  policy  or 
services  and  are 
these  appropriately 
the  proposal?  Does  the 
identified 
proposal  provide  a  suitable  outline  of  how 
the commercialisation or the generation of 
the above-listed benefits will be achieved?  

in 

the 
Proof  of  Concept  Grant:  Does 
proposal  provide  a 
reasonable  and 
acceptable  plan  of  activities  against 
clearly  identified  objectives  and  towards 
establishing the feasibility of the project? 

Enhancing 
integration of new knowledge; 

innovation 

capacity 

and 

Strengthening  the  competitiveness  and 
growth  of  companies  by  developing 
innovations meeting the needs of European 
and  global  markets,  and,  where  relevant, 
by  delivering  such  innovations  to  the 
markets; 

54 

 
 
 
 
 
 
 
 
 
Fast Track to Innovation 

in 

been 

tested 

The  objectives  of  the  proposal  are 
defined  in  a  clear  and  pertinent  way, 
support  Horizon  2020  objectives,  and 
are  directed  towards  fast,  wide  market 
uptake. 
The  proposed  activities  to  be  executed 
will contribute to a credible, realistic and 
optimal  development  of  the  innovation 
to the level of market uptake. 
jointly  developed 
The  underlying, 
business  innovation  concept  of  the 
proposed  innovation  is  sound,  and  has 
already 
an 
operational/production  environment.  It 
has  a  potential  to  bring  important 
progress to or revolutionise an existing 
industrial  sector,  business  practice 
and/or societal challenge. 
The  proposed  innovation  is  ambitious 
and  is  set  to  add  substantial  value  to 
Europe (e.g. considerably contribute to 
Europe's  industrial  leadership  or  the 
solution  of  Horizon  2020  societal 
challenges),  and  this  is  well  identified 
and elaborated in the proposal. 
has 
innovation 
The 
successfully 
an 
tested 
operational  or  production  environment 
(stage  of  development  at  technology 
readiness  level  (TRL)  6  or  similar  for 
non-technological  innovations)  and  can 

proposed 

been 

in 

Any  other  environmental  and  socially 
important  impacts  (not  already  covered 
above); 

Effectiveness of the proposed measures to 
exploit and disseminate the project results 
(including  management  of 
to 
communicate  the  project,  and  to  manage 
research data where relevant 

IPR), 

the 

innovation  will 
innovation  capacity  of 

The objectives of the proposed action are in 
line  with  the  expected  impacts  of  the  FTI 
and  with  the  objectives  of  the  European 
Innovation Council Pilot; 
lead  to 
The  proposed 
enhanced 
the 
consortium  partners,  and  in  particular  of 
the industry partners; 
The  proposed  innovation/solution  has  a 
clear European or global dimension; 
The  proposal  provides  a  realistic  and 
convincing  analysis  of 
targeted 
market(s) and client/user base and how the 
innovation will meet their needs; 
The  way  the  project  will  strengthen  the 
growth/scale-up and competitiveness of the 
industry 
is  well 
documented; 
Framework  conditions,  such  as  regulation 
and  standards,  market  size,  prospects  for 
growth,  competitive  edge  and  intended 
positioning of the solution towards possible 
others 
are 
documented,  and  the  outlook  can  be 
described  as  positive  for  market  launch 
within 3 years' time; 
The commercialisation plan is realistic and 
convincing; 
The  proposed  innovation  is  expected  to 
generate a positive impact at the European 
level  other 
(societal, 
than  economic 
environmental, scientific, etc.); 

(competitor 

analysis), 

partners 

involved 

of 

The work plan is coherent and effective; 
Tasks  and  resources  are  allocated  in  an 
appropriate and cost-effective way; 
The  proposal  demonstrates  that  the 
are 
partners 
complementary, and together can deliver 
on  ground-breaking  innovation  and  fast, 
wide  market  uptake;  Implementation 
risks  and  threats  are  well  identified;  the 
proposal  contains  a  risk  mitigation  plan, 
with detailed actions 

consortium 

the 

55 

 
 
 
 
move  to  market  take-up  (B2B  or  B2C) 
within maximum 36 months. 
The  proposal  demonstrates  that  the 
intrinsic quality of the innovation will be 
significantly  higher  than  current  state-
of-the-art  solutions,  in  terms  of  value 
for  money,  problems  solved,  new 
applications, sustainability, etc. 

The proposed financing plan for further roll-
out  of  the  innovation  is  realistic  and 
convincing and offers a sufficient guarantee 
and coverage to allow for further scale-up 
of the action and companies involved 

56 

 
 
 
Following individual assessment, reviewers are asked to submit their evaluation in the form 
of individual evaluation reports (IERs). The IER template provides reviewers with guidance 
on the scoring criteria for the different actions, as well as specific thresholds and weighting, 
when applicable. IERs collect information on the call (such as the programme, topic, action 
and proposal ID), the proposal itself (such as duration of the project, applicants, project 
abstract), and the name of evaluators. The reports include a score between 0 and 5, and 
they provide comments for each of the three criteria. 

0:  The  proposal  fails  to  address  the  criterion  or  cannot  be  assessed  due  to  missing  or 
incomplete information. 

1 Poor: The criterion is inadequately addressed or there are serious inherent weaknesses. 

2 Fair: The proposal broadly addresses the criterion, but there are significant weaknesses. 

3  Good:  The  proposal  addresses  the  criterion  well,  but  a  number  of  shortcomings  are 
present. 

4  Very  good:  The  proposal  addresses  the  criterion  very  well,  but  a  small  number  of 
shortcomings are present. 

5 Excellent: The proposal successfully addresses all relevant aspects of the criterion. Any 
shortcomings are minor. 

Each  proposal  is  reviewed  as  submitted.  Reviewers  are  instructed  not  to  recommend 
modifications. In order to be considered for funding, proposals must score above a certain 
threshold  for  each  individual  criterion  and  above  an  overall  threshold  for  all  criteria.  A 
summary of the feedback and scoring guidance for the different actions is provided in Table 
17.

57 

 
 
Table 17: Summary of the evaluation guidance for reviewers in H2020 

Actions 

Characteristic 

Description 

For  all  except  MSCA, 
FPA,  SME 
instrument 
and FTI 

Feedback requested 

Scoring 
weighting 

threshold 

and 

MSCA actions 

Feedback requested 

Stage 1 proposal: Only excellence and impact 
are reviewed 
Full proposal: Reviewers are asked to provide 
comments on their assessment of excellence, 
impact  and  quality  and  efficiency  of  the 
implementation  

Stage  1  proposal:  The  threshold  is  4/5  for 
each  criterion;  the  overall  threshold  is  not 
considered in stage 1 proposals 
Full proposal: Threshold default is 3/5; other 
thresholds may apply if foreseen in the call 

Reviewers  are  asked  to  provide  strengths  of 
the proposal (in bullet points); weaknesses of 
the  proposal  (in  bullet  points);  and  other 
comments for each of the criteria 

Scoring 
weighting 

threshold 

and 

Reviewers must also provide a summary with 
their  score  for  each  criterion  as  well  as  the 
weighted score; threshold is set at 70% 

FPA actions 

Feedback requested 

Reviewers are asked to provide comments on 
their  assessment  of  excellence,  impact  and 
quality and efficiency of the implementation 
Reviewers are asked to provide comments on 
their  assessment  of  excellence,  impact  and 
quality and efficiency of the implementation 

Scoring 
weighting 

threshold 

and 

Threshold default is 3/5 

SME instrument Phase 1 
and 2 

Feedback requested 

Reviewers are asked to provide comments for 
each criterion 

Scoring 
weighting 

threshold 

and 

Impact score is weighted 50% 
Excellence score is weighted 25% 
Quality  and  efficiency  of  the  implementation 
score is weighted 25% 
The threshold for each criterion score is 4/5 
The threshold for the total score is 13/15 

Fast Track to Innovation 
(FTI) actions 

Feedback requested 

Reviewers are asked to provide comments for 
each criterion 

Scoring 
weighting 

threshold 

and 

Impact score is weighted 50% 
Excellence score is weighted 25% 
Quality  and  efficiency  of  the  implementation 
score is weighted 25% 
The threshold for each criterion score is 4/5 
The threshold for the total score is 13/15 

In addition to scoring the above-mentioned criteria, evaluators are instructed to consider 
additional factors in their evaluation, such as the applicant’s basic operational capacity to 

58 

 
 
 
 
carry  out their proposed activities and  the applicant’s request for  exceptional funding of 
third-country13 participants or international organisations. 

1.6  Consensus group discussion 

After proposals have been evaluated independently, reviewers convene in the consensus 
meeting.  The  consensus  meeting  is  chaired  by  a  rapporteur  or  a  moderator  from  the 
European Commission or the Executive Agency and involves all the individual evaluators 
of  the  proposals.  The  rapporteur  can  also  be  an  external  expert  not  involved  in  the 
evaluation  of  the  proposals.  In  the  case  of  ERC  Investigator  Grants  (i.e.  Starting, 
Consolidator  or  Advanced),  each  proposal  discussion  is  chaired  by  one  of  the  reviewers 
from  the  individual  evaluation.  The  meeting  aims  to  find  an  agreement  on  the  final 
comments and scores of the proposals. Meetings are held on site (in Brussels) or remotely 
(through teleconferences or through a comment box functionality in the evaluation tool, 
which is the tool used for the majority of consensus discussions in case the call is in full 
remote consensus). 

The  rapporteur  drafts  the  consensus  report,  which  is  an  internal  document.  This  report 
summarises the justification of scores,  records any dissenting views, and the consensus 
view that has emerged from the group’s discussion. The consensus report (CR) collects the 
names of evaluators, as well as the same level of information as the IERs on the call and 
proposal.  In  addition  to  this,  the  rapporteur  comments  on  whether  they  believe  the 
proposal is in scope and also notes if any or all experts had considered it ‘out of scope’; if 
the latter, the rapporteur is advised to immediately contact the call coordinator or project 
officer.  Similarly,  the  rapporteur  provides  comments  on  the  operational  capacity  of 
applicants; if this capacity is deemed to be insufficient, they continue with the evaluation 
but disregard the applicant’s activities and estimated budget. 

In 2019 and 2020 calls, several pilots of IER with no score were performed within MSCA, 
in  which  the  score  was  only  agreed  during  the  consensus  meeting,  after  reaching  an 
agreement  on  the  comments.  Feedback  received  by  DG  RTD  stated  that  the  consensus 
meetings  were  smoother  and  that  there  was  less  inconsistency  between  scores  and 
comments.  DG  RTD  is  currently  analysing  whether  the  experts’  feedback  has  evolved 
compared with previous evaluations using the IERs with scores. 

1.7  Panel review 

A panel review ensures consistency in the comments and scores given at the consensus 
stage  for  all  the  submitted  proposals  to  the  call.  It  recommends  a  list  of  proposals  for 
funding  in  priority  order,  based  on  final  scores.  A  panel  review  consists  of  a  subset  of 
experts (e.g. vice-chairs – very experienced scientific evaluators who are not evaluating 
proposals for the call at hand) or external experts not involved in the evaluations of the 
proposals.  Panel  review  leads  to  the  evaluation  summary  report,  which  summarises  the 
evaluation results and includes scores and supporting comments. Interestingly, the ESR is 
almost identical to the CR, only with the names of evaluators removed. The ESR is sent to 
the applicants, and applicants then have 30 days to file a complaint. 

The ERC Investigator Grants evaluation process use 25 panels per call, each with 12 to16 
members. In each step, applications are reviewed by at least three members of one of the 
panels. In the first step of the evaluation, reviewers evaluate only the extended synopsis 
of the research proposal. Applications that are successful in the first step will be assessed 
again by the panel in the second step. In addition, in the second step, the project synopsis 

13  The  term  third  countries  refers  to  countries  that  are  not  members  of  the  EU  and  countries  or 
territories whose citizens do not enjoy the EU right to free movement. The most recent list of eligible 
countries 
2018–2020 
(https://ec.europa.eu/research/participants/data/ref/h2020/other/wp/2018-2020/annexes/h2020-
wp1820-annex-a-countries-rules_en.pdf). 

in  Horizon 

Programme 

–  Work 

found 

2020 

can 

be 

59 

 
 
                                                 
and  full  proposal  are  sent  to  up  to  seven  external  reviewers  in  order  to  ensure  that  an 
application at second stage will be assessed by at least six reviewers. Panel reviews and 
external reviews are then discussed at each panel in order to reach a conclusion on funding 
(Luukkonen,  2012).  A  diagram  of  the  ERC’s  two-step  evaluation  process  is  outlined  in 
Figure 4. 

Figure 4: Diagram of the ERC’s two-step evaluation process. Source: RAND Europe 

60 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Box 2: Exceptions to the standard H2020 proposal evaluation process 

Multi-step evaluation process 

The ERC applies a single-stage submission of the proposal, followed by a two-step evaluation (three 
steps for ERC Synergy calls). At step 1, the extended synopsis and the principal investigator's track 
record and CV are assessed (and not the full scientific proposal). At step 2, the complete version of 
the retained proposals  is assessed (including the full scientific proposal). Proposals are retained for 
step 2 based on the outcome of the evaluation at step 1 and a budgetary cut-off level of three times 
the panel's indicative budget. Principal  investigators whose  proposals are retained  for step 2 of the 
evaluation  are  invited  for  an  interview  to  present  their  project  to  the  evaluation  panel,  meeting  in 
Brussels. Once the evaluation of the proposal has been completed, applicants receive an evaluation 
report, which will include the collection of individual evaluations and a ranking range of their proposal 
among the proposals evaluated by the panel. The report does not include scores. At the end of step 1 
of the evaluation applicants will be informed that their proposal either a) is of sufficient quality to pass 
to step 2 of the evaluation; b) is of high quality but not sufficient to pass to step 2 of the evaluation; 
or c) is not of sufficient quality to pass to step 2 of the evaluation. At the end of step 2 of the evaluation, 
applicants  will  be  informed  that  their  proposal  a)  fully  meets  the  ERC's  Excellence  criterion  and  is 
recommended for funding if sufficient funds are available; or b) meets some but not all elements of 
the ERC's Excellence criterion and will not be funded. 

In the EIC Accelerator pilot (SME instrument Phase 2), proposals are evaluated in two steps. A diagram 
of the two-step evaluation process of the SME instrument Phase 1 and Phase 2 is provided in Figure 
5. Step 1 is a remote evaluation by four evaluators with mixed expertise in technology, industry sector, 
business and finance. The consensus score of a proposal at the level of the three evaluation criteria is 
the median of the scores given by each evaluator. Proposals that pass all quality thresholds will be 
considered  for  step  2.  Starting  with  the  highest-scoring  proposal  after  step  1  and  in  descending, 
sequential order, proposals are passed to step 2 until, as a batch, either the total amount of EU funding 
requested is as close as possible to twice the grant budget available or all proposals eligible for funding 
have been accounted for. The actual threshold to pass to step 2 is therefore dynamic and dependent 
on the volume of proposals received that pass all quality thresholds. Principal investigators in proposals 
that pass to step 2 are invited to a face-to-face interview in Brussels. In exceptional cases, interviews 
may be organised via videoconference. In step 2, proposals receive, in addition to the score in step 1, 
an  'A'  mark  or  a  'B'  mark  from  the  final  panel  review.  Only  proposals  that  have  passed  all  quality 
thresholds  and  receive  an  'A'  mark  are  proposed  for  funding,  including  blended  finance  when 
applicable. 

Automatic consensus phase 

EIC Pathfinder pilot (FET Open)14 calls for collaborative RIA implements an automatic consensus phase 
with  arithmetically  calculated  scores:  at  consensus  stage,  the  consensus  score  for  each  evaluation 
criterion  is  the  median  of  the  corresponding  scores  attributed  by  the  individual  evaluators.  The 
consensus report comprises a collation of the comments from individual reports or extracts from them. 
The final review panel decides on the final score based on its consensus discussions. The panel also 
decides on any additional comments, possibly including advice not to resubmit the proposal. 

Evaluation criteria 

Under  H2020,  there  are  several  exceptions  to  the  evaluation  criteria  applied:  for  ERC,  only  the 
‘Excellence’  criterion  is  applied,  whereas  for  MSCA,  FET  and  the  EIC  pilot,  the  aspects  to  take  into 
account in each of the three criteria are adapted to these specific activities. 

14 EIC FET Open supports the early stages of the science and technology research and innovation 
around new ideas towards radically new future technologies.  

61 

 
 
 
 
 
 
                                                 
 
 
 
Figure 5: Diagram of the SME instrument’s two-step evaluation process. Source: European Commission, n.d. 

62 

 
 
 
CHAPTER 4: LITERATURE REVIEW ON THE CHARACTERISTICS OF 

GRANT PROPOSAL EVALUATION 

In this chapter, we describe the results of our review on the evidence on the characteristics 
of the proposal evaluation landscape. 

Box  3:  Key  findings  from  the  literature  review  on  international  practice  in  proposal 
evaluation 

 

 

The grant funding process is burdensome on applicants, reviewers and coordinating institutions. 

The burden of the funding process is exacerbated by a low success rate. 

  R&I funders have developed strategies to attempt to reduce the burden of the funding process, 
such as including a two-stage application process or having multiple calls throughout the year. 

 

Increased  training  and  support  in  grant  writing  can  lead  to  better  outcomes  and  decrease  the 
perception of burden. 

  Reducing the burden of the funding process may lead to more timely funding decisions. 

  Despite the important role  of feedback as a means to improve the quality of research, there is 

little evidence on what constitutes ‘proper’ feedback in grant peer review. 

  Bias  in  the  evaluation  process  occurs  in  a  range  of  forms  across  the  funding  process,  from 

programme design to funding allocation. 

 

 

 

 

Funders have developed initiatives aimed at increasing fairness and equality of opportunity in the 
proposal evaluation process, such as blind review or lottery funding. 

Lack of clarity in the definition of evaluation criteria decreases reliability of the funding process. 

The use of group discussion can increase the reliability of the peer review process. 

Peer review is ‘anti-innovation’, as innovative research is considered high risk. 

  Bespoke approaches to the evaluation of innovative research have been introduced with the aim 
of increasing funding innovation, but evidence of their effectiveness is limited. Evaluating MIDRI 
presents  specific  challenges  and  requires  adaptation  of  the  process  to  ensure  evaluation  is 
adequate. 

 

 

The ability of the peer review process to predict successful research projects (i.e. predictive power) 
is limited by the reviewer’s understanding of the evaluation criteria. 

Transparency of the funding process may increase trust in the research community. 

1 

The  grant  funding  process  is  burdensome  on  applicants,  reviewers  and 
coordinating institutions 

In the context of proposal evaluation, burden refers to the degree to which the evaluation 
process places a time burden on researchers, reviewers and administrators who participate 
in the peer review process. The burden associated with the application processes may be 
an important driver of such factors as working hours and work–life balance (Herbert et al., 
2014). 

Most (75%) of the burden of the funding award process falls on applicants (Guthrie, Ghiga, 
and  Wooding,  2018).  Grant  preparation  consists  mainly  of  four  activities,  namely 
preparatory work, collaborative work, grant preparation and writing, and quality assurance 
(Kulage et al., 2015). The most time-consuming and therefore costly of these activities are 
grant  preparation  and  writing  (Kulage  et  al.,  2015).  A  study  found  that  academic 
researchers spent an average of 38 days to prepare a new grant application for the National 
Health  and  Medical  Research  Council  (Gurwitz,  Milanesi,  and  Koenig,  2014).  From  an 
economic perspective, another study estimated that application preparation can account 
for  up  to  35%  of  the  application  budget  (Guthrie,  Ghiga,  and  Wooding,  2018).  This  is 
potentially due to the high level of preliminary results thought to be required for successful 
applications (Bolli, 2014). 

63 

 
 
 
 
Institutions  also  face  the  burden  of  grant  preparation,  playing  an  important  role  in 
supporting faculty applying for research funding, through the provision of mentoring and 
administrative  assistance  to  new  PIs  (Specht,  2013),  as  well  as  for  doctoral  candidates 
encouraged  to  pursue  independent  funding.  In  addition,  institutional  review  boards  and 
conflict-of-interest  committees  require  experienced  staff  to  manage  programmes  and 
ensure compliance with the different mandates (Kulage et al., 2015). A study showed that 
research administrators spend 33.9 to 56.4 hours in grant preparation, with an estimated 
cost ranging from US$4 784 to US$13 512 per grant (Kulage et al., 2015). 

Studies have also emphasised the burden of peer review on the reviewers in the context 
of  grant  applications  (Bolli,  2014;  Kitt,  Williamson,  and  Paganetti,  2009;  Snell,  2015; 
Guthrie,  Ghiga,  and  Wooding,  2018).  Reviewers  are  generally  experts  in  their  field  who 
can  critically  assess  the  quality  and  feasibility  of  a  research  proposal.  Therefore,  being 
asked to be a  reviewer in grant proposal evaluation can be considered an achievement, 
although reviewers generally do not receive academic recognition for their efforts in peer 
review. In addition, few institutions provide reviewers with protected time for peer review 
despite  encouraging  the  process  (Schroter,  Groves,  and  Højgaard,  2010).  It  has  been 
shown  that  reviewers  are  usually  motivated  to  participate  in  peer  review  by  a  sense  of 
professional duty and fairness (Schroter, Groves, and Højgaard, 2010). However, the lack 
of recognition or (financial) reward for their time commitment has been shown to have a 
negative impact on reviewers’ ability to deliver, with peer reviews often being conducted 
in  their  own,  personal  time  and  participants  pointing  to  their  sense  of  professional  duty 
and  fairness  as  being  the  main  reason  that  they  continue  to  support  peer  review 
(Gluckman, n.d.; Schroter, Groves, and Højgaard, 2010). 

2 

The burden of the funding process is exacerbated by a low success rate 

In recent decades, we see an increase internationally in the number of proposals submitted 
for many major research funders (Vrieze, 2017). However, funding for research is being 
reduced, leading to a decrease in the success rate of applications. As mentioned previously, 
the  grant  application  process  is  time  consuming  and  costly,  and  when  these  factors  are 
combined with the fact that the time spent is less likely to lead to funding, the perceived 
burden  of the process increases.  In addition, the higher number of submitted proposals 
and reduced funding make the peer review process less effective, as it is no longer a matter 
of  funding  the  ‘fundable’  proposals,  but,  rather,  of  assessing  the  finer  details  and  being 
able  to  distinguish  between  very  good  proposals  and  outstanding  proposals.  Both 
applicants  and  reviewers  may  find  that  they  invest  a  large  amount  of  time  writing  and 
reviewing  grant  applications  for  research  that  has  already  been  performed  or  that  may 
never be done (Bolli, 2014). 

3 

R&I funders have developed strategies in attempts to reduce the burden of 
the funding process 

In recent years, different strategies have been developed by R&I funders to try to reduce 
the burden of grant applications. These measures could allow researchers to focus on their 
research, save reviewers time, and potentially reduce the cost of grant review by reducing 
the labour required to review grant applications  (Bolli, 2014). With regard to burden on 
applicants, funders have introduced limits on the number of times an application can be 
submitted  (Rockey,  2012),  availability  of  multiple  calls  for  applicants  per  year  (Guthrie, 
Ghiga, and Wooding, 2018; Herbert et al., 2014), extending the application window (Bolli, 
2014),  and  streamlining  CV  generation  (Guthrie,  Rodriguez-Rincon,  et  al.,  2019a).  In 
addition, as Guthrie, Ghiga, and Wooding (2018) suggest, increased transparency of the 
review  process  could  impact  the  perceived  burden,  as  applicants  will  be  better  able  to 
understand the process that leads to the outcome of their application (Gurwitz, Milanesi, 
and Koenig, 2014). 

With the aim of reducing burden on reviewers, some organisations have introduced a two-
stage application process or a multi-step evaluation process that can eliminate applications 
at an initial stage (Guthrie, Rodriguez-Rincon, et al., 2019; Barnett et al., 2015); having 

64 

 
 
multiple  calls  per  year  (Guthrie,  Rodriguez-Rincon,  et  al.,  2019b;  Herbert  et  al.,  2014); 
limiting  the  number  of  grants  a  recipient  can  hold  (Guthrie,  Rodriguez-  Rincon,  et  al.,, 
2019b); limiting the number of times panellists serve, as well as alternating the years in 
which they do so (Guthrie, Rodriguez-Rincon, et al., 2019b); and implementing a lottery 
system within the top 20% high-scoring applications (Fang and Casadevall, 2016) or for 
proposals of medium and unknown merit (Avin, 2015). 

4 

Reducing the burden of the funding process may lead to more timely funding 
decisions 

A high number of applications results in a long time between application submission and 
outcome notification. This is because funders seek to ensure that all applications receive a 
high-quality review, which takes considerable time, as mentioned previously. For example, 
the ERC process may take up to 9 months between the application deadline and the time 
to notify applicants of the outcome of the evaluation, and up to 13 months for successful 
applicants to receive their grant. Similarly, the National Institutes of Health (NIH) process 
may  take  up  to  10  months  before  funding  is  awarded  (NIH,  n.d.).  This  delay  between 
application and funding decision is a particular issue in the context of innovation funding, 
where rapid change in disciplines may lead to results that change the course of a proposed 
project, making previously highly fundable projects less attractive. 

Funders  have  developed  strategies  to  reduce  the  time  delay  between  application  and 
funding, mainly aimed at reducing the number of applications reviewed. For example, in 
2009,  the  NIH  incorporated  a  clause  into  their  grant  policy  limiting  applicants  to  two 
submissions of a research proposal, which led to a reduction in the average time to award 
from submission (Rockey, 2012). Another approach was taken by the National Institute of 
Health Research, who adopted a two-stage application process, decreasing the number of 
applications sent for peer review, leading to a shorter time between application submission 
and outcome notification (Guthrie, Rodriguez-Rincon, et al., 2019). 

5 

Increased training and support in grant writing can lead to better outcomes 
and decrease the perception of burden 

One  of  the  factors  contributing  to  the  perception  of  burden  of  the  grant  application  and 
review  process  is  the  low  success  rate.  Therefore,  several  studies  have  focused  on 
improved outcomes for the process, including through better training (Kulage et al., 2015; 
Kupfer et al., 2014); prior experience in the funding scheme (Enger and Castellacci, 2016); 
and a change from dichotomous funding mechanisms to continuous funding mechanisms, 
in  which  a  degree  of  support  beyond  a  ‘fund/no-fund’  decision  is  provided  to  applicants 
(Mutz, Bornmann, and Daniel, 2016; Bollen et al., 2019, 2017). 

Better training and support in grant writing and applications  (Kulage et al., 2015) could 
lead  to  increased  success  rates  and  therefore  reduce  the  impact  of  perceived  burden 
(Guthrie, Rodriguez-Rincon, et al., 2019b). For example, with the aim of increasing success 
rates  in  NIH  applications,  the  Department  of  Psychiatry  at  the  University  of  Pittsburgh 
established  a  faculty  research  review  committee  for  peer  review  of  external  research 
proposals to ensure that research proposals prepared by investigators affiliated  with the 
department meet the highest scientific and ethical standards (Kupfer et al., 2014). In this 
system, grant applicants are advised to submit their proposals to the committee at least 1 
month  prior  to  the  submission  date,  and  the  committee  will  then  review  and  provide 
constructive  feedback  on  the  proposal.  The  internal  review  process  is  associated  with 
success in obtaining research support, as well as with increasing the levels of collaborative 
scientific work (Kupfer et al., 2014). However, as mentioned previously, one of the main 
reasons for the lower success rate of the funding process is limited funding available given 
the high number of applications. Improving the quality of applications may in fact add to 
the burden, since funding will still be available for only a limited number of proposals. In 
addition, not all institutions will be able to provide the same quality of training and support, 
which leads to a risk of unfairness in the process. Therefore, training and support may not 

65 

 
 
reduce  burden  at  a  system  level  –  although  they  could  offer  improvements  in  proposal 
quality and benefits for individuals. 

Prior  participation  in  a  specific  funding  process  has  also  been  associated  with  better 
outcomes  (Enger  and  Castellacci,  2016).  In  a  study  looking  at  the  outcomes  in  the  EU 
Framework  Programmes  (FP)  in  Norway,  the  authors  found  that  two  main  factors 
strengthen the likelihood that a research organisation will submit a successful application, 
namely prior participation in EU FPs and scientific reputation (Enger and Castellacci, 2016). 
Prior  participation  indicates  persistence  and  learning  effects  from  previous  EU  projects, 
which may contribute to a better assessment of a proposal in the review phase. In addition, 
the authors found no significant correlation between publication numbers and probability 
of success, suggesting that scientific reputation (quality, impact and citation indices) were 
more important than scientific productivity (number of publication points) in securing EU 
FP funding. 

Some papers proposed an alternative to funding schemes in which applications are either 
approved  or  rejected.  One  such  alternative  is  the  distribution  of  continuous  funding,  in 
which a degree of support beyond a ‘fund/no-fund’ decision is provided to applicants (Mutz, 
Bornmann, and Daniel, 2016)). The current funding system of the Austrian Science Fund 
(known  by  its  German  acronym,  FWF)  is  a  mixture  of  a  continuous  and  dichotomous 
system, in which elements of a dichotomous system predominate but different proposals 
receive  different  funding  based  on  the  requested  sum  and  the  achieved  score.  In  their 
study, Mutz et al. (2016) looked at the success rate at FWF in Austria, and found that with 
its current system, success rate was 44.5%. They estimated that this rate would increase 
to 58.4% if the organisation used a continuous funding mechanism and decrease to 32.1% 
if the system was purely dichotomous. The authors found that due to the lack of reliability 
of  the  peer  review  system,  even  proposals  estimated  to  have  very  good  merit  had  an 
average probability of success of only 0.70, while the average probability of success with 
a continuous system would reach 0.80 to 0.90. However, the overall higher probability of 
success of a proposal in a continuous funding system applies also to proposals that receive 
only an average to below-average score from the reviewers.  In contrast, a dichotomous 
system led to high-quality proposals not being funded. Therefore, by including a continuous 
aspect in the funding process, funders must accept a few more ‘false positives’ to avoid 
‘false negatives’. 

Another  approach  to  funding  that  has  been  proposed  is  a  system  in  which  all  scientists 
individually  participate  in  the  allocation  of  research  funding  (Bollen  et  al.,  2017).  This 
system has been termed the self-organised fund allocation. In this system, all participants 
receive an equal portion of all yearly funding, but they are then required to anonymously 
donate a fraction of everything that they have received to peers. These distributed systems 
incorporate  the  opinions  of  the  entire  scientific  community  in  a  way  that  encourages 
fairness, robustness and efficiency. The authors state that in this way, funding would go 
to  people  rather  than  projects.  In  a  follow-up  study,  Bollen  et  al.  (2019)  acknowledged 
that  practical  implementation  of  such  a  funding  system  requires  consideration,  such  as 
decisions  on  who  can  participate  in  this  system  and  how  to  prevent  conflicts  of  interest 
(Bollen et al., 2019). 

6 

Despite the important role of feedback as a means to improve the quality of 
research,  there  is  little  evidence  on  what  constitutes  ‘proper’  feedback  in 
grant peer review 

Constructive feedback is an important aspect of the learning process, both for the person 
who  receives  the  feedback  and  for  the  person  who  provides  the  feedback,  and  it  is  an 
essential component of performance improvement processes (Mulder and Ellinger, 2013). 
Feedback helps to ensure that the reviewer understands the information provided and the 
receiver has a chance to review, reflect and identify any changes needed to ensure that 
the  message  is  understood.  However,  there  is  little  evidence  on  what  constitutes  ‘good’ 
feedback in peer review, and we found no evidence in particular on providing feedback for 

66 

 
 
grant applications. However, some evidence is available on peer review feedback in relation 
to publication review, which may be of relevance here. 

Our study found that reviewers can be classified according to the type of feedback they 
provide. A study by Dobele (2015) analysed the feedback provided by reviewers for two 
conferences, one edited book and six journals, with a view to assessing the usefulness of 
the review, represented by the authors' ability to utilise the feedback effectively. The study 
found that feedback could be classified as constructive if it provides direction or assistance 
to the author or unconstructive if does not provide direction or assistance for the author. 
Both constructive and unconstructive criticism can be positive or negative. Feedback can 
also be neutral in cases where comments are neither constructive or unconstructive nor 
negative or positive (e.g. ‘ok’). 

Dobele (2015) also assessed the importance  of quality versus quantity  of the feedback. 
The  author  found  that  reviewers  provided  a  higher  quantity  of  feedback  when 
recommending acceptance with major changes (with 43.18% of reviewers providing more 
than one page  of  feedback, 22.73% providing at least one page, and  31.82% providing 
less  than  one  page)  than  when  recommending  for  rejection  or  for  acceptance  without 
changes. However, in terms of quantity, less feedback was provided to papers commended 
for rejection (38% provided more than one page of feedback and 15% provided at least 
one  page).  The  majority  of  the  feedback  was  constructive  negative  (81%),  followed  by 
constructive  positive  (10%).  Only  4%  of  reviewers  provided  unconstructive  negative 
feedback  to  rejected  papers,  and  2%  provided  unconstructive  positive  feedback.  In 
addition, when accepting a manuscript for publication without change, reviewers generally 
included a few lines to support their recommendation, even though supporting information 
was not required. 

Another study used LCA to assess reviewer feedback on submitted manuscripts and found 
there were three types of reviewers (Sueur et al., 2020). The first subtype, termed ‘the 
nurturing’ reviewer, responded quickly and gave more detailed reports, which were overall 
more constructive and positive. Nurturers were the least prominent group among reviewers 
(22.3%)  and  were  considered  to  do  ‘justice  to  the  peer  review  process’,  showing  they 
understand their role in improving the study by providing constructive criticism. The second 
subtype,  termed  ‘the  begrudged’,  produced  shorter,  less  detailed,  less  constructive  and 
harsher reports. This group made up 33.4% of reviewers, and although their feedback was 
considered  to  be  valid,  it  could  benefit  from  more  constructive  wording  and  less 
discourteous  language.  The  third  subtype,  termed  ‘the  blasé’,  represented  the  highest 
percentage  of  reviewers  (44.3%).  This  group  consisted  of  reviewers  who  were  quick  to 
submit their reviews; provided shorter, less detailed reports; and were generally positive 
but not very constructive. The authors found that the level of detail, number of comments 
and  length  of  review  all  positively  correlated  with  each  other  and  that  constructiveness 
strongly correlated with the level of detail within a given review (Sueur et al., 2020). 

Studies  providing  guidance  on  how  to  write  a  constructive  feedback  letter  have  mainly 
focused on peer review in the context of manuscript submission to journals. In addition, 
these studies are mainly based on the authors’ opinion and expertise rather than objective 
evidence.  Two  studies  that  were  included  in  our  review  provided  recommendations  on 
for  manuscript  submissions,  although  the 
writing  a  constructive  peer  review 
recommendations may also apply to grant review (Table 18). 

67 

 
 
 
 
 
 
 
Table 18: Guidance for a good peer review 

Aspect of evaluation 

Reviewer decision  

Lazarides, Georgiadis, Papanas et al. 
(2020) 

A  reviewer  should  not  include  in  their 
review  whether  they  are  accepting  or 
rejecting  the  manuscript,  as  the  final 
decision lies with the editor.  

Timeliness 

The  review  should  be  completed  in  a 
timely manner.  

Quality of the manuscript 

review 

the 
The 
methodology,  as  this  may  impact  the 
validity and merit of the findings. 

focus  on 

should 

Language quality check  

The  decision  to  accept  or  reject  a 
manuscript  should  not  be  based  on 
linguistic presentation of the manuscript, 
but, 
the 
rather,  on 
methodology and the data. 

the  concept, 

Nicholas and Gordon (2011) 

Implication for grant application review 

Reviewers  should  assess  whether  a 
manuscript  is  publishable  in  principle  (in 
which  case  they  should  carry  out  a  full 
review), whether it is flawed in a way that 
cannot be fixed, or whether a manuscript 
or 
a  major 
contains 
methodological 
be 
flaw 
addressed in principle.  

theoretical 
can 
that 

Grant applications are generally reviewed 
by  multiple  reviewers 
independently, 
followed  by  a  consensus  meeting. 
Reviewers  may  change  their  views  on  a 
proposal based on the consensus meeting 
and therefore should be willing to change 
their view on acceptance/rejection during 
the meeting.  

Reviewers  should  assess  whether  the 
central  question  asked  by  the  paper  is 
interesting and relevant, and whether the 
methods  are  appropriate  to  address  the 
question. 
Reviewers should evaluate the magnitude 
of  the  advance  within  the  field  that  the 
paper provides.  

Grant applications are a lengthy process. 
During this time, new evidence may arise 
that  changes  the  focus  of  research 
proposals.  Delays  to  the  process  may 
hinder  progress  made  within  a  field  by 
making certain proposals obsolete.  

Methodology  can  be  the  sole  reason  a 
proposal is funded. For example, Proof of 
Concept  Grants  generally  have  fewer 
previous findings to support an idea, and 
therefore  funding  decisions  may  place 
greater weight on the method.  

English is the main language of working in 
research. However, many researchers will 
not  have  English  as  their  first  language 
and  use  English  only  in  a  professional 
environment.  To  ensure  fairness  in  the 
evaluation  of  research  proposals,  a 
reviewer should focus mainly on the idea 
and method rather than proper use of the 
language.  This  is  specifically  true  in  the 

68 

 
 
 
 
 
 
 
Quality of the reviewer  

A  reviewer  should  not  agree  to  assess  a 
manuscript  that  is  outside  their  field  of 
knowledge. 

context of EU-level funding that seeks to 
fund  researchers  across  all  Member 
States.  

interdisciplinary 

One  of  the  main  challenges  in  grant 
proposal evaluation is the assessment of 
multi-  and 
research 
proposals,  as  academics  tend  to  be 
knowledgeable 
field. 
in  a 
Reviewers should be open about their lack 
of knowledge in a specific field or method 
and  about  their  limitations  in  assessing 
multi-  and 
interdisciplinary  research. 
Funders  can  take  additional  steps  to 
ensure  a  fair  assessment  of  multi-  and 
interdisciplinary research.  

specific 

Transparency 

Reviewers should avoid violation of ethical 
rules  in  scientific  writing,  including  citing 
unnecessary  references;  breaches  of 
confidentiality;  and  using  ideas,  data  or 
methods without permission. 

Structure of the review 

The  review  should  start  with  a  brief 
description of the manuscript, followed by 
more  specific  comments  following  the 
IMRaD  (introduction,  method,  results, 
discussion)  format.  This  structure  helps 
the authors in responding. 

The first paragraph should state the main 
question  addressed  by  the  research  and 
summarise  the  goals,  approaches,  and 
conclusions  of  the  paper.  The  second 
paragraph of the review should provide a 
conceptual overview of the contribution of 
the paper. Read the manuscript a second 
time  and  draft  the  main  points  of  the 
review,  including  both  the  positive  and 
negative aspects of the paper. 

Usefulness of the review 

Reviewers  should  try  to  improve  the 
manuscript with their suggestions even if 
the final decision will be rejection. 

Reviewers  should  carefully  explain  their 
reasoning  and  provide  clear  evidence 
(including  citations  from  other  scientific 

Researchers  may  propose  ground-
breaking ideas in their grant applications, 
which get reviewed by other researchers 
in  their 
field.  To  avoid  researchers 
preventing  funding  to  be  awarded  to  a 
competitor,  funders  have  implemented 
the  disclosure  of  conflict  of  interest  in 
their proposal evaluation processes. 

Reviewers  should  be  provided  with  clear 
guidelines  on  how  to  structure  their 
review in order to ensure comparability of 
assessment  for  groups  discussions  (e.g. 
consensus  meetings)  and  usefulness  by 
the applicant receiving the feedback. The 
review  could  include  a  summary  box  at 
the  beginning  followed  by  specific  points 
with regard to the evaluation criteria. 

One of the main purposes of peer review 
is  learning,  both  for  reviewers  and  for 
authors/applicants. In order to maximise 
learning  to  authors/applicants,  reviewers 

69 

 
 
 
 
 
in  their 
Reviewers  should  be  polite 
suggestions  and  avoid  embarrassing  and 
humiliating 
rather, 
provide respectful and polite criticisms. 

comments,  but, 

papers  and  books)  in  support  of  their 
comments (including for rejections). 
If  making  a  criticism,  reviewers  should 
offer  concrete  and  actionable  ways  to 
address  a  problem.  Reviewers  may 
alternative 
provide 
approach.  

appropriate 

an 

should provide constructive feedback that 
is actionable and concrete.  

70 

 
 
 
 
 
 
 
 
7 

Bias in the evaluation process occurs in a range of forms across the funding 
process, from programme design to funding allocation 

Grant  application  processes  must  be  fair  and  objective  in  the  distribution  of  funding. 
However, an increasing number of studies suggest that systematic bias occurs in a range 
of forms across the grant proposal evaluation process, from programme design to funding 
distribution  (Guthrie,  Ghiga,  and  Wooding,  2018).  In  the  context  of  grant  proposal 
evaluation,  we  define  bias  as  any  difference  in  the  chance  of  success  for  individuals, 
assuming applications of the same quality, based on their characteristics (Guthrie et al., 
forthcoming). In contrast, we define ‘fairness’ of the grant proposal application as the fair 
and  equitable  treatment  of  all  applicants  and  awardees  in  the  way  in  which  funding 
processes  are  set  up  and  administered,  enabling  equality  of  opportunity  across  various 
characteristics  (i.e.  career  stage,  gender,  ethnicity  or  research  discipline)  (Guthrie, 
Rodriguez-Rincon, et al., 2019a). 

Bias  can  unfairly  confer  an  advantage  or  disadvantage  to  a  subset  of  applicants, 
undermining the robustness of the peer review process, and sometimes leading to change 
in  application  outcomes  (Tamblyn  et  al.,  2018a).  Bias  can  occur  in  relation  to  the 
applicant’s characteristics, career stage, research field, or institution (Guthrie, Rodriguez-
Rincon,  et  al.,  2019a).  It  can  manifest  itself  in  the  evaluation  criteria  for  different  calls 
(Kaatz et al., 2015), and even in the language used to advertise a call (van der Lee and 
Ellemers, 2015). 

Reviewers can also be sources of bias due to their intellectual closeness to an application 
(Gallo,  Sullivan,  and  Glisson,  2016),  anchoring  bias  (Sattler  et  al.,  2015),  or  cronyism 
(Jang et al., 2017). Additionally, conflicts of interest are potentially extreme forms of bias, 
which  are  primarily  avoided  through  self-identification  and  disclosure  (Abdoul,  Perrey, 
Tubach, et al., 2012). The formation of a panel is also subject to bias, which is compounded 
by the challenge of finding sufficient panellists (Guthrie, Rodriguez-Rincon, et al., 2019). 

Bias in the proposal evaluation process has been mainly assessed ex ante, that is, prior to 
funding being awarded. However, Mutz, Bornmann, and Daniel (2015) state that although 
ex ante analyses provide valuable information on the implementation of funding processes, 
they do not provide insight into real biases and violation of fairness in the review process. 
The authors consider that in order to assess the level of fairness of the process, ex post 
evaluation should be included, ‘because even if proposals submitted by women receive less 
favourable reviews than proposals submitted by men in the ex ante, for example, we could 
speak  of  actual  failure  to  fulfil  the  criteria  of  fairness  (i.e.  gender  bias)  only  if  women 
received  the  same  or  better  evaluations  than  men  in  the  ex  post  of  the  funded  project’ 
(Mutz, Bornmann, and Daniel, 2015). However, this approach does not consider the fact 
that biases may be present ex ante, leading to unfunded projects, which are then excluded 
from ex post assessment. Using both ex ante and ex post evaluation to determine whether 
biases  exist  in  the  FWF  funding  decision  process,  Mutz  et  al.  (2015)  found  there  were 
biases associated with scientific discipline. It is important to mention that FWF do not have 
different funding streams for different disciplines, but, rather, have one funding round to 
include  all  disciplines.  As  a  result  of  their  findings,  the  authors  proposed  that  the  FWF 
should  consider  grouping  applications  for  review  and  discussion  according  to  discipline 
(Mutz, Bornmann, and Daniel, 2015). 

8 

Funders have developed initiatives aimed at increasing fairness and equality 
of opportunity in the proposal evaluation process 

Research funders have taken multiple steps to develop fairer processes and programmes 
in recent years. For  example, many funders,  such as the  NIH, the UK Medical Research 
Council, and the ERC, now have targeted grant streams to support applicants who were 
found to be disadvantaged by biases in peer review or programme structure, such as early-
career researchers (Guthrie, Rodriguez-Rincon, et al., 2019). Other funders, such as the 
Canadian  Institutes  for  Health  Research,  have  introduced  a  training  programme  for 
reviewers,  which  was  associated  with  equal  success  rate  between  female  and  male 

71 

 
 
scientists  (Guglielmi,  2018).  In  addition,  Witteman  et  al.  (2019)  suggest  that  having 
explicit review criteria that focus on the project rather than the applicant reduces bias. 

Another approach taken to reduce bias in peer review consists of blinding the identity of 
applicants from reviewers. This approach has mainly been used in the context of journal 
peer review, although it has also been trialled for grant applications. For example, the US 
National  Science  Foundation  trialled  a  blinded  application  process  called  ‘The  Big  Pitch’, 
which  involved  applicants  submitting  an  anonymised,  two-page  research  proposal 
alongside a full conventional proposal (Bhattacharjee, 2012). In this trial applicants were 
asked to submit a two-page synopsis, which omitted the names and affiliation of applicants, 
in  addition  to  the  traditional  full  proposal.  Two  review  panels  were  assembled,  one  that 
was asked to rate traditional full proposals and another that was asked to review only the 
two-page synopsis. Each panel was asked to provide a list of their ‘high-priority’ proposals. 
The two lists differed greatly, and a comparative analysis showed that there was only ‘a 
weak  correlation’  between  the  success  outcomes  of  the  two  applications,  as  the  panels’ 
assessment strongly diverged (Bhattacharjee, 2012). 

Authors have also suggested the inclusion of a lottery system for peer review as a way to 
reduce bias. Following a study that showed that the productivity of grants is not predicted 
by  peer  review  scores,  Fang,  Bowen,  and  Casadevall  (2016)  and  Fang  and  Casadevall 
(2016)  suggested  a  modified  lottery  system  to  allocate  funding  among  meritorious 
applications. Prior to the random selection, reviewers screen applications to ensure they 
meet a minimum standard. Projects are then given numbers and selected at random by a 
computer until all the funding has been allocated. This system has been implemented by 
a number of funders, including the SNSF and the New Zealand Science for Technological 
Innovation  (SfTI)  National  Science  Challenge  (Adam,  2019).  According  to  one  of  the 
members who oversees the SfTI fund, applicants are less disappointed by rejection if their 
application  has  been  included  in  the  ballot,  as  it  means  they  were  good  enough  to  get 
funding but were unlucky at the draw (Adam, 2019). 

9 

Lack of clarity in the definition of evaluation criteria decreases reliability of 
the funding process 

Reliability refers to the extent to which the judgements of peer reviewers are consistent 
with one another, in terms of both the consistency of scoring between different reviewers 
for a single proposal and the consistency of scoring by an individual reviewer over different 
reviews.  Consistent  scoring  by  reviewers  is  important  in  enabling  fairness  across  the 
process (Guthrie, Rodriguez-Rincon, et al., 2019a). However, studies have found that the 
level  of  consistency  within  the  peer  review  process  is  highly  variable  (Fogelholm  et  al., 
2012; Jayasinghe, Marsh, and Bond, 2003; Jerrim and de Vries, 2020; Pier et al., 2018). 
Variation in reviewer scores may be due to a lack of clarity around the scoring process or 
criteria (Abdoul, Perrey, Amiel, et al., 2012; Sattler et al., 2015; Gregorius et al., 2018); 
reviewer  bias  (Jerrim  and  de  Vries,  2020);  reviewer  discipline  (Lobb  et  al.,  2013);  or 
reviewer experience (Jayasinghe, Marsh, and Bond, 2003). Nonetheless, as peer review is 
based on expert opinion, it may be that two experts have genuine differences of opinion 
and that, in fact, differences of opinion may suggest an innovative aspect in the proposal 
(Guthrie, Rodriguez-Rincon, et al., 2019b). 

A study that assessed the funding process for 10 UK-Africa consortia, found that there was 
substantial  variation  in  the  way  different  applications  were  presented  and  therefore 
discussed by panel members (Gregorius et al., 2018). Although most panel members spoke 
to several assessment criteria, not all criteria were discussed for each application. This was 
due to lack of guidance on the format for presenting applications. Following introduction of 
more detailed guidance, there was less variation in the next round with regard to the focal 
areas of the discussions on scientific strength (Gregorius et al., 2018). 

In the context of H2020, different actions have variations on the evaluation criteria to be 
considered, in terms of both focus and wording. For example, reviewers for ERC grants are 
instructed to focus solely on ‘Excellence’ for the selection process. Although some guidance 

72 

 
 
is provided as to what constitutes ‘Excellence’, one study found that the guidance is not 
operationalised  (van  den  Besselaar,  Sandström,  and  Schiffbaenker,  2018).  The  lack  of 
clarity is intentional, as being more prescriptive might constrain reviewers (who are leading 
scholars in their field and will therefore know how best to assess the proposed research) 
in  their  freedom  to  do  the  selection  in  their  own  way.  Nonetheless,  the  lack  of  clear 
guidance  can  lead  to  reviewers  providing  arguments  that  are  contrary  to  the  suggested 
ERC review guidelines. A second study, assessing interpretation of the assessment criteria 
for the ERC, found that only half of the interviewed panellists for the study thought there 
was  a  distinct  difference  between  excellence  and  frontier  research  (Luukkonen,  2012), 
which may inherently lead to varying reviews. 

However,  variation in reviewer  scores is not necessarily a bad thing, as it may reflect a 
high level of innovation in the proposal  reviewed (as we  explain in further detail below) 
(Linton, 2016; Giraudeau et al., 2011). Therefore, funders may seek to identify the source 
of variation and address variation that refers to bias or a lack of clarity on scoring criteria, 
while promoting variation that arises from a perception of the proposal being high risk. 

10  The  use  of  group  discussion  can  increase  reliability  of  the  peer  review 

process 

In order to address inter-reviewer discrepancies, many R&I funders have introduced a form 
of  reviewer discussion (i.e. panel review meetings and consensus  meetings) at different 
stages  of  the  process.  For  example,  for  its  three  main  funding  schemes  (Starting, 
Advanced, and Consolidator Grants), the ERC use a two-step evaluation process, with the 
panels meeting in both steps. In addition being reviewed by panellists, in the second step 
of  the  evaluation,  proposals  are  also  reviewed  by  external  experts.  In  this  step,  panel 
reviews  and  external  reviews  are  then  discussed  at  each  panel  in  order  to  reach  a 
conclusion on funding (Luukkonen, 2012). 

Although  meeting  discussions  are  frequently  used  in  funding  processes,  there  are 
conflicting findings on the usefulness of meeting discussion. One study found that meeting 
discussions can have a positive effect on 13% of applications (Martin, Kopstein, and Janice, 
2010). However, other studies have not been able to discern an overall improvement in 
agreement rates (Pina, Hren, and Marušić, 2015; Fogelholm et al., 2012). In the case of 
the ERC, panels usually achieve consensus through deliberation and intellectual discussions 
(Luukkonen,  2012).  However,  panels  use  different  means  to  achieve  consensus,  with 
varying styles of deliberation, and different ways of voting as a means to reach agreement 
(Luukkonen, 2012). 

The ERC’s use of multiple reviewers is consistent with evidence on the optimal number of 
reviewers  to  ensure  consistency  in  the  process.  A  study  by  Snell  (2015)  evaluated  the 
review  process  from  a  Canadian  Institutes  of  Health  Research  post-doctoral  fellowship 
programme  and  found  that  the  optimal  number  of  reviewers  was  five  in  order  to  avoid 
random effects of reviewer scoring. However, as noted earlier, there is a limited pool of 
reviewers, which, together with a limit on the number of times reviewers can serve on a 
panel (Luukkonen, 2012), may present a problem for the long term if a high number of 
reviewers continue to be used. 

An  alternative  to  panel  review  meetings  is  consensus  meetings,  which  are  used,  for 
example,  in  the  Marie  Skłodowska-Curie  Actions  (MSCA)  in  European  Commission’s 
Seventh  Framework  Programme  (FP7)  (Pina,  Hren,  and  Marušić,  2015).  In  the  MSCA 
evaluation  process,  there  was  no  common  panel  assessing,  but,  rather,  small  groups  of 
three experts who reviewed and discussed their evaluation for each individual proposal. A 
study  of  the  MSCA  process  found  that  agreement  among  reviewers  was  generally  high 
when comparing the scores from their individual assessment with the scores reached after 
the consensus meeting discussion (Pina, Hren, and Marušić, 2015). However, agreement 
was less high for proposals with lower scores. In addition, disagreement at the consensus 
meeting stage influenced the final score of a proposal negatively, resulting in lower scores 
(Pina,  Hren,  and  Marušić,  2015).  This  may  be  a  disadvantage  to  innovative  proposals, 

73 

 
 
which tend to lead to higher levels of disagreement among scoring (Guthrie, Rodriguez-
Rincon, et al., 2019b). 

11  Peer review is described as ‘anti-innovation’ 

Innovative research is difficult to define, and therefore designing a system that seeks to 
fund innovation can be a challenge (Guthrie, Rodriguez-Rincon, et al., 2019a). Approaches 
to identify and define innovation are varied and include defining innovation based on expert 
opinion (Marks, 2011) or as research that does not require extensive preliminary results 
(Spier, 2002). 

Many  have  described  peer  review  as  ‘anti-innovation’,  providing  strong  incentives  for 
incremental  research  and  discouraging  research  into  new,  unexplored  approaches 
(Langfeldt and Kyvik, 2011; Azoulay, Graff Zivin, and Manso, 2013; Fang and Casadevall, 
2016;  Guthrie,  Ghiga,  and  Wooding,  2018).  Novel  research  approaches  are  considered 
risky and may be more likely to have unsuccessful outcomes (Bornmann and Marx, 2012). 
Therefore, reviewers face the risk of losing their reputation by not highlighting weaknesses 
in innovative proposals (Tversky and Kahneman 1991). In addition, innovative proposals 
from young researchers are at risk of greater disadvantage, since the applicant will lack 
the track record to support funding the risk of the novelty of the research (Guthrie, Ghiga, 
and  Wooding,  2018).  Low  success  rates  of  the  funding  process  may  exacerbate  the 
situation, leading to conservatism also among applicants, reviewers and funders (Guthrie, 
Ghiga, and Wooding, 2018). 

To systematise the emphasis on innovation and therefore reduce conservatism, Liaw et al. 
(2017) propose a formal assessment criterion on innovation as part of the decision-making 
process.  Such  a  formal  assessment  criterion  could  include  such  things  as  whether  the 
project  employs  novel  technologies  or  techniques.  Such  a  criterion  is  already  used  by 
several funders. For example, the NHMRC’s Ideas Grant, which seeks to support innovative 
research, has included a criterion on ‘Innovation and creativity’, which accounts for 25% 
of  the  total  score  for  a  proposal.  In  this  context,  the  NHMRC  define  ‘Innovation  and 
creativity’  as  ‘health  and  medical  research  that  seeks  to  challenge  and  shift  current 
paradigms  and/or  have  a  major  impact  on  a  health  research  area  through  one  or  more 
studies  that:  develop  or  use  novel  and  creative  research  concepts,  approaches, 
methodologies,  technologies  or  interventions;  propose  a  reinterpretation,  refinement, 
improvement  or  new  application  of  existing  theoretical  concepts,  approaches, 
methodologies,  technologies  or  interventions;  or  integrate  and  adapt  concepts, 
methodologies, technologies and approaches from other research fields or disciplines for a 
new purpose or in a new way’ (Guthrie, Rodriguez-Rincon, et al., 2019, 29). However, we 
are not aware of any evaluations testing the effectiveness of this approach in supporting 
innovative R&I. 

Azoulay, Graff Zivin and Manso (2009) suggest that the novelty of keywords used to tag 
the publications of the  applicants relative to the  field, cited literature,  or the applicant’s 
previous work as a possible measure to assess conservatism, which can be used to assess 
an  applicant’s  track  record  for  innovative  research.  The  authors  also  identify  change  in 
research agenda of the PI after the funding being awarded as an indicator of innovation 
(Azoulay, Graff Zivin, and Manso, 2009).  However, this  indicator is difficult to assess at 
the proposal stage and would be more useful in an ex post assessment. 

Bespoke approaches to the evaluation of innovative research have been introduced with 
the  aim  of  increasing  funding  innovation,  but  evidence  of  their  effectiveness  is  limited. 
Funding innovative proposals is important for technological change and economic growth 
(Braben,  2004)  because,  although  they  tend  to  be  considered  high  risk,  they  may  also 
offer higher rewards. As a result, many funders aim to support innovative research as part 
of their funding portfolio (NHMRC, n.d). 

In recent years, different strategies have been developed to improve the assessment of 
innovation in grant proposal evaluation. For example, the NIH in the USA adopted an ‘out-

74 

 
 
of-order funding’ approach, in which a number of applications for innovative research are 
chosen for funding despite receiving lower scores than other funded research based purely 
on  the  peer  review  process  (Lindner  and  Nakamura,  2015).  The  authors  found  that 
approximately 15% of funded applications were selected ‘out of order’, meaning that 15% 
of the funded applications included in the study would not have been funded if the decision 
had  purely  been  based  on  peer  review  scores.  In  addition  to  the  out-of-order  funding 
approach,  the  NIH  introduced  various  changes  to  the  peer  review  process  aimed  at 
increasing the focus on innovation (Lindner et al., 2016), including reducing the length of 
the methodological proposal from 25 to 12 pages. Including innovation as a criterion for 
grant  assessment  could  incentivise  researchers  to  include  innovative  ideas  and  new 
approaches  into  their  proposals  (Guthrie  et  al.,  2018).  To  this  effect,  the  NIH  asked 
applicants to detail what makes their application innovative. In addition, the NIH reduced 
the  range  of  scoring  options  available  to  reviewers  to  five  criteria  (‘Significance’, 
‘Investigators’,  ‘Innovation’, ‘Approach’, and  ‘Environment’) as  well as having an overall 
impact score. The authors found that all the scored review criteria, including ‘Innovation’, 
were related to the overall impact score, and that good scores were necessary on all five 
scored review criteria to achieve a good overall impact score. 

Many  funding  agencies  have  also  adopted  the  strategy  of  having  a  separate  scheme  to 
fund  innovative  research,  allocating  smaller  funds  with  a  shorter  time  frame  to  these 
specific streams, as suggested by Alberts (2009). For example, the Dutch organisation for 
health research and development, ZonMw, designed an ‘off-road’ programme seeking to 
fund  high-risk,  high-reward  projects,  providing  EUR  100 000  for  1.5  years  (Guthrie, 
Rodriguez-Rincon, et al., 2019a). 

12  Evaluating  multi-  and  interdisciplinary  research  and  innovation  presents 
specific  challenges  and  requires  adaptation  of  the  process  to  ensure 
evaluation is adequate 

The extent to which an applicant’s research crosses disciplinary boundaries can potentially 
indicate  innovativeness  in  a  funding  application.  Oviedo-García  (2016)  distinguishes 
among multi-, trans-, and interdisciplinary research. Based on a publication by Lyall et al. 
(2011), Oviedo-García defines multidisciplinary research as a juxtaposition of disciplines 
that lacks interchanges or the integration that characterises interdisciplinary research. The 
author defines transdisciplinary  research  as  research that represents  a different  form  of 
knowledge  production  (concepts  and/or  methodological  frameworks),  which  draws  on 
expertise  from a wide range of  organisations and collaborative partnerships and  defines 
interdisciplinary research as the collaboration and the integration of concepts and methods, 
which may in turn lead to the generation of new concepts and knowledge (Oviedo-García, 
2016). However, the issues related to the assessment of cross-disciplinary proposals are 
similar, if not the same, across multi-, trans-, and interdisciplinary research, and therefore 
we will use the commonly used term ‘multi- and interdisciplinary research and innovation’, 
or MIDRI, to refer to all three types of research in this study. 

MIDRI  has  the  ability  to  address  the  complex  research  questions  facing  society  and  the 
potential  to  deliver  societal  benefits  that  single-field  research  cannot,  by  combining 
expertise  from  different  fields  to  tackle  a  research  question  (Davé  et  al.,  2016). 
Additionally,  by  bringing  together  different  fields,  MIDRI  can  reach  a  wider  audience, 
potentially leading to higher impact (Gold, 2016). 

However, the current research culture remains discipline-oriented, which poses challenges 
for  funding,  conducting,  and  assessing  MIDRI  (McLeish  and  Strang,  2016).  Those 
evaluating MIDRI are faced with lack of consensus on how to objectively evaluate MIDRI 
(McLeish and Strang, 2016). Peer reviewers are usually experts from a single discipline, 
and therefore creating MIDRI peer review panels and allocating reviewers for proposals is 
challenging. 

The suggested frameworks for evaluation of MIDRI proposals have generally attempted to 
identify the holistic structures of MIDRI through the formulation of questions to address in 

75 

 
 
relation to the proposal, project or output. McLeish and Strang (2016) summarise examples 
of good practice in the evaluation of MIDRI, presented in Box 4. 

Box 4: Good practice in the evaluation of MIDRI (adapted from McLeish and Strang, 2016) 

  Reviewers should communicate throughout the review process and produce a single assessment 

of an MIDRI proposal rather than producing individual assessments. 

 

 

 

Applicants should be able to address comments by individual reviewers prior to the proposal being 
discussed or ranked by a panel. 

Panels should include user-community or other non-academic reviewers. 

The process for evaluation of MIDRI proposals should be a one-stage review process. 

  Reviewers  and  panel  members  should  communicate  to  ensure  that  they  individually  and 

 

 

 

 

 

 

corporately understand the process and criteria of MIDRI. 

Track record of the applicants should be assessed for MIDRI experience. 

The evaluation process for MIDRI should avoid reliance on quantitative publication measures, such 
as citation rates and impact factors. 

The evaluation process should probe beyond the research proposals or programmes themselves 
to  consider  the  support  and  development  structures,  such  as  centres  and  institutes,  of  the 
institution(s) in which the research will be pursued. 

There  should  be  one  or  more  explicitly  interdisciplinary  panels,  either  welcoming  any  MIDRI-
flagged  outputs  or  focusing  on  integrative  topics,  such  as  ‘energy’,  ‘security’,  ‘global  policy’, 
‘biophysical sciences’. 

Funders should identify and deploy a pool of panel members with strong interdisciplinary expertise 
and experience, either within a focused MIDRI panel and/or with members on all subject panels. 

In  the  case  of  interdisciplinary  research  (IDR)  with  non-academic  partners,  there  should  be 
structures for evaluation that combine the categories of ‘output’ and ‘impact’ in ways that respond 
to the non-linear nature of MIDRI that involves partners outside of academia. 

  When evaluating the research environment of institutions, funders should include an explicit and 

detailed examination of the incentives for building MIDRI. 

Some funders have modified their  evaluation process to account for the challenges that 
evaluating MIDRI present by introducing MIDRI-specific funding schemes and/or changes 
to their assessment processes. For example, the NHMRC have the Synergy Grants, which 
seek to  support outstanding multidisciplinary teams of investigators to work together to 
answer  major  questions  that  cannot  be  answered  by  a  single  investigator  (Guthrie, 
Rodriguez-Rincon,  et  al.,  2019).  In  addition,  the  NHMRC  use  an  evaluation  process  for 
Synergy  Grants  in  which  individuals  with  broad  expertise  are  used  to  assess  criteria  of 
‘Synergy’  and  ‘Knowledge  gain’,  while  discipline-specific  experts  are  used  to  assess  the 
track record of each chief investigator (Guthrie, Rodriguez-Rincon, et al., 2019). 

In  the  case  of  the  ERC,  there  are  no  interpretative  and  customary  rules  about  the 
assessment  of  MIDRI  proposals;  instead,  they  are  subject  to  a  separate  evaluation 
according to the criteria of each discipline to which they are related and are evaluated by 
experts within each of these related areas (Lamont, 2009; Huutoniemi et al., 2010). Thus, 
the boundaries, criteria and considerations of each established discipline still determine the 
outcome in the evaluation of interdisciplinary proposals (Luukkonen, 2012). In 2010, the 
ERC reserved a specific procedure for clearly MIDRI proposals, accounting for 13% of total 
funds to be awarded (Luukkonen, 2012). In this process, MIDRI proposals that passed the 
threshold for the potentially supported projects were examined and deliberated upon by a 
panel comprised of the chairpersons of the domain-specific panels. In a qualitative study 
of the ERC funding process, Luukkonen (2012) found that highly ranked MIDRI proposals 
were  assessed  and  selected  for  funding  within domain-specific  panels  –  standing  out  as 
excellent work regardless of being MIDRI. However, the less highly ranked MIDRI proposals 
were more likely to be considered difficult to assess and sent to the interdisciplinary final 
panel  consisting  of  the  panel  chairs.  This  suggests  that  experts  in  the  domain-specific 
panels did not necessarily ‘downgrade’  MIDRI  proposals and could still identify excellent 
research when it fell across disciplinary boundaries. 

76 

 
 
 
 
13  The ability of the peer review process to predict successful research projects 
(i.e.  predictive  power)  is  limited  by  the  reviewers’  understanding  of  the 
evaluation criteria 

Predictive power refers to the extent to which the peer review process funds the ‘best’ or 
‘more successful’ research. These terms can be defined in many ways, and the definition 
will depend on the discipline in question and the aims of the specific funding call. These 
terms are often used to refer to research that produces multiple or highly cited publications, 
patents,  technology  development,  or  data  sharing  among  the  research  community.  The 
peer review process can usually identify the highest-quality 20–30% of grant applications, 
but it is less well suited to identifying the top 10% (Fang and Casadevall, 2012), meaning 
that,  as  success  rates  fall,  peer  review  becomes  less  effective  in  ensuring  the  best 
applications are funded. Several studies have challenged the idea that peer review is able 
to  select  the  best  research  (Danthi  et  al.,  2015;  Doyle  et  al.,  2015;  Fang,  Bowen,  and 
Casadevall, 2016; Kaltman et al., 2014; Reinhart, 2009). However, these studies consider 
successful  research  to  be  that  which  leads  to  good  bibliometric  performance,  despite 
evidence that bibliometric analysis does not cover all disciplines equally, as it only provides 
an indication of the academic output of research as measured by journal articles (Belter, 
2015). There are very few studies using alternative measures of project outcomes. 

The predictive power of the peer review process is limited by multiple factors. For example, 
track record is commonly used as a proxy for predicting successful outcomes of research. 
However, assessment of track record as an evaluation criterion must account for the fact 
that  previous  funding  has  an  impact  on  future  success  of  an  applicant.  Track  record  is 
commonly associated with bibliometric performance, which is also used as a measure of 
outcome in assessments of predictive power  (Belter, 2015). The reliance on bibliometric 
analysis  varies  by  discipline  (Luukkonen,  2012).  For  example,  representatives  of  the 
sciences and engineering from ERC panels were more inclined to use quantitative measures 
of excellence, such as bibliometric indicators, to judge the excellence of the applicant than 
those  in  the  humanities  (Luukkonen,  2012).  Therefore,  the  use  of  discipline-specific 
evaluation criteria may be required for funding schemes that comprise multiple disciplines. 
In addition, because the definition of success may vary depending on the funder and field 
of research, bibliometric analysis may not be the best measure in some cases. However, 
in funding streams that are primarily aiming to deliver research excellence and academic 
outcomes, use of bibliometric analysis may be appropriate. In addition, in the absence of 
alternative outcome measures that can be consistently applied across a large number of 
awards, bibliometrics provide our only way to assess predictive power of funding decisions 
at present. 

14  Transparency may be related to the design of the funding process, but this 

relationship has not been measured extensively 

Transparency can be defined as the extent to which information associated with the process 
is  disclosed  (Ho  et  al.,  2013;  Murray  et  al.,  2016a;  Tamblyn  et  al.,  2018b;  Gurwitz, 
Milanesi, and Koenig, 2014). Transparency can also refer to design aspects of the process, 
such as the use of panels to assess applications (rather than just individual peer reviewers) 
(Miner,  2011;  Mow,  2010).  Although  experts  who  participate  in  panels  understand  the 
benefit of panel discussions, the wider academic community may have less understanding 
of these processes and hence see panel discussions as a ‘black box’, which may be random 
or  biased  (Mow,  2010).  Moreover,  there  is  the  concept  of  ‘self-service’,  in  which 
researchers who participate in panel reviews can learn the system and therefore be able 
to  write  to  more  successful  grant  applications  of  their  own  (van  den  Besselaar,  2012). 
Blinding  might  also  seem  counter  to  the  idea  of  transparency,  but  it  is  suggested  that 
hiding the identity of the applicants helps to focus the review exclusively on the merits and 
characteristics of the proposed research and drive a more transparent and application-led 
review  process  (Solans-Domènech  et  al.,  2017).  It  should  be  noted  that  neither  panel 
review nor blinding have been shown experimentally to deliver transparency benefits. 

77 

 
 
Increased transparency of the review process could impact the perceived burden (Gurwitz, 
Milanesi,  and  Koenig,  2014)  because  it  would  allow  applicants  to  better  understand  the 
process that has led to the outcome of their application and it would allow  reviewers to 
obtain recognition for their work. 

Few studies have actively attempted to measure transparency of funding processes. We 
identified two approaches in the literature. One study analysed transparency by looking at 
whether,  when  and  to  whom  information  regarding  different  elements  of  the  funding 
process, along with its outcomes, were made available by different funding organisations 
(Gurwitz,  Milanesi,  and  Koenig,  2014).  To  do  this,  the  study  categorised  a  range  of 
‘knowledge  items’,  which  included  a  description  of  the  evaluation  procedure;  evaluation 
details (questions to the reviewers); names of applicants; names of funded PIs; proposals 
submitted  to  call;  proposals/lay  abstracts  of  funded  projects;  names  of  reviewers  (at 
minimum, in aggregated form); review summaries of all proposals; review summaries of 
funded  proposals;  funding  budget  granted  to  projects;  call  success  rate;  ranking  of 
proposals;  and  final  report.  They  then  grouped  these  into  the  following  four  categories: 
rules, content, expertise and produce. Using comparative analysis, funders were analysed 
to assess the extent to which these different elements are made available – open, restricted 
or closed – and if so, when (i.e. before, after or during the procedure). Another study  – 
looking  at  journal  rather  than  funding  peer  review,  but  using  a  transferable  approach  – 
surveyed  individuals  subject  to  peer  review  (in  this  case,  authors  hoping  to  have  their 
article published) (Ho et al., 2013). The survey aimed to capture the views of academics 
on  the  journal  review  process  generally,  but  included  a  question  asking  respondents  to 
rate transparency on a Likert scale. 

78 

 
 
 
CHAPTER 5: ANALYSIS OF THE H2020 PROPOSAL EVALUATION 

SYSTEM 

This chapter sets out our analysis of the H2020 proposal evaluation system. It consists of 
an overview of work across the following four tasks: 

  analysis of proposal scoring data for H2020 applications provided by DG RTD, 

  analysis of data from a survey of evaluators conducted by DG RTD, 

  data from a new survey we have conducted capturing information from applicants 

and evaluators on the burden of the proposal evaluation process, 

  analysis of the transparency of H2020 processes conducted based on desk research. 

In each case, we set out in detail the evidence and data generated by each of these tasks. 

1 

Proposal scoring data 

This section presents the findings to date from our analysis of the proposal scoring data 
for H2020. 

1.1  Proportion of reviewers by characteristic 

For the actions outlined in Table 7, our analysis shows that the majority of reviewers across 
all  actions  are  from  the  EU  (approximately  70%  EU  reviewers  for  all  actions),  with  a 
minority of reviewers (14.1%) being from widening participation group Member States15. 
In  terms  of  gender,  there  is  an  overall  predominance  of  male  reviewers  (Figure  6). 
However, at the action level, there is a more even gender balance for certain actions, such 
as  SME  (53%  male  and  47%  female  reviewers),  CSA  (53.8%  male  and  46.2%  female 
reviewers)  and  MSCA  (55.5%  male  and  44.5%  female  reviewers).  There  is  also  an 
approximately even distribution of reviewers over and under 50 years old for most actions, 
except  for  ERC  (63.4%  over  50  years  old)  and  for  actions  grouped  under  the  ‘other’ 
category (68.3% over 50 years old). In terms of affiliation, there is a higher proportion of 
reviewers affiliated with a higher or secondary education establishment across most actions 
except SME actions for which reviewers are predominantly affiliated with private for-profit 
organisations (51.4%) in comparison with other types of affiliation (Figure 6). 

15  The widening participation  group EU  countries are: Bulgaria, Croatia, Cyprus,  Czechia, Estonia, 
Hungary, Latvia, Lithuania, Luxembourg, Malta, Poland, Portugal, Romania, Slovakia and Slovenia. 

79 

 
 
                                                 
Figure 6: Summary of reviewer characteristics 

To  assess  whether  there  are  associations  between  the  demographic  characteristics 
provided in the data, we performed LCA. This statistical technique enables identification of 
patterns  in  the  data  and  determination  of  ‘profiles’  (latent  classes)  consisting  of  similar 
associations of characteristics. This methodology allows for a broader overview of the data, 
and it provides a means of reducing its complexity for further statistical analysis. 

LCA is more straightforward to interpret with binary input variables; therefore, we grouped 
reviewer affiliation as private sector versus non-private sector, as this provided the most 
balanced  sample  for  the  LCA.  The  sample  size  of  the  category  ‘organisation  type  not 
defined’ was not sufficient to consider it as a separate group, and therefore it was included 
in the ‘non-private sector’ group. Here, the category private sector corresponds to private 
for-profit entities, whereas the category non-private sector includes higher or secondary 
education  establishments,  research  organisations,  public  bodies,  and  the  category 
‘organisation type not defined’. 

We identified two classes of  reviewers across the H2020 actions analysed, distinguished 
by gender and age (Figure 7 and Table 19). 

1.  ‘Predominantly  male  over  50  years  old’  class:  These  reviewers  have  a  68% 
probability  of  being  male,  a  88%  probability  of  being  over  50  years  old,  a  69% 
probability of being EU nationals and a 85% probability of being affiliated with an 
organisation outside the private sector. 

2.  ‘Gender-diverse below 50 years old’ class: These reviewers have a 56% probability 
of  being  male,  a  89%  probability  of  being  younger  than  50  years  old,  a  73% 
probability of being EU nationals and a 81% probability of being affiliated with an 
organisation outside the private sector. 

Reviewers taking part in the H2020 actions assessed have a 56% probability of belonging 
to the ‘predominantly male over 50 years old’ class and a 44% probability of belonging to 
the ‘gender-diverse below 50 years old’ class. 

80 

 
 
Figure 7: Latent class analysis of reviewer data 

Table 19: Class membership and item response probabilities for reviewers 

Category 

Gender-diverse below 50 years old 
class 

Predominantly male over 50 
years old class 

Class membership 

Male 

Over 50 years old 

Non-EU 

Non-private organisation 

0.44 

0.56 

0.11 

0.27 

0.81 

0.56 

0.68 

0.88 

0.31 

0.85 

1.2  Proportion of applicants by characteristic 

Analyses on applicant characteristics were performed solely on MSCA IF, ERC and SME, as 
these are the main mono-beneficiary grants and therefore the data held on the applicant 
correspond to the beneficiary (for other H2020 actions, the data held corresponds mainly 
to the coordinator of the application rather than all beneficiaries). Our analysis shows that 
there is a clear imbalance in the gender for MSCA IF, ERC and SME actions (Figure 8), with 
71.3%  of  ERC  applicants  being  male  and  28.7%  female,  and  82.6%  of  SME  applicants 
being male and 17% female (Table 20). For MSCA IF actions, 59.2% of applicants were 
male  and  40.8%  were  female.  There  is  also  an  imbalance  with  regard  to  nationality  of 
applicants (Figure 8), with the majority being EU nationals, as would be expected (52.4% 
of MSCA IF applicants, 69.4% of ERC applicants, and 75.1% of SME applicants) (Table 20). 
The minority of EU applications received were from widening participation group Member 
States16  (8.4%  of  MSCA  IF  applicants,  6.7%  of  ERC  applicants,  and  15.6%  of  SME 

16  The widening participation group  EU  countries are: Bulgaria, Croatia, Cyprus, Czechia,  Estonia, 
Hungary, Latvia, Lithuania, Luxembourg, Malta, Poland, Portugal, Romania, Slovakia and Slovenia. 

81 

 
 
 
 
                                                 
applicants).  In  terms  of  affiliations,  most  applicants  to  MSCA  IF  and  ERC  come  from 
higher/secondary education establishments (76.2% of MSCA IF applicants and 76.9% of 
ERC  applicants),  whereas  for  SME  most  applicants  come  from  private  for-profit 
organisations  (97.6%)  (Figure  8  and  Table  20),  reflecting  the  objectives  and  target 
audiences of the different actions. 

Figure 8: Summary of applicant characteristics 

82 

 
 
 
 
Table 20: Summary of applicant characteristics 

Variable 

Levels 

MSCF IF 

ERC 

SME instrument 

 n 

64 519 

57 185 

65 059 

Female 

26 333 (40.8) 

16 438 (28.7) 

11 079 (17.0) 

Gender (%) 

Male 

38 185 (59.2) 

40 747 (71.3) 

53 753 (82.6) 

Missing 

1 (0.0) 

0 (0.0) 

227 (0.4) 

EU Member States 

33 780 (52.4) 

39 683 (69.4) 

48 886 (75.1) 

Nationality (%) 

EU-associated 
countries 

4 509 (7.0) 

4 860 (8.5) 

8 251 (12.7) 

Third countries 

26 230 (40.7) 

12 642 (22.1) 

7 922 (12.2) 

EU category (%) 

Organisation (%) 

Widening 
participation  group 
Member States 

Other 
States 

Member 

Research 
organisations 

Public 
bodies 
(excluding  research 
and 
organisations 
secondary  or  higher 
education 
establishments) 

for-profit 
Private 
entities 
(excluding 
higher  or  secondary 
education 
establishments) 

Higher  or  secondary 
education 
establishments 

5 437 (8.4) 

3 847 (6.7) 

10 140 (15.6) 

28 343 (43.9) 

35 836 (62.7) 

38 746 (59.6) 

12 701 (19.7) 

12 057 (21.1) 

0 (0.0) 

251 (0.4) 

191 (0.3) 

      407 (0.6) 

2 240 (3.5) 

854 (1.5) 

63 467 (97.6) 

49 152 (76.2) 

43 997 (76.9) 

509 (0.8) 

Other 

175 (0.3) 

86 (0.2) 

676 (1.0) 

As  with  the  reviewer  data,  we  assessed  potential  associations  between  demographic 
characteristics in applicants using the LCA method, using the same affiliation segmentation 
of private sector versus non-private sector. Our analysis shows that there are three main 
latent classes of applicants, distinguished mostly by gender and nationality (Figure 9 and 
Table 21). 

83 

 
 
 
 
 
1.  ‘Predominantly  male  non-EU’  class:  These  applicants  have  a  91%  probability  of 
being male, a 41% probability of being non-EU nationals and an 87% probability of 
being affiliated with an organisation outside the private sector. 

2.  ‘Gender-diverse’ class: These applicants have a 58% probability of being male,  a 
70% probability of being EU nationals and a 98% probability of being affiliated with 
an organisation outside the private sector. 

3.  ‘Predominantly  male  and  EU’  class:  These  applicants  have  a  77%  probability  of 
being male, a 90% probability of being EU nationals and a 70% probability of being 
affiliated with an organisation outside the private sector. 

Applicants taking part in the H2020 actions assessed have a 28% probability of belonging 
to  the  ‘predominantly  male  non-EU’  class,  33%  probability  of  belonging  to  the  ‘gender-
diverse’ class and 39% probability of belonging to the ‘predominantly male and EU’ class. 

Figure 9: Latent class analysis of applicant data 

Table 21: Class membership and item response probabilities for applicants  

Category 

Predominantly male 
non-EU class 

Gender-diverse class 

Predominantly male EU 
class 

Class membership 

Male 

Non-EU 

Non-private 
organisation 

0.28 

0.91 

0.41 

0.87 

1.3  Success rate by characteristic 

0.33 

0.58 

0.30 

0.98 

0.39 

0.77 

0.10 

0.70 

The overall success rate for H2020 was 11.4%. Success rates for the different actions in 
Table 22 are higher than expected given the overall success rate for H2020. This is due to 
differences  in  the  two  data  sets  held  for  the  overall  programme  applicants  and  the 
proposals assessed by actions in terms of the proposal numbers provided. Because of these 

84 

 
 
 
 
differences, we were unable to reconcile unsuccessful proposals across the  data set and 
provide and accurate success rate by action, except for ERC and MSCA IF. 

Table 22: Success rate by action 

Action 

ERC 

SME instrument 

IA 

RIA 

CSA 

MSCA IF 

Other 

Success rate (%) 

12.84 

8.76 

16.35 

13.57 

28.4 

14.85 

73.38 

Success  rates  by  applicant  characteristics  were  assessed  for  mono-beneficiary  actions 
(MSCA IF, ERC and the SME instrument). Success rates were similar for male and female 
applicants for MSCA IF, ERC and the SME instrument (Table 23). Greater variation is seen 
across actions with regard to nationality (Table 24). For ERC grants, EU applicants were 
found to have a success rate of 12.71%, whereas applicants from EU-associated countries 
and third countries had a success rate of 15.66% and 12.15%, respectively. For MSCA IF, 
applicants from EU-associated countries had lower success rates (11.13%) than applicants 
from  the  EU  (16.96%)  or  third  countries  (12.77%).  For  the  SME  instrument,  applicants 
from  EU-associated  countries  had  the  highest  success  rate  (10.27%),  compared  with 
applicants  from  the  EU  (8.71%)  and  third  countries  (7.43%).  For  all  three  actions, 
applicants from widening participation group Member States17 had lower success rates than 
applicants  from  other  Member  States  (10.37%  success  rate  for  applicants  to  ERC  from 
widening participation group Member States, compared with 12.96% for applicants from 
other  Member  States;  14.11%  success  rate  for  applicants  to  MSCA  IF  from  widening 
participation  group  Member  States,  compared  with  17.51%  for  applicants  from  other 
Member States; and 5% success rate for applicants to the SME instrument from widening 
participation  group  Member  States,  compared  with  9.69%  for  applicants  from  other 
Member  States).  Similarly,  success  rates  according  to  applicant  affiliation  also  varied 
across actions (Table 25), with research organisations (14.74%), public bodies (21.11%), 
and private for-profit organisations (8.85%) having the highest success rate for ERC, MSCA 
IF and the SME instrument, respectively. In contrast, applicants from the ‘other’ category 
had the lowest success rate across ERC (4.6%), MSCA IF (13.14%) and the SME instrument 
(3.99%).

17  The widening participation group  EU  countries are: Bulgaria, Croatia, Cyprus, Czechia, Estonia, 
Hungary, Latvia, Lithuania, Luxembourg, Malta, Poland, Portugal, Romania, Slovakia and Slovenia. 

85 

 
 
 
                                                 
 
Table 23: Success rate by applicant gender (mono-beneficiary grants only)18 

Action 

MSCA IF 

MSCA IF 

ERC 

ERC 

SME instrument 

SME instrument 

Gender 

Female 

Male 

Female 

Male 

Female 

Male 

Success rate (%) 

15.84 

14.17 

12.58 

12.94 

8.02 

8.93 

18 ERC grants are excluded from this analysis because there were no gender data available. 

86 

 
 
 
                                                 
Table 24: Success rate by applicant nationality (mono-beneficiary grants only) 

Action 

Nationality 

Success rate (%) 

ERC 

ERC 

ERC 

ERC 

ERC 

EU Member State 

Widening  participation  group 
Member State 

Other Member State 

EU-associated country 

Third country 

MSCA IF 

EU Member State 

MSCA IF 

MSCA IF  

MSCA IF  

MSCA IF 

Widening  participation  group 
Member State 

Other Member State 

EU-associated country 

Third country 

SME instrument 

EU Member State 

SME instrument 

Widening  participation  group 
Member State 

SME instrument 

Other Member State 

SME instrument 

EU-associated country 

SME instrument 

Third country 

12.71 

10.37 

12.96 

15.66 

12.15 

16.96 

14.11 

17.51 

11.13 

12.77 

8.71 

5 

9.69 

10.27 

7.43 

87 

 
 
 
Table 25: Success rate by applicant affiliation (mono-beneficiary grants only) 

Action 

Organisation type 

Success rate (%) 

ERC 

ERC 

ERC 

ERC 

ERC 

MSCA IF 

MSCA IF 

MSCA IF 

MSCA IF 

MSCA IF 

Higher  or  secondary  education 
establishments 

Research organisations 

Private for-profit entities  

Public bodies  

Other 

Higher  or  secondary  education 
establishments 

Research organisations 

Private for-profit entities  

Public bodies  

Other 

SME instrument 

Higher  or  secondary  education 
establishments 

SME instrument 

Research organisations 

SME instrument 

Private for-profit entities  

SME instrument 

Public bodies  

SME instrument 

Other 

12.39 

14.74 

11.60 

4.71 

4.65 

14.41 

16.41 

15.18 

21.11 

13.14 

5.3 

n/a 

8.85 

5.9 

3.99 

1.4  Outcome by different applicant characteristics and reviewer characteristics 

To  determine  whether  the  proposal  evaluation  process  is  fair,  we  assessed  whether  the 
outcomes were influenced by the demographic characteristics of applicants and reviewers. 
We identified multilevel logistic regression as the most appropriate method to perform this 
analysis, given that the outcome variable is categorical with only two possible outcomes 
(proposal  funded  or  not  funded)  and  given  that  most  reviewers  assess  more  than  one 
proposal. 

Our analysis focused on the impact of individual characteristics of reviewers and applicants 
on  the  outcome  of  proposal  evaluation  (Table  26,  Table  27,  Table  28).  In  all  cases,  we 
present the likelihood of a proposal being selected as an odds ratio (OR). Additionally, we 
used  confidence  intervals  (CI)  to  estimate  the  precision  of  the  OR,  with  a  smaller  CI 
indicating higher precision in comparison to a larger CI. The binary logistic regression also 
allows investigating whether the joint effect  of applicant and reviewer characteristics on 

88 

 
 
 
 
the funding outcome is larger than the effect of either applicant characteristics or reviewer 
characteristics in isolation. This is achieved by fitting an interaction term between applicant 
and reviewer classes. If the combined effect is larger (or smaller) than expected, this could 
indicate some form of bias. 

For MSCA IF actions, there is some evidence to suggest that male applicants have lower 
odds  (1 − 0.79 = 0.21)  of  success  than  female  applicants.  Furthermore,  proposals 
submitted by male applicants have lower odds (1 − 0.8 = 0.20) of success when assessed 
by  male  reviewers  (Table  26).  For  ERC  actions,  there  is  no  evidence  to  suggest  that 
reviewer or applicant gender influences the outcomes of proposals (Table 26). Additionally, 
our analysis for MSCA IF and ERC actions shows that applicants from EU-associated and 
third countries have significantly lower odds of success in comparison with EU applicants 
(0.33 and 0.28 lower odds, respectively). Furthermore, there are significantly lower odds 
of success for applicants from EU-associated and third countries when their proposals are 
assessed by reviewers from third countries (0.40 and 0.25, respectively) (Table 27). There 
is no evidence to indicate that affiliation influences the outcomes of proposal evaluation in 
MSCA IF and ERC actions (Table 28). 

For SME actions, there is some evidence to suggest that male applicants have 1.13 higher 
odds  of  success  in  comparison  with  female  applicants  (Table  26).  However,  there  is  no 
evidence to suggest that the gender of reviewers influences the outcomes of the proposal 
evaluation.  In  terms  of  nationality,  there  is  evidence  to  suggest  that  applicants  in  SME 
actions from EU-associated countries have 1.19 higher odds of success than EU applicants 
(Table  27).  In  terms  of  affiliation,  the  evidence  suggests  that  applicants  in  non-private 
organisations  have  0.5  lower  odds  of  success  than  applicants  in  private  organisations 
(Table 28). Furthermore, proposals submitted by  applicants in non-private organisations 
have  lower  odds  of  succeeding  when  assessed  by  reviewers  affiliated  with  non-private 
organisations (Table 28). 

Table 26: Outcome of mono-beneficiary grants by applicant and reviewer gender19 

Action 

Reviewer class 

Applicant class 

MSCA 

ERC 

Female applicant 
(reference class) 

Male applicant 

Female 
(reference class) 

reviewer 

1 

0.79 (0.71–0.87); 
p <0.001 *** 

Male reviewer 

0.93 (0.84–1.04); 
p = 0.20 

0.80 (0.72–0.89); 
p <0.001 *** 

Female 
(reference class) 

reviewer 

1 

0.97 (0.84–1.12); 
p = 0.67 

Male reviewer 

1.05 (0.89–1.24); 
p = 0.56 

1.12 (0.96–1.31); 
p = 0.14 

SME instrument 

Female 
(reference class) 

reviewer 

1 

1.13 (1.01–1.27); 
p=0.04 * 

Male reviewer 

0.85 (0.70–1.03); 
p = 0.10 

0.94 (0.79–1.11); 
p = 0.48 

19  Multilevel  logistic  regression.  OR  adjusted  odds  ratio,  95%  confidence  intervals  (LCI:  lower 
confidence interval; UCI; upper confidence interval), p: p-value. MSCA-IF n = 21 100. ERC = 9 726. 
SME n = 65 059. *** p <0.001; ** p <0.01; * p <0.05. 

89 

 
 
                                                 
  
 
 
 
 
 
 
Table 27: Outcome of mono-beneficiary grants by applicant and reviewer nationality20  

Action 

Reviewer class 

Applicant class 

EU Member State 
applicant 
(reference class) 

EU-associated-
country applicant  

Third-country applicant 

MSCA 
IF 
and ERC21 

EU Member State 
(reference class) 

1  

0.67 (0.61–0.75); 
p <0.001 *** 

0.72 (0.67–0.76); 
p <0.001 *** 

EU-associated 
country  

0.84 (0.69–1.02); 
p = 0.08 

0.96 (0.66–1.41); 
p = 0.85 

0.85 (0.67–1.09); 
p = 0.19 

Third country  

0.96 (0.87–1.07); 
p = 0.45 

0.60 (0.49–0.73) 
 p <0.001 *** 

0.75 (0.66–0.84); 
p <0.001 *** 

SME 
instrument 

EU Member State 
(reference class) 

1  

1.19 (1.07–1.31); 
p = 0.001** 

0.90 (0.80–1.00); 
p = 0.05 

EU-associated 
country  

0.83 (0.61–1.14); 
p = 0.25 

1.06 (0.66–1.72); 
p = 0.80 

0.86 (0.52–1.42); 
p = 0.55 

Third country  

1.09 (0.91–1.29); 
p = 0.34 

1.11 (0.89–1.39); 
p = 0.36 

0.89 (0.69–1.15); 
p = 0.38 

20  Multilevel  logistic  regression.  OR  adjusted  odds  ratio,  95%  confidence  intervals  (LCI:  lower 
confidence interval; UCI; upper confidence interval), p: p-value. MSCA-if and ERC n= 40 826. SME 
n=65 059. *** p < 0.001; ** p < 0.01; * p <0.05. 

21  For  the  purpose  of  this  analysis  MSCA-IF  and  ERC  were  combined,  as  they  are  both  mono-
beneficiary grants with similar objectives.  

90 

 
 
 
 
                                                 
 
 
 
 
 
 
Table 28: Outcome of mono-beneficiary grants by applicant and reviewer organisation22 

Action 

Reviewer class 

Applicant class 

Applicant in private 
organisation (reference 
class) 

Applicant in non-
private organisation 

MSCA IF and ERC23 

in  private 

Reviewer 
organisation  
(reference class) 

1  

1.18 (0.85–1.66); 
p = 0.32 

Reviewer in non-private 
organisation 

0.56 (0.39–0.82); 
p = 0.003 

0.88 (0.62–1.25); 
p = 0.47 

SME instrument 

in  private 

Reviewer 
organisation  
(reference class) 

1  

0.50 (0.36–0.70); 
p <0.001 *** 

Reviewer in non-private 
organisation 

0.98 (0.86–1.13); 
p = 0.83 

0.543 (0.38–0.79); 
p = 0.001 ** 

1.5  Scores by assessment criterion by characteristic 

To determine whether there is variation in the proposal scores across evaluation criteria 
and  applicant  characteristics,  we  determined  the  median  scores  and  interquartile  range 
(IQR) for the ‘Excellence’, ‘Impact’ and ‘Quality and efficiency of implementation’ criteria 
across  the  different  applicant  demographics  for  the  mono-beneficiary  grants  (Table  29, 
Table 30 and Table 31). 

22  Multilevel  logistic  regression.  OR  adjusted  odds  ratio,  95%  confidence  intervals  (LCI:  lower 
confidence interval; UCI; upper confidence interval), p: p-value. MSCA-IF and ERC n= 40 826. SME 
n= 65 059. *** p < 0.001; ** p < 0.01; * p <0.05. 

23  For  the  purpose  of  this  analysis  MSCA-IF  and  ERC  were  combined,  as  they  are  both  mono-
beneficiary grants with similar objectives. 

91 

 
 
 
 
                                                 
 
 
 
 
 
 
 
 
 
 
Table 29: Median scores and IQR by evaluation criteria and by applicant gender 

Action 

MSCA  

MSCA  

MSCA  

MSCA  

MSCA  

MSCA  

ERC 

ERC 

ERC 

ERC 

Criterion 

Gender 

Median 

IQR 

Excellence 

Female 

3.80 

Excellence 

Male 

3.60 

Impact 

Impact 

Female 

3.80 

Male 

3.70 

Quality and efficiency 

Female 

3.60 

Quality and efficiency 

Male 

3.50 

Excellence 

Female 

Excellence 

Male 

Impact 

Impact 

Female 

Male 

3.0 

3.0 

3.0 

3.0 

SME instrument 

Excellence 

Female 

3.67 

SME instrument 

Excellence 

Male 

3.70 

SME instrument 

Impact 

Female 

3.70 

SME instrument 

Impact 

Male 

3.73 

SME instrument 

Quality and efficiency 

Female 

3.80 

SME instrument 

Quality and efficiency 

Male 

3.82 

2.60 

2.60 

3.00 

2.60 

3.10 

2.80 

1.5 

1.0 

1.0 

1.0 

1.46 

1.46 

1.50 

1.44 

1.36 

1.40 

92 

 
 
 
Table 30: Median scores and IQR by evaluation criteria and applicant nationality 

Action24 

Criterion25 

Nationality 

EU category 

Median 

IQR 

ERC 

Excellence 

EU Member State 

ERC 

Excellence 

EU Member State 

Widening 
participation 
group  Member 
States 

Other  Member 
States 

2.50 

1.50 

3.00 

1.00 

ERC 

Excellence 

EU-associated 
countries 

n/a 

3.00 

1.00 

ERC 

Excellence 

Third countries 

n/a 

3.00 

1.00 

ERC 

Impact 

EU Member State 

ERC 

Impact 

EU Member State 

Widening 
participation 
group  Member 
States 

Other  Member 
States 

2.50 

1.50 

3.00 

1.00 

ERC 

Impact 

EU-associated 
countries 

n/a 

3.50 

1.50 

ERC 

Impact 

Third countries 

n/a 

3.00 

1.00 

MSCA 

Excellence 

EU Member State 

Widening 
participation 
group  Member 
States 

3.6 

2.1 

MSCA 

Excellence 

EU Member State 

Other  Member 
States 

4.0 

2.2 

MSCA 

Excellence 

EU-associated 
countries 

n/a 

3.2 

2.7 

MSCA 

Excellence 

Third countries 

n/a 

3.5 

3.0 

MSCA 

Impact 

EU Member State 

Widening 
participation 
group  Member 
States 

3.7 

2.5 

24 ERC Proof of Concept (POC) grants were excluded from scoring analysis as scoring is not based on 
a 1-5 scale as is the case for the Investigator (Starting, Consolidator and Advanced) and Synergy 
Grants, but rather use a pass/fail system. 

25 With the exception of POC grants, the criterion 'Quality and efficiency of implementation’ is not 
evaluated for ERC grants. 

93 

 
 
 
                                                 
MSCA 

Impact 

EU Member State 

Other  Member 
States 

4.0 

2.3 

MSCA 

Impact 

EU-associated 
countries 

n/a 

3.1 

2.8 

MSCA  

Impact 

Third countries 

n/a 

3.5 

3.5 

MSCA  

Quality and efficiency 

EU Member State 

Widening 
participation 
group  Member 
States 

3.5 

2.5 

MSCA  

Quality and efficiency 

EU Member State 

Other  Member 
States 

3.9 

2.6 

MSCA  

Quality and efficiency 

EU-associated 
countries 

n/a 

3.0 

3.1 

MSCA  

Quality and efficiency 

Third countries 

n/a 

3.2 

3.4 

SME 
instrument 

SME 
instrument 

Excellence 

EU Member State 

Widening 
participation 
group  Member 
States 

3.51 

1.55 

Excellence 

EU Member State 

Other  Member 
States 

3.79 

1.34 

SME 
instrument 

Excellence 

EU-associated 
countries 

n/a 

3.70 

1.58 

SME 
instrument 

Excellence 

Third countries 

n/a 

3.64 

1.48 

SME 
instrument 

Impact 

EU Member State 

Widening 
participation 
group  Member 
States 

3.50 

1.64 

SME 
instrument 

Impact 

SME 
instrument 

Impact 

SME 
instrument 

Impact 

EU Member State 

Other  Member 
States 

3.80 

1.33 

EU-associated 
countries 

n/a 

3.80 

1.47 

Third countries 

n/a 

3.60 

1.53 

SME 
instrument 

SME 
instrument 

SME 
instrument 

SME 
instrument 

Quality and efficiency 

EU Member State 

Widening 
participation 
group  Member 
States 

3.62 

1.58 

Quality and efficiency 

EU Member State 

Other  Member 
States 

3.90 

1.34 

Quality and efficiency 

EU-associated 
countries 

n/a 

3.90 

1.40 

Quality and efficiency 

Third countries 

n/a 

3.75 

1.50 

94 

 
 
Table 31: Median scores and IQR by evaluation criteria and applicant affiliation 

Action26 

Criterion 

Organisation 

Median 

IQR 

ERC 

ERC 

ERC 

ERC 

ERC 

ERC 

ERC 

ERC 

ERC 

ERC 

Excellence 

Higher  or  secondary  education 
establishments 

3.00 

1.0 

Excellence 

Other 

2.50 

2.0 

Excellence 

Excellence 

Private 
entities 
for-profit 
(excluding  higher  or  secondary 
education establishments) 

2.50 

1.5 

Public  bodies  (excluding  research 
organisations  and  secondary  or 
higher education establishments) 

2.50 

1.5 

Excellence 

Research organisations 

3.00 

1.0 

Impact 

Impact 

Impact 

Impact 

Higher  or  secondary  education 
establishments 

3.00 

1.0 

Other 

3.00 

1.5 

Private 
entities 
for-profit 
(excluding  higher  or  secondary 
education establishments) 

2.50 

2.0 

Public  bodies  (excluding  research 
organisations  and  secondary  or 
higher education establishments) 

2.50 

1.5 

Impact 

Research organisations 

3.00 

1.5 

MSCA IF 

Excellence 

Higher  or  secondary  education 
establishments 

3.70 

2.6 

MSCA IF 

Excellence 

Other 

3.30 

2.9 

MSCA IF 

Excellence 

MSCA IF 

Excellence 

Private 
entities 
for-profit 
(excluding  higher  or  secondary 
education establishments) 

3.00 

2.7 

Public  bodies  (excluding  research 
organisations  and  secondary  or 
higher education establishments) 

3.90 

2.2 

MSCA IF 

Excellence 

Research organisations 

3.90 

2.2 

MSCA IF 

Impact 

Higher  or  secondary  education 
establishments 

3.70 

2.8 

26 ERC Proof of Concept (POC) grants were excluded from scoring analysis as scoring is not based on 
a 1-5 scale as is the case for the Investigator (Starting, Consolidator and Advanced) and Synergy 
Grants, but rather use a pass/fail system. 

95 

 
 
 
                                                 
MSCA IF 

Impact 

Other 

3.00 

3.1 

MSCA IF 

Impact 

MSCA IF 

Impact 

Private 
entities 
for-profit 
(excluding  higher  or  secondary 
education establishments) 

3.10 

3.3 

Public  bodies  (excluding  research 
organisations  and  secondary  or 
higher education establishments) 

4.00 

2.8 

MSCA IF 

Impact 

Research organisations 

4.00 

2.7 

MSCA IF 

Quality and efficiency 

Higher  or  secondary  education 
establishments 

3.50 

3.0 

MSCA IF 

Quality and efficiency 

Other 

2.80 

3.0 

MSCA IF 

Quality and efficiency 

MSCA IF 

Quality and efficiency 

Private 
entities 
for-profit 
(excluding  higher  or  secondary 
education establishments) 

3.00 

3.1 

Public  bodies  (excluding  research 
organisations  and  secondary  or 
higher education establishments) 

3.90 

2.7 

MSCA IF 

Quality and efficiency 

Research organisations 

3.80 

2.6 

SME instrument 

Excellence 

Higher  or  secondary  education 
establishments 

3.40 

1.8 

SME instrument 

Excellence 

Other 

3.33 

1.6 

SME instrument 

Excellence 

SME instrument 

Excellence 

Private 
entities 
for-profit 
(excluding  higher  or  secondary 
education establishments) 

3.70 

1.4 

Public  bodies  (excluding  research 
organisations  and  secondary  or 
higher education establishments) 

3.50 

1.8 

SME instrument 

Impact 

Higher  or  secondary  education 
establishments 

3.38 

1.9 

SME instrument 

Impact 

Other 

3.31 

1.8 

SME instrument 

Impact 

SME instrument 

Impact 

Private 
entities 
for-profit 
(excluding  higher  or  secondary 
education establishments) 

3.73 

1.4 

Public  bodies  (excluding  research 
organisations  and  secondary  or 
higher education establishments) 

3.31 

1.9 

SME instrument 

Quality and efficiency 

Higher  or  secondary  education 
establishments 

3.40 

2.0 

SME instrument 

Quality and efficiency 

Other 

3.44 

1.8 

SME instrument 

Quality and efficiency 

Private 
entities 
for-profit 
(excluding  higher  or  secondary 
education establishments) 

3.83 

1.4 

96 

 
 
SME instrument 

Quality and efficiency 

Public  bodies  (excluding  research 
organisations  and  secondary  or 
higher education establishments) 

3.44 

2.0 

To determine the statistical significance of the differences detected in the median scores, 
we  used  a  Kruskal-Wallis  test,  as  it  can  compare  median  scores  across  more  than  two 
groups. This is a non-parametric test, which allows considering the median score values 
as  ranked  data  without  assuming  that  these  are  normally  distributed.  A  Bonferroni 
correction  was  applied  to  adjust  for  multiple  testing  when  considering  the  statistical 
significance of the results. 

Our analysis shows that there is no significant difference in median scores for any of the 
demographic characteristics considered (i.e. gender, nationality and affiliation) within each 
criterion (Table 32). 

Table 32: Significance testing of differences between median scores for applicant 
gender, nationality and organisation, paired by evaluation criteria27  

Criterion 

p-value 

Gender 

Excellence 

Impact 

Quality and efficiency 

Nationality 

Excellence 

Impact 

Quality and efficiency 

Organisation 

Excellence 

Impact 

Quality and efficiency 

1.6  Consistency of scoring 

Range of reviewer median and average scores 

0.68 

0.68 

0.31 

0.94 

0.81 

0.26 

0.80 

0.90 

0.59 

To determine the level of variability in the scores provided by reviewers, we assessed the 
range  of  reviewer  average  and  median  scores  across  evaluation  criteria  and  for  the 
different H2020 actions (range of scores for criteria 0–5). This part of the analysis aimed 

27 Differences for gender, nationality and organisation were assessed using the Kruskal-Wallis test. 
A  Bonferroni  correction  was  applied to  adjust  for  multiple  testing  when  considering  the  statistical 
significance of the results. p: p-value. n = 1 637 154 *** p <0.0003; ** p <0.003; * p <0.017. 

97 

 
 
 
 
                                                 
 
 
 
 
 
 
 
to obtain an overview on whether any of the evaluation criteria or the H2020 actions show 
considerable variation in scoring. 

Our  analysis  shows  that  the  median  score  across  all  criteria  and  actions  is  3.53,  with 
‘Quality  and  efficiency  of  implementation’  having  the  highest  median  scores  (3.75), 
whereas ‘Excellence’ and ‘Impact’ have a median score of 3.5 (Table 33). 

Table 33: Median scores provided by Reviewers across H202028 

Criteria 

Median score 

IQR 

All criteria 

Excellence 

Impact 

Quality and efficiency 

3.53 

3.50 

3.50 

3.75 

1.68 

1.80 

1.70 

1.59 

We also note some other differences by action. 

  ERC grants29 overall had the lowest median score across criteria (Figure 10). 

  Actions that make up the ‘other’ category (PCP, PPI, EJP Cofund) have the highest 

median score across criteria (Figure 10). 

28 The analysis considered proposals with a score range of 0–5 for each evaluation criterion.  

29 ERC Proof of Concept (POC) grants were excluded from scoring analysis as scoring is not based on 
a 1-5 scale as is the case for the Investigator (Starting, Consolidator and Advanced) and Synergy 
Grants, but rather use a pass/fail system. 

98 

 
 
 
 
                                                 
Figure 10: Median and range of scores by action and evaluation criterion 

We  then  proceeded  to  analyse  the  level  of  variation  in  scores  provided  by  different 
reviewers  per  proposal.  To  this  end,  we  calculated  a  coefficient  of  variation  (standard 
deviation/mean score), which allowed us to identify the proposals with a high, medium and 
low  level  of  divergence  between  reviewer  mean  scores.  Specifically,  the  coefficient  of 
variation  was  segmented  in  three  categories:  high  (coefficient  of  variation  ≥10.48), 
medium (coefficient of variation between 3.49 and 10.48) and low (coefficient of variation 
≤3.49).  The  subset  of  proposals  with  the  highest,  medium  and  lowest  coefficients  of 
variation were assessed in terms of applicant, reviewer and proposal characteristics (Table 
34,  Table  36  and  Table  38).  The  segmentation  of  high,  medium  and  low  coefficient  of 
variation was based on having a balanced number of applications per category. Statistical 
significance  in  the  differences  between  the  proportions  was  assessed  using  chi-square 
tests. 

For ERC, MSCA IF and SME actions, our analysis shows that there is a higher percentage 
of male applicants across all coefficients of variation (Table 34). No significant differences 
were found in this trend between high, medium and low coefficients of variation, except 
for MSCA IF (Table 35). In terms of nationality, all coefficients of variation showed a higher 
percentage  of  EU  applicants  across  the  actions  analysed,  except  for  MSCA  IF  proposals 
with  high  coefficient  of  variation  of  scores,  where  there  was  a  higher  percentage  of 
applicants  from  third  countries  (45.8%)  (Table  34).  For  MSCA  IF  and  SME  actions,  the 
differences  in  nationality  trends  were  significant  across  the  different  coefficients  of 
variation (Table 35). Regarding applicants’ affiliations, ERC and MSCA IF actions show a 
higher percentage of applicants from higher or secondary education establishments (over 
70%) in comparison with other organisation types across all coefficients of variation (Table 
34). On the other hand, the majority of  applicants within SME actions (over 90%) were 

99 

 
 
 
 
 
affiliated with private for-profit entities across all coefficients of variation (Table 34). These 
differences  in  affiliation  were  found  to  be  statistically  significant  between  different 
coefficients of variation for each of the actions analysed (p-value <0.0001) (Table 35). 

100 

 
 
Table 34: Comparison of demographic characteristics of applicants in proposals with low, medium and high coefficient of variation30 
(percentages calculated across demographics) 

Low coefficient of variation 

Medium coefficient of variation 

High coefficient of variation 

Variable 

Level 

ERC 

MSCA IF 

SME 
instrument 

ERC 

MSCA IF 

SME 
instrument 

ERC 

MSCA IF 

SME 
instrument 

6 776 

6 049 

29 753 

5 407 

6 067 

20 936 

5 394 

4 651 

14 225 

n 

Gender 
(%) 

Female 

Male 

2 161 
(31.9) 

4 615 
(68.1) 

Nationality 
(%) 

Member 

EU 
States 

4 731 
(69.8) 

Widening 
participation 
group  Member 
States   

490  
(7.2) 

Other  Member 
States  

4 241 
(62.6) 

EU-associated 
countries 

Third countries 

585  
(8.6) 

1 460 
(21.5) 

2 565 
(42.4) 

3 484 
(57.6) 

3 522 
(58.2) 

530  
(8.8) 

2 992 
(49.5) 

373  
(6.2) 

2 154 
(35.6) 

4 988 
(16.8) 

24 691 
(83.0) 

22 402 
(75.3) 

4 022 
(13.5) 

18 380 
(61.8) 

3 944 
(13.3) 

3 407 
(11.5) 

1 619 
(29.9) 

3 788 
(70.1) 

3 711 
(68.6) 

2 430 
(40.1) 

3 637 
(59.9) 

3 297 
(54.3) 

3 566 
(17.0) 

17 297 
(82.6) 

15 786 
(75.4) 

1 773 
(32.9) 

3 621 
(67.1) 

3 710 
(68.8) 

1 761 
(37.9) 

2 890 
(62.1) 

2 017 
(43.4) 

2 503 
(17.6) 

11 645 
(81.9) 

10 605 
(74.6) 

323  
(6.0) 

564  
(9.3) 

3 384 
(16.2) 

410  
(7.6) 

422  
(9.1) 

2 702 
(19.0) 

3 388 
(62.7) 

473  
(8.7) 

1 223 
(22.6) 

2 733 
(45.0) 

485  
(8.0) 

2 285 
(37.7) 

12 402 
(59.2) 

2 553 
(12.2) 

2 597 
(12.4) 

3 300 
(61.2) 

449  
(8.3) 

1 235 
(22.9) 

1 595 
(34.3) 

503  
(10.8) 

2 131 
(45.8) 

7 903 
(55.6) 

1 737 
(12.2) 

1 883 
(13.2) 

30 ERC Proof of Concept (POC) grants were excluded from scoring analysis as scoring is not based on a 1−5 scale as is the case for the Investigator (Starting, 
Consolidator and Advanced) and Synergy Grants, but rather use a pass/fail system. 

101 

 
 
                                                 
 
 
 
 
 
 
 
 
Organisation 
(%) 

or 

Higher 
secondary 
education 
establishments 

for-
Private 
profit entities  

Public bodies  

Research 
organisations 

Other 

5 295 
(78.1) 

4 499 
(74.4) 

185  
(0.6) 

4 202 
(77.7) 

4 578 
(75.5) 

167  
(0.8) 

4 198 
(77.8) 

3 573 
(76.8) 

155  
(1.1) 

85  
(1.3) 

19  
(0.3) 

1 365 
(20.1) 

12  
(0.2) 

210  
(3.5) 

36 
(0.6) 

1 292 
(21.4) 

12  
(0.2) 

29 190 
(98.1) 

137  
(0.5) 

0  
(0.0) 

241  
(0.8) 

41  
(0.8) 

10  
(0.2) 

1 154 
(21.3) 

0  
(0.0) 

320  
(5.3) 

25  
(0.4) 

1 118 
(18.4) 

26  
(0.4) 

20 401 
(97.4) 

143  
(0.7) 

0  
(0.0) 

225  
(1.1) 

90  
(1.7) 

29  
(0.5) 

1 064 
(19.7) 

13  
(0.2) 

305  
(6.6) 

20  
(0.4) 

728  
(15.7) 

25  
(0.5) 

13 741 
(96.6) 

127  
(0.9) 

0  
(0.0) 

202  
(1.4) 

102 

 
 
 
 
 
 
 
Table 35: Significance of differences in proportions for applicant demographic 
characteristics with a high, medium and low coefficient of variation31 

Proportions 

Chi-square 

df 

p-value 

Applicants 

Gender 

ERC  

11.164 

MSCA IF 

22.765 

SME instrument 

5.2959 

Nationality 

ERC 

4.1968 

MSCA IF 

260.74 

SME instrument 

41.992 

Organisation 

ERC 

39.418 

MSCA IF 

117.08 

SME instrument 

95.041 

2 

2 

2 

4 

4 

4 

8 

8 

6 

p = 0.004 

p <0.0001 
*** 

p = 0.07 

p = 0.15 

p <0.0001 
*** 

p <0.0001 
*** 

p <0.0001 
*** 

p <0.0001 
*** 

p <0.0001 
*** 

For reviewers, there is a more balanced distribution of gender and age across the different 
coefficients of variation (Table 36). Regarding nationality, most reviewers were found to 
be  from  the  EU  (over  70%)  across  all  coefficients  of  variation.  In  terms  of  reviewer 
affiliation,  there  is  a  higher  percentage  of  reviewers  affiliated  with  higher  or  secondary 
education  establishments  across  all  coefficients  of  variation  (Table  36).  The  differences 
found between coefficients of variation were found to be not statistically significant for the 
reviewer demographics assessed (p-value <0.0001) (Table 37). 

31 p: p-value. *** p <0.0001; ** p <0.001; * p <0.006. 

103 

 
 
 
 
                                                 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Table 36: Comparison of demographic characteristics of reviewers in proposals with low, 
medium and high coefficient of variation (percentages calculated across demographics) 

Variable 

Level 

Low coefficient of 
variation 

Medium coefficient 
of variation 

High coefficient of 
variation 

n 

24 495 

26 938 

25 071 

Gender (%) 

Female 

9 572 (39.1) 

10 373 (38.5) 

9 858 (39.3) 

Male 

14 923 (60.9) 

16 565 (61.5) 

15 213 (60.7) 

Age (%) 

Under 50 

10 657 (43.5) 

11 974 (44.5) 

11 164 (44.5) 

Over 50 

13 838 (56.5) 

14 964 (55.5) 

13 907 (55.5) 

Nationality (%) 

EU Member State 

17 366 (70.9) 

19 137 (71.0) 

17 841 (71.2) 

Widening 
participation 
group  Member 
State 

Other  Member 
State 

EU-associated 
country 

3 902 (15.9) 

4 320 (16.0) 

4 082 (16.3) 

13 464 (55.0) 

14 817 (55.0) 

13 759 (54.9) 

1 458 (6.0) 

1 580 (5.9) 

1 466 (5.8) 

Third country 

5 671 (23.2) 

6 221 (23.1) 

5 764 (23.0) 

Organisation (%) 

Higher 
secondary 
education 
establishment 

or 

10 288 (42.0) 

11 276 (41.9) 

10 530 (42.0) 

Private  for-profit 
organisation 

Public 
organisation 

Research 
organisation 

Organisation 
type not defined 

4 785 (19.5) 

5 199 (19.3) 

4 977 (19.9) 

1818 (7.4) 

2 025 (7.5) 

1 942 (7.7) 

3 316 (13.5) 

3 714 (13.8) 

3 427 (13.7) 

4 288 (17.5) 

4 724 (17.5) 

4 195 (16.7) 

104 

 
 
 
 
 
 
 
 
 
 
Table 37: Significance of differences in proportions for reviewer demographic 
characteristics, between proposals with a high, medium and low coefficient of variation32  

Chi-square 

df 

p-value 

Reviewers 

Gender 

Age 

3.8354 

6.5397 

Nationality 

0.53212 

Organisation 

10.618 

2 

2 

4 

8 

p = 0.147 

p = 0.038 

p = 0.970 

p = 0.224 

Regarding  proposal  characteristics,  several  actions  had  a  higher  proportion  of  proposals 
with medium coefficient of variability (IA, MSCA, RIA and ‘other’) (Table 38). ERC and SME 
actions had a higher proportion of proposals with low coefficient of variability in comparison 
with medium and high coefficient of variability, whereas only CSA had a higher proportion 
of proposals with a high coefficient of variability (Table 38). This suggests that proposals 
within CSA tend to have a higher variability in reviewer scores in comparison with proposals 
in  other  actions.  On  the  other  hand,  proposals  within  SME  actions  tend  to  have  lower 
variability between  reviewer scores compared with other actions. Additionally, there is a 
higher  percentage  of  selected  proposals  with  low  coefficient  of  variation  (21.8%), 
compared with proposals with a medium (18.5%) and high (7.3%) coefficient of variation 
(Table 38). These differences were found to be statistically significant (p-value <0.0001) 
(Table 39), and they may indicate conservatism in the evaluation process. 

32 p: p-value. *** p <0.0001; ** p <0.001; * p <0.006. 

105 

 
 
 
 
                                                 
 
 
Table 38: Comparison of characteristics of proposals with low, medium and high 
coefficient of variation (percentages for actions calculated across coefficients of 
variability) 

Low coefficient of 
variability 

Medium 
coefficient of 
variability  

High coefficient of 
variability 

n 

55 371 

53 174 

44 953 

Action (%) 

CSA 

1 329 (0.19) 

2 744 (0.38) 

3 069 (0.43) 

ERC33 

IA 

MSCA 

RIA 

6 776 (0.39) 

5 006 (0.28) 

5 795 (0.33) 

2 979 (0.23) 

5 375 (0.41) 

4 870 (0.37) 

1 185 (0.31) 

1 600 (0.42) 

1 069 (0.28) 

7 946 (0.27) 

11 352 (0.38) 

10 568 (0.35) 

SME instrument 

29 753 (0.46) 

20 936 (0.32) 

14 225 (0.22) 

Other 

30 (0.19) 

80 (0.52) 

44 (0.29) 

Outcome (%) 

Funded 

12 080 (21.8) 

9 821 (18.5) 

3 279 (7.3) 

Not funded 

43 291 (78.2) 

43 353 (81.5) 

41 674 (92.7) 

Table 39: Significance of differences in proportions of proposal characteristics, between 
proposals with a high, medium and low coefficient of variation 

Proportion 

Chi-square 

df 

p-value 

Proposal 

Action 

6 939.1 

Outcome 

4 068.7 

14 

2 

p <0.0001 *** 

p <0.0001 *** 

Relationship between score by criterion and funding decision 

To assess the relationship between scores by criterion and funding decision, we calculated 
the median scores of selected and rejected proposals for each evaluation criteria by action 
(Figure 11). 

33 ERC Proof of Concept (POC) grants were excluded from scoring analysis as scoring is not based on 
a 1–5 scale as is the case for the Investigator (Starting, Consolidator and Advanced) and Synergy 
Grants, but, rather, use a pass/fail system. 

106 

 
 
 
 
 
                                                 
 
 
 
 
Our results show that for the ‘Excellence’ criterion, the selected proposals have a similar 
median score across all actions (4.5), except for ERC actions, for which selected proposals 
have  a  lower  median  score  (3.5).  There  is  some  variation  of  the  median  score  of  the 
‘Excellence’ criterion for rejected proposals, with ERC scoring a median of 2.5, CSA and 
MSCA  scoring  a  median  of  3.0,  and  IA,  RIA,  the  SME  instrument  and  ‘other’  scoring  a 
median of 3.5. 

For  the  Impact  criterion,  selected  proposals  also  had  a  median  score  of  4.5  for  most 
actions,  except  ERC  (3.5)  and  ‘other’  (4.0).  There  is  variation  in  the  median  scores  of 
rejected proposals, with the CSA, ERC and MSCA showing a median score of 3.0 for rejected 
proposals, IA, RIA and the SME instrument with a median of 3.5 and ‘other’ with a median 
of 4.0. 

Regarding the ‘Quality and efficiency of the implementation’ criterion, selected proposals 
for  CSA,  IA  and  ‘other’  actions  scored  a  median  of  4.0,  whereas  selected  proposals  in 
MSCA, RIA and the SME instrument scored a median of 4.5. For rejected proposals, CSA 
and MSCA scored a median of 3.0, whereas IA, RIA and ‘other’ scored a median of 3.5 and 
the SME instrument just above 3.5. 

107 

 
 
 
Figure 11: Distribution of median scores by evaluation criterion and action34 

34  The  box  in  this  ‘box-and-whisker’  plot  represents  the  interquartile  range  defined  by  the  75th 
percentile  and  the  25th  percentile,  with  the  median  being  defined  as  the  50th  percentile  and 
represented by the line dividing  the  box.  The top  whisker represents the largest value within 1.5 
times the interquartile range above the 75th percentile, whereas the bottom whisker represents the 
smallest value within 1.5 times the interquartile range below the 25th percentile. The points outside 

108 

 
 
 
 
                                                 
Characteristics  of  more  and  less  ‘generous’  reviewers,  i.e.  those  who  typically 
give higher scores and lower (by institution type, age and gender) 

To  assess  the  characteristics  of  more  and  less  ‘generous’  reviewers,  we  averaged  the 
reviewer scores across the three criteria and grouped reviewers with a mean score equal 
to or above 4 as generous and  reviewers with a mean score equal to or below 1 as less 
generous (range of scores for criteria: 0–5). 

Our findings show that generous reviewers are predominantly male (60.8%), EU nationals 
(70.9%)  and  affiliated  with  higher/secondary  education  establishments  (40.1%  –  most 
abundant  class  compared  with  the  other  four  types  of  affiliations),  whereas  in  terms  of 
age, there is an approximately even balance between under and over 50 years old (Table 
40).  On  the  other  hand,  there  is  a  more  even  gender  balance  among  less  generous 
reviewers  (scoring  mean  equal  to  or  below  1),  with  45.5%  female  and  54.5%  male 
reviewers. Less generous reviewers are predominantly under 50 years old (65.7%), of EU 
nationality (69.2%) and affiliated with higher/secondary education establishments (45.6% 
– most abundant class compared with the other four types of affiliations) (Table 40). 

We  used  chi-square  tests  to  assess  the  statistical  significance  of  the  differences  in  the 
proportions of demographic characteristics between the generous and the less generous 
reviewer groups. Our analysis shows that the differences between more generous and less 
generous reviewers in terms of gender, age and organisation are significant (Table 41). 

the whiskers represent outliers with values more than 1.5 times or less than 3 times the interquartile 
range beyond either end of the box. 

109 

 
 
 
                                                 
Table 40: Characteristics of generous reviewers (scoring mean equal to or above 4) and 
less generous reviewers (scoring mean equal to or below 1)35  

Variable 

Level 

Generous reviewers 

Less generous reviewers 

n 

12 268 

1 850 

Gender (%) 

Female 

4 807 (39.2) 

841 (45.5) 

Male 

7 461 (60.8) 

1 009 (54.5) 

Age (%) 

Under 50 

5 778 (47.1) 

1 215 (65.7) 

Over 50 

6 490 (52.9) 

635 (34.3) 

Nationality (%) 

EU Member State 

8 694 (70.9) 

1 281 (69.2) 

Widening  participation 
group Member State  

2 013 (16.4) 

273 (14.8) 

Other Member State 

6 681 (54.5) 

1 008 (54.5) 

EU-associated country 

751 (6.1) 

123 (6.6) 

Third country 

2 823 (23.0) 

446 (24.1) 

Organisation (%) 

Higher  or 
secondary 
education establishment 

4 921 (40.1) 

843 (45.6) 

Private 
organisation 

for-profit 

2 309 (18.8) 

186 (10.1) 

Public organisation 

936 (7.6) 

291 (15.7) 

Research organisation 

1 923 (15.7) 

156 (8.4) 

Other 

2 179 (17.8) 

374 (20.2) 

35 ERC Proof of Concept (POC) grants were excluded from scoring analysis as scoring is not based on 
a 1-5 scale as is the case for the Investigator (Starting, Consolidator and Advanced) and Synergy 
Grants, but rather use a pass/fail system. 

110 

 
 
 
                                                 
 
 
Table 41: Significance of differences in proportions for generous and less generous 
reviewers36 

Proportion: 
generous versus 
less generous 

Chi-square 

df 

p-value 

Reviewers 

Gender 

Age 

Nationality 

Organisation 

26.12 

221.2 

2.157 

98.79 

1 

1 

2 

4 

p <0.001 *** 

p <0.001 *** 

0.3401 

p <0.001 *** 

1.7  Conservatism and interdisciplinarity 

Based on the analysis developed in the section titled ‘Replicability’ (Subsection 1.6, ‘Range 
of reviewer median and average scores’), we determined whether the level of variability of 
proposal  scores  influences  the  outcomes  of  the  evaluation  process.  As  before,  the 
coefficient  of  variation  was  segmented  in  three  categories:  high  (coefficient  of  variation 
≥10.48), medium (coefficient of variation between 3.49 and 10.48) and low (coefficient of 
variation  ≤  3.49),  whereas  the  reviewer  number  was  segmented  in  the  following 
categories: high (5 or more reviewers per proposal), medium (4 reviewers per proposal) 
and low (3 or fewer reviewers per proposal). This segmentation allowed for the most even 
split  between  categories.  The  data  show  that  most  proposals  are  assessed  by  four 
reviewers  and  that  there  are  also  several  cases  where  proposals  are  evaluated  by  five 
reviewers (Figure 12). 

Figure 12: Number of reviewers by proposal across the H2020 programme 

36 p: p-value. *** p <0.001; ** p <0.01; * p <0.05. 

111 

 
 
 
 
 
                                                 
 
Figure 13 shows the distribution of the coefficient of variation across reviewer categories 
for each action analysed in this study. To determine if there are any statistically significant 
differences in the trends depicted in  Figure 13, we fitted a simple logistic regression for 
each action, whereby we fitted the outcome of the proposal evaluation process (funded/not 
funded) as a function of the coefficient of variation calculated in the previous section and  
the  number  of  reviewers  per  proposal  and  the  interaction  between  these  two  variables. 
These  predictors  were  fitted  as  a  multilevel  categorical  variables.  It  was  not  possible  to 
include proposals in the ‘other’ category in this analysis due to the small sample size. 

Figure 13: Distribution of the coefficient of variation across reviewer categories for the actions analysed 

112 

 
 
 
 
 
Our analysis indicates that ERC proposals with a medium and low coefficient of variation 
have 1.72 and 1.69 higher odds of being selected, respectively, than proposals with a high 
coefficient  of  variation  (Table  42).  This  indicates  conservatism,  as  proposals  with  less 
differences in mean scores of reviewers have a higher chance of being successful. In terms 
of number of reviewers per ERC proposal, there is evidence to suggest that proposals with 
a medium and low number of reviewers have lower odds of being successful. Specifically, 
proposals  with  a  medium  number  of  reviewers  have  2.27  (1:0.44)  lower  odds  of  being 
selected, whereas proposals with a low number of reviewers have 7.69 (1:0.13) lower odds 
of being selected (Table 42). Additionally, our analysis shows that the interaction between 
number  of  reviewers  and  the  coefficient  of  variation  is  associated  with  the  outcome  of 
proposal  evaluation  in  ERC  actions.  Namely,  proposals  with  a  medium  coefficient  of 
variation have 1.15 higher odds of being selected when evaluated by a medium number of 
reviewers and 1.92 (1:0.52) lower odds of being selected when assessed by a low number 
of  reviewers.  These  odds  are  also  low  for  proposals  with  a  low  coefficient  of  variation, 
namely 1.07 (1:0.93) lower odds of being selected when reviewed by a medium number 
of  reviewers  and  2.56  (1:0.39)  lower  odds  of  being  selected  when  assessed  by  a  low 
number of reviewers (Table 42). 

Similar results were obtained for SME actions in terms of the influence of the variation in 
reviewer scores and the number of reviewers in the proposal outcome. Specifically, SME 
instrument proposals with a medium and low coefficient of variation have 2.64 and 13.36 
higher  odds  of  being  selected,  respectively,  than  proposals  with  a  high  coefficient  of 
variation (Table 43). Additionally, proposals with a medium number of reviewers have 1.49 
(1:0.67) lower odds of being selected, whereas proposals with a low number of reviewers 
have  1.82  (1:0.55)  lower  odds  of  being  selected  (Table  43).  However,  the  interaction 
between the number of reviewers and the coefficient of variation influences the outcome 
in  a  different  way,  in  comparison  with  ERC  actions.  Namely,  proposals  with  a  medium 
coefficient  of  variation  have  1.54  higher  odds  of  being  selected  when  evaluated  by  a 
medium number of reviewers and 1.64 higher odds of being selected when assessed by a 
low  number  of  reviewers.  These  odds  are  higher  for  proposals  with  a  low  coefficient  of 
variation, namely 4.32 higher odds of being selected when reviewed by a medium number 
of reviewers and 5.03 higher odds of being selected when assessed by a low number of 
reviewers (Table 43). 

CSA  show  a  similar  pattern  in  terms  of  the  influence  of  the  coefficient  of  variation  in 
proposal outcome, as proposals with a medium and low coefficient of variation have 3.33 
and 2.43 higher odds of being selected, respectively, than proposals with a high coefficient 
of  variation  (Table  44).  In  terms  of  reviewer  numbers,  proposals  with  a  low  number  of 
reviewers  have  1.45  higher  odds  of  being  selected,  whereas  the  impact  of  a  medium 
number of reviewers on the proposal outcome was found to be not statistically significant 
(Table  44).  The  interaction  between  number  of  reviewers  and  coefficient  of  variation  is 
similar to SME actions, with proposals with a medium coefficient of variation having 3.69 
higher odds of being selected when evaluated by a medium number of reviewers and 4.36 
higher odds of being selected when assessed by a low number of reviewers. CSA proposals 
with a low coefficient of variation have 2.74 higher odds of being selected when reviewed 
by a medium number of reviewers and 3.02 higher odds of being selected when assessed 
by a low number of reviewers (Table 44). 

For  MSCA,  the  evidence  suggests  that  proposals  with  a  medium  and  low  coefficient  of 
variation have 4.00 and 10.48 higher odds of being selected, respectively, than proposals 
with  a high  coefficient of  variation  (Table  45).  In  terms  of  reviewer  numbers,  proposals 
with  a  low  number  of  reviewers  have  2.21  higher  odds  of  being  selected  in  comparison 
with proposals with a high number of reviewers, whereas the impact of a medium number 
of reviewers on the proposal outcome was found to be not statistically significant (Table 
45).  In  terms  of  interactions  between  coefficient  of  variation  and  number  of  reviewers, 
MSCA  proposals  with  a  medium  coefficient  of  variation  have  4.41  higher  odds  of  being 
selected when evaluated by a medium number of reviewers and 4.80 higher odds of being 
selected when assessed by a low number of reviewers. These odds are higher for proposals 

113 

 
 
with a low coefficient of variation, as proposals reviewed by a medium and low number of 
reviewers have 10.21 and 15.00 higher odds of being selected, respectively (Table 45). 

Within RIA actions, our analysis indicates that proposals with a medium and low coefficient 
of variation have 3.50 and 2.43 higher odds of being selected, respectively, than proposals 
with a high coefficient of variation (Table 46). On the other hand, proposals with a medium 
and low number of reviewers have 1.14 (1:0.88) and 1.075 (1:0.93) lower odds of being 
selected,  respectively,  in  comparison  with  proposals  with  a  high  number  of  reviewers 
(Table  46).  The  interactions  between  coefficient  of  variation  and  number  of  reviewers 
suggest  that  proposals  with  a  medium  coefficient  of  variation  have  2.75  higher  odds  of 
being selected when evaluated by a medium number of reviewers and 2.71 higher odds of 
being selected when assessed by a low number of reviewers. These odds are also higher 
for proposals with a low coefficient of variation, with proposals reviewed by a medium and 
low number of reviewers having 1.88 and 1.51 higher odds of being selected, respectively 
(Table 46). 

Proposals in IA follow a similar pattern to the one described for RIA, with proposals with a 
medium and low coefficient of variation having 3.21 and 2.39 higher odds of being selected, 
respectively,  than  proposals  with  a  high  coefficient  of  variation  (Table  47),  whereas 
proposals  with  a  medium  and  low  number  of  reviewers  have  1.075  (1:0.93)  and  1.19 
(1:0.84)  lower  odds  of  being  selected,  respectively.  In  terms  of  interactions  between 
coefficient of variation and number of reviewers, our findings suggest that proposals with 
a medium coefficient of variation have 2.01 higher odds of being selected when evaluated 
by a medium number of reviewers and 1.97 higher odds of being selected when assessed 
by  a  low  number  of  reviewers.  These  odds  are  also  higher  for  proposals  with  a  low 
coefficient of variation, with proposals reviewed by a medium and low number of reviewers 
having 2.22 and 1.63 higher odds of being selected, respectively (Table 47). 

Overall, this analysis indicates that there is evidence to suggest that the variation in mean 
scores  between  reviewers  and  the  number  of  reviewers  assessing  each  proposal  is 
associated with the outcome of proposal evaluation. Specifically, proposals with medium 
and low variation in scores  have significantly higher odds  of success  in comparison with 
proposals with high variation in scores across all actions. In terms of number of reviewers, 
the  trend  is  less  consistent;  for  some  actions,  a  lower  number  of  reviewers  assessing  a 
proposal negatively impacts the success probability (ERC, the SME instrument, RIA, IA) in 
comparison with proposals assessed by a higher number of reviewers, whereas for other 
actions we see the opposite trend (CSA, MSCA). For all actions, we found that there is a 
significant interaction between the coefficient of variation and the number of reviewers per 
proposal.  Specifically,  for  all  actions  except  ERC,  there  is  evidence  to  suggest  that 
proposals  with  medium  and  low  coefficients  of  variation  in  scores  have  higher  odds  of 
success when reviewed by a medium or low number of reviewers. 

114 

 
 
Table 42: Simple logistic regression of evaluation outcome as a function of number of 
reviewers per proposal and coefficient of variation between reviewer mean scores per 
proposal for ERC actions37,38 

Number of reviewers 

Coefficient of variation between reviewer mean scores  

High (reference class) 

Medium 

Low 

High (reference class) 

1 

1.72 (1.58–1.87); 
p <0.001 *** 

1.69 (1.55–1.83); 
p <0.001 *** 

Medium 

Low 

0.44 (0.40–0.49); 
p <0.001 *** 

1.15 (1.05–1.26); 
p <0.001 *** 

0.93 (0.86–1.02); 
p <0.001 *** 

0.13 (0.12–0.14); 
p <0.001 *** 

0.52 (0.48–0.57); 
p <0.001 *** 

0.39 (0.36–0.43); 
p <0.001 *** 

Table 43: Simple logistic regression of evaluation outcome as a function of number of 
reviewers per proposal and coefficient of variation between reviewer mean scores per 
proposal for SME actions39 

Number of reviewers 

Coefficient of variation between reviewer mean scores 

High (reference class) 

Medium 

Low 

High (reference class) 

1 

2.64 (2.23–3.13); 
p <0.001 *** 

13.36 (11.45–15.60); 
p <0.001 *** 

Medium 

Low 

0.67 (0.55–0.82); 
p <0.001 *** 

1.54 (1.30–1.81); 
p <0.001 *** 

4.32 (3.69–5.04); 
p <0.001 *** 

0.55 (0.47–0.66); 
p <0.001 *** 

1.64 (1.40–1.92); 
p <0.001 *** 

5.03 (4.32–5.86); 
p <0.001 *** 

37 ERC Proof of Concept (POC) grants were excluded from scoring analysis as scoring is not based on 
a 1-5 scale as is the case for the Investigator (Starting, Consolidator and Advanced) and Synergy 
Grants, but rather use a pass/fail system. 

38  OR  adjusted  odds  ratio,  95%  confidence  intervals  (LCI:  lower  confidence  interval;  UCI;  upper 
confidence interval), p: p-value. n=74 095. *** p <0.001; ** p <0.01; * p <0.05. 

39  OR  adjusted  odds  ratio,  95%  confidence  intervals  (LCI:  lower  confidence  interval;  UCI;  upper 
confidence interval), p: p-value. n= 279,944. *** p < 0.001; ** p < 0.01; * p <0.05. 

115 

 
 
 
 
 
                                                 
  
 
  
 
Table 44: Simple logistic regression of evaluation outcome as a function of number of 
reviewers per proposal and coefficient of variation between reviewer mean scores per 
proposal for CSA actions40 

Number of reviewers 

Coefficient of variation between reviewer mean scores  

High (reference class) 

Medium 

Low 

High (reference class) 

1 

3.33 (2.93–3.79); 
p <0.001 *** 

2.43 (2.07–2.84); 
p <0.001 *** 

Medium 

Low 

0.90 (0.78–1.03); 
p = 0.12 

3.69 (3.27–4.18); 
p <0.001 *** 

2.74 (2.37–3.17); 
p <0.001 *** 

1.45 (1.28–1.64); 
p <0.001 *** 

4.36 (3.86–4.91); 
p <0.001 *** 

3.02 (2.64–3.46); 
p <0.001 *** 

Table 45: Simple logistic regression of evaluation outcome as a function of number of 
reviewers per proposal and coefficient of variation between reviewer mean scores per 
proposal for MSCA actions41 

Number of reviewers 

Coefficient of variation between reviewer mean scores  

High (reference class) 

Medium 

Low 

High (reference class) 

1 

4.00 (3.26–4.91); 
p <0.001 *** 

10.48 (8.51–12.91); 
p <0.001 *** 

Medium 

Low 

1.18 (0.96–1.45); 
p = 0.12 

4.41 (3.64–5.35); 
p <0.001 *** 

10.21 (8.41–12.40); 
p <0.001 *** 

2.21 (1.70–2.89); 
p <0.001 *** 

4.80 (3.81–6.04); 
p <0.001 *** 

15.00 (11.77–19.10); 
p <0.001 *** 

40  OR  adjusted  odds  ratio,  95%  confidence  intervals  (LCI:  lower  confidence  interval;  UCI;  upper 
confidence interval), p: p-value. n= 31 519. *** p < 0.001; ** p < 0.01; * p <0.05. 

41  OR  adjusted  odds  ratio,  95%  confidence  intervals  (LCI:  lower  confidence  interval;  UCI;  upper 
confidence interval), p: p-value. n= 19 676. *** p < 0.001; ** p < 0.01; * p <0.05. 

116 

 
 
 
                                                 
  
 
  
 
Table 46: Simple logistic regression of evaluation outcome as a function of number of 
reviewers per proposal and coefficient of variation between reviewer mean scores per 
proposal for RIA actions42 

Number of reviewers 

Coefficient of variation between reviewer mean score 

High (reference class) 

Medium 

Low 

High (reference class) 

1 

3.50 (3.28–3.73); 
p <0.001 *** 

2.43 (2.26–2.61); 
p <0.001 *** 

Medium 

Low 

0.88 (0.82–0.96); 
p = 0.07 

2.75 (2.57–2.94); 
p <0.001 *** 

1.88 (1.75–2.02); 
p <0.001 *** 

0.93 (0.85–1.01); 
p <0.001 *** 

2.71 (2.52–2.91); 
p <0.001 *** 

1.51 (1.39–1.65); 
p <0.001 *** 

Table 47: Simple logistic regression of evaluation outcome as a function of number of 
reviewers per proposal and coefficient of variation between reviewer mean scores per 
proposal for IA actions43 

Number of reviewers 

Coefficient of variation between reviewer mean scores  

High (reference class) 

Medium 

Low 

High (reference class) 

1 

3.21 (2.95–3.49); 
p <0.001 *** 

2.39 (2.16–2.64); 
p <0.001 *** 

Medium 

Low 

0.93 (0.83–1.03); 
p = 0.17 

2.01 (1.83–2.20); 
p <0.001 *** 

2.22 (1.99–2.47); 
p <0.001 *** 

0.84 (0.76–0.94); 
p <0.001 *** 

1.97 (1.80–2.15); 
p <0.001 *** 

1.63 (1.47–1.81); 
p <0.001 *** 

2 

Evaluator survey data 

This section presents the findings to date from our analysis of the evaluator survey data 
for H2020. 

2.1  Demographic characteristics 

42  OR  adjusted  odds  ratio,  95%  confidence  intervals  (LCI:  lower  confidence  interval;  UCI;  upper 
confidence interval), p: p-value. n= 147 531. *** p < 0.001; ** p < 0.01; * p <0.05. 

43  OR  adjusted  odds  ratio,  95%  confidence  intervals  (LCI:  lower  confidence  interval;  UCI;  upper 
confidence interval), p: p-value. n= 62 666. *** p < 0.001; ** p < 0.01; * p <0.05. 

117 

 
 
 
 
 
 
                                                 
  
 
  
 
Our  analysis  of  the  demographic  characteristics  of  reviewers  shows  that  most  survey 
respondents were male (58.6%), between 35 and 49 years old (51.4%) and from higher 
education establishments (42.7%) (Figure 14). Additionally, most responses were related 
to  the  Excellent  Science  Pillar  (43%).  The  predominant  nationality  was  Italian  (12.2%), 
followed by Spanish (10.0%) (Figure 15). 

Figure 14: Demographic characteristics of reviewers responding to the survey assessing 2016/2017 H2020 calls 

118 

 
 
 
 
 
Figure 15: Distribution of nationalities of reviewers responding to the survey assessing 2016/2017 H2020 calls. FYROM = North 
Macedonia 

119 

 
 
 
 
 
2 : Background materials 

3 : Evaluation criteria 

4 : Session 

6 : Evaluators 

7 : Exceeded pages 

8 : Remote process 

9 : Timelines 

11 : Communication 

12 : Connection 

13 : Interaction 

14 : Number of experts 

15 : Questions 

17 : Moderator 

18 : Organisation 

19 : Panel members 

20 : Process 

21 : Timelines 

23 : Criteria 

24 : Evaluators 

25 : Information 

26 : Interpretation 

27 : Overlap 

28 : Process 

29 : Scoring 

30 : Time commitment 

31 : Timelines 

Male reviewer 

Briefing sessions 

49.68% 

35.56% 

84.07% 

Consensus meetings 

49.07% 

69.8% 

51.06% 

62.16% 

Hearings meetings 

0% 

100% 

39.02% 

100% 

31.25% 

Panel meetings 

20.08% 

31.46% 

18.62% 

83.38% 

49% 

Peer review 
58.29% 

66.7% 

52.26% 

53.12% 

15.94% 

73% 

97.26% 

59.21% 

40.34% 

Female reviewer 

50.32% 

64.44% 

15.93% 

50.93% 

30.2% 

48.94% 

37.84% 

100% 

0% 

60.98% 

0% 

68.75% 

79.92% 

68.54% 

81.38% 

16.62% 

51% 

41.71% 

33.3% 

47.74% 

46.88% 

84.06% 

27% 

2.74% 

40.79% 

59.66% 

Figure 16: Demographics (gender) for feedback provided as free text in the survey.  

120 

 
 
 
 
 
 
 
 
 
Age = under 35 

Age = 35–49 

Age = 50–59 

Age = over 60 

2 : Background materials 

3 : Evaluation criteria 

4 : Session 

6 : Evaluators 

7 : Exceeded pages 

8 : Remote process 

9 : Timelines 

11 : Communication 

12 : Connection 

13 : Interaction 

14 : Number of experts 

15 : Questions 

17 : Moderator 

18 : Organisation 

19 : Panel members 

20 : Process 

21 : Timelines 

23 : Criteria 

24 : Evaluators 

25 : Information 

26 : Interpretation 

27 : Overlap 

28 : Process 

29 : Scoring 

30 : Time commitment 

31 : Timelines 

5.25% 

0% 

0% 

2.5% 

11.57% 

2.17% 

0% 

0% 

0% 

0% 

0% 

0% 

9.45% 

0% 

0% 

1.63% 

13.55% 

5.1% 

4.59% 

8.01% 

0% 

0% 

8.6% 

0% 

0% 

0% 

Briefing sessions 

59.39% 

42.78% 

32.96% 

Consensus meetings 

61.14% 

45.93% 

61.08% 

76.02% 

Hearings meetings 

0% 

0% 

0% 

0% 

0% 

Panel meetings 

54.33% 

46.41% 

84.48% 

86.65% 

84.46% 

Peer review 
50.52% 

54.54% 

63.07% 

35.48% 

54.11% 

53.53% 

23.95% 

93.42% 

86.1% 

26.75% 

55.56% 

38.52% 

27.11% 

23.33% 

27.88% 

7.73% 

100% 

100% 

60.98% 

100% 

68.75% 

36.22% 

53.59% 

15.52% 

2.45% 

0% 

21.56% 

33.05% 

26.13% 

37.2% 

38.16% 

36.93% 

68.11% 

0% 

10.17% 

Figure 17: Demographics (age) for feedback provided as free text in the survey 

8.6% 

1.67% 

28.52% 

9.26% 

19.17% 

8.87% 

16.25% 

0% 

0% 

39.02% 

0% 

31.25% 

0% 

0% 

0% 

9.26% 

1.99% 

22.82% 

7.82% 

2.79% 

27.31% 

7.73% 

0.93% 

7.94% 

6.58% 

3.73% 

121 

 
 
 
 
 
Public 
research 
centres, 
among 
which 
international 
research 
centres, 
among 
which joint 
research 
centres 

Higher 
education 
establishments 

Non-
research 
public 
sector 

Non-research 
commercial sector – 
including SMEs, among 
which consultancy 
firms 

Other 
organisation 
type 

Private/commercial 
research centres 

Private non-
profit 
research 
centres 

Non-
research 
private non-
profit 

Non-research 
international 
organisations  

Briefing sessions 

2: Background materials 

3: Evaluation criteria 

4: Session 

Consensus meetings 

6: Evaluators 

7: Exceeded pages 

8: Remote process 

9: Timelines 

Hearings meetings 

11: Communication 

12: Connection 

13: Interaction 

14: Number of experts 

15: Questions 

Panel meetings 

17: Moderator 

18: Organisation 

19: Panel members 

17.68% 

30.56% 

0% 

14.03% 

9.04% 

7.51% 

25.45% 

100% 

0% 

60.98% 

0% 

68.75% 

12.6% 

0% 

7.24% 

53.03% 

12.22% 

30.74% 

57.83% 

23.87% 

68.24% 

46.82% 

0% 

0% 

0% 

0% 

0% 

7.17% 

52.78% 

0% 

2.69% 

0% 

0.74% 

8.52% 

0% 

0% 

0% 

100% 

0% 

54.72% 

50.49% 

65.86% 

0% 

17.09% 

11.38% 

12.1% 

1.67% 

33.7% 

8.02% 

43.94% 

8.71% 

3.86% 

0% 

0% 

39.02% 

0% 

31.25% 

32.68% 

32.43% 

0% 

0% 

2.78% 

28.15% 

2.36% 

3.07% 

3.53% 

5.8% 

0% 

0% 

0% 

0% 

0% 

0% 

0% 

0% 

5.1% 

0% 

7.41% 

5.87% 

10.13% 

1.98% 

0% 

0% 

100% 

0% 

0% 

0% 

0% 

0% 

0% 

4.94% 

0% 

0% 

4.71% 

0% 

6.77% 

2.39% 

0% 

0% 

0% 

0% 

0% 

0% 

0% 

15.52% 

0% 

0% 

0% 

4.5% 

9.95% 

0% 

7.16% 

0% 

0% 

0% 

0% 

0% 

0% 

0% 

0% 

0% 

0% 

0% 

0% 

0% 

2.52% 

0% 

0% 

0% 

0% 

0% 

0% 

0% 

0% 

0% 

122 

 
 
 
 
20: Process 

21: Timelines 

Peer review 

23: Criteria 

24: Evaluators 

25: Information 

26: Interpretation 

27: Overlap 

28: Process 

29: Scoring 
30: Time commitment 

31: Timelines 

8.45% 

0% 

23.27% 

25.98% 

22.07% 

4.73% 

22.71% 

4.53% 

0% 

13.16% 
33.56% 

70.16% 

70.92% 

37.4% 

52.62% 

41.81% 

63.01% 

20.77% 

67.93% 

55.41% 

86.84% 
33.22% 

6.27% 

18.73% 

4.09% 

0.71% 

0% 

0% 

34.3% 

6.53% 

0% 

0% 
0% 

15.12% 

10.36% 

19.13% 

12.51% 

28.46% 

23.23% 

0% 

8.13% 

23.38% 

0% 
30.85% 

0% 

0% 

7.82% 

2.83% 

6.27% 

0% 

20.77% 

0% 

0% 

0% 
2.37% 

0% 

0% 

0.99% 

1.51% 

1.39% 

0% 

0% 

3.93% 

0% 

0% 
0% 

0% 

0% 

1.01% 

3.28% 

0% 

9.03% 

0% 

3.67% 

0% 

0% 
0% 

0% 

0% 

6.28% 

0% 

0% 

0% 

0% 

5.27% 

21.21% 

0% 
0% 

0% 

0% 

0% 

0.55% 

0% 

0% 

1.45% 

0% 

0% 

0% 
0% 

Figure 18: Demographics (organisation) for feedback provided as free text in the survey 

123 

 
 
 
Previous 
experience = No 

Briefing sessions 
49.68% 

66.67% 

73.33% 

Consensus meetings 
55.64% 

87.52% 

82.5% 

70.68% 

Hearings meetings 
100% 

100% 

100% 

100% 

100% 

Panel meetings 
74.8% 

63.11% 

62.07% 

71.8% 

78.49% 

Peer review 
63.1% 

55.5% 

46.46% 

90.11% 

69.08% 

49.07% 

26.12% 

65.79% 

74.58% 

2 : Background materials 

3 : Evaluation criteria 

4 : Session 

6 : Evaluators 

7 : Exceeded pages 

8 : Remote process 

9 : Timelines 

11 : Communication 

12 : Connection 

13 : Interaction 

14 : Number of experts 

15 : Questions 

17 : Moderator 

18 : Organisation 

19 : Panel members 

20 : Process 

21 : Timelines 

23 : Criteria 

24 : Evaluators 

25 : Information 

26 : Interpretation 

27 : Overlap 

28 : Process 

29 : Scoring 

30 : Time commitment 

31 : Timelines 

Previous 
experience = Yes 

50.32% 

33.33% 

26.67% 

44.36% 

12.48% 

17.5% 

29.32% 

0% 

0% 

0% 

0% 

0% 

25.2% 

36.89% 

37.93% 

28.2% 

21.51% 

36.9% 

44.5% 

53.54% 

9.89% 

30.92% 

50.93% 

73.88% 

34.21% 

25.42% 

Figure 19: Demographics (previous experience) for feedback provided as free text in the survey 

Our analysis shows that there are two main classes within the pool of survey respondents. 
These classes are mostly distinguished by age and, to a lesser extent, by gender and by 
experience with being an evaluator for an EU research programme. 

  Gender-diverse, below 50 years old and diverse experience class: These reviewers 
have  a  53%  probability  of  being  male,  a  11%  probability  of  being  over  50  years 
old, a 56% probability of being affiliated with a non-HEE organisation and a 57% 
probability of not being a first-time evaluator. 

  Predominately  male,  over  50  years  old  and  experienced  evaluator  class:  These 
reviewers have a 68% probability of being male, a 99% probability of being over 

124 

 
 
 
 
 
 
50 years old, a 59% probability of being over 50 years old and an 83% probability 
of not being a first-time evaluator. 

Reviewers taking part in the survey have a 64% probability of belonging to the  ‘gender-
diverse,  below  50  years  old  and  diverse  experience’  class  and  a  36%  probability  of 
belonging to the ‘predominately male, over 50 years old and experienced evaluator’ class 
(Figure 20). 

Figure 20: Latent class analysis of reviewers responding to the survey assessing 2016/2017 H2020 calls. HEE = Higher 
Education Establishment 

2.2  Survey analysis 

In general, the survey respondents had mostly positive feedback on the H2020 proposal 
evaluation process. Specifically, most respondents mentioned that they would participate 
again as an evaluator, that the invitation to review was received in sufficient time and that 
there  was  also  sufficient  time  to  evaluate  the  proposals.  Additionally,  most  reviewers 
responding to the survey felt that proposals fell within their area of expertise. However, 
most respondents felt that the days fees did not reflect the time needed to evaluate the 
proposals (Figure 21). 

125 

 
 
 
 
 
Figure 21: Survey answers on different aspects of the proposal evaluation process 

Regarding the different stages of the evaluation process, the briefings meetings, hearing 
meetings, peer review and panel meetings received mostly positive scores. On the other 
hand, there is more variation in scores for consensus meetings, with approach for exceeded 

126 

 
 
 
 
pages  limits44,  other  evaluators  rating45  and  process  rating46  scoring  low.  There  is  also 
some  more  negative  feedback  on  the  logistics  of  the  evaluation  in  terms  of  the 
reimbursement process47. This is set out in more detail in the following sections. However, 
in terms of the overall process, most respondents were largely satisfied, as shown in Figure 
22. 

44 Survey question: “How would you rate the evaluation approach for proposals that exceeded page 
limits (i.e. watermarked)?” 

45 Survey question: “How would you rate…: the contribution of the other evaluators in the consensus 
discussion?” 

46 Survey question: “How would you rate…: the process of reaching consensus?” 

47 Survey question: “To what extent were you satisfied with the overall reimbursement process?” 

127 

 
 
 
                                                 
Figure 22: Survey answers on the level of satisfaction with the proposal evaluation process overall 

2.3  Analysis by stage of the process 

To  obtain  a  more  granular  view  on  the  feedback  provided  by  reviewers,  we  performed 
qualitative  analysis  on  a  sample  of  free-text  answers  from  the  survey.  Our  sample 
consisted  of  the  free-text  answers  associated  with  the  25%  lowest-  and  25%  highest-
scoring quantitative answers. Our analysis identified that the vast majority of the free-text 
answers were associated with the lowest-scoring quantitative answers. For this reason, our 
analysis focused on the negative feedback provided by reviewers. 

The  free-text  answers  were  analysed  in  the  NVIVO  software,  using  an  iterative 
categorization  approach,  whereby  codes  were  developed  to  represent  the  key  themes 
emerging from the textual answers. Our analysis shows that consensus meetings received 
the most feedback, particularly on the evaluators and remote processes. The peer review 
stage  also  received  considerable  feedback,  in  terms  of  criteria,  evaluators  and  process. 
Hearings and panel meetings received the least feedback in the survey (Figure 23). 

128 

 
 
 
 
Figure 23: NVIVO analysis of the number of mentions of issues related to the coded themes at each stage of the proposal 
evaluation process 

In the following sections, we present the findings by each stage of the process, presenting 
the overall scoring, as well as the concerns and issues raised in relation to aspects of the 
process from those who were less satisfied. 

Briefing sessions 

As we noted above, briefing sessions primarily received positive scores in the quantitative 
aspect of the survey (Figure 24). This is reflected in the limited number of comments on 
this topic. However, we do see some issues raised by those least satisfied with this aspect 
of the process. The main issues mentioned by reviewers relate to the background materials 
provided, the evaluation criteria and the session it itself. Reviewers  raising these issues 
specifically mentioned the following: 

 

lack  of  useful  and  sufficient  information  provided  to  evaluators  at  the  briefing 
session, 

  no information on how to harmonise different evaluations, 

  materials overwritten, redundant and contradictory at times, requiring considerable 

time to read and understand, 

 

 

lack  of  clear  information  on  the  evaluation  criteria  and  rules,  which  introduces 
variability among evaluators, 

lack of organisation and coordination in the session itself, which made it take longer 
than necessary, 

 

technical issues related to information technology (IT) and sound quality. 

129 

 
 
 
 
 
 
Figure 24: Survey answers on the level of satisfaction with the briefing sessions 

Consensus meetings 

Consensus meetings were one of the areas where the quantitative scoring was more mixed 
(Figure  25),  and  this  is  reflected  in  the  qualitative  responses,  with  many  comments 
received,  in  particular  in  relation  to  the  evaluators  and  the  remote  process.  These 
comments mentioned the following: 

 

 

lack  of  expertise  and  communication  ability  of  reviewers,  as  well  as  the  lack  of 
seniority, 

lack  of  neutrality  in  the  scoring  and  undue  influence  from  the  moderator  in  the 
decision-making process, 

  uneven contributions to the discussion, with stronger personalities dominating the 
conversation, and a hostile environment, namely in terms of the unwillingness to 
compromise, 

 

lack of quality and transparency in consensus reports, and lack of clarity in the role 
of the participants in the meeting. 

There  was  extensive  criticism  on  the  remote  setting  for  consensus  meetings.  Reviewers 
mentioned the following regarding the remote setting: 

 

chaotic,  biased  and  unfair  process,  as  well  as  inefficient,  unstructured,  time 
consuming, with significant over-run of evaluator time and effort, particularly when 
there were discrepancies between individual evaluations, 

  very susceptible to misunderstandings, and lacking a clear procedure, 

  online tool cumbersome in supporting the remote process. 

There were also mentions of issues related to timelines, specifically regarding the lack of 
time to discuss and reach consensus for the number of proposals. Additionally, there were 

130 

 
 
 
 
also issues with reviewers not initiating work on time and not replying in a timely manner 
to  comments,  causing  delays  in  the  process.  A  number  of  issues  were  also  raised 
concerning exceeded page limits, with the reviewers feeling there was no standard process 
or guidance for handling them. 

However, it is important to note that remote consensus was introduced for the first time 
in H2020 calls in 2016 and that changes have been made since then to improve the system, 
including  further  guidance  and  support  to  reviewers  during  the  process.  Therefore,  it  is 
possible that our analysis, which focuses on surveys from 2016/2017, reflects issues with 
implementing  a  new  process  and  that  are  not  a  reflection  of  the  process  at  the  end  of 
H2020. 

Figure 25: Survey answers on the level of satisfaction with consensus meetings 

131 

 
 
 
 
 
Hearings meetings 

Hearings meetings were largely positively received in the quantitative scoring (Figure 26), 
and this is reflected in low number of text comments on those issues. Among the comments 
made, most feedback focused on issues around the communication abilities of the people 
representing the consortium, the lack of discussion with applicants, the small number of 
to 
experts  participating,  and 
misunderstandings. Additionally, the poor network connection was also mentioned as an 
issue at these meetings. 

the  generic  questions  being  asked,  which 

led 

Figure 26: Survey answers on the level of satisfaction with hearings meetings 

Panel meetings 

Panel  meetings  were  generally  positively  received  in  terms  of  the  quantitative  scores 
received in the survey (Figure 27), and this is reflected in the relatively low number of text 
comments on this area. Among the comments received, reviewers mentioned that there 
was not enough time to evaluate and discuss all proposals, while too much time was spent 
on reading documents. There was a lack of guidance on the criteria and process for scoring, 
as well as a lack of information on what to expect from the meeting. As in the consensus 
meetings, issues were raised regarding the lack of expertise of the reviewers and the bias 
and lack of engagement of some participants, while others dominated the meeting. There 
were  also  mentions  of  undue  pressure  from  moderators  on  reviewers  regarding  funding 
decisions  and  a  poor  communication  style  in  moderating  the  meeting.  Reviewers  also 
mentioned  that  there  were  difficulties  with  harmonising  the  opinions  of  too  many 
evaluators. 

132 

 
 
 
 
Figure 27: Survey answers on the level of satisfaction with panel meetings 

Peer review 

In the quantitative aspect of the survey, peer review largely received positive scores from 
respondents  (Figure  28).  However,  among  those  less  satisfied  with  this  aspect  of  the 
process,  we  do  see  a  large  number  of  comments,  particularly  focused  on  the  criteria 
reviewed. A range of issues were raised. Those with a negative view of this aspect of the 
process mentioned the following about the process. 

  The process was complex, inefficient, lacking in fairness, and time consuming. 

  The  evaluation  was  too  focused  on  the  formal  aspects  of  the  application  and  too 

restrictive, bureaucratic and formalistic. 

Additionally, a number of issues were flagged regarding the criteria, evaluators, timelines 
and time commitment. Specifically, reviewers mentioned the following. 

  There were difficulties in discriminating between proposals that reached top scores, 

and it was challenging to apply the criteria to middle range proposals. 

 

It was difficult to homogenise criteria for proposals from diverging fields, and the 
criteria for judging a proposal were not always clearly applicable in the context of a 
particular proposal. 

 

In some cases, rigid criteria led proposals to be unfairly penalised. 

133 

 
 
 
 
  Evaluation  criteria  were  worded  in  an  ambiguous  way  and  lacked  specific 
information,  with  no  clear  instructions  on  scoring  or  on  handling  scoring 
discrepancies between evaluators. 

  Resolution  of  the  scores  was  lacking  in  granularity,  and  their  interpretation  was 
subjective,  depending  on  the  background  of  each  expert.  This  led  to  arbitrary 
interpretations  and  difficulty  in  consistently  applying  criteria  across  different 
evaluations.  The  different  approaches  to  the  same  criteria  eventually  led  to 
difficulties in reaching consensus at a later stage of the proposal evaluation. 

  There  was  overlap  between  the  different  criteria  and  difficulty  in  distinguishing 

between the numerous subcomponents. 

  There was a lack of expertise and low seniority of  reviewers, as well as a lack of 
impartiality in scoring, bias and unwillingness to compromise. There was also a lack 
of command of English and weak communication skills. 

 

It  was a time-consuming and repetitive process, not allowing for the  appropriate 
amount of time to evaluate proposals. 

Figure 28: Survey answers on the level of satisfaction with peer review 

134 

 
 
 
 
Logistics 

In the quantitative scoring, the logistics of the proposal evaluation process received mixed 
responses  (Figure  29  and  Figure  30).  There  was  also  some  specific  feedback  received 
regarding  these  logistical  aspects  of  the  proposal  evaluation  process.  Specifically, 
reviewers felt that there were instances where conflict of interest and bias were particularly 
evident from reviewers regarding sector or nationality of applicants or  regarding specific 
projects; however, no appropriate action was taken. There were issues raised regarding 
the payment process; reviewers mentioned that it was not clear which documents needed 
to  be  uploaded,  and  some  mentioned  difficulties  with  uploading  the  documents. 
Additionally, it was reported that there was a lack of instructions or guidance, the payment 
process was too lengthy, the IT system was not user-friendly and the IT help desk did not 
provide  adequate  support.  Some  reviewers  also  mentioned  that  the  buildings  at  which 
meetings took place had issues with internet connection and there was a lack of directions 
or  information  to  guide  the  evaluator  through  the  building,  which  was  particularly 
challenging for visually impaired evaluators. 

Figure 29: Survey answers on the level of satisfaction with issues related to conflict of interest 

135 

 
 
 
 
 
Figure 30: Survey answers on the level of satisfaction with logistics of the evaluation for evaluation carried out centrally in 
Brussels 

136 

 
 
 
 
 
 
Demographics 

Gender 

Our analysis shows that there were  some differences in the opinion of female and male 
reviewers across the different stages of the proposal evaluation process. 

  Regarding  briefing  sessions,  most  of  the  negative  feedback  on  the  evaluation 
criteria came from female reviewers, whereas most of the feedback on the session 
itself was provided by male reviewers. 

  Regarding consensus meetings, male reviewers provided more negative feedback 
across all the main issues,  except  on the  evaluators, for which the feedback was 
provided in almost equal amounts by male and female reviewers. 

  For the hearings meetings, issues with communication were raised solely by female 
reviewers,  whereas  issues  with  connection  and  number  of  experts  were  reported 
solely  by  male  reviewers.  Female  reviewers  provided  more  feedback  on  issues 
related to interactions. 

  For panel meetings, most issues were raised by female reviewers, namely regarding 
the moderator, the organisation and the panel members. On the other hand, male 
reviewers provided more feedback on the process. Both genders provided a similar 
amount of feedback on the timeline issues. 

  Regarding the peer review process, most issues were reported by male reviewers, 
except  the  overlap  in  criteria  and  the  timelines  in  the  process,  which  were  more 
commented on by female reviewers. 

Figure 31: NVIVO analysis of the number of mentions of issues related to the coded themes at each stage of the proposal 
evaluation process, by reviewer gender 

137 

 
 
 
 
 
 
Age 

In terms of age, our analysis shows that reviewers aged 35–49 and 50–59 provided most 
of the feedback. 

  For briefing sessions, both age groups provided similar amounts of feedback on the 

issues raised. 

  For consensus meetings, most issues were raised by reviewers aged 35–49. 

  For  hearings  meetings,  the  majority  of  the  feedback  was  provided  by  reviewers 

aged 50–59. 

  For panel meetings and peer review, most feedback was provided by the 35–49 age 

group. 

Figure 32: NVIVO analysis of the number of mentions of issues related to the coded themes at each stage of the proposal 
evaluation process, by reviewer age 

Affiliation 

In  terms  of  affiliation,  overall  most  feedback  was  provided  by  reviewers  affiliated  with 
higher  education  establishments.  However,  some  specific  issues  were  raised  more 
frequently by reviewers with other affiliations. 

  Reviewers  in  public  research  centres  reported  most  significantly  on  hearings 

meetings, raising issues about communication, interaction and questions. 

  Reviewers  in  the  non-research  public  sector  raised  issues  about  the  number  of 

experts at hearings meetings. 

  Reviewers  in  private/commercial  research  centres  raised  issues  with  the  network 

connection at the hearings meetings. 

138 

 
 
 
 
 
Figure 33: NVIVO analysis of the number of mentions of issues related to the coded themes at each stage of the proposal 
evaluation process, by reviewer affiliation 

Experience 

In terms of previous experience with evaluating, most feedback was provided by reviewers 
with  no  previous  experience  of  the  proposal  evaluation  process.  This  feedback  was 
particularly frequent regarding consensus meetings and hearings meetings. Reviewers with 
previous experience provided the most feedback on the scoring process of the peer review. 

Figure 34: NVIVO analysis of the number of mentions of issues related to the coded themes at each stage of the proposal 
evaluation process, by reviewer experience 

139 

 
 
 
 
 
 
 
 
 
3 

Reviewers survey and applicants surveys on burden 

This section sets out the findings of the survey of applicants and evaluators conducted on 
the burden of the proposal evaluation process. 

3.1  Respondent characteristics 

We  sent  the  survey  to  58 770  individuals,  a  group  comprised  of  39 068  applicants  and 
19 702  evaluators.  The  response  rate  varied  across  stakeholder  groups,  with  the  final 
sample having 5 369 applicants (response rate = 14%), and 7 049 evaluators (response 
rate = 36%). Of the applicants, 27% were female, 67% were male; <1% preferred to self-
describe (as non-binary, gender-neutral, or other); and 2% preferred not to say. For 4%, 
the gender information was missing. Of the evaluators, 40% were female; 55% were male; 
<1%  preferred  to  self-describe  (as  non-binary,  gender-neutral  or  other);  and  <2% 
preferred not to say. For 3%, the information was missing. 

Figure 35: Number of survey participants across the two stakeholder groups 

Figure 36: Number of survey participants across the two stakeholder groups, by gender 

The  distribution  of  survey  participants  across  different  country  groups  is  highlighted  in 
Figure 37 and Figure 38. 

140 

 
 
 
 
 
Figure 37: Applicant survey participants by current country of work 

141 

Country groupingCountryNumber of participantsAustria144Belgium142Bulgaria33Cyprus28Croatia24Czech Republic60Denmark130Estonia28Finland142France391Germany465Greece160Hungary33Ireland122Italy807Latvia17Lithuania25Luxembourg19Malta8Netherlands227Poland77Portugal183Romania49Slovakia25Slovenia51Spain766Sweden136United Kingdom300Unspecified2Total4594Armenia2Albania4Bosnia and Herzegovnia3Georgia4Faroe Islands1FYROM4Iceland12Israel76Moldova3Montengro1Norway147Serbia20Switzerland144Turkey91Unspecified2Total514Third CountriesTotal17Missing244Total5369EU Member States or the UKAssociated Countries 
 
 
 
Figure 38: Evaluator survey participants by current country of work 

142 

Country groupingCountryNumber of participantsAustria168Belgium186Bulgaria83Cyprus20Croatia96Czech Republic79Denmark92Estonia37Finland143France630Germany584Greece267Hungary89Ireland154Italy698Latvia19Lithuania53Luxembourg30Malta24Netherlands231Poland278Portugal323Romania122Slovakia42Slovenia64Spain700Sweden170United Kingdom634Unspecified6Total6022Armenia0Albania5Bosnia and Herzegovnia7Georgia1Faroe Islands1FYROM3Iceland4Israel55Moldova2Montengro3Norway83Serbia52Switzerland113Tunisia3Turkey106Ukraine15Unspecified2Total455Third CountriesTotal347Missing255Total7049EU Member States or the UKAssociated Countries 
 
 
 
 
 
3.2  Burden across key tasks in the proposal evaluation process 

We  asked  survey  respondents  across  the  two  stakeholder  groups  (applicants  and 
evaluators) to provide an estimation of the time spent (in hours or days, as appropriate) 
on a range of different, specific tasks associated with the proposal evaluation process. To 
facilitate  our  analysis  of  burden,  we  grouped  tasks  into  key  stages  across  the  two 
stakeholder groups. We examined how burden was distributed within stakeholder groups 
across  tasks  and,  where  information  was  available,  across  funding  stream  clusters  and 
demographic groups. 

3.3  Burden across applicant tasks 

Overall, on average, applicants spend around 25 days on a single-stage application and 47 
days on a multi-stage application (including co-applicant time). The majority of this time 
was spent on preparing written proposals, as shown in Figure 39. The following sections 
set out how this time breaks down across tasks in detail, and they highlight any differences 
in time taken between groups of applicants by gender, country and prior experience. 

Figure 39: Burden across the application process 

143 

 
 
 
 
 
 
Written proposal 

When considering the written proposal submission, we see that the highest burden fell on 
applicants  when  writing  the  full  proposal,  where  they  spent  on  average  20  days  per 
application  within  the  single-stage  submission  and  on  average  25  days  per  application 
within  the  multi-stage  submission.  The  second-highest  burden  fell  on  applicants  when 
writing the first-stage proposal, where they spent on average 15 days per application. Co-
applicants spent approximately 2 days contributing to the first-stage proposal and 3.3 days 
contributing to the full proposal48. 

Figure 40: Median time spent on written proposal 

Comparison between male and female applicants found that, on average, female applicants 
spent  20  days  writing  the  first-stage  proposal,  which  was  significantly  longer  when 
compared  with  male  applicants,  who  spent  approximately  14  days  on  average  (Figure 
41)49. Female applicants also spent significantly longer on the full application as part of the 
multi-stage  submission  process,  spending  on  average  30  days,  compared  with  male 
applicants,  who  spent  on  average  20  days50.  Regarding  the  single-stage  applications, 
female applicants spent significantly more time writing the application, on average, than 
male  applicants,  spending  30  days  per  application,  compared  with  20  days  for  male 
applicants51. 

Comparison between applicants working in different countries found that applicants from 
EU  Member  States  or  the  UK  spent  the  same  time  on  average  writing  the  first-stage 

48 It should be noted that applicants were asked to estimate time spent by the co-applicants on the 
proposal and this may be hard to estimate for the applicant; therefore reported time taken by co-
applicants is likely to be an underestimate.  

49 T(592) = 2.24, p = 0.0257 

50 T(283) = 2.53, p = 0.012 

51 T(1785) = 5.17, p  <0.001 

144 

 
 
 
 
                                                 
proposal (median = 15 days) when compared with applicants from associated counties52. 
For the full proposal within the multi-stage process, applicants from EU Member States or 
the UK spent the most time on average when writing the full proposal (median = 27 days) 
compared  with  applicants  from  associated  countries  (median = 20  days).  Finally,  when 
considering  full  proposal  submissions  to  the  single-stage  process,  applicants  from  all 
countries  spent  approximately  the  same  amount  of  time  on  their  applications:  for  EU 
Member States or the UK, the median was 21 days and for EU-associated countries, the 
median was 20 days. Comparison between applicants working within the Member States 
grouped under widening participation group53 or ‘other’ showed that despite spending the 
same  amount  of  time  on  their  first-stage  proposal  (median = 15  days),  applicants  from 
‘other’ Member States spent more time on their full proposal (median = 29 days) compared 
with  applicants  from  widening  participation  group  Member  States  (median = 22.5  days) 
within  the  multi-stage  process,  although  the  mean  average  was  not  statistically 
significant54. Time spent on the full proposal within the single-stage application was about 
the same. 

Comparison between applicants with experience of H2020 and those without found that for 
the first-stage proposal, both applicants with and without experience of H2020 spent on 
average the same  amount of time on their proposals (median = 15 days).  However,  for 
the  full  proposal  within  the  multi-stage  submission,  applicants  with  prior  experience  of 
H2020 spent longer than those with no experience (median = 29 days, compared with 20 
days), although the mean average time was not statistically significant55. For the single-
stage proposal submission, applicants who had experience with H2020 spent on average 
the same amount of time compared with applicants who had no experience (median = 20 
days). 

52  Applicants  from  third  countries  (e.g.  outside  of  the  EU  Member  States,  UK  or  EU–associated 
countries) were excluded from the analysis as the sample size was very small (N = 17).  

53  Member  States  belonging  to  the  widening  participation  group  are  Bulgaria,  Croatia,  Cyprus, 
Czechia,  Estonia,  Hungary,  Latvia,  Lithuania,  Luxembourg,  Malta,  Poland,  Portugal,  Romania, 
Slovakia and Slovenia. 

54 T(66) = −0.5, p = 0.6181 

55 T(221) = 0.74, p = 0.46 

145 

 
 
                                                 
Figure 41: Median time spent on written proposal, by gender, country of work, and previous experience of H202056. MS = Member State 

56 Applicants from third countries (e.g. outside of the EU Member States, UK or EU–associated countries) were excluded from the analysis as the sample size was 
very small (N = 17). 

146 

 
 
 
                                                 
Interview 

Regarding time spent on the interview, there is a statistically significant difference between 
how long applicants spent on the different tasks (Figure 42)57. Applicants spent the longest 
time  preparing  for  their  interview,  spending  on  average  30  hours  per  application. 
Applicants spent on average 6 hours travelling to the interview and 2 hours attending the 
interview. 

Figure 42: Time spent on interview 

Comparison between male and female applicants found that both spent the same amount 
of time attending their interviews (2 hours), but male applicants spent on average slightly 
more time travelling to their interview (8 hours, compared with 5 hours). Both male and 
female applicants spent the same amount of time on average preparing for their interview 
(median = 30 hours) (Figure 43)58. Applicants from EU-associated countries spent slightly 
longer both on preparing for the interview (median = 32.5 hours) and on travelling to their 
interview (median = 8 hours) when compared with applicants from the EU Member States 
or  the  UK  (median = 30  hours  and  6  hours,  for  preparation  time  and  travel  time, 
respectively)59.  Time  spent  attending  the  interview  was  the  same  for  both  groups 
(median = 2  hours).  Comparison  between  applicants  working  within  the  Member  States 
grouped  under  widening  participation  group60  or  ‘other’  found  that  applicants  grouped 
within the ‘Other Member States’ spent longer preparing for their interview (median = 30 
hours)  compared  with  applicants  from  Member  States  within  the  Widening  Participation 
Group  (median = 18  hours);  however,  the  difference  in  means  was  not  statistically 

57 ANOVA (F2 866) = 164, p <0.001 

58 ANOVA (F2 842) = 0.326, p = 0.722 

59  Applicants  from  ‘third  countries’  (e.g.  outside  of  the  EU  Member  States,  the  UK  or  associated 
countries) were excluded from the analysis as the sample size was very small (N = 17). 

60  Member  States  belonging  to  the  widening  participation  group  are  Bulgaria,  Croatia,  Cyprus, 
Czechia,  Estonia,  Hungary,  Latvia,  Lithuania,  Luxembourg,  Malta,  Poland,  Portugal,  Romania, 
Slovakia and Slovenia. 

147 

 
 
 
 
                                                 
significant61. Finally, applicants who had prior experience with H2020 spent slightly more 
time preparing for their interview, spending 30 hours compared with 27 hours. Time spent 
attending did not differ depending on whether applicants had experience or not (median + 
2 hours for both groups), and time spent travelling to the interview differed slightly, with 
applicants  who  were  experienced  spending  6  hours,  compared  with  9.5  hours  for  those 
without prior experience. 

61 T(35) = −0.12, p = 0.904 

148 

 
 
 
                                                 
Figure 43: Median time spent on interview, by gender, country of work, and previous H2020 experience62. MS = Member State 

62 Applicants from third countries (e.g. outside of the EU Member States, UK or Associated Countries) were excluded from the analysis as the sample size was 
very small N = 17. 

149 

 
 
 
                                                 
Perceptions of burden 

When survey participants were asked which parts of the proposal submission process they 
found  burdensome,  76%  of  applicants  (4 094  applicants  out  of  5 369)  stated  that  they 
found the full submission process burdensome, and 21% of applicants (1 121 out of 5 369) 
stated that they found the first-stage submission process burdensome (Figure 44). Only 
2%  of  applicants  (109  out  of  5 369)  stated  that  they  found  the  interview  process 
burdensome;  however,  this  may  be  reflective  of  the  fact  that  for  many  of  the  survey 
participants,  an  interview  did  not  feature  in  their  application  process  (of  the  survey 
participants, 827 had a possible interview as part of their submission, and of these, 321 
were invited to interview). 

Figure 44: Applicant perception of burden 

Submission tool and proposal feedback 

Regarding the proposal submission tool, 44% of applicants stated that they found the tool 
extremely user-friendly or very user-friendly (2 221 applicants) (Figure 45). In addition, 
41% of applicants stated that they found the tool somewhat user-friendly. A further 15% 
of applicants found the tool not so user-friendly or not at all user-friendly. These results 
together suggest that the proposal submission tool does not put additional burden on the 
applicants. 

150 

 
 
 
 
Figure 45: Applicant perception of the proposal submission tool 

Applicant  perception  on  the  usefulness  of  feedback  differed  between  successful  and 
unsuccessful  applicants  (Figure  46).  Where  applicants  had  been  successful,  58%  found 
feedback was either extremely useful or very useful. This was in contrast to unsuccessful 
applicants where only 20% thought so. In addition, 41% of unsuccessful applicants found 
the feedback not so useful or not at all useful, compared with 10% of successful applicants. 

151 

 
 
 
 
Figure 46: Applicant perception of the usefulness of feedback 

1.1  Burden across evaluator tasks 

Overall,  evaluators  collectively  spend  43.6–58.6  hours  on  the  evaluation  of  each 
application from start to finish (depending on whether single- or multi-stage). Time spent 
is  slightly  lower,  at  31.3–46.3  hours,  for  processes  without  an  interview  step.  Between 
35% and 51% of that time is spent on review of the proposal text. 

152 

 
 
 
 
Figure 47: Evaluator burden across the proposal submission process 

Evaluating proposal text 

For  each  application,  evaluators  spent  the  most  time  on  average  evaluating  the  full 
proposals  within  the  multi-stage  process  (median = 6  hours)  or  single-stage  process 
(median = 5 hours). On average, evaluators spent 4 hours reviewing first-stage proposals. 
The  median  number  of  reviewers  per  proposal  was  3,  and  therefore  this  equated  to 
approximately  12  hours  of  collective  evaluator  time  spent  on  first-stage  proposals;  18 
hours of collective evaluator time on full proposals within the multi-stage process, and 15 
hours of collective time on full proposals within the single-stage process. 

153 

 
 
 
 
 
Figure 48: Median time spent on evaluation of written proposal 

There are no differences between time spent on reviewing written proposals and the gender 
of the evaluator. Female and male evaluators spent the same amount of time reviewing 
first-stage  proposals  within  the  multi-stage  submission  process  (median = 4  hours),  full 
proposals  within  the  multi-stage  submission  process  (median = 6  hours),  and  full 
proposals within the single-stage submission process (median = 5 hours). 

There  were  minimal  differences  between  time  spent  on  reviewing  written  proposals  and 
the country in which the evaluator was working. Evaluators working in EU Member States 
or the UK spent on average 6 hours reviewing full applications as part of the multi-stage 
process, 4 hours reviewing first-stage applications, and 5 hours reviewing full applications 
as  part  of  the  single-stage  process.  Comparison  between  reviewers  working  within  the 
Member  States  grouped  under  widening  participation  group63  or  ‘other’  found  that 
reviewers from Member States grouped under widening participation group spent slightly 
longer  evaluating  first-stage  proposals  (median = 5  hours,  compared  with  4  hours),  full 
proposals within the multi-stage process (median = 8 hours, compared with 6 hours), and 
full proposals within the single-stage process (median = 6 hours, compared with 5 hours). 
Evaluators from EU-associated countries spent on average 5 hours reviewing full proposals 
(as part of either the single- or the multi-stage process) and 4 hours reviewing first-stage 
applications.  Reviewers  from  third  countries  spent  the  least  time  evaluating  proposals, 
spending  4  and  5  hours  reviewing  full-time  applications  from  multi-  or  single-proposal 
submissions, respectively, and 3 hours reviewing first-stage proposal submissions. 

There  were  minimal  differences  between  the  time  spent  on  reviewing  written  proposals 
and  whether  the  reviewers  had  prior  experience  of  H2020  or  not.  Reviewers  with  prior 
experience of H2020 or previous EU framework evaluation programmes spent on average 
4 hours reviewing first-stage proposals (compared with 4 hours spent by evaluators with 
no prior experience), 6 hours reviewing full proposals as part of the multi-stage submission 
(compared  with  5  hours  spent  by  evaluators  with  no  prior  experience),  and  5  hours 

63 Member States belonging to the widening participation group include: Bulgaria, Croatia, Cyprus, 
Czechia,  Estonia,  Hungary,  Latvia,  Lithuania,  Luxembourg,  Malta,  Poland,  Portugal,  Romania, 
Slovakia and Slovenia. 

154 

 
 
 
 
                                                 
reviewing  full  proposals  as  part  of  the  single-stage  submission  (compared  with  6  hours 
spent by evaluators with no prior experience). 

155 

 
 
Figure 49: Median time spent on evaluation of written proposal, by evaluator gender, country of work, and prior experience of H2020. MS = Member State 

156 

 
 
 
 
 
Consensus group 

The consensus group is a meeting that aims to find agreement on the final comments and 
scores of the proposal. It is attended by the individual evaluators for proposals, as well as 
a moderator from the EC or the Executive Agency. Meetings may be held on site or virtually. 

Per application, evaluators spent on average 0.43 hours preparing for the consensus group 
meeting,  0  hours  travelling  to  the  consensus  group  meeting  (the  meetings  were  held 
virtually),  and  0.75  hours  attending  the  consensus  group  meeting.  Taking  into 
consideration  the  number  of  evaluators  per  application,  this  amounted  to  evaluators 
collectively spending 1.6 hours preparing for the consensus group and 3 hours attending 
the consensus group per application. 

Figure 50: Median time spent on consensus group discussion 

Comparison between male and female evaluators found that on average equal amounts of 
time  were  spent  on  attending  the  consensus  group  (median = 0.75  hours  per  evaluator 
per  application)  and  preparing  for  the  consensus  group  (median = 0.44  for  female 
evaluators and median = 0.43 for male evaluators). 

Comparison between evaluators based in different countries found that evaluators from EU 
Member  States  or  the  UK  spent  approximately  0.44  hours  preparing  for  the  consensus 
group  per  evaluator  per  application  (compared  with  0.43  hours  for  evaluators  from  EU-
associated countries and 0.33 hours for evaluators from third countries). Comparison on 
time spent attending the meeting found that evaluators from EU Member States or the UK 
spent  approximately  0.75  hours  attending  the  meeting  per  evaluator  per  application, 
compared  with  0.87  hours  for  evaluators  from  EU-associated  countries  and  0.8  for 
evaluators from third countries. Comparison between reviewers working within the Member 
States grouped under widening participation group64 or ‘other’ found that reviewers from 
Member States grouped under widening participation group spent a similar amount of time 
both attending the consensus group meeting (median = 0.67 hours per application) and 

64 Member States belonging to the widening participation group include: Bulgaria, Croatia, Cyprus, 
Czechia,  Estonia,  Hungary,  Latvia,  Lithuania,  Luxembourg,  Malta,  Poland,  Portugal,  Romania, 
Slovakia and Slovenia. 

157 

 
 
 
 
                                                 
preparing for the meeting (median = 0.5 hours per application) compared with reviewers 
working  in  Other  Member  States  (median = 0.75  and  0.43  hours  for  attending  and 
preparation, respectively). Comparison between evaluators with prior experience of H2020 
indicated that evaluators with no prior experience spent longer preparing for the consensus 
group discussion meeting (median = 0.56 hours per evaluator per application) compared 
with  evaluators  who  had  prior  experience  (median = 0.4  hours  per  evaluator  per 
application).  Evaluators  with  no  prior  experience  also  stated  that  they  spent  longer 
attending the consensus group discussion (1 hour per evaluator per application, compared 
with 0.67 hours per evaluator per application for those with prior experience). 

158 

 
 
Figure 51: Median time spent on consensus group discussion, by gender, by country of work, by prior H2020 experience. MS = Member State 

159 

 
 
 
 
Panel review 

The  panel  review  aims  to  ensure  consistency  in  the  comments  and  scores  given  at  the 
consensus  stage  for  submitted  proposals  to  each  call.  It  then  recommends  a  list  of 
proposals  for  funding.  It  consists  of  experts  not  involved  in  the  individual  evaluation  of 
proposals. 

The  burden  of  the  panel  review  session  was  highest  on  evaluators  when  attending  the 
session, with each evaluator spending approximately 0.5 hours per application on session 
attendance; 0.25 hours preparing for the panel review; and 0 hours travelling to the panel 
review (meetings were virtual)65. Multiplying this by the number of reviewers per session 
resulted in evaluators collectively spending 7.98 hours attending the panel review meeting 
per application and 3.75 hours preparing for the panel review meeting. 

Figure 52: Median time spent on panel review 

Comparison between male and female evaluators found a similar time burden across panel 
review tasks. Female evaluators spent on average 0.27 hours per evaluator per application 
preparing for the panel review and 0.5 hours attending the panel review session, compared 
with  male  evaluators,  who  spent  0.25  hours  preparing  for  and  0.5  hours  attending  the 
session. Comparison between evaluators based in different countries found that evaluators 
from the EU or UK and EU-associated countries spent the same amount of time attending 
the panel review session (median = 0.5 hours per evaluator per application for EU and EU-
associated countries, respectively), with evaluators from ‘other countries’ stating that they 
spent  0.8  hours  per  evaluator  per  application  attending  the  session.  There  were  also 
minimal differences in time spent preparing for the session, with evaluators spending 0.33 
hours, 0.25 hours and 0.3 hours per evaluator per application for EU-associated countries, 
EU  Member  States  or  the  UK,  and  ‘other  countries’,  respectively.  Comparison  between 

65 Meetings were organised virtually due to the COVID-19 pandemic, and therefore travel time is 0 
hours.  

160 

 
 
 
 
                                                 
reviewers working within the Member States grouped under widening participation group66 
or Other found that reviewers from Member States grouped under widening participation 
group spent the same amount of time attending the panel review session (median = 0.5 
hours per application) and slightly longer preparing for the meeting (median = 0.29 hours 
per application, compared with 0.25 hours). 

Finally,  comparison  between  evaluators  with  prior  experience  of  H2020  found  that 
evaluators  with  no  prior  experience  of  H2020  spent  slightly  longer  preparing  the  panel 
review  meetings  compared  with  evaluators  with  no  prior  experience  (median = 0.33 
hours, compared with 0.25 hours) and slightly longer attending the panel review session 
(median = 0.62 hours, compared with 0.5).

66 Member States belonging to the widening participation group include: Bulgaria, Croatia, Cyprus, 
Czechia,  Estonia,  Hungary,  Latvia,  Lithuania,  Luxembourg,  Malta,  Poland,  Portugal,  Romania, 
Slovakia and Slovenia. 

161 

 
 
                                                 
Figure 53: Median time spent on panel review, by gender, by country, by prior H2020 experience. MS = Member State 

162 

 
 
 
 
Interview 

Per  application,  most  of  the  burden  regarding  evaluator  time  was  in  conducting  the 
interviews, with individual evaluators spending approximately 1 hour per application, which 
amounted  to  approximately  8  evaluator  hours  collectively  per  application.  Evaluators 
individually  spent  approximately  0.33  hours  preparing  for  the  interview  per  application, 
which amounted to 2.67 hours collectively. The least amount of time was spent travelling 
to  the  interviews  (median = 0.2  hours  per  application,  which  amounted  to  1.6  hours 
collectively per application). 

Figure 54: Median time spent on interview 

Comparison  between  male  and  female  evaluators  indicated  similar  time  spent  on  the 
different tasks within the interview process. When conducting interviews, male evaluators 
spent 0.95 hours per application, collectively, whereas female evaluators spent 1 hour per 
application; when preparing for interviews, both female and male evaluators individually 
spent  0.33  hours  on  average  per  application.  Comparison  between  time  spent  by 
evaluators based in different countries found that evaluators from ‘other countries’ spent 
longer travelling to the interview when compared with evaluators from EU Member States 
or  the  UK  or  from  EU-associated  countries  (median = 0.6  hours  per  evaluator  per 
application, compared with 0.2 hours for EU/UK and 0.089 for  EU-associated countries). 
Evaluators from EU Member States or the UK, EU-associated countries, and third countries 
spent the  same amount of time preparing for the interview (0.33 hours  on average per 
application). Comparison between  reviewers working within the Member States grouped 
under widening participation group67 or Other found that reviewers from Member States 
grouped  under  widening  participation  group  spent  longer  preparing  for  the  interview 
(median = 0.88  hours  per  application,  compared  with  0.33  hours)  but  felt  they  spent  a 
shorter  amount  of  time  conducting  the  interview  (median = 0.8  hours  per  application, 
compared  with  1  hour  for  reviewers  from  Other  Member  States).  Evaluators  with  prior 
experience of H2020 spent less time preparing for the interview (0.33 hours per evaluator 
per application, compared with 0.38 hours per application for evaluators who had no prior 

67 Member States belonging to the widening participation group include: Bulgaria, Croatia, Cyprus, 
Czechia,  Estonia,  Hungary,  Latvia,  Lithuania,  Luxembourg,  Malta,  Poland,  Portugal,  Romania, 
Slovakia and Slovenia. 

163 

 
 
 
 
                                                 
experience).  Evaluators  with  no  prior  H2020  experience  also  spent  slightly  less  time 
conducting the interview, spending 0.94 hours, compared with 1 hour per application.

164 

 
 
Figure 55: Median time spent on interview, by gender, by country of work, by H2020 experience. MS = Member State 

165 

 
 
 
 
Perceptions of burden 

When evaluators were asked which tasks they found the most burdensome, 67% (4 732 
out  of  7 049)  responded  that  they  found  the  full  proposal  review  process  burdensome; 
21% responded that they found consensus group meeting burdensome; 17% responded 
that they found the first-stage proposal review burdensome; 6% responded that they found 
the  panel  review  meeting  burdensome;  and  only  1%  responded  that  they  found  the 
interview process burdensome. However, again this may be reflective of the fact that many 
of the evaluators would not have participated in interviews. 

Figure 56: Evaluator perception of burden across tasks 

166 

 
 
 
2 

Analysis of feedback letters provided to H2020 applicants 

Table 48 outlines the number of ESRs reviewed per programme and outcome. We aimed 
to  obtain  a  widespread  distribution  of  ESR,  covering  both  successful  (n = 124)  and 
unsuccessful  (n = 122)  applicants  in  order  to  assess  the  extent  to  which  applicants  are 
provided with feedback that ensures benefit to applicants through learning. 

Table 48: Number of evaluation summary reports reviewed per programme and outcome 

Successful applicants 

Unsuccessful applicants 

RIA 

IA 

MSCA 

ERC Proof of Concept 

CSA 

H2020-EEN-SGA  

Total 

26 

25 

25 

25 

17 

6 

124 

26 

22 

25 

25 

24 

0 

122 

Table 49 summarises the number of reviewers contributing to the ESRs, which ranged from 
one to five. Most ESRs were reviewed by one person (78%), followed by five people (20%). 
A very small number were reviewed by three reviewers (1%) or four reviewers (1%). None 
of the ESRs had two reviewers. However, all programmes in H2020 have more than one 
reviewer.  Therefore,  it  is  likely  that  ESRs  that  were  identified  as  being  from  only  one 
reviewer  are  either  the  summary  of  all  reviewer  comments  in  a  single  document  or  the 
review from only one of the reviewers for that application. For the purpose of this study, 
where we were unable to identify the number of reviewers, we have considered the ESR 
as a single review. 

Table 49: Number of reviewers  contributing to the ESRs 

Number of reviewers 
contributing to ESRs 

Number 

Percentage 

1 

2 

3 

4 

5 

Total 

191 

0 

2 

3 

50 

246 

77.6% 

0% 

0.8% 

1.2% 

20.3% 

100% 

167 

 
 
 
 
The  length  of  the  review  is  not  necessarily  linked  to  the  usefulness  of  a  review  or  the 
outcome  of  the  application,  as  this  will  depend  on  the  reviewer’s  style,  the  format 
requirements of the review and the number of criteria that need to be assessed. We found 
that for H2020 programmes, the page length ranged from two to six, with most ESRs being 
three pages long (59%). The length of the review did not seem to reflect the outcome of 
the  application  for  either  the  shorter-than-average  reviews  or  the  longer-than-average 
reviews. It did, however, appear to reflect the number of reviewers. All but one of the ESRs 
that  extended  beyond  four  pages  contained  reviews  of  multiple  reviewers  (11  ESRs 
contained  comments  from  five  reviewers  and  three  contained  comments  from  four 
reviewers), explaining the increased length. Similarly, all ESRs less than three pages long 
comprised a single review. Consistency in length of reviews across ESRs is likely a reflection 
of the structure of the ESRs and, potentially, of guidance provided to reviewers. 

Table 50: Page length of the evaluation summary reports 

Number 

Page length of ESR 

Successful 
applications 

Unsuccessful 
applications 

Total 

Percentage of 
total number of 
ESRs 

2 

3 

4 

5 

6 

14 

61 

41 

7 

1 

15 

85 

15 

6 

1 

29 

146 

56 

13 

2 

11.8% 

59.3% 

22.8% 

5.3% 

0.8% 

Total 

124 

122 

246 

100% 

As with the length of the review, whether a review contains a summary of the proposal is 
dependent on the reviewer’s style, in the absence of clear guidance on the format of the 
review. In an analogous example, some journals ask reviewers to provide a summary of 
the manuscript prior to providing feedback on strengths and weaknesses. This is meant to 
ensure that the reviewer has a correct understanding of the research. However, in H2020 
programmes,  reviewers  are  not  asked  to  provide  a  summary;  therefore  it  is  at  the 
discretion  of  each  reviewer.  Table  51  outlines  the  frequency  with  which  the  reviewers 
summarise the proposal in their feedback (or at least one reviewer summarises it, in cases 
of multiple reviewers). As this table shows, most ESRs did not offer a proposal summary 
(86%). 

168 

 
 
 
 
 
 
 
 
Table 51: Reviewers summarising the proposal 

At least one reviewer 
summarises proposal? 

Number 

Percentage 

Yes 

No 

Total 

35 

211 

246 

14.2% 

85.8% 

100% 

2.1  Score and assessment alignment 

For each ESR, the alignment between the score(s) provided and the written feedback were 
assessed using a RAG rating system. Green indicates the score was well aligned with the 
written feedback, amber suggests some inconsistencies and red indicates a large difference 
between the score and the feedback. 

Table 52 provides the overview of the RAG ratings for alignment of score and assessment. 
As this table shows, most feedback was deemed to be in line with the score (68.3% marked 
as  ‘green’).  However,  for  30.5%  of  ESRs,  there  were  some  inconsistencies  between  the 
scoring  and  assessment,  and  1.2%  had  significant  inconsistencies  between  assessment 
and scoring and so were marked as ‘red’. 

Table 52: RAG rating for alignment of scores and written assessment 

RAG rating 

Number 

Percentage 

Successful 

Unsuccessful 

Total 

Green 

Amber 

Red 

Total 

90 

32 

2 

78 

43 

1 

124 

122 

168 

75 

3 

246 

68.3% 

30.5% 

1.2% 

100% 

For  the  ESRs  marked  as  having  some  inconsistencies  between  the  score  and  the 
assessment (i.e. with an ‘amber’ rating), the themes arising for this are as follows. 

  Written  feedback  highlights  weaknesses,  yet  the  score  is  higher  than  may  be 
anticipated from this feedback. This may be because the weaknesses identified are 
significant  in  comparison  with  the  score,  the  weaknesses  are  described  as  being 
more  significant  than  the  reviewer  actually  interpreted  them  to  be,  and/or  the 
strengths of the proposal justifying the high score are not made clear enough in the 
written feedback. 

 

In  a  smaller  number  of  instances,  the  written  feedback  strongly  highlights  the 
positive aspects of the proposal, but the score is lower than might be expected from 
this  feedback.  In  these  cases,  it  may  be  that  the  reviewer  has  not  made  clear 

169 

 
 
 
 
 
enough  in  their  written  feedback  the  weaknesses  of  the  proposal  and  the 
significance of these. 

  Where  multiple  reviewers  provided  feedback  on  one  proposal,  there  were  some 
instances  where  reviewers  offered  predominantly  positive  feedback  and  others 
negative feedback. It was not always clear how this feedback related to the score 
provided (e.g. for the ERC Proof of Concept programme, it was a simple pass/fail 
for  each  criterion,  yet  it  was  not  always  clear  how  the  predominantly  negative 
feedback from one reviewer was weighed against the positive feedback of others). 

  A small number of reviewers focused primarily on describing the information in the 

proposal rather than providing feedback on strengths and/or weaknesses. 

  Some  reviewers  did  not  offer  the  highest  score  for  a  criterion  yet  focused  their 

written feedback on positives without discussing any weaknesses. 

For  those  ESRs  identified  as  having  large  inconsistencies  between  the  score  and 
assessment (i.e. given a ‘red’ rating), the inconsistency was due to the fact that the ESR 
asked  only  for  main  weaknesses  of  the  proposal;  therefore,  when  a  proposal  passed  a 
threshold it was not possible to assess the positive aspects of the proposal. 

Reviews structured around assessment criteria 

For  each  ESR,  how  well  the  review  was  structured  around  the  assessment  criteria  was 
assessed  using  a  RAG  rating  system.  Green  indicates  the  assessment  addresses  all 
assessment criteria, amber suggests there were some minor inconsistencies in addressing 
the criteria and red indicates that the review does not address the criteria in major ways. 

Table 53 provides the  summary of the RAG  ratings assessing whether the reviews were 
structured  around  the  assessment  criteria.  As  this  table  shows,  over  half  of  ESRs  were 
deemed to be well structured around the criteria (56% marked as ‘green’). However, there 
were also a substantial number of ESRs in which there were some inconsistencies in how 
the review was structured around the criteria (with 40% being marked as ‘amber’). None 
were deemed to have significant issues in their structuring around the criteria (i.e. marked 
as ‘red’). There were nine ESRs (4%) in which no assessment criteria were provided, but 
in which reviewers were asked to comment only on the main weaknesses of the proposal. 
We did not assign a RAG rating to these latter ESRs. 

Table 53: RAG rating for structure of review around assessment criteria 

RAG rating for 
structure of review 
around assessment 
criteria 

Green 

Amber 

Red 

n/a 

Total 

Number 

Percentage 

Successful 

Unsuccessful 

Total 

65 

55 

0 

4 

74 

43 

0 

5 

138 

99 

0 

9 

124 

122 

246 

56.1% 

40.2% 

0% 

3.7% 

100% 

170 

 
 
 
 
For those ESRs assigned an ‘amber’ rating, the themes arising as to the reasons for this 
are as follows. 

  Most  criteria  were  addressed,  but  some  were  not  addressed  or  very  indirectly 

addressed due to briefness of feedback. 

  Feedback was provided against a criterion that would better fit another criterion. 

  Assessment focused more on describing the proposal than on assessing it against 

the criteria. 

  Some unnecessary information was provided. 

  Feedback  was  written 

around 
strengths/weaknesses rather than the criteria, which made it difficult to determine 
if all assessment criteria had been met. 

large  paragraphs 

structured 

or 

in 

2.2  Quality of the review 

Reviewer  comments  on  a  proposal  are  commonly  used  to  help  panel  members  in  their 
assessment of the application. Therefore, it is important for a review to provide a critical 
assessment  of  the  proposal,  including  comments  on  topic  area  as  well  on  methodology. 
Table  54  outlines  whether  the  reviewer  offered  critical  appraisal  of  the  proposal.  In  all 
cases but one, reviewers provided some form of critical appraisal. This includes comments 
on  feasibility  in  terms  of  methodology  and  timeliness,  proposed  impact  and  ways  to 
achieving this, and placement of the research in the broader landscape. Only one ESR did 
not include appraisal of this kind, providing generic comments and not providing context 
for  comments.  However,  this  ESR  belonged  to  a  successful  application,  and  the  ESR 
requested that only main weaknesses be reported. Therefore, it may be that it was a strong 
proposal with no strong weaknesses regarding approach or delivery. 

Table 54: Critical appraisal of proposals 

Reviewer offers critical 
appraisal? 

Number 

Percentage 

Yes 

No 

Total 

245 

1 

246 

99.6% 

0.4% 

100% 

The use of examples to illustrate feedback can further aid others upstream in the proposal 
evaluation process and can provide an applicant with clear guidance on what aspects of 
the  proposal  performed  well  and  what  aspects  could  be  improved.  This  is  particularly 
important in the context of H2020 proposal evaluation, where reviewers are asked to not 
provide ways of improving the proposal, yet the reviews are still expected to be useful to 
applicants. 

For each ESR, the use of examples by the reviewer to illustrate their feedback was assessed 
using a RAG rating system. Green indicates the reviewer used many useful examples to 
illustrate their point, amber suggests there were some (albeit limited) use of examples and 
red  indicates  that  no  (or  a  very  small  number  of)  examples  were  provided.  Table  55 
summarises the RAG assessment for the use of examples. As this  table shows, a similar 
number of ESRs were marked as being ‘green’ (48%) and ‘amber’ (44%). In addition, 8% 
were marked as ‘red’. 

171 

 
 
 
Table 55: RAG rating for provision of examples by reviewers to illustrate feedback 

Number 

Percentage 

Successful 

Unsuccessful 

Total 

58 

55 

10 

61 

53 

9 

124 

122 

119 

108 

19 

246 

48.4% 

43.9% 

7.7% 

100% 

RAG rating for 
provision of 
examples to 
illustrate feedback 

Green 

Amber 

Red 

Total 

2.3  Language use 

In  H2020,  applicants  are  able  to  submit  a  proposal  in  any  of  the  official  EU  languages. 
However, most applicants will choose to submit their proposal in English, as providing a 
proposal in English expands the pool of reviewers and enables more collaborative, cross-
border  working.  Therefore,  reviewers  should  focus  more  on  the  content  of  the  proposal 
and the research and/or innovation proposed rather than the proper use of language and 
grammar. 

For each ESR, whether the reviewer commented on the grammar or language used in the 
proposal was assessed using a RAG rating system. Green indicates the reviewer did not 
comment  on  grammar/language  at  all,  amber  suggests  there  were  some  references  to 
language/grammar  and  red  indicates  that  language/grammar  was  commented  on  to  a 
great extent. Table 56 outlines the RAG rating for feedback on language/grammar. As this 
table shows, almost all reviewers did not comment on language/grammar (99% marked 
as ‘green’). Only two (1%) were assessed as being ‘amber’, based on a single comment 
per reviewer. None of the ESRs were marked as ‘red’. 

For those ESRs assigned an ‘amber’ rating, the reasons for this were as follows. 

  The  reviewer  provided  comments  on  the  use  of  different  conventions  when 
presenting  numbers  (commas,  points  and  decimals),  which  was  identified  as  a 
minor shortcoming. 

  One reviewer commented that the proposal was ‘poorly written’. However, they also 
commented that it seemed rush and unconvincing. Therefore, we cannot confirm or 
deny whether ‘poorly written’ refers to the use of language or grammar. 

172 

 
 
 
 
Table 56: RAG rating for comment on grammar/language by reviewer 

RAG rating for comment on 
grammar/language 

Number 

Percentage 

Green 

Amber 

Red 

Total 

244 

2 

0 

246 

99.2% 

0.8% 

0% 

100% 

The ESRs were also assessed for whether the reviewer used language that was particularly 
positive, neutral or particularly negative (Table 57). As Table 57 shows, almost all of the 
ESRs were deemed to use neutral language (94%). Only a small number used language 
that  was  particularly  positive  (4%)  or  negative  (2%).  Examples  of  positive  language 
include such phrases as ‘radically and convincingly developed new ideas’, ‘outstanding’ and 
‘excellent’. Examples  of negative language include the use  of  such terms as ‘unrealistic’ 
and  ‘oversized  budget’.  It  is  unsurprising  that  ESRs  in  which  reviewers  used  positive 
language all belonged to successful applicants and ESRs in which reviewers used negative 
language all belonged to unsuccessful applicants. 

Table 57: Positivity of language used by reviewer 

Positivity of 
language 

Number 

Percentage 

Successful 

Unsuccessful 

Total 

Positive 

Neutral 

Negative 

Total 

5 

119 

0 

246 

0 

118 

4 

122 

11 

237 

4 

246 

2% 

96.4% 

1.6% 

100% 

2.4  Constructiveness of feedback 

The  ESRs  were  assessed  for  whether  the  review  was  constructive,  neutral  or 
unconstructive. A review was considered constructive if it offered comments that could be 
used by the  applicant for future applicants, offering learning from the proposal process, 
even  if  the  application  had  been  unsuccessful.  In  contrast,  a  review  was  considered 
unconstructive  if  it  offered  harsh  criticism  and  could  not  be  used  for  learning  by  the 
applicant. A review was considered neutral if it commented on strengths and weaknesses 
but provided no specificity on what was lacking or what worked well in an application. Table 
58 shows the number of ESRs that were considered constructive (89.8%), unconstructive 
(0.4%) and neutral (9.7%). The vast majority of ESRs were constructive, and this was the 
case for both successful and unsuccessful applications. In order to benefit applicants, both 
successful  and  unsuccessful,  reviews  must  provide  feedback  on  which  the  applicant  can 

173 

 
 
 
 
 
 
build. This may be particularly important for unsuccessful applicants, who do not gain direct 
benefit (i.e. funding) from the process. 

Table 58: Constructiveness of reviews 

Constructiveness 

Number 

Percentage 

Successful 

Unsuccessful 

Total 

Constructive 

Neutral 

Unconstructive 

Total 

110 

14 

0 

246 

111 

10 

1 

221 

89.8% 

24 

1 

246 

9.7% 

0.4% 

100% 

2.5  Weaknesses and areas for improvement 

Any weaknesses of the reviews were recorded, and ways of improving these were identified 
(Table 59). These weaknesses have been organised into five categories: amount of detail; 
critical assessment; adherence to assessment criteria; score and feedback alignment; and 
structure and language. 

Table 59: Themes related to identified weaknesses and areas for improvement 

Area 

Weakness 

Area for improvement 

Amount of detail 

Critical assessment 

reviewer 

The 
broad/generic 
weaknesses. 

provides 
description 

a 
of 

Where weaknesses are described, 
provide  specific 
feedback  and 
examples on what the issues are.  

The feedback is short/brief. 

Provide 
feedback. 

additional 

detail 

to 

feedback 

The 
on 
summarising  the  proposal  rather 
than offering critical assessment. 

focuses 

Focus  feedback  on  assessing  the 
proposal. 

Adherence 
criteria 

to 

assessment 

The feedback is not relevant to the 
assessment criteria. 

Ensure  feedback  is  applicable  to 
the  assessment  criteria  being 
responded to. 

The  assessment  criteria  are  not 
addressed. 

Ensure all assessment criteria are 
addressed throughout. 

Score and feedback alignment 

There  is  a  mismatch  between 
written feedback and score. 

Ensure  written  feedback  is  in  line 
with 
the  score  provided  (i.e. 
feedback justifies the score). 

Structure and language 

The feedback is provided in a large 
paragraph. 

the 

feedback 
Split 
sections/bullet points. 

into  sub-

174 

 
 
 
 
 
 
 
Grammatical issues or typos make 
to 
the 
understand. 

feedback 

difficult 

Review 
to 
submitted 
understandable. 

feedback  before 

ensure 

it 
it 

is 
is 

The  reviewer  uses  unconstructive 
language. 

Avoid  the  use  of  unhelpful  or 
unconstructive language/tone. 

3 

Transparency analysis 

This section sets out the findings of desk research analysing the transparency  of H2020 
proposal evaluation processes. 

Transparency in proposal evaluation processes can be considered as the extent to which 
information  associated  with  the  process  is  disclosed  (Murray  et  al.,  2016b;  Gurwitz, 
Milanesi, and Koenig, 2014; Tamblyn et al., 2018a). Transparency can have benefits not 
just in terms of openness and understanding of the process, but also in terms of impact on 
the  perceived  burden  (Gurwitz,  Milanesi,  and  Koenig,  2014)  because  it  would  allow 
applicants to better understand the process that has led to the outcome of their application 
and it would allow reviewers to obtain recognition for their work. 

We  have  conducted  an  analysis  of  transparency  by  reviewing  publicly  available 
documentation  on  H2020  funding  processes.  Specifically,  we  have  used  the  framework 
developed by Gurwitz, Milanesi, and Koenig (2014) to assess the transparency of a range 
of  funders.  The  authors  used  this  framework  to  assess  the  transparency  of  27  major 
international  public  funders,  which  serves  as  a  comparison  point  for  H2020  funding 
schemes  regarding  the  level  of  transparency.  This  framework  characterises  a  range  of 
‘knowledge items’, specified in Table 6 across four ‘knowledge categories’: rules, content, 
expertise, and procedure. Through analysis of websites and documentation they categorise 
the following: 

 

 

Information on the level of access: open, if publicly available; restricted, if available 
only to a circumscribed group of people; and closed, if not available at all 

Information  on  the  timing  of  that  access:  providing  access  to  a  knowledge  item 
either before the evaluation procedure, during the procedure, or after the procedure 

The  different  H2020  actions  have  the  same  level  of  transparency  in  their  proposal 
evaluation processes, with most knowledge items being fully ‘open’ (i.e. publicly available 
either before or after the evaluation process has taken place). Our analysis showed that 
almost  all  information  relating  to  descriptions  of  evaluation  procedures  and  evaluation 
details (i.e. the ‘rules’), successful applicants and their proposals, and call rates is openly 
disclosed. Much  of this information is disclosed via the European  Commission’s ‘Funding 
and Tender Opportunities’ portal, where all H2020 calls are listed68. This system acts as a 
centralised  information  source  for  applicants  and,  indeed,  any  other  interested  parties. 
Links to the calls’ success rate and to summaries and proposals of funded projects and PIs 
are  provided  here.  The  ‘rules’  are  typically  disclosed  in  the  work  programmes  of  the 
different actions, and questions are displayed via the portal mentioned above. In regard to 
evaluator  names,  one  interviewee  from  the  scoping  stage  of  the  study  mentioned  that 
these  are  disclosed  annually  in  an  aggregated  form,  but  never  for  individual  proposals 
[INT01]. As noted previously, the names of evaluators are removed from the ESR sent to 
applicants.  For  all  H2020  actions,  information  about  unsuccessful  proposals  –  including 

68https://ec.europa.eu/info/funding-tenders/opportunities/portal/screen/opportunities/topic-search 

175 

 
 
 
                                                 
details, abstracts or names of applicants – are not disclosed at any point, which is typical 
of most current funding systems.

176 

 
 
Figure 57: H2020 Transparency Framework analysis on access to information69 

Figure 58: H2020 Transparency Framework analysis on timing to information70 

69 In blue, items that are publicly available, indicating an ‘open’, or transparent, system; in red, items that are not publicly available, indicating a ‘closed’ system. 

70 In blue, items that are made available prior to the evaluation procedure, indicating an ‘open’, or transparent, system; in grey, items that are made available 
after the evaluation procedure has taken place, indicating a partly ‘open’, or partly transparent, system; in red, items that are not made available, indicating a 
‘closed’ system 

177 

CategoryItemDescription of evaluation procedureEvaluation details (questions to the reviewers)Names of applicantsNames of funded PisProposals submitted to callProposals/Lay Abstracts of funded projectsNames of reviewers (at minimum, in aggregated form)Review summaries of all proposalsReview summaries of funded proposalsFunding budget granted to projectsCall success rateRanking of proposalsFinal report (at minimum, the Final Report Abstract)RIAsIAsCSAsERCERA-Net Cofund MSCACOFUND-EJPSME InstrumentFTIProduceExpertiseContentRulesCategoryItemDescription of evaluation procedureEvaluation details (questions to the reviewers)Names of applicantsNames of funded PIsProposals submitted to callProposals/Lay Abstracts of funded projectsNames of reviewers (at minimum, in aggregated form)Review summaries of all proposalsReview summaries of funded proposalsFunding budget granted to projectsCall success rateRanking of proposalsFinal report (at minimum, the Final Report Abstract)RIAsIAsCSAsERC ERA-Net Cofund MSCACOFUND-EJPSME InstrumentFTIProduceExpertiseContentRules 
 
 
 
                                                 
Such  are  the  similarities  among  actions  that  a  generic  H2020  process  can  be  made,  as 
displayed below, alongside the comparators listed in Gurwitz, Milanesi, and Koenig (2014) 
and the findings from the case studies. 

Several  of  the  case  studies  highlighted  examples  of  actions  that  have  supported  the 
transparency of the proposal evaluation processes. One interesting case is the sharing of 
draft calls for proposals by the CFI for community feedback, which then informs the final 
call design. In addition, when calls are issued, as a first stage, institutions that intend to 
submit  a  proposal  are  required  to  submit  a  notice  of  intent,  which  is  made  publicly 
available.  This  is  intended  to  drive  collaboration  between  institutions,  but  it  is  unusual 
because it is rare  for information about proposals at any level to be made public unless 
and until the proposals are funded. 

As noted above, the DFG has implemented a process of elections for review boards, which 
is intended to contribute to fairness but also provides transparency, as well as community 
ownership,  of  the  process  through  which  key  decision  makers  involved  in  proposal 
evaluation processes are selected. Further, in a survey of reviewers conducted by the DFG, 
there  was  support  for  a  higher  level  of  transparency  in  the  review  process,  notably  the 
open publication of final review reports on applications. 

SNSF have taken a comprehensive look at the transparency, as well as the overall quality, 
of their proposal evaluation system through an evaluation in 2012. They found concerns 
about  the  transparency  of  their  funding  processes,  with  almost  two  thirds  of  applicants 
being unclear on how funding decisions are made, and they have subsequently improved 
their  publicly  available  information,  including  more  targeted  information  for  applicants. 
Their proposal evaluation processes are clearly set out online, and they also make clear 
the rationale around any changes made to the system. 

When  compared  with  the  funding  agencies  discussed  by  Gurwitz,  Milanesi,  and  Koenig 
(2014), we see variation in the level of transparency for our case study examples in terms 
of access to information (Figure 57) and timing of the information (Figure 58). The NRF 
and  UKRI  are  less  transparent  in  terms  of  access  to  information  than  most  of  the 
comparators. In particular, neither agency provides publicly available information on the 
success rate for their calls. The other agencies explored in our case studies (the DFG, the 
NHMRC, the SNSF, the CFI and the Polish National Science Centre (NCN)) have a similar 
level of transparency to H2020 in terms of access to information, although the CFI and the 
NCN differ in what they disclose. For example, the CFI is the only agency that provides the 
names of all applicants and not just of successful applicants. This is provided at the notice-
of-intent stage, where the CFI provide names and descriptions of the proposed projects to 
enable applicants with complementary projects to consider collaborations. However, it does 
not provide the names of reviewers. the NCN is the only agency that publishes the ranking 
of proposals, although it does not make available reviewers’ reports. 

The work by Gurwitz, Milanesi, and Koenig (2014) included the assessment of the level of 
transparency for the DFG and the NHMRC – two case studies included in our study. This 
allows us to determine how these agencies have changed their practice  over time. Both 
the  DFG  and  the  NHMRC  have  improved  in  terms  of  transparency  related  to  access  to 
information.  Whereas  in  2013,  Gurwitz,  Milanesi,  and  Koenig  were  unable  to  determine 
whether either agency published the names of the evaluators, in 2021, we were able to 
rapidly find this information. The DFG has also introduced making the name of reviewers 
and of funded beneficiaries available, as well as the funding budget granted to proposals. 

178 

 
 
Figure 59: Transparency of H2020 compared with other funding agencies with regard to access to information71 

71 In blue, items that are publicly available, indicating an ‘open’, or transparent, system; in red, items that are not publicly available, indicating a ‘closed’ system; 
in white, items whose public availability is unknown. 

179 

CategoryItemDescription of evaluation procedureEvaluation details (questions to the reviewers)Names of applicantsNames of funded beneficiariesProposals submitted to callProposals/Lay Abstracts of funded projectsNames of evaluators (at minimum, in aggregated form)Review summaries of all proposalsReview summaries of funded proposalsFunding budget granted to projectsCall success rateRanking of proposalsFinal report (at minimum, the Final Report Abstract)H2020 (2020)NIH (2013)NSF  (2013-14) Wellcome (2012-13)JSPS (2012-13)DFG (2013)DFG (2021) NSFC (2011)ERC (2013)MRC (2013-14)CONACyT (2012)NSERC (2012-13)CSIC (2013)SNSF (2012)BBSRC (2013–2014)NHMRC (2013)NHMRC (2020)Vetenskapsradet (2012)ANR (2013)Academy of Finland (2014)ZonMW (2011)NCN (2013)FWF (2013)FNRS (2012)DFF (2013)RFBR (2013) Italian Ministry of Health CIRM (2013)ISF  (2014)HFSP (2012-13)SNSF (2021)CFI (2021)NCN (2021)NRF (2021)UKRI (2021)RulesContentExpertiseProduce 
 
 
                                                 
Figure 60: Transparency of H2020 compared with other funding agencies with regard to timing of the information72 

72 In blue, items that are made available prior to the evaluation procedure indicating an ‘open’ or transparent system; in grey, items that are made available 
after the evaluation procedure has taken place indicating a partly ‘open’ or transparent system; in red, items that are not made available indicating a ‘closed’ 
system; in white, unknown 

180 

CategoryItemDescription of evaluation procedureEvaluation details (questions to the reviewers)Names of applicantsNames of funded beneficiariesProposals submitted to callProposals/Lay Abstracts of funded projectsNames of evaluators (at minimum, in aggregated form)Review summaries of all proposalsReview summaries of funded proposalsFunding budget granted to projectsCall success rateRanking of proposalsFinal report (at minimum, the Final Report Abstract)H2020 (2020)NIH (2013)NSF (2013-14) Wellcome (2012-13)JSPS (2012-13)DFG (2013)DFG (2021)NSFC (2011)ERC (2013)MRC (2013-14)CONACyT (2012)NSERC (2012-13)CSIC (2013)SNF (2012)BBSRC (2013–2014)NHMRC (2013)NHMRC (2020)Vetenskapsradet (2012)ANR (2013)Academy of Finland (2014)ZonMW (2011)NCN (2013)FWF  (2013)FNRS  (2012)DFF (2013)RFBR (2013) Italian Ministry of Health (2012)CIRM (2013)ISF  (2014)HFSP (2012-13)SNSF (2021)CFI (2021)NCN (2021)NRF (2021)UKRI (2021)RulesContentExpertiseProduce 
 
 
                                                 
Like all other funding agencies listed in Gurwitz, Milanesi, and Koenig (2014), the general 
H2020 process is fully transparent in the disclosure of ‘rules’, of information pertaining to 
evaluation details, and of procedures. Similarly, much like those of other funding agencies, 
typically H2020 actions do not disclose information on the assessment of funded proposals, 
as these tend to be sent directly to applicants. For all agencies, H2020 actions included, 
no information on unfunded proposals is disclosed; as Gurwitz, Milanesi, and Koenig (2014) 
say: ‘not published at all are the names  of unsuccessful applicants and their proposals’. 
Although most of the agencies surveyed by Gurwitz, Milanesi, and Koenig (2014) did not 
publish reports on their website (only 4 out of 22 did so), typically H2020 actions do publish 
reports and project results on the ‘CORDIS EU research results’ website73. It is not clear to 
what extent H2020 actions are atypical in publishing the names of aggregated reviewers 
annually;  Gurwitz,  Milanesi,  and  Koenig  (2014)  note  only  that  ‘some  funding  bodies’ 
released 
the  Swedish  Research  Council 
(Vetenskapsradet),  at  least  in  2012,  disclosed  the  names  of  the  reviewers  before  the 
evaluation procedure of a given call. 

Interestingly,  and  uniquely, 

this. 

4 

Bibliometric analysis of predictive power and bias 

The aim of this task was to analyse the predictive power of funding decisions by looking at 
whether  proposal  evaluation  processes  are  effective  in  selecting  the  ‘best’  applicants  in 
terms of their subsequent bibliometric performance. The analysis also addresses bias, by 
analysing  differences  in  bibliometric  performance  levels  before  and  after  the  award,  for 
success overall and for groups characterised by gender and country. 

To do this, we identified the publication output of 3 815 applicants between 2009 and 2019 
in  the  Web  of  Science  databases;  output  includes  publications  from  before  and  after 
applications  were  submitted  from  2014  to  2016.  The  research  performance  before  and 
after  application  was  analysed  in  terms  of  output  (number  of  publications)  and  impact 
(metrics based on the number of citations). The change in performance was compared with 
the outcome of the application (successful or unsuccessful). Furthermore, we considered if 
gender or country affected the outcome of applications. 

4.1  Overall bibliometric performance 

Successful  applicants  do  perform  better  bibliometrically  than  unsuccessful 
applicants 

We  find  that  there  is  a  significant  difference  in  performance  bibliometrically  between 
successful and unsuccessful applicants and that success is a good predictor of subsequent 
bibliometric  performance.  Table  60  summarises  bibliometric  performance  of  applicants 
across all three indicators. 

Table 60: Overall performance of successful and unsuccessful applicants post-award 

Number of publications 
per year 

Mean CNCI 

% highly cited papers 

Successful  

6.5 papers 

Unsuccessful  

5.1 papers 

1.62 

1.17 

18.2% 

12.1% 

73 https://cordis.europa.eu/projects/en 

181 

 
 
 
                                                 
 
However,  this  seems  to  largely  be  a  continuation  of  prior  bibliometric 
performance 

Looking at number  of publications, we  find that successful applicants had a significantly 
higher average number of papers per year compared with unsuccessful applicants before 
and after applications, as shown in Figure 61 and Table 61. The difference in pre-application 
output  of  papers  between  successful  and  unsuccessful  applicants  is  significant  (Mann-
Whitney U test p <0.0001), and the difference in post-application output of papers between 
successful  and  unsuccessful  applicants  was  also  significant  (Mann-Whitney  U  test 
p <0.0001). 

Table 61: Number of publications per year before and after application 

Mean output 

Before application 

After application 

Successful  

Unsuccessful  

5.7 papers 

4.5 papers 

6.5 papers 

5.1 papers 

Figure 61: Number of publications per year pre- and post-award for successful and unsuccessful applicants 

Looking at category-normalised citation impact, we find that successful applicants had a 
higher average CNCI compared with unsuccessful applicants before and after application, 
as indicated by the shift in the distribution for successful applications in Figure 62. Papers 
published  after  application  for  both  successful  and  unsuccessful  applicants  had  slighter 
lower average CNCI compared with papers published before application, but the difference 
is  tiny  (see  Table  62).  However,  this  can  be  expected  as  later  papers  have  not  had  the 
time  to  accumulate  citations.  The  difference  in  pre-application  CNCI  between  successful 
and  unsuccessful  applicants  is  significant  (Mann-Whitney  U  test  p   <0.0001),  and  the 
difference in post-application CNCI between successful and unsuccessful applicants is also 
significant (Mann-Whitney U test p  <0.0001).

182 

 
 
 
 
 
Table 62: Category-normalised citation impact before and after application 

Mean CNCI 

Before application 

After application 

Successful  

Unsuccessful  

1.79 

1.24 

1.62 

1.17 

Figure 62: Category-normalised citation impact pre- and post-award for successful and unsuccessful applicants 

Looking at the percentage of highly cited publications, we find that successful applicants 
had  a  higher  percentage  of  highly  cited  papers  compared  with  unsuccessful  applicants 
before  and  after  application,  as  indicated  by  the  shift  of  the  distribution  for  successful 
applications (Figure 63Figure 63, Table 63). The difference in pre-application percentage 
of highly cited papers between successful and unsuccessful applicants is significant (Mann-
Whitney  U  test  p <0.0001),  and  the  difference  in  post-application  percentage  of  highly 
cited papers between successful and unsuccessful applicants was also significant (Mann-
Whitney U test p <0.0001).

183 

 
 
 
 
 
 
Table 63: Percentage of highly cited publications before and after application 

% highly cited papers 

Before application 

After application 

Successful  

Unsuccessful  

22.9% 

15.0% 

18.2% 

12.1% 

Figure 63: Percentage of highly cited papers pre- and post-award for successful and unsuccessful applicants 

Proposal scores are only a weak predictor of future bibliometric performance 

Analysing the relationship between the overall score awarded to proposals in the evaluation 
process  and  the  subsequent  bibliometric  performance,  we  find  a  significant  but  weak 
correlation  between  proposal  score  and  subsequent  CNCI  (Table  64  and  Figure  64). 
However,  we  would  conclude  that  although  there  is  a  relationship,  the  score  given  to  a 
grant is unlikely to be a useful predictor of the future CNCI of applicants.

184 

 
 
 
 
 
 
 
Table 64: Correlation and significance of the relationship between applicants’ category-
normalised citation impact after application and the score give to their application 

All applicants 

Successful applicants 

Unsuccessful applicants 

Person correlation 
coefficient 

0.155 

0.074 

0.030 

Correlation 

Weak positive 

Weak positive 

Weak positive 

p-value 

<0.001 

0.001 

0.190 

Significance 

Significant at 95% 
confidence 

Significant at 95% 
confidence 

Not significant at 95% 
confidence 

I

C
N
C
n
o
i
t
a
c
i
l

p
p
a
-
t
s
o
p
e
g
a
r
e
v
A

25

20

15

10

5

0

0

20

40

60

80

100

Score

Successful

Unsuccessful

Figure 64: Relationship between applicant’s category-normalised citation impact after application and the score given to their 
application 

4.2  Gender analysis 

Bibliometric  performance  of  successful  male  applicants  is  higher,  on  average, 
than that of successful female applicants 

Successful female applicants have a lower pre-application average output of papers and 
citation impact compared with successful male applicants, and the difference is significant 
according  to  the  Mann-Whitney  U  test  (Table  65).  On  average,  from  2010  to  the  year 
before making an application, female applicants published 4.3 papers per year, compared 

185 

 
 
 
 
 
 
 
 
 
with  6.3  for  male  researchers  (Figure  65).  This  could  be  related  to  research  area,  so 
disciplines with higher average outputs may have higher proportions of male researchers. 
Male researchers’ pre-application papers have a higher average CNCI (1.88) than female 
researchers’  (1.66),  although  both  are  above  the  world  average  (1.00)  (Figure  66). 
Further,  24.0%  of  male  researchers’  papers  are  highly  cited  compared  with  22.2%  of 
female  researchers,  although  both  groups  are  again  above  the  world  average  (10.0%) 
(Figure  67).  The  difference  in  CNCI  and  percentage  of  highly  cited  papers  between  the 
genders is very slight – 0.22% for females and 1.8% for males. 

Table 65: Summary of performance indicators pre-application by gender 

Pre-application indicators for 
successful applicants 

Male 

Female 

Average  number  of  papers 
published per year  

Average CNCI 

6.3 

1.88 

% of highly cited papers  

24.0% 

4.3 

1.66 

22.2% 

Figure 65: Average number of papers per year by gender, prior to application 

186 

 
 
 
 
 
Figure 66: Average category-normalised citation impact by gender, prior to application 

Figure 67: Average percentage of highly cited publications by gender, prior to application 

The impact of gender on post-award bibliometric performance is limited 

Looking at the number of publications, we find that there is no significant difference in the 
change  in  output  between  successful  male  and  successful  female  applicants  or  between 
unsuccessful male and unsuccessful female applicants (Table 66). Between successful male 
and  female  applicants,  the  change  in  output  is  not  significant  according  to  the  Mann-
Whitney U test (p = 0.46) (Figure 68). Between unsuccessful male and female applicants 
the change in output is not significant according to the Mann-Whitney U test (p = 0.23) 
(Figure 69). 

Table 66: Change in number of papers published per year after application 

Mean change in output 

Male 

Female 

Successful  

Unsuccessful  

0.9 papers 

0.7 papers 

0.8 papers 

0.6 papers 

Figure 68: Average number of papers per year for successful applicants by gender 

187 

 
 
 
 
 
 
 
Figure 69: Average number of papers per year for unsuccessful applicants by gender 

Looking at normalised citations, again we see little difference between  male and female 
applicants.  There  is  no  significant  difference  in  the  change  in  average  CNCI  between 
successful  male  and  successful  female  applicants  or  between  unsuccessful  male  and 
unsuccessful  female  applicants  (Table  67).  On  average,  changes  in  average  CNCI  were 
negligible. Between successful male and female applications, the change in average CNCI 
is not significant according to the Mann-Whitney U test (p = 0.91) (Figure 70). Between 
unsuccessful male and female applications, the change in average CNCI is not significant 
according to the Mann-Whitney U test (p = 0.07) (Figure 71). 

Table 67: Mean change in average category-normalised citation impact post-award by 
gender 

Mean change in average CNCI  Male 

Female 

Successful  

Unsuccessful  

−0.18 

−0.08 

−0.14 

0.02 

Figure 70: Average category-normalised citation impact for successful applicants by gender 

188 

 
 
 
 
 
Figure 71: Average category-normalised citation impact for unsuccessful applicants by gender

189 

 
 
 
Considering  highly  cited  publications,  we  find  no  significant  difference  in  the  change  in 
percentage  of  highly  cited  papers  between  successful  male  and  successful  female 
applicants or between unsuccessful male and unsuccessful female applicants (Table 68). 
Between successful male and female applications, the change in percentage of highly cited 
papers  is  not  significant  according  to  the  Mann-Whitney  U  test  (p = 0.76)  (Figure  72). 
Between unsuccessful male and female applications, the change in percentage  of highly 
cited  papers  is  not  significant  according  to  the  Mann-Whitney  U  test  (p = 0.50)  (Figure 
73). 

Table 68: Mean change in average percentage of highly cited publications post-award by 
gender 

Mean change in % of highly 
cited papers 

Male 

Female 

Successful  

Unsuccessful  

−4.8% 

−2.9% 

−5.5% 

−2.2% 

Figure 72: Average% of highly cited publications for successful applicants by gender 

Figure 73: Average% of highly cited publications for unsuccessful applicants by gender 

190 

 
 
 
 
 
 
 
We  note  here  that  there  is  a  decrease  in  the  citation  indicators  for  more  than  half  of 
applicants. The average change in CNCI is −0.12, and the percentage of highly cited papers 
is  −3.7%.  There  are  many  reasons  why  citation  indicators  change  over  time,  but  the 
magnitude of the changes observed here is very small and should be considered negligible. 
Considering the small number of papers individual applicants publish in the period before 
or after application, citation indicators can be easily inflated by a single or a few papers 
with  a  high  citation  count.  If  this  occurs  in  the  period  before  or  after  application,  the 
difference  in  average  CNCI  between  these  two  periods  could  easily  flip  from  positive  to 
negative, or vice versa, based on a few papers. We would not recommend any conclusions 
be drawn from these changes, and in general terms we would consider these data to be 
indicative of broadly stable performance. 

4.3  Country analysis 

Successful  applicants  from  EU-associated  countries  require  a  similar  level  of 
performance to those from EU Member States in order to be successful 

We  analysed  pre-application  data  for  successful  applicants  from  different  groups  of 
countries.  The  purpose  of  this  analysis  was  to  see  whether  achieving  a  successful 
application requires the same level of research performance for applicants from all different 
regions. As shown in Table 69, citation indicators for all country groupings are above world 
averages, which for CNCI is 1.00 and  for the  percentage  of highly cited papers is 10%. 
Successful  applicants  from  EU-associated  countries  have  similar  citation  indicators  to 
successful applicants from EU Member States, although they have a higher average annual 
output of papers (6.8 papers compared with 5.6 papers). However, successful applicants 
from  widening  participation  group  Member  States  typically  have  a  lower  level  of 
bibliometric performance than those from other EU Member States. Widening participation 
group Member States have the lowest average pre-application indicators, indicating that 
before application, successful applicants from these countries published fewer papers, and 
that these papers received fewer citations, compared with successful applicants from other 
Member States. 

We  considered  also  conducting  this  analysis  for  individual  countries,  but  the  number  of 
matched applications for each country in our data set was too small for a robust analysis. 

Table 69: Summary of performance metrics pre-application by country group 

Region 

Number of 
successful 
applications 

Pre-application 
average yearly 
output of papers 

Pre-application 
CNCI 

Pre-application% 
of highly cited 
papers  

EU Member States 

1 804 

Widening 
participation  group 
Member States 

Other 
States 

Member 

EU-associated 
countries 

83 

1 721 

136 

5.6 

4.0 

5.7 

6.8 

1.79 

22.8% 

1.39 

18.5% 

1.81 

23.0% 

1.76 

23.7% 

191 

 
 
 
 
 
Looking at post-award performance, we see little difference between successful 
and unsuccessful applicants across the different country groups 

Starting with the number of publications, applicants from all regions had, on average, an 
increase in output of papers after application (Table 70). For EU Member States, application 
outcome does not appear to have much of an effect on average output of papers. For the 
widening participation group, a successful application appears to be associated with a lower 
number of publications. Unsuccessful applicants increased output by 1.7 papers per year 
on  average,  compared  with  just  0.6  papers  for  successful  applicants.  For  EU-associated 
countries,  a  successful  application  appears  to  be  associated  with  a  positive  impact  on 
output.  Successful  applicants  increased  output  by  1.7  papers  per  year  on  average, 
compared  with  just  0.1  papers  for  unsuccessful  applicants.  However,  because  there  are 
relatively few applicants from Associated Countries and the widening participation group, 
we must be cautious in the conclusions we draw. 

Table 70: Mean change in average number of papers published per year post-application 
by country group 

Region 

Total number of 
applications  

All applications 

Successful 
applications  

Unsuccessful 
applications  

EU Member States 

3 479 

Widening 
participation  group 
Member States 

Other 
States 

Member 

EU-associated 
countries 

287 

3 192 

336 

0.7 

1.4 

0.7 

0.5 

0.8 

0.6 

0.8 

1.1 

0.7 

1.7 

0.5 

0.1 

Turning to normalised citations, we find that applicants from all regions had, on average, 
a  small  decrease  in  average  CNCI  after  application  (Table  71).  However,  these  changes 
are  so  small  that  they  are  indicative  of  broadly  stable  performance.  For  all  regions, 
application outcome does not appear to be associated with any effect on average CNCI. 
Differences  in  mean  change  in  average  CNCI  between  successful  and  unsuccessful 
applicants are negligible. 

Table 71: Mean change in average category-normalised citation impact post-application 
by country group 

Region 

Total number of 
applications  

All applications 

Successful 
applications  

Unsuccessful 
applications  

EU 
States 

Member 

Widening 
participation 
group  Member 
States 

Other  Member 
States 

EU-associated 
countries 

3 479 

−0.12 

−0.17 

−0.06 

287 

−0.01 

−0.18 

0.06 

3 192 

−0.13 

−0.17 

−0.08 

336 

−0.17 

−0.19 

−0.15 

192 

 
 
Finally, looking at the percentage of highly cited publications, we find that applicants from 
all  regions  had,  on  average,  a  decrease  in  percentage  of  highly  cited  papers  after 
application  (Table  72).  For  all  regions,  application  outcome  does  not  appear  to  be 
associated with any effect  on percentage of highly cited papers.  The  difference in  mean 
change  in  percentage  of  highly  cited  papers  between  successful  and  unsuccessful 
applicants is negligible.  

Table 72: Mean change in average percentage of highly cited papers post-application by 
country group 

Region 

Total number of 
applications  

All applications 

Successful 
applications  

Unsuccessful 
applications  

EU 
States 

Member 

Widening 
participation 
group  Member 
States 

Other  Member 
States 

EU-associated 
countries 

3 479 

−3.7% 

−4.6% 

−2.7% 

287 

−1.1% 

−4.5% 

0.3% 

3 192 

−3.9% 

−4.6% 

−3.2% 

336 

−4.4% 

−5.7% 

−3.6% 

193 

 
 
 
 
CHAPTER 6: CASE STUDIES 

We  conducted  seven  in-depth,  international  case  studies  on  proposal  evaluation  for 
research and innovation funding. The aim of the case studies was to compare H2020 with 
other proposal submission and evaluation processes, identify best practice internationally, 
and assist in identifying lessons learnt that may be applicable for Horizon Europe. 

In  consultation  with  DG  RTD,  we  selected  seven  case  studies  that  highlight  different 
aspects of proposal evaluation in different contexts, to include funders varying in terms of 
amount  of  annual  available  funding  and  focus  discipline  (for  example,  biomedical, 
innovation, engineering or physical sciences) (Table 73). 

Table 73: Funding agencies selected for case studies 

Funding agency (country) 

Annual budget (EUR) 

Discipline 

Canadian 
Innovation (CFI) (Canada) 

Foundation 

for 

250 million 

Scientific R&I 

German  Research  Foundation 
(DFG) (Germany) 

3.2 billion 

Science research across 
disciplines (humanities and social 
sciences, life sciences, natural 
sciences and engineering)  

Fundamental science, clinical 
science, health services, public 
health.  

540 million 

3.2 billion 

Scientific research 

- 

Basic research  

988.4 million 

All disciplines 

National  Health  and  Medical 
Research  Council 
(NHMRC) 
(Australia)  

National  Research  Foundation 
(NRF) (Singapore) 

National Science Centre (NCN) 
(Poland)  

National 

Swiss 
Foundation 
(Switzerland) 

Science 
(SNSF) 

United Kingdom Research and 
Innovation (UKRI) (UK) 

8.5 billion 

Research disciplines (including 
arts, humanities, economics, 
biosciences, physical sciences) 

The following sections present the seven case studies conducted in detail.74 

74 Note that where we use the code INT followed by a number, this refers to the specific interview 
conducted as part of this study.  

194 

 
 
 
                                                 
1 

Canadian Foundation for Innovation – Canada 

1.1 

Introduction 

The Canadian Foundation for Innovation was established in 1997 and strives to increase 
Canada’s  capacity  to  undertake  world-class  research  and  technology  development  that 
benefits  Canadians  and  the  global  community  (CFI,  n.d.).  The  CFI  aims  to  ensure  that 
Canadian  researchers  have  the  research  infrastructure  –  such  as  labs,  facilities  and 
equipment  –  needed  to  conduct  high-quality  research  (CFI,  n.d.).  Specifically,  the  CFI’s 
objectives are as follows: 

  enhance the capacity of institutions in order to support economic growth and job 

creation, as well as health and environmental quality, through innovation, 

 

carry out world-class scientific research and technology development, 

  expand  research  and  job  opportunities  by  providing  support  through  research 

infrastructure for the development of highly qualified personnel, 

  promote  productive  networks  and  collaboration  among  Canadian  universities, 
colleges, research hospitals, non-profit research institutions and the private sector. 

The  CFI  funds  projects  across  the  sectors,  including  projects  in  health  sciences;  natural 
sciences and engineering; arts and literature; and human and social sciences (CFI, n.d.). 
Only eligible institutions may submit a proposal to the CFI (CFI, n.d.). Eligible institutions 
correspond to Canadian universities, colleges, research hospitals and non-profit research 
organisations. 

Within the 2019/2020 financial year, the CFI approved awards relating to approximately 
CAD  380  million  (approximately  EUR  246  million)  (CFI,  n.d.).  The  CFI  has  five  primary 
funding calls. 

 

 

John  R.  Evans  Leaders  Fund:  This  allocation-based  scheme  provides  funding  for 
institutions in order for them to attract and retain excellent researchers (CFI, n.d.). 

Innovation  Fund:  This  competition-based 
for 
infrastructure  across  the  full  spectrum  of  research,  from  fundamental  through  to 
technology  development.  It  is  the  largest  fund,  with  a  budget  of  up  to  CAN  400 
million  (approximately  EUR  260  million),  and  it  provides  investments  in 
infrastructure across the full spectrum of research. The fund supplies up to 40% of 
a project’s eligible infrastructure costs (CFI, n.d.). 

fund  provides 

investments 

  College-Industry  Innovation  Fund:  This  competition-based  fund  aims  to  enhance 
the capacity of colleges to support business innovation through supporting research 
infrastructure that supports partnerships (CFI, n.d.). 

  Major Science Initiatives Fund: This competition-based fund supports the ongoing 

operating and maintenance needs of national research facilities. 

  Exceptional  Opportunities  Fund:  This  competition-based  fund  is  designed  to  fund 
research where the opportunity might be missed were the project to undergo the 
normal  course  of  a  national  competition.  For  example,  under  the  Exceptional 
Opportunities Fund, the CFI announced a call to provide infrastructure relating to 
COVID-19 research (CFI, n.d.). 

195 

 
 
 
 
1.2  Proposal evaluation processes 

The CFI merit review process 

The  CFI  have  established  a  proposal  evaluation  ‘merit  review’  process  that  rewards 
excellence (CFI, n.d.). All applications are assessed through three broad criteria: a) quality 
of  the  research  and  the  need  for  infrastructure;  b)  contribution  to  strengthening  the 
capacity for innovation; and c) potential benefits of the research to Canada. In addition, 
each  specific  CFI  fund  has  its  own  objectives  and  assessment  criteria  detailed  in  the 
respective calls for proposals. In the case of the competition-based funds (e.g. Innovation 
Fund,  College-Industry  Innovation  Fund,  Major  Sciences  Initiative  Fund),  the  review 
process determines the degree to which a proposal satisfies the assessment criteria relative 
to competing proposals. The  exception to this is the allocation-based  fund (e.g. John R. 
Evans Leaders Fund), where the review process intends to determine the degree to which 
the  proposal  satisfies  the  criteria  relative  to  CFI  standards  of  excellence.  Competition-
based funds typically use a two- or three-stage merit review process, which includes the 
Expert Committee  stage; the Multidisciplinary Assessment  Committee (MAC) stage; and 
the Special Multidisciplinary Assessment Committee (S-MAC) stage. Allocation-based funds 
typically  use  a  one-stage  review  process  conducted  by  external  experts,  although  an 
advisory committee may be used for more complex cases. Figure 74 provides an overview 
of the generic review process at the CFI. 

Figure 74: CFI proposal review process (CFI, n.d.)  

Step 1 involves CFI staff conducting an administrative review to ensure the completeness, 
eligibility  and  readiness  of  applications  for  review.  Step  2  consists  of  the  one-,  two-  or 
three-stage  merit  review.  Step  3  consists  of  receiving  and  preparing  funding 
recommendations (CFI, n.d.). 

196 

 
 
 
 
 
In the next section, we will focus more specifically on the proposal evaluation process for 
the Innovation Fund, seeing as it is the largest programme at the CFI, as well as the one 
that focuses on innovation. Within Horizon 2020, this is most comparable to the ‘Research 
Infrastructures’ funding within the Excellent Science Pillar. 

Innovation Fund proposal evaluation process 

Initially, before the CFI issues a call for proposals, they may issue draft calls, which are 
circulated  among  the  research  community  for  feedback  [INT01-02-03].  Once  they  have 
adapted the call according to the feedback, they issue the final call for proposals. 

For the Innovation Fund, each eligible institution is given an institutional envelope, which 
is the upper limit on the total value that an institution can request (CFI, n.d.). This value 
is based on the average share of research funding that the institution received from the 
three  federal  research  funding  agencies  over  a  certain  period.  The  institution  then 
undergoes  its  own  internal  selection  process  and  puts  forward  proposals  to  the  CFI 
accordingly [INT01-02-03]. 

Once the full call has been announced, institutions must submit a notice of intent. This is 
used by the CFI to identify the expertise required, to recruit the committee members and 
to identify potential eligibility issues that may arise from the application (CFI, n.d.) [INT01-
02-03]. The notice of intent contains project information; a project summary (maximum 
1 500 characters); any collaborating institutions; team; a project description (maximum 4 
pages); and suggested reviewers. In addition, the CFI publish a list of the notices of intent 
online, along with short project summaries to encourage institutions with complementary 
projects to consider potential collaborations or multi-institution initiatives. 

Once  the  notice-of-intent  period  is  over,  full  proposals  are  submitted  through  electronic 
software [INT01-02-03]. For the full proposal, this includes the following sections: project 
information; plain language summary and project summary (maximum 3 pages in length); 
the project team; collaborating institutions; the financial resources required for operating 
and  maintenance;  and  information  relating  to  how  the  proposal  meets  each  of  the 
assessment  criteria  (CFI,  n.d.).  For  the  information  relating  to  each  of  the  assessment 
criteria,  proposals  for  less  than  CAD  2  million  (approximately  EUR  1.3  million)  have  a 
maximum number of 30 pages and proposals for more than CAD 2 million (approximately 
EUR 1.3 million) have a maximum page length of 35 pages. 

Once the full proposals have passed the eligibility check process, they proceed to the merit-
based review process. For the Innovation Fund, the merit review process (step 2) is three-
stage  (CFI,  n.d.).  Figure  75  provides  an  overview  of  the  Innovation  Fund  merit  review 
process. This consists of the Expert Review stage, the MAC stage; the S-MAC stage; and, 
finally, the CFI Board final decision stage. 

Figure 75: Overview of the merit review process for the CFI Innovation Fund (CFI, n.d.) 

197 

 
 
 
In the first stage of the merit-based review, expert committees are assembled to review 
the proposals. Expert committees are assembled according to their expertise each time, 
depending  on  the  expression-of-interest  stage  [INT01-02-03].  The  committees  evaluate 
how well a group of similar or related proposals meet the assessment criteria for the fund 
(CFI, n.d.) [INT01-02-03]. Table 74 lists the latest assessment criteria for the Innovation 
Fund.  Criteria  are  weighed  equally  [INT01-02-03].  Before  the  committee  meeting, 
members evaluate the proposal against the assessment criteria and provide a preliminary 
assessment to the CFI. This preliminary assessment includes the reviewers’ assessment of 
the proposal against the assessment criteria, using the CFI rating scale (Figure 76). Ratings 
are supported by reviewers identifying the strengths and weaknesses of the proposal. Next, 
the committee meeting takes place. The meetings usually have a chair and between two 
to  six  members,  depending  on  the  number  and  breadth  of  proposals  under  review.  The 
committee reaches consensus on a rating for each of the assessment criteria and discusses 
the  strengths  and  weaknesses  for  each  assessment  criterion  that  will inform  the  Expert 
Committee report. In addition, during the Expert Committee stage, for proposals that are 
deemed  particularly  large  or  complex,  interviews  may  occur  (CFI,  n.d.).  However, 
interviews is something that interviewees stated the CFI is moving away from [INT01-02-
03]. After the meeting, CFI staff draft the Expert Committee report for each proposal, and 
then the chair reviews and approves the reports. 

Figure 76: CFI rating scale for expert committees (CFI, n.d.) 

198 

 
 
 
 
 
Table 74: Assessment criteria for the CFI Innovation Fund75 

Criteria 

Description 

Research or technology development 

The  research  or  technology  development  programmes 
are innovative, feasible and internationally competitive. 

Team 

Research capacity 

Infrastructure 

Sustainability 

Benefits 

The  diverse  team  comprised  the  breadth  of  expertise 
needed to conduct the proposed programmes. 

The  institutions  and  their  partners  have  the  necessary 
research capacity on which the proposal will build. 

requested 

The 
appropriate to conduct the proposed programme. 

infrastructure 

is  necessary  and 

The infrastructure will be optimally used,  operated and 
tangible 
sustained  over 
commitments. 

its  useful 

through 

life 

The team and partners will have a well-defined plan to 
transfer  the  results  of  the  research  or  technology 
development programme.  The results are likely to lead 
to  social,  health,  environmental  and/or  economic 
benefits for Canadians.  

After  the  Expert  Committee  stage,  the  CFI  retain  approximately  80%  of  the  proposals 
[INT01-02-03].  These  are  then  evaluated  by  the  MAC,  who  are  experts  with  a  broader 
vision, experience and multidisciplinary knowledge [INT01-02-03]. They assess proposals 
based  on  the  three  competition  objectives  and  are  responsible  for  identifying  proposals 
that demonstrate the highest standard of excellence against three criteria: a) enable global 
leadership by supporting world-class research or technology development; b) enhance and 
optimise  the  capacity  of  institutions  and  research  communities  to  conduct  the  proposed 
research  or  technology  development  programme(s);  and  c)  lead  to  social,  health, 
environmental and/or economic benefits for Canadians (CFI, n.d.). Figure 77 demonstrates 
the interplay between the individual assessment criteria and how these link to the three 
fund objectives. 

75 Source: Innovation Fund call for proposals 2020 (CFI, n.d.)  

199 

 
 
 
 
                                                 
Figure 77: CFI Innovation Fund objectives and assessment criteria (CFI, n.d.) 

MAC members are provided with the Expert Committee reports, the proposals and the MAC 
report template (CFI, n.d.). Each individual proposal is assigned to three MAC members. 
During  the  MAC  meeting,  for  each  proposal,  there  is  discussion,  consensus  ratings  and 
funding recommendation made. At the end of the meeting, each MAC member may identify 
up to two proposals that they deem of exceptional merit. This aids the next stage of the 
merit-based review. 

MAC members compile the MAC report, which contains the MAC consensus ratings for each 
objective, comments on each objective, justification for the funding recommendation and 
justification for exceptional projects (CFI, n.d.). After the MAC review stage, the CFI retain 
approximately 50% of the proposals [INT01-02-03]. For projects recommended for funding 
by the MAC, the project summary is the only section of the proposal that is provided to the 
S-MAC to help with their deliberations. 

In  the  third,  and  final,  stage  of  the  merit  review  process,  the  Special  Multidisciplinary 
Assessment  Committee  review  the  MAC  reports  for  the  proposals  recommended  for 
funding. They make  sure there is consistency among the MAC  reports  and, in instances 
where  the  MAC  recommendations  exceed  the  available  budget,  recommend  to  the  CFI 
Board  of  Directors  which  proposals  best  meet  the  CFI’s  mandate  and  the  competition 
objectives and represent the best portfolio for Canadian investments  (CFI, n.d.). During 
the third stage, approximately 15% of proposals are rejected, leading to an overall success 
rate  of  approximately  36%  [INT01-02-03].  The  final  decision  is  made  by  the  board  of 
directors [INT01-02-03]. The interviewees noted that the Innovation Fund has ambitious 
objectives and that the three stages enable the assessment of the proposals from different 
perspectives in order to permit the best proposals [INT01-02-03]. 

The CFI are committed to providing detailed feedback to applicants [INT01-02-03]. After 
the first stage of the merit review process, the Expert Committee provide a report that is 
approximately 5–7 pages long, which offers detailed feedback on all assessment criteria 
[INT01-02-03]. Both successful and unsuccessful applicants receive these reports. For the 
second stage of the merit review process, the MAC members look at the reports from the 
Expert Committee and provide a further 1–2 pages of feedback. Therefore, proposals that 
have passed through to the second stage of the merit review process will receive 6–9 pages 
of  feedback  [INT01-02-03].  Interviewees  reflected  that  good  feedback  should  be 
constructive, pointing out weaknesses in the original proposal as well as ways to improve 
[INT01-02-03].  In addition, interviewees  stated that the reports were  used in their own 

200 

 
 
 
 
internal  processes  (MAC  assessment  relies  on  the  Expert  Committee  reports)  and  that 
therefore reports should be of a high standard, to ensure that the merit review process is 
rigorous, transparent and fair [INT01-02-03]. 

The  evaluation  process  is  now  done  remotely.  However,  before  COVID-19,  the  Expert 
Committee  met  in  person  [INT01-02-03].  The  CFI  note  that  the  meetings  have  been 
working well virtually and that therefore this may be something they seek to keep in future 
[INT01-02-03].  The  interviewees  also  noted  that  many  of  the  experts  are  from  outside 
Canada and are interested in lowering their environmental footprint, and therefore have 
been keen on the virtual meetings from this perspective [INT01-02-03]. Figure 78 provides 
the call timeline for 2020 in more detail. 

Figure 78: Call timeline for the CFI Innovation Fund 2020 (CFI, n.d.) 

The interviewees noted that institutions often start preparations 6–8 months ahead of the 
next call cycle [INT01-02-03]. A key priority for the CFI is to improve operational efficiency 
and to find ways to shorten timelines [INT01-02-03]. The interviewees noted that virtual 
meetings may be one way of achieving this [INT01-02-03]. 

Other funds within the CFI 

The proposal evaluation processes differ across the funding schemes. For example, another 
competition, the John R. Evans Leaders Fund, aims to provide funding for institutions to 
attract and retain individuals [INT01-02-03]. It is more frequent than the Innovation Fund 
and has several calls per year. Applicants can apply for up to CAD 2 million (approximately 
EUR 1.3 million). Because the call is offered on a more frequent basis, the CFI have two 
experts that review each proposal independently and provide written evaluations. If there 
is agreement between them, a recommendation is made to the Board of Directors and the 
proposal  is  funded.  The  success  rate  for  this  call  is  approximately  80%  [INT01-02-03]. 
Another example of how calls differ is in the feedback supplied to applicants. For example, 
for  the  Exceptional  Opportunities  Fund,  which  provides  funding  for  infrastructure  that 
needs to be used immediately, the CFI have simplified the application to make it shorter 
[INT01-02-03], and they have shortened feedback requirements for applicants, something 
that, the interviewees reflected, the CFI may adopt in the other funding calls to minimise 
evaluator workload while still providing good-quality feedback [INT01-02-03]. 

201 

 
 
 
 
 
1.3  Monitoring, evaluation and learning 

The CFI undergo monitoring, evaluation and learning both formally and informally [INT01-
02-03]. After each competition, the CFI run an internal ‘post-mortem’, during which they 
reflect on the process and gather statistics on the success rate by size of institution or by 
discipline  [INT01-02-03].  The  CFI  also  monitor  and  gather  information  on  gender  and 
career stage of applicants, and the interviewee stated they are confident there is no bias 
within the system [INT01-02-03]. The CFI require their staff to undertake bias training, 
and although this is currently not compulsory for experts and committee members, it is 
recommended; the CFI are considering making it compulsory [INT01-02-03]. In addition, 
the CFI engage with the other stakeholders involved in the process and gather feedback. 
For example, the CFI run a survey of MAC members where they ask for feedback [INT01-
02-03].  In  addition,  they  gather  feedback  on  the  draft  call  and  update  the  full  call 
depending  on  this  feedback  [INT01-02-03].  They  also  run  consultations  with  the 
institutions in order to gather feedback on the process from the institutional perspective 
[INT01-02-03]. 

The  CFI  have  also  commissioned  formal  evaluations  of  individual  funds  conducted  by 
external  organisations.  For  example,  the  CFI  have  commissioned  an  evaluation  of  the 
College-Industry Innovation Fund and have constructed an implementation plan to make 
changes  based  on  the  findings  of  the  evaluation  [INT01-02-03].  There  have  also  been 
wider external evaluations of the CFI. For example, a 2015 performance evaluation of the 
CFI  concluded  that  the  CFI  has  strong  performance  against  each  of  the  outcomes  put 
forward in its logic model (Bell Browne Molnar & Delicate Consulting Inc., n.d.). 

The CFI are keen to learn from the monitoring and evaluation that they do and have made 
several changes to their process [INT01-02-03]. They introduced the elimination of weaker 
proposals  between  the  first  and  second  stages  of  the  merit  review  process  and  of  the 
interviews that had been previously conducted [INT01-02-03]. In addition, they have made 
changes  to  what  they  consider  for  funding.  For  example,  the  concept  of  research 
infrastructure  has  moved  towards  being  more  inclusive  for  space  for  researchers  within 
social  sciences  and  humanities  subjects  rather  than  lab  infrastructure  or  scientific 
equipment [INT01-02-03]. In addition, they have changed the call language to reflect an 
increased emphasis on teamwork. Previously, competitions would have  1 project leader, 
but now there can be 2 team leaders and up to 10 researchers within the team [INT01-
02-03]. 

There is also a team within the CFI that look at performance, analytics and evaluation of 
the  funded  projects  [INT01-02-03].  This  team  are  specialists  in  evaluation  and  seek  to 
measure  the  impact  of  the  funded  projects,  including  the  wider  impacts  on  Canadian 
society, something which is very important to the CFI [INT01-02-03]. Once projects have 
received  funding,  they  are  required  to  report  on  their  progress,  including  research 
outcomes,  spin-offs, patents, publications  and  conferences. The evaluation team use an 
amalgamation of these indicators to determine whether the project has met its objectives 
[INT01-02-03]. 

1.4  Reflections and lessons learnt 

The Canadian Foundation for Innovation funding schemes are most similar to the Research 
Infrastructures funding within the Excellent Science Pillar of H2020, as they are committed 
to  supporting  research  and  technology  development  through  the  provision  of  research 
infrastructure. 

The  CFI  proposal  evaluation  process  has  several  steps,  each  aimed  at  assessing  the 
proposal from different perspectives (e.g. assessing against the six call assessment criteria 
or  against  the  three  call  objectives).  Interviewees  reflected  that  although  the  process 
enabled  a  thorough  examination  of  the  proposals,  considerations  were  currently  being 
undertaken on how to optimise the process further with a view to reducing the burden of 
the process. 

202 

 
 
One interesting aspect of the process at the CFI that may be of wider relevance, and one 
that may reduce the ultimate burden, is the notice-of-intent period that occurs prior to the 
full application process. This enables the CFI to ensure they recruit the relevant expertise 
for the evaluation process; reviewers can also be suggested by the applicants. In addition, 
as the CFI post the notices of intent online and encourage institutions with complementary 
proposals to consider joint initiatives, this may reduce the overall volume of proposals the 
CFI receive. Other measures during the application and evaluation process may also reduce 
burden further. For example, proposals contain a section where the applicant must suggest 
the relevance of the proposal to the assessment criteria, which may serve as a useful tool 
for reviewers when evaluating the proposals against these criteria. Furthermore, the CFI 
have introduced the elimination of weaker proposals between the different stages of the 
proposal evaluation process, resulting in reduced burden at each stage as fewer proposals 
progress forward. 

203 

 
 
2 

Deutsche Forschungsgemeinschaft (German Research Foundation) 

2.1 

Introduction 

The  Deutsche  Forschungsgemeinschaft (DFG,  the  German  Research  Foundation)  is 
Germany’s national funding organisation, investing across a wide range of subject areas, 
from  the  humanities  to  materials  science  and  engineering,  via  a  range  of  funding 
instruments (DFG, n.d.). The DFG provide grants for individual researchers in Germany or 
at German research institutions abroad, funding for institutions, and prizes to individuals 
for outstanding individual research activities (Figure 79). Funding at the DFG is organised 
under four categories. 

Investigators: This is relevant only to individual researchers, with a strong focus on early-
career researchers. 

Themes:  This  applies  to  both  individuals  and institutions,  and  is  used  to  fund  individual 
research projects or series of projects with a clearly defined programme structure in terms 
of content, duration and personnel. 

Forum:  This  supports  collaborative  research  either  at  the  individual  or  the  institutional 
level,  facilitating  ‘open  platforms’,  and,  as  such,  proposals  are  not  required  to  outline 
structured research content, duration or personnel, only the general research idea. 

Infrastructure: This supports either individuals or institutions, for projects that will either 
develop research and information infrastructure or acquire new technical equipment. 

Figure 79: DFG funding portfolio (DFG) 

The DFG’s annual budget is EUR 3.3 billion, distributed across individual grants, research 
units, priority programmes,  and so forth (Figure 80Figure 80) (DFG,  n.d.). In 2019, the 
DFG funded 17 100 projects, with a success rate of 33.3% in individual grant programmes 
(DFG,  n.d.).  The  highest  success  rate  was  seen  in  ‘Natural  Sciences’  and  the  lowest  in 
‘Engineering  Sciences’.  However,  ‘Engineering  Sciences’  had  the  highest  success  with 
regard to amount of funding received versus amount of funding requested (Figure 81). 

204 

 
 
 
 
 
 
Figure 80: DFG research funding by programme for 2019 (in million EUR and %) (DFG) 

Figure 81: Funding rates in the individual DFG grant programmes by scientific discipline, 2017–2019 (DFG) 

2.2  Proposal evaluation processes 

The proposal submission and evaluation process for the DFG depends on the programme 
and  subject  area.  However,  there  are  some  fundamental  features  of  the  proposal 

205 

 
 
 
 
 
 
 
evaluation process that are common to all DFG funding schemes (DFG, n.d.). A summary 
of the basic proposal evaluation system at the DFG is displayed below, in Figure 82. 

Figure 82: DFG decision-making process (DFG) 

Proposal submission process 

The DFG’s proposal submission process may consist of one or two stages, depending on 
the  programme.  In  a  one-stage  submission  process,  applicants  submit  a  full  proposal 
detailing the proposed project researchers and institutions (including publication list and 
CVs);  the  proposal’s  innovative,  ‘state-of-the-art’  aspects;  the  project’s  objectives  and 
work  programme  (including  project  duration);  the  proposed  methodology;  and  the 
requested  modules  and  funds  (DFG,  n.d.).  In  addition,  to  maximise  the  ‘diversity 
dimensions in research’ – a key organisational principle – applicants are required to fill in 
a section on sex, gender and/or diversity. In a two-stage submission process, applicants 
first  submit  a  draft  of  the  proposal;  they  are  then  invited  to  complete  a  full  proposal  if 
successful (DFG, n.d.). Draft proposals vary in length depending on the programme, from 
8  to  100  pages,  and  include  information  on  the  proposed  project  staff  and  institutions, 
proposed research programme, and objectives (DFG, n.d.). For typical project funding, the 
maximum length of a full proposal is 25 pages in A4 format, with the proposed research 
being described in no more than 15 pages and supplementary information on the ethical 
and  legal  aspects  of  the  project  in  no  more  than  10  pages  (DFG,  n.d.).  However,  some 
funding streams allow for a much longer draft proposal, of 100 pages. 

206 

 
 
 
 
Proposals  are  submitted  via  elan,  the  DFG’s  web-based  electronic  proposal  processing 
system, and processed by the DFG head office, whose staff check the proposal to ensure 
formal requirements have been met and that that no information is missing (DFG, n.d.). 

Review process 

Once proposals have been checked for eligibility, expert peer reviewers (who are unpaid 
and work in an honorary capacity) are selected by programme directors from the DFG head 
office  (DFG,  n.d.).  These  programme  directors  are  responsible  for  ensuring  that  the 
selected reviewers have expertise in all important aspects of the relevant proposal and are 
‘capable of giving an objective appraisal of the proposal’ (DFG, n.d.). Although there is no 
official  guidance  on  the  selection  criteria  for  reviewers,  certain  characteristics  are 
considered, as illustrated in  Figure 83, which shows the results of a 2007 survey of 577 
DFG  head  office  staff  on  which  selection  criteria  they  deemed  important  for  reviewers 
(Hornbostel and Olbrecht, 2007). 

Generally,  the  programme  directors  seek  three  independent  reviews;  however,  more 
individuals are involved in the panel discussion. The panels are organised by subject area, 
with  panel  members  selected  based  on  their  expertise  and  independence  from  the 
proposals76. The exact number of panellists varies. There are approximately 10 panellists 
for the Collaborative Research Centres (CRCs), while for the Priority Programmes, a much 
larger group, of 30 scientists, can be invited for a round-table discussion (Hotopp, 2005). 

Figure 83: Importance of selection criteria for DFG reviewers (DFG, n.d.) 

All  DFG  proposals  are  evaluated  against  the  following  criteria,  plus  specific  additional 
considerations associated with the individual programme  (DFG, n.d.)77: i) Quality of the 
project;  ii)  Objectives  and  work  programme;  iii)  Applicants’  qualifications;  iv)  Work  and 
research environment; v) Opportunities for employment in a scientific environment; and 
vi) Planned allocation of funding. The weighting of these criteria is not clearly specified in 
public documentation. Additionally, reviewers are not asked to provide a metric score on 
each component, suggesting these are not formally weighted, but, rather, that reviewers 
are asked to take them into consideration when reviewing and writing a summary report 

76 At least one member of DFG’s central decision-making team also participates in the panel review 
as a quality assurance step.  

77 For co-ordinated programmes, the programme-specific criteria and the quality and added value of 
the collaboration are also considered.  

207 

 
 
 
 
                                                 
of their assessment, and when giving a draft recommendation as to whether the proposal 
should be considered for funding or not. 

The  reviewers’  recommendations  and  appraisal  are  then  passed  to  the  review  boards. 
There are 49 review boards, with a total of 632 members, elected by the German scientific 
community (in the 2019 elections, 53 300 researchers cast votes), who have responsibility 
for  ensuring  the  quality  of  the  review  process  and  ensuring  that  reviewers  have  been 
appropriately selected. They also check that fair, uniform standards have been applied to 
all proposals. If this is the case, they prepare a funding decision based on comparison of 
all proposals received within the relevant board area (DFG, n.d.). The boards meet four to 
six times a year, either face-to-face or via teleconferencing, to discuss and jointly assess 
proposals received for each call. In the main, they discuss proposals where a unanimous 
decision was not reached by the expert reviewers. In these cases, the proposal is assigned 
a rapporteur from among the review board members, who presents the research project 
to the group for discussion, after which a final recommendation is made. 

After submission of this final recommendation, the proposal is once again checked by the 
DFG  head  office  for  errors  and  potential  conflicts  of  interest  before  being  passed  to  the 
DFG Joint Committee, the DFG’s main decision-making body (DFG, n.d.). This is comprised 
of scientific members, representatives from the federal government and the federal states, 
and two representatives from a German non-profit association promoting science and the 
humanities (DFG, n.d.). The DFG Joint Committee makes cross-subject comparisons and 
final funding decisions, sending the decision letter and the reviewers’ anonymised reports 
to applicants via the DFG’s head office (DFG, n.d.). 

The  above  describes  the  basic,  two-step  proposal  evaluation  procedure  at  the  DFG. 
However, there are exceptions and flexibilities made for certain programmes (DFG, n.d.). 
For  example,  the  coordinated  programmes  have  a  distinct  evaluation  process.  The 
coordinated  programmes  are  those  funding  instruments  that  ‘promote  cooperation  and 
structural  innovation  by  encouraging  national  and  international  collaboration  in  areas  of 
current  relevance  and  by  concentrating  scientific  potential  at  a  university’  and  have 
separate evaluation criteria. There is no individual review stage, but, rather, review takes 
place  via  review  panels  in  which  members  of  the  review  board  are  also  involved.  In 
addition,  coordinated  programmes  have  an  on-site  evaluation  component,  whereby  the 
review  panel  speak  directly  to  applicants  and  the  proposed  project  team,  as  well  as 
assessing  the  research  environment  (DFG,  n.d.)78.  For  some  coordinated  programmes, 
such as the CRCs, the page limit is extended to 100 pages to provide additional details on 
institutional conditions and how the network will work. For the ‘Forum’ funding streams, 
proposal  evaluation  focuses  on  the  ‘overall  research  idea’  and  the  applicants’  expertise 
rather  than  requiring  detailed  information  on  the  research  content.  Similarly,  the 
evaluation of ‘Infrastructure’ focuses on the contribution of the proposed project to improve 
research and information infrastructure. 

Monitoring, evaluation and learning 

Although the DFG can be seen to follow traditional peer review procedures in most ways, 
there are specificities to its evaluation process, such as the use of review boards. Review 
boards were introduced in 2004 following recommendations of a 1999 external evaluation 
of  the  DFG  and  the  Max  Planck  Society  (Schiermeier  and  Weydt,  2000).  Prior  to  2004, 
expert  reviewers  were  charged  with  both  reviewing  proposals  and  formulating  funding 
recommendations  (Hornbostel  and  Olbrecht,  2007).  If  reviewers  though  that  additional 
expertise was needed, they could consult a ‘special reviewer’ at their discretion. The latter 
were ‘neither regulated nor legitimised by the statutes of the DFG’ but were increasingly 
central to proposal evaluation. In this way, proposal review, the comparative assessment 

78 These are long-term collaborations (up to 12 years) between research institutions, university and 
non-university collaborators, who work together on a multidisciplinary research programme.  

208 

 
 
                                                 
of  proposals  for  funding  recommendations,  and  the  final  funding  decision  are  separated 
into different stages. 

Reviewers’ perceptions of the introduction of review boards were collected in a 2006 survey 
(Hornbostel and Olbrecht, 2007). The study found broad approval among reviewers and 
found  that  they  were  generally  resistant  to  further  structural  changes,  such  as 
standardisation and formalisation. The study found that reviewers were mostly critical of 
‘double-blind’ evaluations, as they believed that assessment of the applicant and their track 
record  to  be  an  important  aspect  of  evaluation  and  that,  in  any  case,  it  could  not  be 
achieved  in  practice,  as  the  cited  literature  would  disclose  their  identity.  Conversely, 
reviewers  broadly  supported  greater  transparency  in  the  process,  advocating  for  final 
review  reports  to  be,  at  least  in  part,  published  online.  In  addition,  more  than  half  of 
reviewers then surveyed said they would welcome a standardised and open collection of 
reports. 

Another characteristic of the DFG’s funding process is equal opportunities and diversity in 
research, which have a direct bearing on proposal review. Although the DFG state there is 
‘no  room  for  any  non-scientific  criteria  in  the  review  process’,  they  also  state  that  ‘due 
consideration must be given to individual circumstances’ in the review stage (DFG, n.d.). 
For  example, the DFG  recommend  that all researchers indicate in their CVs  career gaps 
related to childcare, so that this can be considered by reviewers to potentially ‘compensate 
for any disadvantages’  (DFG, n.d.). Even if formal programme criteria are not met  as a 
result of such delays, the DFG make clear that possible allowances can still be made (DFG, 
n.d.).  For  example,  the  Emmy  Noether  Programme,  an  early-career  programme  that 
enables  post-doctorates  with  up  to  4  years  of  experience  to  lead  an  independent  junior 
research group, allows the years of experience to be extended by 2 years for every child 
under the age of 12 (DFG, n.d.). The number of female applicants and the representation 
of women in the German research landscape more broadly has improved since this review, 
but the DFG have urged caution in making causal relationships. 

The DFG has also taken steps to increase  equal opportunities among reviewers. A 2007 
study of ‘Gender Equality in DFG Research Funding’ commissioned by the DFG, found that 
women  were  underrepresented  among  peer  reviewers,  and  that  female  reviewers  were 
significantly  less  likely  than  their  male  peers  to  say  that  peer  reviewers  are  generally 
objective and treat men and women equally (Güdler and Reinhardt, n.d.). As a result of 
this  study,  the  DFG  established  the  Research-Oriented  Standards  on  Gender  Equality in 
2008,  which  has  increased  representation  of  women  at  all  levels  of  the  review  process 
(DFG, n.d.). 

2.3  Lessons learnt 

The DFG’s focus on individual grant programmes enables this case study to offer insight 
particularly to the ERC and MSCA IF programmes. However, the DFG funds a wide variety 
of  activities,  including  major  collaborative  institutional  grants,  offering  learnings  for  the 
broader H2020 portfolio and its proposal evaluation process. For example, unlike the ERC, 
where experts are recruited to form panels before proposals are submitted, the DFG selects 
expert reviewers only after proposals are received. Some other H2020 actions are similar 
to the DFG’s programmes in this respect, especially multi- and interdisciplinary research 
proposals  where  proposal  content  would  be  difficult  to  predict  a  priori.  Selecting  expert 
reviewers  after  applicants  have  submitted  proposals  enables  the  DFG  to  more  precisely 
match reviewers to proposals. 

The DFG have, as an organisational priority, taken steps to address gender inequalities in 
the  research  system,  such  as  accommodating  family-related  career  gaps  and  paying 
reviewers  with  family  obligations  (up  to  EUR  60  allowance),  to  increase  the  female 
representation  of  reviewers  and  number  of  female  applicants.  Our  findings  from  the 
analysis  of  the  proposal  evaluation  process  for  H2020  show  that  across  the  framework 
programme, female applicants comprise only 25.5% of applications, with the exception of 
‘Science  with and for Society’ pillar, where 62%  of applicants are female. This suggests 
more work is needed to encourage and support female researchers and innovators to apply. 

209 

 
 
Introducing  strategies  to  address  family-related  disruptions  could  lead  to  an  increase  in 
submissions by female applicants. 

Much like H2020, the DFG have a clear commitment to evaluating their processes and use 
monitoring  and  evaluation  to  make  improvements.  Unlike  H2020  programmes,  the  DFG 
have a dedicated evaluation partner to offer learnings and improvements of the system. 
This was particularly evident following major reforms of proposal evaluation in 2004, with 
the introduction of the review boards. The review boards are in many ways similar to the 
panel review stage used widely across H2020 programmes. However, review boards at the 
DFG  are  arranged  prior  to  calls  for  proposals  (as  is  also  the  case  for  ERC  grants),  and 
elections are used to appoint members  of these review boards.  In incorporating elected 
reviewers into the process and demarcating the responsibilities of funding and review, the 
DFG have sought to create a legitimate and fair proposal evaluation system that excludes 
bias (Hornbostel and Olbrecht, 2007).

210 

 
 
3 

National Health and Medical Research Council – Australia 

3.1 

Introduction 

The NHMRC is the Australian government’s primary health and medical research funding 
agency, investing in research across four main areas of health and medical research: Basic 
science, Clinical medicine and science, Public health, and Health services research (NHMRC, 
2021a). the NHMRC administers the Medical Research Endowment Account (MREA), aimed 
at  providing  support  to  institutions  and  people  engaged  in  medical  research,  as  well  as 
encouraging  training  and  capacity  building.  In  2019,  the  NHMRC  committed  AUD  923.2 
million (approximately EUR 565 million) in funding across 857 awards. Success rates for 
applicants range from 13.2% for investigator awards to 11.1% for Ideas Grants and 15.6% 
for Synergy Grants (see below for more information on funding streams). 

The NHMRC’s grant programme comprises four funding streams, summarised in Table 75. 
Individuals  can  hold  a  maximum  of  two  grants  simultaneously,  with  the  following 
exceptions and considerations: 1) individuals may apply for an Investigator Grant together 
with an Ideas Grant, but will only be awarded the Investigator Grant if successful at both; 
2) individuals can hold two Ideas Grants and apply and receive an Investigator Grant, but 
their funding from the Investigator Grant will be discounted until the Ideas Grants have 
been completed; and 3) individuals can hold two Ideas Grants and a Synergy Grant. 

211 

 
 
Table 75: Overview of the NHMRC’s restructured grant programme79  

Grant type 

Purpose 

Investigator 
Grants  

To support the 
research of 
outstanding 
investigators at 
all career stages 

Strategic and 
Leveraging Grants 

To support 
research that 
addresses 
identified national 
priorities 

Synergy Grants 

Ideas Grants  

To support 
focused 
innovative 
research projects 
addressing a 
specific question  

To support 
outstanding 
multidisciplinary 
teams of 
investigators to 
work together to 
answer major 
questions that 
cannot be 
answered by a 
single investigator 

Duration 

5 years 

5 years 

Up to 5 years 

Number of chief 
investigators 

1 

4–10 

1–10 

Funding 

Research support 
package plus 
optional salary 
support 

Grant of a set 
budget (AUD 5 
million) 

Based on the 
requested budget 
for research 
support 

Maximum number 
of applications 
allowed per round 

1 

1 

2 

Maximum number 
of each grant type 
that can be held 

1 

1 

2 

Varies with 
scheme 

Dependent on 
individual scheme 

Dependent on 
individual scheme 

Not capped 
relative to 
Investigator, 
Synergy and 
Ideas Grants; 
Dependent on 
individual scheme 

Not capped 
relative to 
Investigator, 
Synergy and 
Ideas Grants; 
Dependent on 
individual scheme 

Indicative 
proportion of all 
funding 

40% 

5% 

25% 

30% 

79 Source: NHMRC (NHMRC, n.d.) 

212 

 
 
 
 
 
                                                 
3.2  Proposal Evaluation Processes 

The  NHMRC’s  principles  of  peer  review  (NHMRC,  2021b)  are  fairness,  transparency, 
independence,  appropriateness  and  balance, 
research  community  participation, 
confidentiality,  impartiality,  and  quality  and  excellence.  Their  proposal  evaluation 
processes  follow  broadly  standard  peer  review  processes,  including  submission  of  a 
proposal,  expert  review,  panel  discussion  and  selection  of  applications  for  award.  The 
specifics  of  this  process  and  the  criteria  used  for  assessment  vary  across  the  different 
funding mechanisms outlined above, as summarised in Table 76. 

Table 76: Overview of THE NHMRC proposal evaluation processes by grant type80 

Grant type 

Assessment criteria 

Summary of assessment process 

Investigator 
Grants 

Track record (70%) 
Publications (35%) 
Leadership (15%) 
Research impact (20%) 
Knowledge gain (30%) 

Synergy 
Grants 

Track record (40%) 
Knowledge gain (30%) 
Synergy of the team (30%) 

Ideas 
Grants 

Research quality (35%) 
Innovation and creativity 
(25%) 
Significance (20%) 
Feasibility (20%) 

  Submit application 
  Five-member  panels  assess  applications  on  all 

criteria 

  Panel discussion by exception (i.e. for applications 
where  there  is  disagreement  only),  conducted  by 
teleconference or videoconference 

  Ranked  list  based  on  overall  score  produced  to 

inform funding allocation  

  Submit application 
  Stage 1 assessment 
  Assessment  by  5  panel  members  with  broad 
expertise on ‘knowledge gain’ and ‘synergy’ criteria 

  Shortlisting of applications 
  Panel discussion by exception (i.e. for applications 
where  there  is  disagreement  only),  conducted  by 
teleconference or videoconference 

  Stage 2 assessment 
  Assessment  of  track  record  for  chief  investigators 
(5  assessors  per  chief  investigator,  with  specific 
disciplinary expertise) 

  Panel discussion by exception (i.e. for applications 
where  there  is  disagreement  only),  conducted  by 
teleconference or videoconference 

  Ranked  list  produced  based  on  overall  score  to 

inform funding allocation 

  Submit application 
  Review  by  discipline  panels,  four  spokespersons 

score each application on all criteria 

  Not  for  further  consideration  (NFFC)  and  rescue: 
Refers  to  process  in  which  the  bottom  50%  least 
competitive  applications  (based  on  initial  scores) 
are  removed  from  consideration,  but  reviewers 
have  the  opportunity  to  ‘rescue’  one  application 
from that list for discussion at the panel meeting 
  Panel  meeting,  all  members  score  applications  on 

all criteria 

  Ranked  list  produced  based  on  overall  score  to 

inform funding allocation 

80 Source: Guthrie, Rodriguez-Rincon, et al. (2019) 

213 

 
 
 
 
                                                 
3.3  Monitoring, evaluation and learning 

Until  recently,  expenditure  of  the  MREA  was  spread  across  a  variety  of  grant  schemes, 
including  Project  Grants,  Program  Grants,  Development  Grants,  Partnership  Projects, 
Centres of Research Excellence, and Targeted Calls for Research. MREA funding was also 
directed  at  scholarships  and  fellowships  and  at  collaborative  schemes  with  international 
partners. 

Feedback  from the research  community revealed that the work required to prepare and 
evaluate unfunded grant applications was becoming increasingly burdensome, with many 
early-career researchers being discouraged from exploring new avenues of research and 
many even becoming discouraged from pursuing a career in research (NHMRC, 2015). This 
reflects wider international trends. Namely, there has been a funding crisis for biomedical 
research, not just in Australia but internationally, partly due to an increase in the number 
of researchers. This has led to a decline in success rates for applications (De Vrieze, 2017; 
Schroter, Groves and Højgaard, 2010; Snell, 2015) and an increase in conservatism in all 
aspects of the peer review and funding application process (Guthrie, Ghiga and Wooding, 
2018). 

In  2016,  the  NHMRC  conducted  a  review  of  the  grant  programme,  with  the  aim  of 
determining whether the funding schemes of the NHMRC could be restructured to optimise 
the public investment in health and medical research (NHMRC, 2016). This review led to 
restructuring  of  the  NHMRC’s  grant  programme,  with  changes  aimed  at  encouraging 
greater creativity and innovation in research, minimising burden on researchers and peer 
reviewers, and providing opportunities for talented researchers at all stages in their career. 
The new grants programme is structured around three key funding streams, as outlined 
above (plus strategic awards), and is intended to reduce the number of awards any one 
individual  holds  –  in  order  to  reduce  the  burden  of  applications  –  and  to  support  more 
innovative work and researchers across career stages. Alongside the refresh of their grant 
award  structure,  the  NHMRC  also  conducted  a  review  of  their  proposal  evaluation 
processes,  which  have  been  refined  alongside  the  funding  system  itself.  Key  changes 
include the following (Kelso, 2019). 

Changing criteria for assessment: The criteria for each stream have been adapted to better 
reflect  the  aims  of  each  funding  stream.  Changes  to  the  assessment  of  track  record  for 
investigator awards are particularly notable, with a new framework developed intended to 
focus more on outcomes than, for example, on previous funding awards. 

Streamlining  processes  to  reduce  burden:  All  funding  streams  are  now  streamlined  to 
include  panel  review  only,  with  no  interview  stage.  Applications  are  reviewed  by  panel 
members  (typically  five  or  more),  and  panel  discussions  for  two  of  the  schemes  are 
conducted only ‘by exception’ – i.e. in those cases where there is disagreement between 
reviewers.  Discussions  are  conducted  remotely  by  teleconference  and  videoconference, 
which has been shown to have little impact on outcomes (Gallo, Carpenter, and Glisson, 
2013).  This  streamlined  review  process  allows  opening  and  closing  dates  for  different 
funding streams to be staggered, reflecting evidence that concurrent application dates can 
impact on applicant burden and work–life balance (Herbert et al., 2014). 

Improving confidence in the process: There have also been a number of changes intended 
to improve confidence in the review process and its outcomes. The first is to increase the 
number of panel members reviewing applications in detail – with at least five reviews for 
each  application  across  all  funding  streams.  There  have  also  been  changes  in  the 
assessment  process  to  ensure  different  criteria  are  assessed  by  those  with  the  most 
relevant knowledge. For example, for Synergy Grants, individuals with broad expertise are 
used to assess criteria of ‘Synergy’ and ‘Knowledge gain’, while discipline experts are used 
to assess the track record of each chief investigator. 

Supporting  innovative  research:  A  specific  funding  stream  has  been  dedicated  to 
supporting  innovative  and  creative  research  –  the  Ideas  Grants.  These  have  a  specific 
criterion for assessment of ‘Innovation and creativity’, which is intended to encourage fresh 

214 

 
 
thinking  and  encompasses  a  broad  definition  of  innovation  –  not  limited  to  commercial 
innovation – with the aim to cover concepts, approaches, methodologies and interventions 
in  all  research  areas.  This  funding  stream  retains  panel  meetings,  unlike  the  other  two 
schemes,  and  it  also  retains  two  previous  features  of  NHMRC  funding,  namely  ‘Not  for 
Further  Consideration’  and  ‘Rescue’.  The  Not  for  Further  Consideration  process  was 
introduced with the aim of reducing burden by removing the least competitive applications 
before the panel meeting – initially the lowest-scoring 30% and, since 2014, the lowest-
scoring 50%. The list is based solely on the scores of spokespersons. Panel members are 
also  given  the  opportunity  to  ‘Rescue’  one  application  from  the  list  of  NFFC  applications 
that they feel has potential, which will then be fully reviewed by the panel. 

These changes were developed on the basis of an extensive consultation exercise, with the 
aim of addressing some of the challenges in the existing proposal evaluation system. As 
shown below, the framework for assessment of track record is intended to be holistic and 
forward  looking,  capturing  information  on  outcomes  and  achievements  from  research 
(Figure 84). 

Figure 84: Framework for track record assessment (Kelso, 2019) 

3.4  Reflections and lessons learnt 

The NHMRC provides an interesting example of a funding organisation that has identified 
and  recognised  some  issues  with  its  existing  proposal  evaluation  system  and  looked  to 
learn and adapt, building on existing evidence and in consultation with the sector. Some 
of  the  observed  challenges  –  particularly  around  the  burden  of  the  system  and  the 
challenges  in  supporting  innovative  and  creative  research  –  are  likely  of  relevance  to 
Horizon  Europe  and,  indeed,  most  major  R&I  funders.  There  are  also  some  interesting 
ideas  in  terms  of  changes  in  their  process  that  could  be  considered  by  other  funders  to 
address some of these challenges. 

215 

 
 
 
 
a.  Discussion ‘by exception’ – reducing burden by discussing only those applications 
where there is disagreement and conducting those discussions remotely. This could 
be reflected in the way consensus meetings are used in the framework programmes. 

b.  Streamlining  of  processes  –  aligning  similar  funding  schemes  into  the  same 
assessment process. For example, all fellowship schemes are now assessed in the 
same way as Investigator Grants. 

c.  Tailoring of approaches to support innovative research – specific funding streams 
to target innovative research with different criteria (excluding track record and with 
specific focus on innovation and creativity) and with different processes that allow 
for  more  disagreement  (panel  discussion  of  all  applications,  option  for  individual 
reviewers to ‘rescue’ one application). 

d.  Staggering of applications dates for different funding streams. 

e.  Matching of assessors to application criteria – for example, different assessors for 
track record for multidisciplinary applications. This could be particularly useful for 
complex, multi-applicant bids, which are common in the framework programmes. 

Although the NHMRC are a health research–specific funder, their approaches likely have 
wider applicability, and aspects of this approach would likely translate to a larger, cross-
disciplinary funding context. However, the new grants programme only completed its first 
round of funding in late 2019, and there is not yet any evaluative evidence to assess the 
effectiveness  of  these  changes.  Many  of  these  changes  draw  on  prior  evidence  in  the 
literature; however, their application and effectiveness in this real-world setting are yet to 
be  established.  The  NHMRC  have  developed  a  framework  to  assess  the  success  of  their 
new  grant  programme  (Guthrie,  Rodriguez-Rincon,  et  al.,  2019),  and  therefore  we  may 
expect to see more evidence on the effectiveness of these changes in achieving the aims 
of reducing burden, supporting more innovative research, and ensuring opportunities are 
available for researchers at all career stages in the future. 

216 

 
 
4 

National Research Foundation – Singapore 

4.1 

Introduction 

The  National  Research  Foundation  (NRF)  was  established  in  2006  and  is  a  department 
within  the  Prime  Minister’s  Office  of  Singapore  (NRF,  n.d.).  The  NRF  sets  the  national 
direction for research and development within Singapore by developing policies, plans and 
strategies  for  research,  innovation  and  enterprise,  as  well  as  funding  initiatives  and 
supporting R&D capability. The NRF aims to set the national direction for R&D by: 

  developing policies, plans and strategies for research, innovation and enterprise; 

 

funding initiatives that strengthen research and scientific capabilities and that achieve 
economic and national impact; 

  building up R&D capability and capacity through nurturing people and attracting foreign 

researchers and scientists; 

 

coordinating the research agenda of different  agencies to transform Singapore into a 
knowledge-intensive, innovative and entrepreneurial economy. 

The NRF has several opportunities for grant funding, ranging from grants for researchers 
across differing career stages, as well as thematic calls and international calls jointly run 
through the NRF and funding agencies outside of Singapore (NRF, n.d.). An example of a 
grant aimed at researchers at a particular career stage is the NRF Fellowship, which aims 
to provide the opportunity for early-career researchers to carry out independent research 
in Singapore over a five-year period, with each NRF Fellowship offering a research grant 
of up to SGD 3 million (approximately EUR 1.86 million) over a 5-year period (NRF, n.d.). 

The NRF also offer grants aimed to  support research at differing stages of technological 
readiness. For example, the Central Gap Fund aims to support the translation of research 
outcomes into useful products, processes and services that might generate economic and 
societal  benefits  (NRF,  n.d.).  This  fund  supports  project  teams  to  develop  early 
technologies  into  working  prototypes  or  functional  processes  to  attract  private  sector 
investment.

217 

 
 
Table 77: Funding and grants available from the NRF81 

Grant 

Description 

NRF Fellowship 

The  Fellowship  programme  provides  opportunities  for 
early-career  researchers  to  carry  out  research  in 
Singapore. 

NRF Fellowship for Artificial Intelligence 

The  NRF  Fellowship  for  Artificial  Intelligence  provides 
opportunities 
established 
early-career 
researchers to carry out artificial intelligence research in 
Singapore.  

and 

for 

NRF Investigatorship 

The  Investigatorship  grant  provides  opportunities  for 
established, innovative and active researchers to pursue 
ground-breaking, high-risk research. 

Competitive Research Programme 

This  programme  funds  use-inspired  basic  research 
based on scientific excellence. 

Central Gap Fund 

This  scheme  supports  the  translation  of  research 
outcomes  into  useful  products  and  services  that 
generate social and economic outcomes. 

International Grant Calls 

The  NRF  collaborates  with  international  partners  for 
joint funding of research projects. 

4.2  Proposal evaluation processes 

The proposal evaluation processes differ across the different funding schemes.  Calls are 
announced  through  the  NRF  website  and  posted  on  the  Integrated  Grant  Management 
System, which holds information on both closed and open calls. The number of calls varies 
per fund. For example, the NRF Fellowships programme invites applications once per year; 
the Competitive Research Programme invites applications twice per year, and the Central 
Gap Fund invites applications as part of a rolling call. Proposals are submitted through the 
Integrated Grant Management System. 

In the following sections, we describe the proposal evaluation process for NRF Fellowships, 
the Central Gap Fund and the Competitive Research Programme. 

NRF Fellowships 

For the NRF Fellowship, the NRF invite applications once a year, providing opportunities for 
early-career  researchers  to  conduct  independent  research  in  Singapore.  Each  NRF 
Fellowship  provides  a  research  grant  of  up  to  3  million  SGD  (approximately  EUR  1.86 
million) over 5-year period (NRF, n.d.). For the NRF Fellowship, the application guide states 
that applicants should meet the following requirements (NRF, n.d.): 

  Have excellent scientific credentials as an outstanding young researcher in the early 

stage of their research career, 

81 Source: NRF (NRF, n.d.) 

218 

 
 
 
 
                                                 
  Be ready to take on an independent research appointment, 

  Submit  an  internationally  competitive  research  proposal  demonstrating  scientific 

excellence and potentially impactful outcomes, 

  Have the potential and aspiration to become a next-generation research leader in 

their field, 

  Be committed to carry out impactful research on a full-time basis in Singapore. 

The  NRF  proposes  an  application  template,  which  contains  the  following  sections:  title 
page; table of contents; executive summary (approximately 300 words); main body of the 
proposal  (maximum  8  pages,  consisting  of  specific  aims,  reasons  for  wanting  to  do  the 
research  in  Singapore,  and  where  research  will  be  carried  out;  background  and 
significance;  research  design  and  methods;  milestones  and  deliverables);  and  annexes 
(includes  references,  publication  list,  industry-link  research/patents)  (NRF,  n.d.).  In 
addition,  applicants  must  submit  their  CV  and  complete  the  NRF  Fellowship  registration 
form. 

The selection of NRF Fellows involves a three-stage process. In the first stage, all eligible 
applications are sent to all prospective host institutions in Singapore for longlisting. In the 
second stage, candidates are shortlisted by the NRF Fellowship evaluation panel, which is 
comprised of eminent scientists within the field. The panel look for individuals with a track 
record of early achievement, evidence of maturity, readiness to lead independent research, 
and  scientific  merit  of  the  proposed  research  (NRF,  n.d.).  Finally,  in  the  third  stage, 
shortlisted  candidates  are  invited  to  Singapore  for  an  interview  (NRF,  n.d.).  Before  the 
interview, applicants are encouraged to meet with all prospective participating institutions 
before  deciding  on  a  host  institution.  In  addition,  all  applicants  are  required  to  obtain  a 
letter of support from their host institution of choice describing the level of commitment to 
applicants (if successful) from the host institution. The interviews are undertaken by the 
NRF Fellowship evaluation panel. Applicants are notified of selection outcomes through the 
NRF,  and  further  discussion  may  be  taken  to  finalise  research  budgets  at  this  stage.  A 
formal letter of award is then issued via the selected host institution. No individual feedback 
is given at any stage of the selection process (NRF, n.d.). 

Central Gap Fund 

The  Central  Gap  Fund  aims  to  support  the  translation  of  research  outcomes  into  useful 
products,  processes  and  services  that  generate  economic  and  societal  benefits.  Projects 
must  aim  to  develop  early  technologies  into  working  prototypes  or  functional  processes 
that are closer to market, as well as demonstrating market need and demand (NRF, n.d.). 
The  fund  aims  to  provide  a  central  platform  to  resource  collaborating  institutes,  and 
collaborative  projects  are  preferred.  The  Central  Gap  Fund  provides  funding  support  for 
projects requiring 250 000 SGD (approximately EUR 155 000) or more for a period of up 
to 2 years. Applications are submitted by project teams, consisting of a project lead who 
has experience in leading translation/gap-funded projects; team members with relevant 
technical  expertise;  and  team  members  with  experience  in  business  management. 
Proposals are able to be submitted throughout the year, and the Central Gap Fund operates 
as a rolling call. Proposals are processed by the Central Innovation and Enterprise Office 
at the NRF. Proposals must provide impact metrics and propose key performance indicators 
as a means of tracking the success of the project (NRF, n.d.). The NRF outline examples 
of impact metrics and tracking indicators (described in Table 78).

219 

 
 
Table 78: Examples of impact metrics and tracking indicators provided by the NRF82 

Impact metric 

Possible tracking indicators 

Direct contribution to economy 

 
 

 
 

 

Productivity increase (labour and capital efficiency) in identified sector. 
Imputed value of IP from projected product/service sales and/or licensing revenue 
(local and overseas). 
Valuation of company / IP assets (assessed by independent valuer). 
Foreign  direct  investment  into  Singapore  (e.g.  foreign  licensing  revenue,  foreign 
research funding). 
Imputed value of goods and services created in adjacent/complementary industries. 

Contribution  to  key  sectors  of  Singapore’s  economic,  security  and  social 
interests 
Examples  of  key  sectors  include  water  management,  clean  energy, 
advanced  manufacturing,  deep  tech/hardware,  cybersecurity,  fintech, 
biotechnology and medical technology, materials engineering etc. 

  Disruption of identified key sector leading to productivity leap, self-sufficiency and/or 

creation of new markets. 

  Creation of sustainable competitive advantages for Singapore in identified key sector. 
Increased efficiency in coordination and usage of resources (e.g. transport systems). 
 
Improved  health  and  well-being  of  general/ageing  population,  improved  medical 
 
outcomes. 

Creation / introduction of new industries 

  Creation of new industry (e.g. additive manufacturing when it was first introduced). 
 

Introduction of new industry in Singapore. 

82 Source: NRF Central Gap Fund guidelines for applicants 

220 

 
 
 
                                                 
Job creation 

  Number of downstream jobs created (direct or indirect). 
  Quality of downstream jobs created (labour productivity).

Increased  partnerships  with  local  companies  and  other  public  research 
performers 

  Number and value of downstream research collaborations with local companies and 

other public research performers. 
Increased productivity/ competitiveness of local company as an outcome of project.

 

Development of I&E capabilities and professionals 

Enlarging Singapore’s global footprint in innovation and technology 

  Meaningful deployment of trained I&E manpower to industry. 
  Meaningful deployment of improved technology commercialisation, IP management 

and other I&E capabilities within public research organisations. 

  Number of STEM professionals trained in I&E skillsets.

  Number of notable, world-class innovative tech products/companies (and “unicorns”) 

generated. 

  Notable awards and media publicity. 
  Commercial agreements with notable global companies.

221 

 
 
 
 
 
 
 
 
the  office 

The proposal evaluation process for the Central Gap Fund is outlined in Figure 85. Before 
submitting a proposal, project leads are encouraged to approach potential collaborators, 
conduct a preliminary technology/intellectual property scan and develop an initial business 
plan  (NRF,  n.d.).  After  submission,  the  proposal  is  evaluated  by  the  Innovation  and 
Enterprise  Office  (IEO)83, 
transfer  and 
commercialisation  functions  for  the  host  institution.  For  each  application,  the  host 
institution must designate an IEO or equivalent that will evaluate the proposal based on 
the proposal criteria. Once the IEO has assessed the proposal, it provides an enforcement 
if satisfied. Proposals are then sent to the NRF Central IEO, which checks for formalities. 
After this, proposals are evaluated by the Central Gap Standing Oversight Committee, who 
evaluate proposals against the criteria listed in Table 79, with inputs from subject matter 
experts (NRF, n.d.). As part of the evaluation process, shortlisted teams may be requested 
to  present  their  proposals  to  the  Standing  Oversight  Committee  and  subject  matter 
experts. The Central IEO inform applicants of the outcome through their respective IEO. 

technology 

takes  on 

that 

Figure 85: Proposal evaluation process for the Central Gap Fund (NRF, n.d.) 

83 ‘The Host Institution’ refers to the body or institution employing or hosting the project lead. 

222 

 
 
 
 
 
                                                 
Table 79: Evaluation criteria for Central Gap Fund proposals84 

Criterion 

Guiding questions 

Project Scope 

Commercialisation Potential 

Team 

  Does the project fit the scope and timeframe of a 

Central Gap Project? 

  Does  the  expected  impact  of  the  project  accrue 

 

 

 

socio-economic benefits to Singapore? 
Is  the  project  in  an  area  where  Singapore  has  or 
could have a competitive advantage or where there 
is a national need or challenge? 
Is  the  expected  impact  and/or  deliverables  of  the 
project commensurate with the requested funding? 
Are  the  proposed  deliverables  and  milestones 
commensurate with the expected impact? 

  Does  the  project  address  an  important  unmet 

need? 

  Does  the  proposed  technology  have a  sustainable 
competing 

competitive 
patents/research labs/products/companies? 

advantage 

against 

  What is the addressable market size? 
  Does  the  project  have  a  viable  business  model 
(licensing,  spin-offs,  co-development,  etc.)?  Does 
the  project  have  a  clear  market  development 
roadmap? What are the possible barriers to entry? 
  What  is  the  time  to  market?  Is  the  project 
sustainable  e.g.  plans  have  been  made  for  long-
term funding beyond the Central Gap? 

  What  is  the  manufacturability/scalability  of  the 
technology?  How  easy  is  it  to  scale  up  the 
technology for mass production? 
Is there interest from potential customers? 

 

  Does  the  team  have  the  necessary  technical, 
business management and development expertise 
and resources? 

  Does  the  team  have  key/potential  partnerships 
(collaborators  from  industry  etc.)  that  will  enable 
faster development of the technology and shorten 
the time to market? 

Technical Innovation 

 

  What 

is  the  current  TRL  of  the  proposed 
technology?  What  TRL  would  the  team  like  to 
advance  the  proposed  technology  to  with  the 
Central Gap? 
Is  the  proposed  approach  unique,  innovative  and 
disruptive? 
Is the proposed approach technically sound? 

 
  Does  it  have  a  clear  technology  development 

Institutional Support 

Competitive Research Programme 

roadmap? 

  What is the IP position/strategy of the technology? 

 

Are there programmes & infrastructure available at 
the Host Institution or collaborators to adequately 
support the project? 

84 Source: NRF Central Gap Fund guidelines for applicants 

223 

 
 
 
                                                 
The  NRF  Competitive  Research  Programme  (CRP)  seeks  to  foster  the  formation  of 
multidisciplinary teams to conduct cutting-edge research projects. The CRP offers funding 
support to programmes of varying sizes, for a period of 3–5 years (NRF, n.d.). Calls for 
proposals  are  launched  twice  a  year  to  solicit  research  ideas  (NRF,  n.d.).  The  CRP  call 
process has two stages. First, there is the white paper stage, where applicants must submit 
their ideas to the NRF in no more than 5 pages and describe the research programme being 
proposed.  Specifically,  the  white  paper  should  contain  a  description  of  the  project’s 
objectives, a brief overview of the proposal, novelty of the proposed research, expected 
research outputs, budget, timeline and team expertise (NRF, n.d.). The NRF then shortlist 
ideas  through  the  NRF  expert  panels  to  be  developed  into  full  proposals.  Full  proposals 
must not exceed 20 pages and should include sections on research objectives, approach, 
programme  plan,  role  of  team  members,  and  outcomes  and  deliverables.  In  addition, 
applicants must submit a 1 page rebuttal highlighting how they have addressed reviewers’ 
comments from the white paper stage. For the section on outcomes and deliverables, the 
team  must  provide  an  impact  statement  and  a  pathway  to  impact.  Each  full  proposal 
submission may also include recommendations for international peer reviewers who could 
be  requested  to  evaluate  the  full  proposal  (NRF,  n.d.).  These  should  not  have  any 
relationship, direct or otherwise, with the PIs, co-PIs and/or collaborators that would create 
a real or apparent conflict of interest (NRF, n.d.). 

Full  proposals  are  evaluated  by  international  peer  reviewers  and  the  CRP  international 
evaluation panel. The evaluation criteria include a) significance of  research; b) potential 
for  scientific  breakthroughs/disruptive  innovation;  c)  programme  management  and 
budget; d) technical competencies of the research team; and e) potential impact/relevance 
to Singapore (NRF, n.d.). 

4.3  Monitoring, evaluation and learning 

To the best of our knowledge, there are no publicly available formal evaluations of the NRF 
process. However, certain funds have monitoring requirements for the progress of funded 
awards, which may be used for internal evaluation purposes. For example, the Central Gap 
Fund  requires  the  project  teams  to  provide  half-yearly  progress  reports;  final  progress 
reports;  and  post-project  review  reports  using  NRF  templates  (NRF,  n.d.).  Post-project 
review  reports  aim  to  capture  commercialisation  efforts,  outcomes  and  impacts  of  the 
project  1  year  after  the  project  end  date.  The  CRP  also  has  requirements  for  progress 
reports, including regular progress reports from the project lead to their host institution, 
at least once a year. The host institution must then provide consolidated progress reports 
(once a year) to the NRF containing details of all CRP projects. Final reports on the overall 
project  achievements  are  also  required  within  3  months  of  the  programme  completion 
dates.  Progress  reports  are  reviewed  by  an  NRF-appointed  review  panel,  and  the  NRF 
organises  a  mid-term  review  of  each  funded  programme  (NRF,  n.d.).  These  evaluation 
activities  at  the  award  level  provide  some  check  that  the  funding  awarded  is  being 
appropriately  used  and  may  provide  insights  into  the  achievements  of  those  awards. 
However,  at  present,  this  is  not  linked  to  funding  mechanisms  so  does  not  provide 
significant insight on the effectiveness of the proposal evaluation process. 

4.4  Reflections and lessons learnt 

The NRF has a strong emphasis on funding innovative and translational research, and many 
of the funds share similarities with aspects of H2020, such as the EIC Accelerator Pilot. The 
NRF has tailored parts of its process to better fund innovative research, and there may be 
points for learning from this for other funders. 

The  NRF  has  specific  research  grants,  and  assessment  criteria,  aimed  at  funding 
translational  research.  For  example,  to  support  proposals  with  high  potential  for 
technological  innovation,  the  NRF  specify  ‘Technological  readiness  level’  as  one  sub-
criterion that should be considered when assessing applications. 

The NRF encourage applicants to consider the potential impacts of their work from early 
on. For example, applicants are encouraged to consider impact metrics when developing 

224 

 
 
their proposals and to include metrics, as well as possible indicators to track performance, 
within  the  proposal  themselves.  Examples  of  metrics  include  direct  contribution  to  the 
economy, as well as job creation. This strong focus on economic impact at the proposal 
stage may increase the likelihood of beneficial economic outcomes from funded projects 
and  may  be  something  that  other  funders,  such  as  the  EU,  wish  to  include  within  their 
applicant guidelines. Funding proposals that supported these economic impacts would also 
support the EU to strengthen its industrial competitive position, as outlined in the H2020 
intervention logic. 

The NRF  collaborates  on the proposal evaluation process both with host institutions and 
related offices that have experience with technology transfer (such as the Innovation and 
Enterprise  Offices).  For  example,  host  institutions  may  participate  in  the  selection  of 
proposals  for  longlisting  before  they  are  evaluated  by  the  NRF  panels.  Ultimately,  this 
enables  some  of  the  burden  of  proposal  evaluation  to  be  transferred  onto  other 
organisations, as well as enabling the engagement of broader expertise during the process 
itself. 

225 

 
 
5 

Narodowe Centrum Nauki (National Science Centre) – Poland 

5.1 

Introduction 

The National Science Centre (NCN) in Poland is a government agency that is supervised by 
the Ministry of Science and Higher Education and was established in 2011 to support basic 
research in Poland (NCN, n.d.). The NCN funds projects in the arts; the humanities; the 
social, life, and physical sciences; and engineering. The mission of the NCN is to support 
the development of Polish research on the international arena and leverage the quality and 
effectiveness  of research through a competitive grant system  (NCN,  2019a). To achieve 
this mission, the NCN has set five primary goals: 

 

 

 

 

 

fund excellent research projects in basic research, 

support early-stage researchers, 

inspire  the  creation  of  large,  interdisciplinary  research  teams  that  are  able  to 
compete internationally, 

foster international cooperation in research, 

create new job opportunities in NCN-funded projects. 

The NCN supports basic research by funding research projects at  both the domestic and 
the international level through funding calls. In 2019, the NCN committed approximately 
1.28  billion  PLN  (approximately  EUR  0.28  billion)  to  fund  basic  research  under  their 
domestic  calls,  launching  a  total  of  13  domestic  calls,  which  received  a  total  of  10 438 
proposals,  of  which  2 533  were  recommended  for  funding  (success  rate = 22%)  (NCN, 
2019a)85. 

The organisational structure of the NCN includes four primary levels: the NCN director, the 
NCN  council,  the  NCN  coordinators  and  the  NCN  office  (NCN,  2020).  The  NCN  director 
oversees  the  NCN’s  statutory  tasks  as  well  as  financial  management.  The  NCN  council, 
made  up  of  24  academics  from  varying  disciplines,  define  the  research  priority  areas, 
determine  the  terms  and  conditions  of  proposal  calls,  allocate  funding,  and  appoint  the 
members of the expert teams who then evaluate the research proposals. Every 2 years, 
50% of the council are rotated [INT04]. The NCN coordinators organise the expert teams; 
conduct calls for research proposals; moderate panel meetings; and work to ensure that 
the peer review process is correct, impartial and fair [INT04]. Finally, the NCN office are 
the administrative core of the NCN; they provide information and support to applicants, as 
well as support for ongoing calls (NCN, 2020). 

The  NCN  supports  basic  research  through  financing  research  projects,  single  research 
activities, scholarships and post-doctoral scholarships (NCN, 2019a). The funding schemes 
aim to match the diverse needs of the research community and offer calls for researchers 
across the different stages of their research career [INT04]. As such, NCN funding calls are 
researcher-led,  ‘bottom-up’  calls,  where  researchers  may  specify  the  activities  they  are 
applying for funding for, as opposed to thematic, ‘top-down’ funding [INT04]. An exception 
to this are the calls run in partnerships with other international organisations [INT04]. 

Funding  calls  are  categorised  into  domestic  funding  calls  and  international  funding  calls 
(NCN,  2020)86.  Domestic  calls  are  regular,  NCN-funded  calls  for  research  proposals, 
scholarships,  fellowships  and  research  activities  that  contribute  to  basic  research,  and 

85  Success rate  was calculated as the  ratio of the  number of proposals recommended for funding 
against the number of proposals submitted, expressed as a percentage.  

86 In 2020, the NCN had 9 active domestic calls. 

226 

 
 
                                                 
these  are  launched  in  accordance  with  the  call  schedule,  once  or  twice  per  year. 
International calls are organised within a framework for international cooperation and may 
be  bilateral  (launched  in  cooperation  with  foreign  research  funding  institutions  and 
organisations)  or  multilateral  (typically  funded  from  the  resources  of  the  EU  framework 
programme), launched in cooperation between the NCN and foreign funding agencies. NCN 
grantees must be employed by a Polish host institution (NCN, n.d.). Table 80 provides an 
overview of the domestic funding schemes. 

Table 80: NCN domestic funding schemes87 

Funding scheme 

Description 

Preludium 

Preludium Bis 

Sonatina 

Sonata 

Sonata Bis 

Miniatura 

Maestro 

Tango 

OPUS 

Call  for  research  projects,  targeted  at  researchers 
without a PhD 

Call  for  research  projects  addressed  at  institutions 
operating doctoral schools 

Call for research projects, targeted at researchers who 
earned  their  PhD  no  earlier  than  3  years  before  the 
submission of the proposal 

Call for research projects, targeted at researchers who 
earned  their  PhD  between  2 and  7  years  before  the 
submission of the proposal 

Call  for  research  projects  aimed  at  setting  up  a  new 
research  team,  targeted  at  researchers  who  earned 
their  PhD  between  5  and  12  years  before  the 
submission of the proposal 

Call for research activities, targeted at researchers who 
earned their PhD up to 12 years before the submission 
of the proposal 

Call  pioneering 
experienced researchers 

research  projects, 

targeted  at 

Call to enable researchers to turn the results of their 
basic  research  projects  into  solutions  and  products 
capable  of  being  directly  socially  and  economically 
beneficial 

Call  for  research  projects,  targeted  at  all  researchers 
irrespective of research experience 

Although there is no specific funding instrument for innovative R&I, novelty and originality 
are considered within the criteria that proposals are assessed on [INT01]. 

5.2  Proposal evaluation processes 

The objective of the NCN is to grant funding to the top research projects that are selected 
through  a  peer  review  process.  The  interviewee  stated  that  the  ERC  model  had  guided 
initial discussion around the funding and proposal evaluation processes that was adopted 
by the NCN [INT04]. They stated that the NCN had been an instrument to enable peer-

87 Source: NCN information brochure (2020) 

227 

 
 
 
                                                 
reviewed  competitive  funding  and  transparency  of  process,  aspects  which  were  not 
prevalent within the Polish funding system previously [INT04]. 

The  NCN  process  considers  both  the  quality  of  the  project  and  the  achievements  of  the 
researcher involved. Proposal evaluation begins with an eligibility check conducted by the 
NCN  coordinators  [INT04],  which  involves  checking  proposals  for  completeness  and 
correctness. Around 2% of proposals are rejected at this stage, and there is the possibility 
for appeal [INT04].  Eligible  proposals then undergo a two-stage peer  review process by 
dedicated expert teams  (NCN, 2020). The  NCN council assemble a new expert team for 
every funding call [INT04], comprising of academics across Europe and worldwide based 
on  the  topics  of  the  submitted  proposals  to  that  call  [INT04].  Experts  are  paid  for  their 
time [INT04]. 

The  first  stage  involves  a  ‘qualification  check’,  where  proposals  are  assessed  by  two 
independent experts from the expert review group ‘expert teams’. Figure 86 provides an 
overview  of  the  qualification  check  process.  During  the  qualification  check,  the  experts 
evaluate a short proposal, which involves a short project description of up to five pages 
[INT04]88. Once the experts have conducted their independent evaluations, there is a panel 
session where all experts across the proposals discuss the individual evaluations and the 
decision to reject or approve a proposal is taken collectively by the group as a whole. The 
panel members prepare a shortlist of projects that are then admitted to the second stage 
of the peer review process (NCN, 2020). Approximately 50% of proposals are rejected at 
this stage [INT04]. Applicants whose proposal has been rejected receive their individual 
reviews,  as  well  as  a  consensus  report  compiling  information  from  the  panel  session 
[INT04]. Again, the applicants may appeal the decision if they wish [INT04]. 

Figure 86: Overview of the first-stage proposal evaluation process of the NCN (NCN Annual Review 2019) 

The  second  stage  of  the  peer  review  process  is  the  ‘specialist  evaluation’.  Figure  87 
provides an overview of the specialist evaluation process. 

88 Applicants submit the 5-page summary and 15-page full project description at the same time in a 
single submission stage [INT04]. 

228 

 
 
 
 
                                                 
Figure 87: Overview of the second-stage proposal evaluation process of the NCN (NCN Annual Review 2019) 

The specialist evaluation process varies depending on the type of call (NCN, 2020). For the 
funding calls OPUS, Preludium, Preludium Bis and Sonata (Table 80), this is done through 
evaluation  by  external  experts.  These  experts  are  nominated  by  the  NCN  coordinators, 
based on the recommendations of the expert group members. The external experts review 
the full proposal, which includes a 15 page project description, and then provide their own 
independent, anonymised reviews [INT04]. For the funding calls Sonatina, Sonata Bis, and 
Maestro (Table 80), the specialist evaluation is through an evaluation by external experts 
as well as an interview with the principal investigator conducted by the expert group (NCN, 
2019b).  Generally,  interviews  are  a  part  of  the  review  process  for  schemes  where  the 
researcher  is  applying  to  be  a  team  leader  or  where  the  scheme  involves  a  leadership 
element [INT04]. Finally, for the funding call Etiuda (Table 80), the specialist evaluation is 
through an interview with the principal investigator only. The three methods of specialist 
evaluation  described  above  apply  to  all  domestic  calls  except  for  Miniatura  (Table  80), 
which  has  a  simplified  one-stage  review  process  where  the  proposal  gets  sent  to  three 
reviewers.  Miniatura  is  a  relatively  new  call,  providing  an  opportunity  for  funding  for  a 
single research task to support researchers, particularly those who may not have a large 
track  record  [INT04].  In  addition,  proposals  submitted  in  the  Tango  funding  call  are 
evaluated by the Narodowe Centrum Badań i Rozwoju (National Centre for Research and 
Development),  as  this  is  a  joint  call  organiser89.  Proposals  in  international  calls  are 
assessed  based  on  separate  principles  described  in  the  specific  call  documents  (NCN, 
2020). 

After the second stage, applicants whose applications are rejected receive the individual 
reviews,  as  well  as  a  consensus  report  compiling  feedback  [INT04].  The  interviewee 
reflected that the feedback to the applicant provided transparency on the decision made 
and  provided  constructive  criticism  on  the  application,  giving  the  applicant  a  chance  to 
improve if they wished to reapply at a later stage [INT04]. In addition, they stated that 
good feedback should consider all aspects of the proposal, as well as pointing out the main 
weaknesses and potential ways in which the applicant could improve [INT04]. One way in 
which  the  NCN  strives  to  enable  good  feedback  is  to  provide  reviewers  with  a  good 
assessment and evaluation form, which leads the reviewer through the feedback process 

89 This is a call for research projects enabling basic research results to be turned into economic and 
social practice, jointly organised by the National Science Centre (NSN) and the National Centre for 
Research and Development (NCBR). 

229 

 
 
 
 
                                                 
[INT04]90. In addition, more generally, expert reviewers can find guidance online, and the 
NCN offer briefings prior to the initial expert group meeting, as well as providing guidance 
via email on the process as a whole and on the expectations [INT04]. 

Generally, the proposal evaluation criteria are the same across the different domestic calls; 
however, the weighting of the criteria may differ across the different types of call [INT04]. 
Table 81 provides information on the weighting across three of the domestic calls. 

Table 81: Assessment criteria and weighting for three domestic NCN funding calls 

Funding call 

Overarching criterion 

Weighting 

OPUS 

A. Project Assessment 

A1.  Scientific  quality  of  research 
project 

A2.  Potential  impact  of  research 
project 

A3. Feasibility of research project 

B.  Scientific  qualifications  of  the 
principal investigator 

B1. Scientific achievements of the 
principal investigator 

Principal 

B2. 
investigator’s 
involvement  in  research  projects 
not  co-financed  from  the  Polish 
budget,  selected 
in  EU  R&I 
programmes 

Preludium Bis 

A. Project assessment 

A1.  Scientific  quality  of  research 
project 

A2.  Potential  impact  of  research 
project 

A3. Feasibility of research project 

B.  Scientific  qualifications  of  the 
principal investigator 

B1. Scientific achievements of the 
principal investigator 

Principal 

investigator’s 
B2. 
involvement  in  research  projects 
not  co-financed  from  the  Polish 

55% 

25% 

15% 

15% 

45% 

35% 

10% 

35% 

15% 

10% 

10% 

65% 

55% 

10% 

90 The evaluation form is attached to each funding call documentation. An example of the form for 
the Opus call can be found on the NCN website. 

230 

 
 
                                                 
budget,  selected 
programmes 

in  EU  R&I 

Sonata 

A. Project assessment 

A1.  Scientific  quality  of  research 
project 

A2.  Potential  impact  of  research 
project 

A3. Feasibility of research project 

B.  Scientific  qualifications  of  the 
principal investigator 

60% 

30% 

15% 

15% 

40% 

Before  COVID-19,  proposals  were  submitted  electronically,  and  meetings  were  held  in 
person  [INT04].  Since  COVID-19,  everything  is  now  done  virtually  [INT04].  The 
interviewee  reflected that  the NCN are now  considering moving to a hybrid model (post 
COVID-19), which would involve the first panel meeting being conducted face-to-face, with 
the  second  panel  meeting  held  remotely  to  enable  the  benefits  of  in-person  meetings 
[INT04]. 

The average time to grant is approximately 6 months [INT04]. The interviewee reflected 
that the NCN try to balance the quality of the review process against the time it takes, with 
reviewers  not  given  more  than  15  proposals  each  [INT04].  In  2019,  there  were  1 648 
expert group members appointed to review a total number of 10 771 proposals at the first 
stage of evaluation. In addition, 8 112 external experts participated in the second stage of 
the  evaluation,  delivering  9 886  reviews.  The  NCN  provide  renumeration  ‘payment’  for 
experts.  For  the  first  stage  of  the  review,  members  of  the  expert  review  group  ‘expert 
team’  receive  renumeration  for  each  review  undertaken  in  the  first  stage  and  for  their 
presence in the panel review session. In the second stage, the external reviewers receive 
renumeration for each review undertaken [INT04]. 

5.3  Monitoring, evaluation and learning 

The  NCN  have  not  conducted  a  formal  review  of  their  proposal  evaluation  processes 
[INT04]. However, the interviewee stated that the NCN constantly reflect on what they do 
[INT04]. 

The NCN monitor gender, age, and geographical dispersion of applicants, and information 
on this is provided in their annual report [INT04]. For example, the NCN have several calls 
dedicated  to  early-career  researchers;  within  the  domestic  calls  that  closed  (for 
application) in 2019, 60% of the proposals recommended for funding involved a principal 
investigator  who  was  an  early-career  researcher  (NCN,  2019a).  In  terms  of  gender 
balance,  comparison  of  proposals  in  2019  found  that  female  applicants  submitted 
approximately 51% of the proposals (5 270 out of 10 391) and that approximately 47% of 
the  successful  proposals  were  submitted  by  female  applicants  (1 087  out  of  2 305), 
resulting in a success rate of approximately 21% for female applicants and 24% for male 
applicants. The NCN encourage a greater gender balance for applicants by increasing the 
publication time included in applications: female applicants are able to add 18 months to 
their publication time for each child born or adopted (in general, applicants are asked to 
provide a list of publications from the  past 10 years) [INT04]. Men are also able to add 
publication time if paternal leave was taken. The NCN also monitor the gender balance of 
reviewers.  Between  2011  and  2020,  the  NCN  had  approximately  29.5%  female  expert 
panel members compared with 70.43% male expert panel members [INT04]. 

231 

 
 
 
The review is single-blind in order to support independent proposal evaluation across all 
stages of the proposal evaluation process for domestic calls; the identities of the experts 
and external reviewers are not disclosed to the applicants [INT04]. In addition, the NCN 
coordinator role is one way in which the NCN attempt to minimise and eliminate bias within 
the proposal evaluation process [INT04]. Coordinators are able to un-invite reviewers who 
do not adhere to NCN standards [INT04]. 

Since  their  establishment,  the  NCN  have  undergone  change  through  the  introduction  of 
new calls, such as the introduction of Miniatura to provide the opportunity to obtain funding 
for a single research task. This has been well received by the research community [INT04]. 
The  NCN  have  also  closed  certain  calls,  such  as  Harmonia,  whereby  the  call  became 
redundant due to the presence of other international cooperation calls [INT04]. In addition, 
there had been changes to the evaluation criteria for certain calls. For example, in the 20th 
edition of the Opus funding call, there was a bonus criterion to encourage applicants to be 
more active in other projects not co-financed from the Polish budget (i.e. ERC,  Seventh 
Framework Programme, Horizon 2020 and Horizon Europe) [INT04]. 

5.4  Reflections and lessons learnt 

The NCN offer several interesting aspects for consideration to other funding agencies and 
potential points of learning for Horizon Europe. 

One such aspect is the use of targeted grants for discrete research tasks. Miniatura is an 
example  of  this,  providing  small  amounts  of  funding  for  activities  that  serve  as  part  of 
broader basic research. Within the NCN, Minatura is targeted at researchers who have not 
been NCN grantees previously. These smaller pots of funding enable more projects to be 
funded and are less of a commitment for the funder, as both the budget and timescale of 
the projects are shorter. In addition, where funders are particularly interested in funding 
innovative proposals, this type of small grant could enable proof-of-concept-type activities, 
which pose a higher risk for funders and applicants. 

The NCN assembles their expert review panels after receiving proposals. This ensures the 
expertise of the panel matches the proposals submitted, enabling a more optimised review 
process and potentially increasing the quality of the reviews. Although this may have the 
potential to increase the elapsed time from the submission of proposals to the awarding of 
grants,  the  NCN  seem  to  have  balanced  this  with  fairly  short  time-to-grant  times,  of 
approximately 6 months. 

Finally, the NCN provide detailed feedback on all aspects of the proposal to the applicants, 
including weaknesses and recommendations for improvements. This enables applicants to 
learn and benefit from the process, and it supports applicants to submit better proposals 
in the future. 

232 

 
 
6 

Swiss National Science Foundation 

6.1 

Introduction 

The  Swiss  National  Science  Foundation  is  Switzerland’s  national  science  research 
organisation,  funding  across  all  academic  disciplines,  from  history  to  medicine  to 
engineering  sciences  (SNSF,  n.d.).  The  SNSF  has  a  particular  focus  on  supporting  and 
funding basic research, having identified this as their ‘core mission’, and this focus is even 
listed in their funding regulations as a priority (SNSF, n.d.). However, the SNSF also funds 
major  mission-led  programmes,  as  outlined  below,  as  well  as  research  that  has  an 
anticipated practical impact (SNSF, n.d.). The SNSF does not fund applied research with 
an immediate commercial output. 

In regard to proposal evaluation procedures, the SNSF’s funding portfolio can be divided 
into four separate streams. 

  Project funding. This is SNSF’s ‘main funding scheme’ and supports researchers to 
conduct research projects with ‘topics and goals of their (researchers’) own choice’ 
(SNSF,  n.d.).  Funding  is  available  for  1  to  4  years,  with  grants  available  for  a 
minimum of CHF 50 000 (approx. EUR 46 410). There is no maximum funding limit, 
but it generally varies from approximately CHF 1.5–2 million (approx. EUR 1.4–1.86 
million). In 2020, the success rates for project funding was 34% (SNSF, n.d.). 

  Careers.  This  funds  the  salaries  and  living  costs  of  early-career  researchers, 
primarily at the post-doctorate level. Funding amounts vary depending on childcare 
needs  and  travel  costs,  but  typically  range  between  40 000  and  70 000  CHF 
(approx. EUR 37 000–64 000) (for professorships, more funding is available), and 
funding is available for 2 to 5 years, depending on the scheme (SNSF, n.d.). Success 
rates depend on the exact funding instrument; these are lower for professorships 
(12–18%), but are around 40–50% for post-doctorate schemes (SNSF, n.d.). 

  National Research Programmes (NRP). This funds large, interdisciplinary research 
projects  that  are  solution  oriented  and  aim  to  solve  ‘Switzerland’s  most  pressing 
problems’ (SNSF, n.d.). Examples of current NRPs include ‘COVID-19’ and ‘Digital 
transformation’. Funding is available of CHF 10–20 million (approx. EUR 9.3–18.5 
million) and lasts between 6 and 7 years. The success rates depends on the exact 
NRP, but they are generally more competitive than other SNSF grants, with success 
rates varying from approximately 20% to 30% (Kolarz et al., 2018). 

  National  Centres  of  Competence  in  Research  (NCCR).  These  grants  are  aimed  at 
established researchers and funding collaborative research on a ‘theme of strategic 
importance’  that  will  enhance  the  Swiss  research  landscape  (SNSF,  n.d.).  SNSF 
provides CHF 3–5 million (approx. EUR 2.8–4.6 million) of funding to NCCRs each 
year, with a maximum duration of 12 years. For the most recent NCCR call, outline 
proposals had a success rates of 11%, while for those applications reaching the full 
proposal review stage, the success rate was 26% (Langfeldt and Borlaug, 2016). 

6.2  Proposal evaluation processes 

Proposal development 

First, applicants are tasked with writing a proposal, submitted as separate documents on 
SNSF’s online portal, mySNSF. Applicants write a research plan, with details on the current 
state  of  research  in  the  field;  overall  objectives  and  aims;  methods;  and  impacts.  For 
project  funding,  this  section  is limited  to  20  pages  and  80 000  characters  (SNSF,  n.d.). 
CVs are submitted as a separate document, detailing scientific qualifications and previous 
research results. For project funding, this is limited to a total of four pages. Applicants also 

233 

 
 
 
must write a data management plan for project funding and some careers funding, but this 
plan currently excluded from the evaluation process. 

Evaluation and standard decision-making processes 

After submission, proposals are first quality assured by SNSF’s administrative offices, who 
check whether formal requirements have been met and whether the proposal ‘contravenes 
the  rules  of  scientific  integrity’.  The  administrative  offices  may  contact  applicants  for 
clarification, but if the above requirements have not been met, the application will not be 
considered further. If formal requirements have been met, the proposal is passed on for 
peer review. As can be seen from Table 82, several groups and individuals play a key role 
in evaluating proposals and the decision-making processes (SNSF, n.d.). 

  The  internal  referees  are,  in  the  main,  drawn  from  the  Research  Council,  itself 
composed of 100 researchers, who have oversight of the entire evaluation process. 
Referees  are  chosen  by  administrative  offices  on  the  basis  on  their  scientific 
expertise and, to a lesser extent, their workload and capacity to review proposals. 

  The  Research  Council  is  supported  by  evaluation  bodies,  which  are  organised  by 
discipline and by call type. There are 90 such bodies, with a combined total of more 
than 700 members, a third of whom are based at non-Swiss institutions. 

  The Specialised Committee Careers is chaired by a member of the Research Council 
(who is also part of the SNSF Presiding Board)  and has  specific responsibility for 
the evaluation of career proposals and the managing of the long-term strategy of 
careers funding. 

  External  referees  are  independent  of  SNSF  and  generally  come  from  outside 
Switzerland.  They  are  chosen  by  internal  referees  with  assistance  from  the 
administrative  offices.  In  addition  to  this,  applicants  are  entitled  to  provide  a 
‘negative  list’  of  persons  who  should  not  be  asked  for  review,  which  the 
administrative officer can consider (SNSF, n.d.). 

The  standard  peer  review  processes  for  each  of  SNSF’s  funding  streams  is  highlighted 
below in Table 82. Particular schemes may have slight differences. 

234 

 
 
 
 
 
 
 
 
 
 
 
 
 
Table 82: Overview of proposal evaluation processes by SNSF grant type 

Funding streams  

Summary of assessment process 

Project Funding 

 
 

External review by at minimum of two assessors 
Internal  review  by  a  referee  and  co-referee  from 
the SNSF Research Council 

Careers 

One-stage 
 

Internal  review  by  a  referee  and  co-referee  from 
the  funding  scheme–specific  panel  set  up  by  the 
Specialised Committee Career 

  Quality 

external 

assurance 

reviewers’ 
of 
assessments, and assess applications in relation to 
others,  providing  a  comparative  rating  score 
between 1 (‘poor’) to 6 (‘outstanding’) 
Evaluation  bodies  approve  or  reject  applications 
based  on  these  reviews  and  provide  provisional 
funding recommendations 
Approval  by  the  Presiding  Board  of  the  Research 
Council and final decision made 

  Optional external review 
 

Evaluation  bodies  provisionally  approve  or  reject 
applications based on these reviews 
Specialised  Committee  Careers  provides  a 
provisional funding recommendation 
Approval  by  the  Presiding  Board  of  the  Research 
Council and final decision made 

 

 

 

 

Two-stage 
 

Internal  review  by  a  referee  and  co-referee  from 
the  funding  scheme–specific  panel  set  up  by  the 
to  create  a 
Specialised  Committee  Career 
candidate shortlist 
  Optional external review 
 

Evaluation bodies discuss and further shortlist the 
most promising candidates depending on available 
budget 
The  Specialised  Committee  Careers  make  a  final 
decision on interviews 

 

National Research Programmes (NRPs) 

  Mandatory  external  review,  with  the  exception  of 

 
 

 

 

 
 

 

 
 

 

their humanities doctorate programme 
Interview with the relevant evaluation body 
Evaluation  bodies  provisionally  approve  or  reject 
applications based on these reviews 
Specialised  Committee  Careers  provides  a 
provisional funding recommendation 
Approval  by  the  Presiding  Board  of  the  Research 
Council and final decision made 

Submission of brief project outline 
Internal  review  by  a  referee  from  the  SNSF 
Research Council 
If  approved,  applicants  go  on  to  submit  a  full 
application 
External review 
Evaluation  bodies  approve  or  reject  applications 
based on these reviews and split available budget 
across  applications;  provide  a  provisional  funding 
recommendation 
Approval  by  the  Presiding  Board  of  the  Research 
Council and final decision made 

National  Centres  of  Competence  in  Research 
(NCCRs) 

 

Submission of letter of intent 

235 

 
 
 
 

 
 
 
 

Submission of outline proposal, detailing the vision 
of the proposed NCCR and main research questions 
External review 
Submission of full proposal 
External Review 
5. Approval by the Presiding Board of the Research 
Council and final decision made 

Principles and assessment criteria 

Regardless  of  the  funding  stream  or  project,  the  SNSF  apply  a  basic  set  of  assessment 
criteria to applicants themselves and to their proposed projects. Applicants are assessed 
against  their  scientific  track  record  and  expertise  in  regard  to  the  proposed  project, 
education,  and  previous  scientific  achievements  (SNSF,  n.d.).  For  some  careers-funding 
schemes, teaching activities and location may also be considered (SNSF, n.d.). Proposed 
projects  are  assessed  against  their  scientific  relevance,  originality  and  topicality;  the 
suitability of the proposed methods; and, for ‘use-inspired’ projects (meaning those with a 
practical dimension), the potential broader impact. For NRPs, projects are assessed against 
their scientific quality and how they fit against programme objectives (SNSF, n.d.). 

The  SNSF  also  have  a  core  set  of  funding  principles  that  apply  to  all  their  proposal 
evaluations and decision making (SNSF, n.d.). 

  Excellence  through  competition.  The  SNSF  seek  to  fund  high-quality  science  that 
will bring ‘added value for a scientific or academic career’ and  to those proposals 
that make a ‘better impression’ than their direct rivals. 

  Fairness and equal opportunities. The SNSF treat applicants equally regardless of 
‘personal  features’  and  set  ‘attainable’  criteria.  For  careers-funding  grants,  this 
might mean reducing the required work-time percentages, usually 100%, in case 
of family-care duties. 

  Transparency. The SNSF make decisions on ‘clearly defined procedures and rules’ 

that are publicly available. 

 

Integrity.  The  SNSF  ‘respects  national  and  international  ethical  standards  in  its 
decision-making’. 

  Confidentiality. All applicant data, information and documents submitted to SNSF 

are confidential. 

Funding-by-lottery pilot 

Although  the  SNSF  has  typically  relied  upon  peer  review  for  funding  allocation,  it  has 
recently sought to experiment with a random selection element. In 2019, they introduced 
a  lottery  component  for  one  of  their  post-doctorate  programmes,  Postdoc.Mobility 
(University of Groningen, 2020). These grants are 1 to 2 years long and fund post-doctoral 
researchers to complete a research stay abroad. They were selected for the trial because, 
following the simplification of their portfolio, the SNSF anticipated an approximately three-
fold increase in the number of applications to the programme and knew some adaptations 
would in any case be necessary. The SNSF were also aware of what they perceived as the 
‘limitations’ of the peer review system: its bias against innovative or ‘risky’ research; poor 
predictability of future research performance; and limited consistency between experts in 
their  appraisals  (Egger,  n.d.).  Therefore,  the  SNSF  used  the  need  to  reform 
Postdoc.Mobility Grants to try an alternative; they introduced a lottery component for those 
applicants  who  were  equally  qualified,  had  received  a  very  similar  average  score  from 
reviewers, and were at the threshold of funding (University of Groningen, 2020). In these 
cases, panels had the option to incorporate a lottery and draw lots on those they would 

236 

 
 
 
actually  fund.  Figure  88  shows  how  SNSF  reviewers  incorporated  a  lottery  into  their 
decision making (the remote evaluation was used only for simulation purposes). 

Figure 88: Decision making in the Postdoc.Mobility pilot (Egger, n.d.) 

Of the 134 applications considered, 8 (6%) proposals were decided by lottery (Bieri et al., 
2020).  Although  applicants  had  lots  of  technical  questions  around  the  randomisation 
process, no objections were made, and the funding decisions were not contested [INT06]. 
Similarly, despite some initial concerns around the potential to cause reputational damage 
and  undermine  the  standard  peer  review  process,  reviewers  ultimately  accepted  the 
lottery.  The  SNSF  Presiding  Board  were  also  impressed  by  the  results  of  the 
Postdoc.Mobility  pilot,  and  they  have  since  decided  to  introduce  the  option  of  a  random 
selection for all funding schemes, as it is now anchored in the Organisational Regulations 
of the National Research Council [INT06]. 

‘Triage’ experiment 

As  part  of  the  Postdoc.Mobility  pilot,  SNSF  also  trialled  a  ‘triage’  approach  to  proposal 
evaluation. In a first step, each proposal was independently reviewed and scored by two 
panel members using a 6-point scale: outstanding = 6 points, excellent = 5 points, very 
good = 4  points,  good = 3  points,  mediocre = 2  points,  poor = 1  point.  Following  this, 
applications were then allocated to three groups based on the ranking of the mean scores 
given to each proposal: fund without further discussion (F); discuss in panel meeting (D); 
and  reject  (R),  as  highlighted  in  Figure  88.  The  D  proposals  were  then  discussed  in  the 
panel meeting (although all panel members were free to request that applications in the F 
or R group be reallocated to D) and discussed to be ranked and either funded or rejected. 
In focusing on a subsection of proposals this way, the panel discussion could be focused 
on borderline proposals that ‘needed discussion’, as opposed to clearly excellent or clearly 
uncompetitive proposals. As a result, an interviewee claimed, the SNSF ‘could cut down a 
6-hour meeting to a 3 to 4-hour meeting’ [INT06]. Following the Postdoc.Mobility pilot, the 
triage  system  has  been  used  in  the  Sinergia  and  Agora  research  grants  [INT06].  As  a 
further  trial,  the  SNSF  ran  a  simulated,  hypothetical  evaluation  with  no  triage  or  panel 
meeting stages, but only individual assessor scores (shown on the right-hand side of Figure 
88). In 80.6% of cases, this simplified procedure aligned with the final funding outcome; 

237 

 
 
 
 
 
it thus could have cut evaluation costs in half and could have saved 31 hours of meeting 
time (Bieri et al., 2020). 

Double-blind Pilot 

To reduce bias in the peer review process, SNSF is also trialling a double-blind evaluation 
process, in a new project funding scheme, known as Spark (SNSF, n.d.). The scheme aims 
to fund ‘unconventional ideas’ that follow a ‘unique approach’. Although formal conclusions 
cannot yet be drawn – SNSF are carrying out a formal evaluation of Spark in 2021 – they 
claim  that  it  is  already  ‘contributing  to  innovation  and  diversity’.  However,  ‘insufficient 
anonymisation’ has been an issue for Spark, as some  applicant details were revealed in 
the  ‘properties  of  the  project  description  PDF  file’.  In  one  call  for  funding,  30%  of 
applications were found to be insufficiently anonymised for this reason, and so SNSF were 
forced to cancel the call entirely, inviting applicants to submit applications in a follow-up 
call, with full anonymisation. 

6.3 

 Monitoring and evaluation 

SNSF  have  commissioned  evaluations  of  many  of  their  funding  activities,  as  well  as 
completing  internal  studies  on  their  proposal  evaluation  processes.  They  have  also 
demonstrated  a  clear  commitment  to  learning  from  monitoring  and  evaluation,  learning 
from evidence and often adapting their processes appropriately. 

Holistic review of the proposal evaluation process at SNSF 

In 2012, SNSF commissioned an evaluation of the transparency and overall quality of their 
proposal evaluation system. The evaluation, conducted by Western Michigan University's 
Evaluation Center, found that while SNSF is meeting its objectives, improvements could 
be made. The evaluation listed eight recommendations, the most pertinent of which are 
listed below, together with SNSF’s response (Coryn et al., 2012). 

  The  evaluation  identified  a  clear  need  for  SNSF  to  be  more  ‘transparent  and 
understandable’, as they found that ‘almost two thirds of applicants indicated that 
they  are  ‘very  unclear’  or  ‘unclear’  as  to  how  ‘funding  decisions  are  made’.  In 
response  to  this,  SNSF  created  a  renewed  website  with  more  ‘targeted 
communication with researchers’ and better access to applicant guidelines. Indeed, 
their  proposal  evaluation  processes  are  now  clearly  detailed,  and  they  publicly 
release strategic thinking and rationale around any adaptations. 

  The evaluation also found that peer  review processes placed a significant burden 
on  reviewers  from  the  SNSF  Research  Council.  Sourcing  external  reviewers  was 
found  to  be  a  particularly  time-consuming  process.  To  reduce  the  burden,  SNSF 
scientific officers at the administrative offices now provide an initial list of reviewers 
to the evaluation bodies, who make the final decision. 

  The evaluation recommended that the SNSF conduct regular, systematic reviews of 
their  funding  schemes  and  ensure  that  funding  is  not  ‘misaligned  with  their 
intentions’. SNSF has heeded this call, and has since commissioned reviews of the 
NRP funding scheme (Kolarz et al., 2018); a survey of Swiss researchers’ view on 
SNSF  project  funding  (Langfeldt,  Ramberg,  and  Gunnes,  2014);  as  well  as  a 
dedicated evaluation of the NCCR selection scheme. The latter evaluation found that 
NCCR’s selection processes had, overall, been a success, but that there was scope 
for improvement. In particular, they recommended that they increase the number 
of  reviewers  and  allow  rebuttals  from  applicants.  The  SNSF  implemented  these 
recommendations in 2016 (SNSF, n.d.). 

238 

 
 
 
 
Learnings from the Postdoc.Mobility lottery pilot 

Although the pilot was largely deemed a success by SNSF as they aim to expand the use 
of lotteries in research funding, valuable lessons were learnt from the adapted processes. 
First, effective communication of the rationale for randomisation was seen as essential by 
at least one  scientific officer at the SNSF  (University of  Groningen, 2020). As this novel 
aspect  of  the  evaluation  was  clearly  communicated  to  applicants  in  the  call  description, 
‘the (funding) decision was not contested’ by applicants, even if they were rejected through 
randomisation. In regard to improving the process, the SNSF seek to improve the selection 
process for randomisation, because  they struggled to find ‘objective criteria’ and ‘it is not 
easy to do randomisation based on [reviewer] score’ [INT06]. In particular, because of the 
limited numbers of reviewers and panel members, it is difficult to statistically determine 
whether two scores above and below the funding threshold are different. In an attempt to 
resolve  this,  the  SNSF  have  experimented  with  a  Bayesian  hierarchical  model  based  on 
expected  ranks,  which  incorporate  the  magnitude  and  the  uncertainty  of  the  estimated 
difference between proposals (Heyard et al., 2021). 

Biases in peer review 

In 2017, SNSF staff and partner academics completed an analysis of 38 250 external peer 
review reports to examine whether gender of applicants and other factors influenced the 
peer review of grants (Severin et al., 2020). The reports covered grant applications across 
multiple disciplines, from medicine to architecture, and covered reports written between 
2009 and 2016. The authors concluded that SNSF ‘may be prone to biases stemming from 
different applicant and reviewer characteristics’. In particular, they found a gender bias: 
male  reviewers  systematically  tended  to  rate  male  applicants  higher  than  female 
applicants.  Interestingly,  the  authors  found  that  peer  reviewers  selected  by  applicants 
consistently  gave  a  higher  rating  than  those  selected  by  SNSF  evaluation  bodies.  As  a 
result of this finding, the SNSF abandoned the nomination of peer reviewers by applicants, 
permitting  only  the  ‘negative  list’  mentioned  in  the  section  titled  ‘Proposal  evaluation 
processes’. 

6.4  Lessons learnt 

The  SNSF  can  provide  many  relevant  lessons  to  various  programmes  under  H2020, 
especially  as  many  similarities  between  them  are  apparent:  both  are  transparent, 
publishing  much  of  their  internal  documents  online;  both  have  a  clear  commitment  to 
monitoring  and  evaluation;  and,  at  a  basic  level,  both  have  similar  proposal  evaluation 
processes,  especially  for  the  project  and  career  streams  of  funding.  However,  the  SNSF 
has recently explored adaptations to traditional peer review processes. 

The SNSF is one of the first few science funding bodies to experiment with randomisation 
in their proposal evaluation, introducing a pilot lottery in 2019 for one scheme, and it is an 
aspect of evaluation that they seek to extend. Although the pilot was deemed successful, 
interestingly,  it  did  not  do  much  to  alleviate  the  burden  of  peer  review  (University  of 
Groningen, 2020). Reviewers still had to screen and quality assure applications, and only 
very few applications actually went through the lottery. A scientific officer of the SNSF even 
said that ‘we do not want to give the impression this (randomisation) makes our life easier’ 
and noted the scope for further efficiencies. Interestingly, this scientific officer claimed that 
extending  randomisation  would  be  more  ‘politically  acceptable’  for  high-risk  innovation 
projects,  where  failure  is  an  accepted  possibility,  than  for  fellowships,  where  you  are 
funding people. This could be of note to the European Innovation Council (EIC) pilot if it 
sought to experiment with randomisation in its funding decisions. 

In  addition  to  this,  SNSF  is  also  trialling  a  double-blind  evaluation  on  Spark,  a  new 
programme and approach for them. Although the full evaluation of this trial has not yet 
been completed, it is already clear that ensuring anonymity is difficult to achieve in practice 
for this scheme. SNSF decided to re-issue a call on Spark due to insufficient anonymisation 
largely because of metadata associated with the project description containing references 
to  the  identity  of  the  applicant  (SNSF,  n.d.).  As  double-blind  evaluations  will  be  re-

239 

 
 
introduced as a pilot in Horizon Europe, it is worth reflecting on and learning from SNSF’s 
early experiences. 

240 

 
 
7 

United Kingdom Research and Innovation 

7.1 

Introduction 

UKRI  is  the  UK’s  national  funding  agency  investing  in  science  and  research,  and  is 
sponsored by the Department for Business, Energy and Industrial Strategy (known by the 
acronym  BEIS)  (UKRI,  n.d.).  Launched  in  April  2018,  the  organisation  brought  together 
seven previously discrete disciplinary research councils91, as well as Research England (a 
public  body  responsible  for  supporting  research  and  knowledge  exchange  at  UK  higher 
education institutions in England) and Innovate UK (the UK’s dedicated innovation agency). 
UKRI have a wide variety of funding instruments and streams, totalling almost GBP 8 billion 
in 2019, as  shown in Figure 89 (UKRI, 2020a). UKRI’s investments are  classed into  the 
following five discrete categories (UKRI, 2020b): 

 

Ideas – which supports research council programmes and funding for research at 
universities, 

 

Infrastructure – ranging from supercomputers to biomedical labs, 

  Business Environment and Place – supporting innovation through the private sector 

and to address industrial and societal challenges, 

 

International – developing R&I collaborations abroad, 

  People – including studentship and fellowship funding. 

91  The  seven  councils  are:  Arts  and  Humanities  Research  Council  (AHRC);  Biotechnology  and 
Biological  Sciences  Research  Council;  Economic  and  Social  Research  Council  (ESRC);  Engineering 
and  Physical  Sciences  Research  Council  (EPSRC);  Medical  Research  Council  (MRC);  Natural 
Environment Research Council (NERC); Science and Technology Facilities Council (STFC).  

241 

 
 
                                                 
Figure 89: UKRI councils’ investment, by foundation, in 2019/2020 (UKRI 2020a) 

As a relatively new agency (less than 3 years old at the time of writing) that brings together 
funding  activities  from  multiple  research  and  innovation  funders  with  well-established 
processes,  UKRI  does  not  currently  have  a  standard  proposal  evaluation  process  across 
the constituent organisations. Taking this into consideration, as well as the lack of available 
evidence on innovation-specific funding from the other case study comparators, this case 
study  will  focus  on  Innovate  UK.  However,  reference  to  broader  UKRI  processes  will  be 
made throughout the case study. 

The  aim  of  Innovate  UK  is  to  ‘de-risk,  enable  and  support  innovation’  for  businesses, 
particularly micro-businesses and SMEs, to develop commercially successful products and 
foster economic growth. As can be seen in Figure 89Figure 89, the majority of Innovate 
UK’s  funding  falls  under  the  Business  Environment  and  Place  funding  stream  of  UKRI 
(Innovate  UK,  n.d.).  Such  is  the  focus  on  business-led  innovation  that,  for  many  calls, 
academic  institutions  cannot  lead  or  work  alone,  and  must  partner  with  an  SME.  In 
2019/2020, Innovate UK had a budget of GBP 900 million, delivered through a wide variety 
of funding instruments, such as the cross-UKRI Industrial Strategy Challenge Fund (ISCF), 
UKRI’s  ‘flagship’  challenge-led  innovation  programme,  which  supports  industry-led, 

242 

 
 
 
 
mission-oriented R&D focused on the UK’s Grand Challenges92, and the more open-ended 
Smart Grants, which fund ‘commercially viable or innovative ideas’ that need not fall under 
these Grand Challenges (Innovate UK, n.d.). Although success rates for Innovate UK calls 
are  not  centrally  compiled93,  applicants  can  have  access  to  information  on  specific  calls 
through a freedom-of-information request. 

7.2  Proposal evaluation processes 

UKRI 

Although  exact  processes  vary  across  the  constituent  organisations  of  UKRI,  the 
organisation  has  publicly  outlined  broad  principles  of  proposal  assessment  and  decision 
making that apply across the board (UKRI, 2020c). 

First,  applicants  are  tasked  with  writing  a  proposal,  which  is  assessed  either  via 
independent review or via collective panel assessment. The reviews are typically experts 
or respected peers from industry, academia or other sectors, such as  the charity sector. 
They anonymously assess the merit of proposals against published criteria, which vary by 
programme and institution but typically focus on excellence and, if applicable, impact. In 
a step designed ‘to reduce bias’, reviewers do not consider journal-impact factors and other 
journal-based metrics ‘as a surrogate measure of the quality of individual research articles’ 
when reviewing an individual researcher’s contributions or making funding decisions (UKRI, 
2020c). 

In  the  case  of  independent  review,  reviewers  judge  proposals  against  the  assessment 
criteria  alone  and  do  not  consider  the  strength  of  other  proposals.  In  the  case  of  panel 
review, ‘benchmarking’ does occur, as assessors are tasked with judging the applications 
collectively with other panellists and developing a consensus (UKRI, 2020c). In many calls, 
especially for applications to the research councils, a two-stage evaluation process takes 
place, where independent review is followed by panel review. In all cases, the assessment 
of  applications  is  separated  from  final  funding  decisions,  which  are  ultimately  made  by 
senior staff within UKRI constituent organisations. 

Innovate UK 

The broad principles and features of UKRI’s funding decision making apply to the proposal 
evaluation process and decision making at Innovate UK, albeit with specific adaptations. 

  Applicants submit an application in one stage, comprising a full proposal organised 

into 10 equally weighted sections, plus appendices (Innovate UK, n.d.). 

  Business opportunity. Applicants outline the business need and customer demand 

for their research idea or innovation and explain its commercial outcomes. 

  The market. Applicants explain the size and dynamics of the relevant market and 

what share they could reasonably capture with the proposed innovation. 

  Exploitation.  Applicants  outline  the  business  plan  and  how  outputs  will  be 

commercialised. 

  Benefits. Applicants explain how the project will foster wider economic, social and 
environmental impacts in the UK beyond any immediate commercial benefits. 

92  These  are  a)  Artificial  intelligence  and  data,  b)  Ageing  society,  c)  Clean  growth,  d)  Future  of 
mobility.  

93 They appear to have last done this in 2014/2015. 

243 

 
 
                                                 
  Project plan. Applicants detail the project timelines and outline key milestones. 

 

Innovation. Applicants explain the novelty of their proposed project and how it will 
drive improvements in multiple different applications. 

  Risks.  Applicants  outline  the  risks  involved  with  their  proposed  project,  from 

commercial to environmental, and how they will be mitigated. 

  Skills/project  consortia.  Applicants  detail  how  the  proposed  team  has  the  skills, 

experience, resources and facilities to exploit the results of the innovation. 

  Finances. Applicants outline all project costs in this section. 

  Additionality. Applicants outline why public finance is required, and why internal or 

commercial finance is not available. 

  Appendices. Applicants typically submit graphic summaries, such as a Gantt chart, 
or organograms in this section. These have page limits (typically between two to 
four pages) and are mandatory for certain sections only, depending on the call. 

Submissions  are  then  assessed  for  eligibility,  scope,  and  ‘guidance  of  the  funding 
opportunity’,  such  as  whether  the  competition  is  arranged  in  phases  (for  example,  a 
‘technical feasibility’ phase followed by a ‘prototype development and evaluation’ phase) 
(Innovate UK, n.d.). However, an interviewee has said that Innovate UK ‘tend not to throw 
things out’ and that assessing scope can be challenging, as many funding competitions are 
open and non-prescriptive in terms of scope [INT05]. Those proposals considered eligible 
are  passed  on  for  assessment.  For  individual  review,  applications  are  assessed  by  a 
minimum of three reviewers, although in practice, typically five reviewers assess proposals. 
Reviewers are experts from both industry and academia and are typically recruited through 
Innovate UK’s professional network [INT05]. However, a minority of reviewers are selected 
through  competitive  application  for  entry  into  a  pool  of  reviewers.  The  exact  selection 
process varies by programme and funding stream, but for one scheme on offshore wind 
energy,  reviewers  required  5  years’  technical  experience  (30%);  demonstratable 
experience  in  the  offshore  wind  sector  (20%);  5  years’  experience  in  another  relevant 
sector (20%); previous review experience (20%); and an understanding of barriers to the 
UK supply chain (10%) (Offshore Wind Growth Partnership, 2019). Potential reviewers are 
given a score between 0 and 8, and those who have received a score of less than 4 are 
rejected. 

Once  the  experts  have  been  selected  as  a  reviewer,  they  are  drawn  from  this  pool  to 
evaluate proposals depending on their expertise and conflicts of interest. For each of the 
10  sections  detailed  above,  reviewers  provide  written  comments  along  with  a  score 
between  0  and  10.  Reviewers  are  provided  with  training,  scoring  criteria  and  reviewer 
briefings prior to their evaluation, but this information is not publicly available. 

Following the assessors’ assessment, assessor scores are averaged and the proposals are 
compiled in a ranked order into a report called the panel sheet. This sheet is moderated 
and reviewed by an assessment moderation panel, who discuss applications that require 
clarification  around  scope,  quality,  feasibility  and  fundability.  They  also  check  for  any 
outlying assessor scores, which may be removed if ‘their assessment is not robust’ [INT05]. 
After this moderation, the panel recommend a ranked list of applications to be funded by 
Innovate UK. To receive funding, proposals must exceed a minimum threshold on the total 
average score of reviewers. Although this varies from call to call, the threshold is typically 
between 70 and 80. 

This recommended list of proposals is then passed to the Innovate UK Funder’s Panel for 
final approval for funding. The Funder’s Panel is composed of senior leadership at Innovate 
UK,  operation  staff  and  industry  experts  [INT05].  Although  Innovate  UK  ‘engages 
assessors  to  act  on  its  behalf’,  the  Funder’s  Panel  ultimately  make  the  final  decision  on 
whether  a  proposal  is  in  scope,  so  that  it  fits  with  the  specific  themes  and  research 

244 

 
 
categories  of  the  call,  and  they  may  also  make  modifications  to  ensure  that  funding  is 
allocated evenly across strategic areas and, more rarely, regions of the UK [INT05]. 

The above describes the basic evaluation procedure at Innovate UK, but certain funding 
competitions, such as those under ISCF, have exceptions. For example, some competitions 
include interview panels, and some have a two-stage application, where applicants write a 
brief  expression  of  interest  before  being  invited  to  submit  a  full  proposal,  if  selected 
(Innovate UK, n.d.). 

7.3  Monitoring, evaluation and learning 

UKRI has a strong commitment to monitoring and evaluation (M&E) of all aspects of their 
portfolio,  commissioning  holistic  process  and  impact  evaluations  of  their  major 
investments. Indeed, in their 2020/2021 corporate plan, UKRI listed developing a ‘What 
Works’  evidence  programme  by  launching  a  call  for  research-on-research-innovation 
proposals  as  a  priority  objective.  This  commitment  to  M&E  extends  to  UKRI  constituent 
organisations, and Innovate UK is no exception. 

Smart Grant evaluation 

In 2014, Innovate UK commissioned an evaluation of their Smart Grant programme, with 
a sub-focus on assessment processes. Although the evaluation found that the application 
and  assessment  process  of  Smart  Grants  was  broadly  fit  for  purpose,  challenges  were 
encountered  (SQW,  Cambridge  Econometrics,  and  BMG  Research,  2015).  First,  the 
evaluation suggested that burden was an issue for applicants, some of whom felt that the 
time input required for the application was disproportionate to the scale and nature of the 
award. In particular, applicants identified the strict, limiting character limits in application 
forms  as  a  source  of  burden.  Second,  although  applicants  generally  perceived  feedback 
positively  and  noted  that  it  helped  them  to  inform  subsequent  applications,  they 
highlighted  inconsistency  of  feedback  as  an  issue.  One  applicant  noted  that  ‘some 
examiners supported an aspect of the project  which others  criticised’, a consequence of 
having  several,  independent  reviews  as  part  of  the  assessment  process.  The  fact  that 
assessors  can  change  between  submission  and  resubmission  of  initially  unsuccessful 
proposals can also add to inconsistencies in feedback (Lawrence-Berkeley, 2017). On this 
note, one applicant said they had endeavoured to address the first round of comments and 
suggestions,  only  to  receive  ‘a  different  emphasis  on  marking’  in  feedback  following 
resubmission.  This  led  the  applicant  to  conclude  that  ‘the  goal  posts  moved’,  creating 
uncertainty around the proposal’s strengths and weaknesses. Inconsistent feedback was a 
point highlighted by both successful and unsuccessful applicants. 

The evaluation also found evidence of burden among reviewers, as some interviewed spoke 
of a burden in the review stage, especially in relation to reviewing the appendix sections, 
which were ‘more complex and time consuming’. To improve on process evaluation, the 
authors of this evaluation suggested introducing a quality-assurance phase prior to expert 
review and ensuring consistency of reviewers across submission and resubmission, among 
other  modifications.  These  modifications  were  suggested  with  the  aim  of  lessening  the 
burden on both applicants and assessors, as well as improving the consistency of feedback. 

Since this 2014 evaluation, Innovate UK have made adaptations and improvements to the 
process. In regard to reviewer burden, there is now a page limit set on appendices (varying 
from 2 to 4 pages). In regard to applicant burden, applicants need no longer adhere to a 
character  limit  for  sections;  instead,  they  need  to  adhere  to  a  word  limit.  However,  an 
interviewee thought that, more than adjusting page limits, confusion around Innovate UK’s 
role and funding scope created undue burden for applicants and reviewers. They said that 
a ‘huge number of people that apply to us […] that never stood a chance’, as their proposed 
idea ‘doesn’t need public money’ and would be better suited to other funding routes, such 
as private equity or tax credits [INT05]. 

To address the issue of inconsistencies in feedback between submission and resubmission, 
Innovate UK have clarified in their guidance to applicants that different assessors ‘will have 

245 

 
 
no prior knowledge of the original application or its feedback’ and that addressing the initial 
assessors’  feedback  was  no  guarantee  of  funding.  Innovate  UK  do  have  a  moderation 
phase, whereby outlying assessor scores are checked. In addition, on an ad hoc basis, they 
monitor  reviewers  who  consistently  give  unduly  harsh  or  positive  score.  However,  an 
interviewee has said that disagreement between reviewers ‘is the nature of the beast’ and 
that Innovate UK would not want five identical reviews from their independent reviewers.  

University of Bath evaluation 

Further to the 2014 evaluation, Innovate UK commissioned a broader review of their entire 
grant assessment process in 2020, led by researchers at the University of Bath (Salandra 
and Salter, 2019). This study is still ongoing and aims to gather insight on sources of bias 
in the review system and on the effectiveness of the assessment system as a whole. At 
the time of writing this case study, the evaluation had found that although more applicants 
are male, there is ‘no evidence that women-led applications have a lower  success  rate’. 
Similarly,  although  there  is  a  strong  concentration  of  applications  from  London  and  the 
South-East  of  England,  the  evaluation  found  no  geographical  preference  in  selection. 
However,  company  size  did  have  an  impact  on  the  selection  of  proposal.  Smaller  firms 
typically submitted more applications and had a lower success rate than other firms, which 
the authors explained through the apparently enhanced ‘administrative capabilities of large 
and medium size firms to develop applications’. For the assessors’ pool, similar imbalances 
in  gender  and  location  were  found,  as  male  and  London-based  applicants  were  more 
successful, but these  imbalances did not lead to ‘structural disadvantages for women or 
for applications from outside London’. 

This review of Innovate UK’s proposal evaluation system is ongoing, and the research will 
continue to look at the workload of assessors and the factors that shape their evaluation 
more broadly. 

Machine learning in reviewer allocation 

In  2017,  Innovate  UK  issued  the  first  phase  of  a  call  for  proposals  ‘to  develop  machine 
learning solutions that will improve Innovate UK’s operations’ (Innovation Funding Service, 
n.d.). In particular, Innovate UK sought to use machine learning to improve the efficiency 
of  reviewer  allocation,  as  this  was  a  significant  burden  for  Innovate  UK  staff  following 
simplification  of  their  portfolio.  According  to  an  interviewee,  this  change  had  ‘hugely 
increased the number of applications we would get’ and meant that staff would spend days 
allocating assessors to proposals manually. To reduce burden, Innovate UK commissioned 
a business to replace the manual process with a key-word identification system through 
machine learning. According to an interviewee, this new system is a major improvement 
and  has  ‘reduced  a  huge  burden’  on  Innovate  UK  staff,  but  is  nonetheless  imperfect; 
occasionally, assessors are inappropriately selected, as they lack the requisite skills, but 
this has only been a minority of cases. 

Internal review of assessment criteria 

To ensure that they are funding the most innovative business ideas, Innovate UK are in 
the process of internally reviewing their application processes and criteria. An interviewee 
said  that  it  was  ‘absurd’  that  Innovate  UK  did  not  apply  weighting  to  different  proposal 
sections, as equal weighting means that the ‘quality of the innovation’ is scored the same 
as  other,  arguably  less  important  sections.  From  internal  modelling  of  previous  calls  for 
proposals, the most innovative proposals tended to be ‘dragged down by criteria we don’t 
see as important’. In addition to this, they have found some initial evidence that suggests 
some questions are overlapping and that a high score on one section almost guaranteed a 
high  score  on  another  section.  In  response  to  these  internal  findings,  Innovate  UK  will 
‘develop and enhance our current assessment process to ensure we identify the best ideas 
from businesses with the ambition and potential to grow through innovation’. 

246 

 
 
 
7.4  Lessons learnt 

Given Innovate UK’s focus on funding industry-led innovations and SMEs, this case study 
can be thought of as offering direct learnings to the enhanced EIC pilot. In examining how 
Innovate UK operates within a much larger funding body (i.e. UKRI), the case study can 
be  useful  to  consider  how  the  EIC  pilot  fits  within  the  broader  EU  R&I  framework 
programme. 

Innovate UK and the EIC pilot are similar in many respects, from their total budgets (GBP 
0.9 billion and  EUR 1.3 billion, respectively, in 2019/2020), to their proposal evaluation 
processes,  to  their  structure,  in  that  they  both  sit  within  a  much  broader  R&I  funding 
programme  (UKRI  and  H2020,  respectively).  However,  there  are  illustrative  differences 
between them, and lessons can be learnt from experiences at Innovate UK. 

First, it is clear that feedback to applicants is an area of potential improvement at Innovate 
UK,  particularly  in  regard  to  consistency  of  comments.  This  could  be  a  result  of  the 
independent review process. Unlike in the EIC  pilot, at Innovate UK independent review 
occurs  without  a  consensus  phase,  so  at  no  point  are  assessor  comments  aggregated. 
However, the lack of reviewer consensus is also potentially advantageous to Innovate UK; 
an interviewee associated with the EIC reported that the consensus phase was one major 
aspect they would like to remove in Horizon Europe, as it ‘does not support disruption’ and 
is a ‘barrier to innovation’. 

Second, burden on assessors and applicants appears to be an issue for Innovate UK. For 
the assessors, reviewing lengthy, information-dense appendices was found to be the major 
source  of  burden,  while  for  applicants,  it  was  completing  shorter,  character-limited 
sections. Innovate UK have introduced changes to ameliorate burden, but it is not clear to 
what extent these have been effective. Innovate UK are examining this, however, because 
the University of Bath–led evaluation aims to access information on the workload of current 
assessors. 

247 

 
 
 
REFERENCES 

Abdoul, H., Perrey, C., Amiel, P., Tubach, F., Gottot, S., I. Durand-Zaleski, I. and Alberti, 
C. (2012), ‘Peer review of grant applications: Criteria used and qualitative study of 
reviewer practices’, PLoS ONE, Vol. 7, No 9, 
https://doi.org/10.1371/journal.pone.0046054. 

Abdoul, H., Perrey, C., Tubach, F., Amiel, P., Durand-Zaleski, I. and Alberti, C. (2012), 
‘Non-financial conflicts of interest in academic grant evaluation: A qualitative study of 
multiple stakeholders in France’, PLoS ONE, Vol. 7, No 4, e35247, 
https://doi.org/10.1371/journal.pone.0035247. 

Adam, D. (2019), ‘Science funders gamble on grant lotteries’, Nature, Vol. 575, No 7784, 
pp. 574-575, https://doi.org/10.1038/d41586-019-03572-7. 

Alberts, B. (2009), ‘On incentives for innovation’, Science, Vol. 326, No 5957, 1163, 
https://doi.org/10.1126/science.1184848. 

Avin, S. (2015), ‘Funding science by lottery’, In Recent developments in the philosophy 
of science: EPSA13 Helsinki, edited by Mäki, U., Votsis, I., Ruphy, S. and Schurz, G., pp. 
111-126. European Studies in Philosophy of Science. Springer International Publishing, 
Cham, https://doi.org/10.1007/978-3-319-23015-3_9. 

Azoulay, P., Graff Zivin, J. S. and Manso, G. (2013), ‘National Institutes of Health peer 
review: Challenges and avenues for reform’, Innovation Policy and the Economy, Vol. 13, 
January, pp. 1-22, https://doi.org/10.1086/668237. 

Azoulay, P., Graff Zivin, J. S. and Manso, G. (2009), ‘Incentives and creativity: Evidence 
from the academic life sciences’, RAND Journal of Economics, Vol. 42, No 3, pp. 527-554, 
https://doi.org/10.1111/j.1756-2171.2011.00140.x. 

Barnett, A. G., Herbert, D. L., Campbell, M., Daly, N., Roberts, J. A., Mudge, A. and 
Graves, N. (2015), ‘Streamlined research funding using short proposals and accelerated 
peer review: An observational study’, BMC Health Services Research, Vol. 15, No 1, 55, 
https://doi.org/10.1186/s12913-015-0721-7. 

Bell Browne Molnar & Delicate Consulting Inc. (n.d.), ‘Overall performance evaluation of 
the Canada Foundation for Innovation’, Bell Browne Molnar & Delicate Consulting Inc., 
Ottawa, https://www.innovation.ca/sites/default/files/pdf/2015/Final%20Report-EN-
CFI%20OPE-March2015.pdf. 

Belter, C. W. (2015), ‘Bibliometric indicators: Opportunities and limits’, Journal of the 
Medical Library Association, Vol. 103, No 4, pp. 219-221, https://doi.org/10.3163/1536-
5050.103.4.014. 

Besselaar, P. van den (2012), ‘Selection committee membership: Service or self-service’, 
Journal of Informetrics, Vol. 6, No 4, pp. 580-585, 
https://doi.org/10.1016/j.joi.2012.05.003. 

Besselaar, P. van den and Sandström, U. (2015), ‘Early career grants, performance, and 
careers: A study on predictive validity of grant decisions’, Journal of Informetrics, Vol. 9, 
No 4, pp. 826-838, https://doi.org/10.1016/j.joi.2015.07.011. 

Besselaar, P. van den, Sandström, U. and Schiffbaenker, H. (2018), ‘Studying grant 
decision-making: A linguistic analysis of review reports’, Scientometrics, Vol. 117, No 1, 
pp. 313-329, https://doi.org/10.1007/s11192-018-2848-x. 

Bhattacharjee, Y. (2012), ‘NSF’s “Big Pitch” tests anonymized grant reviews’, Science, 
Vol. 336, No 6084, pp. 969-970, https://doi.org/10.1126/science.336.6084.969. 

248 

 
 
Bieri, M., Roser, K., Heyard, R. and Egger, M. (2020), ‘How to best evaluate applications 
for junior fellowships? Remote evaluation and face-to-face panel meetings compared’, 
BioRxiv, November, 2020.11.26.400028, https://doi.org/10.1101/2020.11.26.400028. 

Bollen, J., Carpenter, S., Lubchenco, J. and Scheffer, M. (2019), ‘Rethinking resource 
allocation in science’, ArXiv, April, 1904.07645, http://arxiv.org/abs/1904.07645. 

Bollen, J., Crandall, D., Junk, D., Ding, Y. and Börner, K. (2017), ‘An efficient system to 
fund science: From proposal review to peer-to-peer distributions’, Scientometrics, Vol. 
110, No 1, pp. 521-528, https://doi.org/10.1007/s11192-016-2110-3. 

Bolli, R. (2014), ‘Actions speak much louder than words’, Circulation Research, Vol. 115, 
No 12, pp. 962-966, https://doi.org/10.1161/CIRCRESAHA.114.305556. 

Bornmann, L. and Marx, W. (2012), ‘The Anna Karenina principle: A way of thinking 
about success in science’, Journal of the American Society for Information Science and 
Technology, Vol. 63, No 10, pp. 2037-2051, https://doi.org/10.1002/asi.22661. 

Braben, D. W. (2004), Pioneering research: A risk worth taking. John Wiley & Sons.  

CFI (n.d.), ‘Canada Foundation for Innovation’, accessed 28 January 2021, 
https://www.innovation.ca. 

Coryn, C., Applegate, B., Schröter, D., Martens, K. and McCowen, R. (2012), ‘An 
evaluation of the transparency and overall quality of evaluation at the Swiss National 
Science Foundation’, Western Michigan University, Kalamazoo, 
https://doi.org/10.22163/fteval.2012.76. 

Danthi, N. S., Wu, C. O., DiMichele, D., Hoots, W. K. and Lauer, M. S. (2015), ‘Citation 
impact of NHLBI R01 grants funded through the American Recovery and Reinvestment 
Act as compared to R01 grants funded through a standard payline’, Circulation Research, 
Vol. 116, No 5, pp. 784-788, https://doi.org/10.1161/CIRCRESAHA.116.305894. 

Davé, A., Hopkins, M., Hutton, J., Krčál, A., Kolarz, P., Martin, B. et al. (2016), 
‘Landscape review of interdisciplinary research in the UK’, 184. 

DFG (n.d.), ‘German Research Foundation’, https://www.dfg.de. 

Dobele, A. R. (2015), ‘Assessing the quality of feedback in the peer-review process’, 
Higher Education Research & Development, Vol. 34, No 5, pp. 853-868, 
https://doi.org/10.1080/07294360.2015.1011086. 

Doyle, J. M., Quinn, K., Bodenstein, Y. A., Wu, C. O., Danthi, N. and Lauer, M. S. 2015. 
‘Association of percentile ranking with citation impact and productivity in a large cohort of 
de novo NIMH-funded R01 Grants’. Molecular Psychiatry, Vol. 20, No 9, pp 1030-1036. 
https://doi.org/10.1038/mp.2015.71. 

Egger, M. (n.d.), ‘Funding by lottery @SNF?’ 
https://www.chess.uzh.ch/dam/jcr:d1817454-3c9a-44ac-bed8-
432c7e2e1181/Egger%20CHESS%2019.11.19 

Enger, S. G. and Castellacci, F. (2016), ‘Who gets Horizon 2020 research grants? 
Propensity to Apply and Probability to Succeed in a Two-Step Analysis’, Scientometrics, 
Vol. 109, No 3, pp. 1611-1638. 

European Commission (2017), ‘MSCA-IF: Evaluation step by step: Guide for evaluators 
2017’, https://ec.europa.eu/info/sites/info/files/guide_for_evaluators_2017_3 

249 

 
 
European Commission (n.d.a), ‘Horizon 2020 – SME instrument: European Innovation 
Council (EIC) Pilot. Experts Briefing Part 2 – The Evaluation Process’, 
https://ec.europa.eu/easme/sites/easme-site/files/expert_briefing_part_2 

European Commission (n.d.b), CORDIS, accessed 17 December 2020, 
https://cordis.europa.eu/en. 

Fang, F. and Casadevall, A. (2012), ‘Reforming science: Structural reforms’, Infection 
and Immunity, Vol. 80, No 3, pp. 897-901, https://doi.org/10.1128/IAI.06184-11. 

Fang, F. and Casadevall, A. (2016), ‘Research funding: The case for a modified lottery’, 
MBio, Vol. 7, No 2, https://doi.org/10.1128/mBio.00422-16. 

Fang, F., Bowen, A. and Casadevall, A. (2016), ‘NIH peer review percentile scores are 
poorly predictive of grant productivity’, ELife, Vol. 5, February, p. e13323, 
https://doi.org/10.7554/eLife.13323. 

Fogelholm, M., Leppinen, S., Auvinen, A., Raitanen, J., Nuutinen, A. and Väänänen, K. 
(2012), ‘Panel discussion does not improve reliability of peer review for medical research 
grant proposals’, Journal of Clinical Epidemiology, Vol. 65, No 1, pp. 47-52, 
https://doi.org/10.1016/j.jclinepi.2011.05.001. 

Gallo, S. A., Carpenter, A. S. and Glisson, S. R. (2013), ‘Teleconference versus face-to-
face scientific peer review of grant application: Effects on review outcomes’, PLoS ONE, 
Vol. 8, No 8, p. e71693, https://doi.org/10.1371/journal.pone.0071693. 

Gallo, S. A., Sullivan, J. H. and Glisson, S. R. (2016), ‘The influence of peer reviewer 
expertise on the evaluation of research funding applications’, PLoS ONE, Vol. 11, No 10, 
e0165147, https://doi.org/10.1371/journal.pone.0165147. 

Giraudeau, B., Leyrat, C., Le Gouge, A., Léger, J. and Caille, A. (2011), ‘Peer review of 
grant applications: A simple method to identify proposals with discordant reviews’, PLoS 
ONE, Vol. 6, No 11, e27557, https://doi.org/10.1371/journal.pone.0027557. 

Gluckman, P. (n.d.), ‘Which science to fund: Time to review peer review?’, Office of the 
Prime Minister’s Science Advisory Committee, London. 

Gold, B. (2016), ‘The 5 Significant advantages of interdisciplinary research’, Institute for 
Humane Studies (blog). 9 November 2016, https://theihs.org/blog/5-advantages-of-
interdisciplinary-research/. 

Gregorius, S., Dean, L., Cole, D. C. and Bates, I. (2018), ‘The peer review process for 
awarding funds to international science research consortia: A qualitative developmental 
evaluation’, F1000Research, Vol. 6, January, pp. 1808, 
https://doi.org/10.12688/f1000research.12496.3. 

Güdler, J. and Reinhardt, A. (n.d.), ‘Gender equality in DFG research funding – Facts and 
assessment’, 9. 

Guglielmi, G. (2018), ‘Gender bias tilts success of grant applications’, Nature, Vol. 554, 
No 2. 

Gurwitz, D., Milanesi, E. and Koenig, T. (2014), ‘Grant application review: The case of 
transparency’, PLoS Biology, Vol. 12, No 12, 
https://doi.org/10.1371/journal.pbio.1002010. 

Guthrie, S., Ghiga, I. and Wooding, S. (2018), ‘What do we know about grant peer 
review in the health sciences?’ F1000Research, Vol. 6, March, 
https://doi.org/10.12688/f1000research.11917.2. 

250 

 
 
Guthrie, S., Rodriguez Rincon, D., McInroy, G., Ioppolo, B. and Gunashekar, S. (2019a), 
‘Measuring bias, burden and conservatism in research funding processes’, 
F1000Research, Vol. 8, June, pp. 851, https://doi.org/10.12688/f1000research.19156.1. 

Guthrie, S., Rodriguez-Rincon, D., McInroy, G., Ioppolo, B. and Gunashekar, S. (2019b), 
Design and development of an evaluation framework for the NHMRC: Review of 
International practice, overview of NHMRC’s data and grant management systems, and 
an evaluation framework and metrics, RAND Corporation, Santa Monica, CA, 
https://www.rand.org/pubs/research_reports/RR2984.html. 

Herbert, D. L., Coveney, J., Clarke, P., Graves, N. and Barnett, A. G. (2014), ‘The impact 
of funding deadlines on personal workloads, stress and family relationships: A qualitative 
study of Australian researchers’, BMJ Open, Vol. 4, No 3, p. e004462, 
https://doi.org/10.1136/bmjopen-2013-004462. 

Heyard, R., Ott, M.,  Salanti, G. and Egger, M. (2021), ‘Rethinking the funding line at the 
Swiss National Science Foundation: Bayesian ranking and lottery’, ArXiv:2102.09958, 
February, http://arxiv.org/abs/2102.09958. 

Ho, R. Chun-Man, Mak, K.-K., Tao, R., Lu, Y., Day, J. R. and Pan, F. (2013), ‘Views on 
the peer review system of biomedical journals: An online survey of academics from high-
ranking universities’, BMC Medical Research Methodology, Vol. 13, No 1, 74, 
https://doi.org/10.1186/1471-2288-13-74. 

Hornbostel, S. and Olbrecht, M. (2007), ‘Peer Review in der DFG: Die Fachkollegiaten’, 
http://www.forschungsinfo.de/Publikationen/Download/working_paper_2_2007 

Hotopp, T. 2005), ‘Basic research funding in Germany: Instruments of the DFG and roles 
of POs for R&D’, https://www.jst.go.jp/po_seminar/pdf/semi2/pre/no_09 

Huutoniemi, K., Thompson Klein, J., Bruun, H. and Hukkinen, J. (2010), ‘Analyzing 
interdisciplinarity: Typology and indicators’, Research Policy, Vol. 39, No 1, pp. 79-88, 
https://doi.org/10.1016/j.respol.2009.09.011. 

Innovate UK (n.d.), ‘Delivery Plan 2019’, https://www.ukri.org/wp-
content/uploads/2020/09/INUK-250920-DeliveryPlan2019  

Innovate UK (n.d.), ‘What is an assessor looking for? A short guidebook for Innovate UK 
competition applicants’, 
https://www.nibusinessinfo.co.uk/sites/default/files/A_Short_Guidebook_for_InnovateUK
_Competition_Applicants 

Innovate UK. (n.d.a), ‘About us’, accessed 28 January 2021, 
https://www.gov.uk/government/organisations/innovate-uk/about. 

Innovate UK (n.d.b), ‘Business Innovation: What funding you can get and how to apply’, 
accessed 28 January 2021, https://www.gov.uk/guidance/innovation-apply-for-a-
funding-award. 

Innovate UK (n.d.c), ‘Competition Overview – ISCF Future Food Production Systems – 
Innovation Funding Service’, accessed 28 January 2021, https://apply-for-innovation-
funding.service.gov.uk/competition/461/overview. 

Innovation Funding Service (n.d.), ‘Competition overview – SBRI Competition: Improving 
Operations through Machine Learning – Innovation Funding Service’, accessed 29 
January 2021, https://apply-for-innovation-
funding.service.gov.uk/competition/47/overview#scope. 

Jang, D., Doh, S. Kang, G.-M. and Han, D.-S. (2017), ‘Impact of alumni connections on 
peer review ratings and selection success rate in national research’, Science, Technology, 

251 

 
 
& Human Values, Vol. 42, No 1, pp. 116-143, 
https://doi.org/10.1177/0162243916665466. 

Jayasinghe, U. W., Marsh, H. W. and Bond, N. (2003), ‘A multilevel cross-classified 
modelling approach to peer review of grant proposals: The effects of assessor and 
researcher attributes on assessor ratings’, Journal of the Royal Statistical Society: Series 
A (Statistics in Society), Vol. 166, No 3, pp. 279-300, https://doi.org/10.1111/1467-
985X.00278. 

Jerrim, J. and de Vries, R. (2020), ‘Are peer-reviews of grant proposals reliable? An 
analysis of Economic and Social Research Council (ESRC) Funding Applications’, The 
Social Science Journal, pp. 1-19, https://doi.org/10.1080/03623319.2020.1728506. 

Kaatz, A., Magua, W., Zimmerman, D. R. and Carnes, M. (2015), ‘A quantitative linguistic 
analysis of National Institutes of Health R01 application critiques from investigators at 
one institution’, Academic Medicine, Vol. 90, No 1, pp. 69-75, 
https://doi.org/10.1097/ACM.0000000000000442. 

Kaltman, J., Evans, F., Danthi, N., Wu, C., DiMichele, D. and Lauer M. (2014), ‘Prior 
publication productivity, grant percentile ranking, and topic-normalized citation impact of 
NHLBI Cardiovascular R01 grants’, Circulation Research, Vol. 115, No 7, pp. 617-624, 
https://doi.org/10.1161/CIRCRESAHA.115.304766. 

Kelso, A. (2019), ‘Peer review for NHMRC’s new grant program’, 
https://www.tri.edu.au/event/nhmrc-funding-video. 

Kitt, C., Williamson, J. and Paganetti, H. (2009), ‘TH-A-BRD-01: The future of NIH 
research funding’, Medical Physics, Vol. 36, No 6, Part 26, pp. 2791-2791, 
https://doi.org/10.1118/1.3182589. 

Kolarz, P., Arnold, E., Fryges, H., Krcal, A., Peter, V. and Rentel, M. (2018), ‘Impact 
evaluation of National Research Programmes 59, 60 and 61’, https://www.technopolis-
group.com/wp-content/uploads/2020/02/Impact-evaluation-of-National-Research-
Programmes-59-60-and-61  

Kulage, K. M., Schnall, R., Hickey, K. T., Travers, J., Zezulinski, K.,Torres, F. et al. 
(2015), ‘Time and costs of preparing and submitting an NIH grant application at a school 
of nursing’, Nursing Outlook, Vol. 63, No 6, pp. 639-649, 
https://doi.org/10.1016/j.outlook.2015.09.003. 

Kupfer, D. J., Murphree, A. N., Pilkonis, P. A., Cameron, J. L., Giang, R. T., Dodds, N. E. 
et al. (2014), ‘Using peer review to improve research and promote collaboration’, 
Academic Psychiatry, Vol. 38, No 1, pp. 5-10, https://doi.org/10.1007/s40596-013-
0027-1. 

Lamont, M. (2009), How professors think: Inside the curious world of academic 
judgment. Harvard University Press, Cambridge, Mass.. 

Langfeldt, L. and Borlaug, S. B. (2016), ‘Swiss National Centres of Competence in 
Research (NCCR): Evaluation of the selection process’, 84. 

Langfeldt, L. and Kyvik, S. (2011), ‘Researchers as evaluators: Tasks, tensions and 
politics’, Higher Education, Vol. 62, No 2, pp. 199-212, https://doi.org/10.1007/s10734-
010-9382-y. 

Langfeldt, L., Ramberg, I. and Gunnes, H. (2014), ‘Swiss research funding – Researcher 
survey for the SNSF’, http://www.snf.ch/SiteCollectionDocuments/Web-
News/researcher_survey_nifu_report_snsf_2014  

252 

 
 
Lanza, S. T. and Rhoades, B. L. (2013), ‘Latent class analysis: An alternative perspective 
on subgroup analysis in prevention and treatment’, Prevention Science, Vol. 14, No 2, 
pp. 157-168, https://doi.org/10.1007/s11121-011-0201-1. 

Lanza, S.T., Coffman, D. L. and Xu, S. (2013), ‘Causal inference in latent class analysis’, 
Structural Equation Modeling, Vol. 20, No 3, pp. 361-383, 
https://doi.org/10.1080/10705511.2013.797816. 

Lawrence-Berkeley, A. (2017), ‘Our unsuccessful Innovate UK funding bid’, Sense Media, 
20 March 2017, https://www.sensemedia-events.com/our-unsuccessful-application-to-
innovate-uk/. 

Lazarides, M. K., Georgiadis, G. S. and Papanas, N. (2020), ‘Do’s and don’ts for a good 
reviewer of scientific papers: A Beginner’s brief decalogue’, The International Journal of 
Lower Extremity Wounds, Vol. 19, No 3, pp. 227-229, 
https://doi.org/10.1177/1534734620924349. 

Lee, R. van der, and Ellemers, N. (2015), ‘Gender contributes to personal research 
funding success in the Netherlands’, Proceedings of the National Academy of Sciences of 
the United States of America, Vol. 112, No 40, pp. 12349-12353, 
https://doi.org/10.1073/pnas.1510159112. 

Liaw, L., Freedman, J. E., Becker, L. B., Mehta, N. N. and Liscum, L. (2017), ‘Peer review 
practices for evaluating biomedical research grants: A scientific statement from the 
American Heart Association’, Circulation Research, Vol. 121, No 4, pp. e9-19, 
https://doi.org/10.1161/RES.0000000000000158. 

Lindner, M. D. and Nakamura, R. K. (2015), ‘Examining the Predictive Validity of NIH 
Peer Review Scores’, PLoS ONE, Vol. 10, No 6, p. e0126938, 
https://doi.org/10.1371/journal.pone.0126938. 

Lindner, M. D., Vancea, A., Chen, M.-C. and Chacko, G. (2016), ‘NIH peer review: Scored 
review criteria and overall impact’, American Journal of Evaluation, Vol. 37, No 2, pp. 
238-249, https://doi.org/10.1177/1098214015582049. 

Linton, J. D. (2016), ‘Improving the peer review process: Capturing more information 
and enabling high-risk/high-return research’, Research Policy, Vol. 45, No 9, pp. 1936-
1938, https://doi.org/10.1016/j.respol.2016.07.004. 

Lobb, R., Petermann, L., Manafo, E., Keen, D., and Kerner, J. (2013), ‘Networking and 
knowledge exchange to promote the formation of transdisciplinary coalitions and levels of 
agreement among transdisciplinary peer reviewers’, Journal of Public Health Management 
and Practice, Vol. 19, No 1, pp. e9-20, https://doi.org/10.1097/PHH.0b013e31823991c2. 

Luukkonen, T. (2012), ‘Conservatism and risk-taking in peer review: Emerging ERC 
practices’, Research Evaluation, Vol. 21, No 1, pp. 48-60, 
https://doi.org/10.1093/reseval/rvs001. 

Lyall, C., Bruce, A., Tait, J. and Meagher, L. (2011), ‘Interdisciplinary research journeys’, 
Bloomsbury Publishing, London, https://www.bloomsbury.com/uk/interdisciplinary-
research-journeys-9781849660143/. 

Marks, A. R. (2011), ‘Repaving the road to biomedical innovation through academia’, 
Science Translational Medicine, Vol. 3, No 89, 
https://doi.org/10.1126/scitranslmed.3002223. 

Martin, M. R., Kopstein, A. and Janice, J. M. (2010), ‘An analysis of preliminary and post-
discussion priority scores for grant applications peer reviewed by the Center for Scientific 
Review at the NIH’, PLoS ONE, Vol. 5, No 11, e13526, 
https://doi.org/10.1371/journal.pone.0013526. 

253 

 
 
McLeish, T. and Strang, V. (2016), ‘Evaluating interdisciplinary research: The elephant in 
the peer-reviewers’ room’, Palgrave Communications, Vol. 2, No 1, pp. 1-8, 
https://doi.org/10.1057/palcomms.2016.55. 

Miner, J. T. (2011), ‘Behind door #3: The hard-to-please grant reviewer’, Research 
Management Review, Vol. 18, pp. 85-107. 

Mow, K. (2010), Inside the black box: Research grant funding and peer review in 
Australian research councils, Lambert Academic Publishing, Chisinau. 

Mulder, R. H. and Ellinger, A. D. (2013), ‘Perceptions of quality of feedback in 
organizations: Characteristics, determinants, outcomes of feedback, and possibilities for 
improvement: Introduction to a special issue’, European Journal of Training and 
Development, Vol. 37, No 1, pp. 4-23, https://doi.org/10.1108/03090591311293266. 

Murray, D. L., Morris, D., Lavoie, C., Leavitt, P. R., MacIsaac, H., Masson, M. E. J. and 
Villard, M.-A. (2016a), ‘Bias in research grant evaluation has dire consequences for small 
Universities’, PLoS ONE, Vol. 11, No 6, p. e0155876, 
https://doi.org/10.1371/journal.pone.0155876. 

Murray, D. L., Morris, D., Lavoie, C., Leavitt, P. R., MacIsaac, H., Masson, M. E. J. and 
Villard, M.-A. (2016b), ‘Bias in research grant evaluation has dire consequences for small 
universities’, PLoS ONE, Vol. 11, No 6, e0155876, 
https://doi.org/10.1371/journal.pone.0155876. 

Mutz, R., Bornmann, L. and Daniel, H.-D. (2016), ‘Funding decision-making systems: An 
empirical comparison of continuous and dichotomous approaches based on psychometric 
theory’, Research Evaluation, Vol. 25, No 4, pp. 416-426, 
https://doi.org/10.1093/reseval/rvw002. 

Mutz, R., Bornmann, L. and Daniel, H.-D. (2015), ‘Testing for the fairness and predictive 
validity of research funding decisions: A multilevel multiple imputation for missing data 
approach using ex-ante and ex-post peer evaluation data from the Austrian Science 
Fund’, Journal of the Association for Information Science and Technology, Vol. 66, No 11, 
pp. 2321-2339, https://doi.org/10.1002/asi.23315. 

NCN (2019a), ‘Annual Report 2019’, 
https://www.ncn.gov.pl/sites/default/files/pliki/centrum-prasowe/NCN_raport_2019_ang 

NCN (2019b). Order No 62/2019 of the Director of the National Science Centre on the 
Detailed Procedure for Evaluating Proposals of 16 September 2019, 
https://ncn.gov.pl/sites/default/files/pliki/zarzadzenia-dyrektora/zarzadzenieDyr-
62_2019_ang 

NCN (2020), ‘Information brochure’, https://ncn.gov.pl/sites/default/files/pliki/centrum-
prasowe/NCN_broszura_2020_ang 

NCN (n.d.), ‘About the NCN’, accessed 28 January 2021, https://ncn.gov.pl/o-
ncn/zadania-ncn?language=en. 

Neale, J. (2016), ‘Iterative Categorization (IC): A Systematic technique for analysing 
qualitative data’, Addiction (Abingdon, England), Vol. 111, No 6, pp. 1096-1106, 
https://doi.org/10.1111/add.13314. 

NHMRC (2016), ‘Structural review of NHMRC’s grant program: Consultation paper’, 
https://consultations.nhmrc.gov.au/public_consultations/nhmrc-grant-program.html. 

NHMRC (2021a), ‘Australian standard research classifications and research keywords’, 
2021, https://www.nhmrc.gov.au/about-us/resources/australian-standard-research-
classifications-and-research-keywords. 

254 

 
 
NHMRC (2021b), ‘Principles of peer review’, accessed 28 January 2021, 
https://www.nhmrc.gov.au/about-us/resources/principles-peer-review. 

NHMRC (n.d.), ‘Overview’, accessed 28 January 2021, 
https://www.nhmrc.gov.au/funding/new-grant-program/overview. 

Nicholas, K. A. and Gordon, W. S. (2011), ‘A quick guide to writing a solid peer review’, 
Eos, Transactions American Geophysical Union, Vol. 92, No 28, pp. 233-234, 
https://doi.org/10.1029/2011EO280001. 

NIH (n.d.), ‘Online survey software | Qualtrics Survey Solutions’, accessed 11 December 
2020, 
https://nihodoercomm.az1.qualtrics.com/jfe/form/SV_eypqaXlx2j1IY9T?Q_CHL=si&Curre
ntPageURL=https%3A%2F%2Fgrants.nih.gov%2Fgrants%2Fgrants_process.htm&PageRe
ferrer=null&Intercept=HTML&Q_CanScreenCapture=1. 

NRF (2009), ‘NRF competitive research programme funding scheme detailed guidelines’, 
http://www.mse.ntu.edu.sg/Research/Documents/researchforms/NRF/NRF%20Detailed
%20Guidelines%20for%20CRP%20-%20Version%204 

NRF (2019a), ‘NRF Central Gap Fund – Detailed guidelines for applicants’, 
https://www.nrf.gov.sg/docs/default-source/default-document-library/central-gap-fund-
application-guidelines-v2 

NRF (2019b), ‘Singapore National Research Foundation (NRF) Fellowship Scheme – 
Application guide’, https://www.nrf.gov.sg/docs/default-source/default-document-
library/nrf-fellowship-application-guide 

NRF (n.d), ‘National Research Foundation’, https://www.nrf.gov.sg. 

Offshore Wind Growth Partnership (2019), ‘OWGP funding competitions – Call for 
external assessors’, 

Oviedo-García, M. Á. (2016), ‘Ex ante evaluation of interdisciplinary research projects: A 
literature review’, Social Science Information 55, No 4, pp. 568-588, 
https://doi.org/10.1177/0539018416658147. 

Palinkas, L. A., Horwitz, S. M., Green, C. A., Wisdom, J. P., Duan, N. and Hoagwood, K. 
(2015), ‘Purposeful sampling for qualitative data collection and analysis in mixed method 
implementation research’, Administration and Policy in Mental Health, Vol. 42, No 5, pp. 
533-544, https://doi.org/10.1007/s10488-013-0528-y. 

Pier, E. L., Brauer, M., Filut, A., Kaatz, A., Raclaw, J. and Nathan, M. J. et al. (2018), 
‘Low agreement among reviewers evaluating the same NIH grant applications’, 
Proceedings of the National Academy of Sciences of the United States of America, Vol. 
115, No 12, pp. 2952-2957, https://doi.org/10.1073/pnas.1714379115. 

Pina, D. G., Hren, D., and Marušić, A. (2015), ‘Peer review evaluation process of Marie 
Curie Actions under EU’s Seventh Framework Programme for Research’, PLoS ONE, Vol. 
10, No 6, p. e0130753, https://doi.org/10.1371/journal.pone.0130753. 

R Core Team (2020), R: A language and environment   for statistical computing. R 
Foundation for   Statistical Computing, Vienna, Austria. https://www.R-project.org/. 

Reinhart, M. (2009), ‘Peer Review of grant applications in biology and medicine: 
Reliability, fairness, and validity’, Scientometrics, Vol. 81, No 3, pp. 789-809, 
https://doi.org/10.1007/s11192-008-2220-7. 

Rockey, S. (2012), ‘The A2 resubmission policy continues: A closer look at recent data’, 
NIH Extramural Nexus (blog). 28 November 2012, 

255 

 
 
https://nexus.od.nih.gov/all/2012/11/28/the-a2-resubmission-policy-continues-a-closer-
look-at-recent-data/. 

Salandra, R. and Salter, A. (2019), ‘Helping Innovate UK understand their grant 
assessment process’, Innovation Caucus (blog). 17 December 2019, 
https://innovationcaucus.co.uk/2019/12/17/helping-innovate-uk-understand-their-grant-
assessment-process/. 

Salandra, R. and Salter, S. (2019), ‘Helping Innovate UK understand their grant 
assessment process’, Innovation Caucus (blog). 17 December 2019, 
https://innovationcaucus.co.uk/2019/12/17/helping-innovate-uk-understand-their-grant-
assessment-process/. 

Sattler, D. N., McKnight, P. E., Naney, L. and Mathis, R. (2015), ‘Grant peer review: 
Improving inter-rater reliability with training’, PLoS ONE, Vol. 10, No 6, p. e0130450, 
https://doi.org/10.1371/journal.pone.0130450. 

Schiermeier, Q. and Weydt, P. (2000), ‘German Research Agency stifles creativity’, 
Nature, Vol. 404, No 6775, 217, https://doi.org/10.1038/35005249. 

Schroter, S., Groves, T. and Højgaard, L. (2010), ‘Surveys of current status in biomedical 
science grant review: Funding organisations’ and grant reviewers’ perspectives’, BMC 
Medicine, Vol. 8, October, 62, https://doi.org/10.1186/1741-7015-8-62. 

Severin, A., Martins, J., Heyard, R., Delavy, F., Jorstad, A. and Egger, M. (2020), ‘Gender 
and other potential biases in peer review: Cross-Sectional Analysis of 38 250 external 
peer review reports’, BMJ Open, Vol. 10, No 8, p. e035058, 
https://doi.org/10.1136/bmjopen-2019-035058. 

Snell, R. R. (2015), ‘Menage a quoi? Optimal number of peer reviewers’, PLoS ONE, Vol. 
10, No 4, https://doi.org/10.1371/journal.pone.0120838. 

SNSF (n.d.), ‘Swiss National Science Foundation’, accessed 28 January 2021, 
http://www.snf.ch. 

Solans-Domènech, M., Guillamón, I., Ribera, A., Ferreira-González, I., Carrion, Gaietà 
Permanyer-Miralda, C. and Pons, J. M. V. (2017), ‘Blinding applicants in a first-stage 
peer-review process of biomedical research grants: An observational study’, Research 
Evaluation, Vol. 26, No 3, pp. 181-189, https://doi.org/10.1093/reseval/rvx021. 

Specht, J. A. (2013), ‘Mentoring relationships and the levels of role conflict and role 
ambiguity experienced by novice nursing faculty’, Journal of Professional Nursing, Vol. 
29, No 5, pp. e25-31, https://doi.org/10.1016/j.profnurs.2013.06.006. 

Spier (2002) Peer review and innovation. Sci Eng Ethics. 2002;8(1):99–108. 
10.1007/s11948-002-0035-0 

SQW, Cambridge Econometrics, and BMG Research (2015), ‘Evaluation of Smart – 
Impact and Process Evaluation’ 

Sueur, H. L., Dagliati, A., Buchan, I., Whetton, A. D., Martin, G. P., Dornan, T. and 
Geifman, N. (2020), ‘Pride and prejudice – What can we learn from peer review?’ Medical 
Teacher, Vol. 42, No 9, pp. 1012-1018, 
https://doi.org/10.1080/0142159X.2020.1774527. 

Tamblyn, R., Girard, N., Qian, C. J. and Hanley, J. (2018), ‘Assessment of potential bias 
in research grant peer review in Canada’, Canadian Medical Association Journal, Vol. 190, 
No 16, pp. e489−499, https://doi.org/10.1503/cmaj.170901. 

256 

 
 
Tversky, A. and Kahneman, D. (1991), ‘Loss aversion in riskless choice: A reference-
dependent model’, Quarterly Journal of Economics, Vol. 106, No 4, pp. 1039-1061, 
https://doi.org/10.2307/2937956. 

UKRI (n.d.), ‘UK Research and Innovation’, accessed 29 January 2021, 
https://www.ukri.org/?_ga=2.149682803.473754006.1635244299-
1250990708.1632144555. 

UKRI (2020a), ‘Corporate Plan 2020-21’, 

UKRI (2020b), ‘Annual report and accounts 2019-20’, https://www.ukri.org/wp-
content/uploads/2020/10/UKRI-050920-AnnualReport2019-2020 

UKRI (2020c), ‘UKRI principles of assessment and decision making’, 

University of Groningen (2020), ‘Recap: Celebrating openness (22 October 2020)’, 
University of Groningen, 16 November 2020. https://www.rug.nl/library/open-
access/blog/recap-celebrating-openness-22-october-2020-16-11-2020. 

Varker, T., Forbes, D., Dell, L., Weston, A., Merlin, T., Hodson, S. and O'Donnell, M. 
(2015), Rapid evidence assessment: Increasing the transparency of an emerging 
methodology. Journal of Evaluation in Clinical Practice, Vol. 21, No 6, pp. 1199-1204, 
https://doi: 10.1111/jep.12405. 

Vrieze, J. de (2017), ‘Funders groan under growing review burden’, Science, Vol. 357, No 
6349, p. 343, https://doi.org/10.1126/science.357.6349.343. 

Witteman, H. O., Hendricks, M., Straus, S. and Tannenbaum, C. (2019), ‘Are gender 
gaps due to evaluations of the applicant or the science? A natural experiment at a 
national funding agency’, The Lancet, Vol. 393, No 10171, pp. 531-540. 
https://doi.org/10.1016/S0140-6736(18)32611-4. 

257 

 
 
ANNEX A. INTERVIEW PROTOCOL WITH EC STAFF 

Interview Protocol 

RAND  Europe  has  been  commissioned  by  DG  RTD  Unit  A.2  to  provide  a  comprehensive 
review of proposal evaluation processes in EU R&I framework programmes. The study will 
feed  into  the  ex-post  evaluation  of  Horizon  2020  and  where  relevant  inform  the 
development and adaptation of the future proposal evaluation approaches. In particular, 
we are seeking to assess (1) the relevance of the current proposal evaluation system to 
achieve the objectives of Horizon 2020; (2) the effectiveness and efficiency of the process; 
and what lessons can be learned in view of adapting the proposal evaluation system in the 
future. 

In the inception stage of the study we are conducting a limited number of key interviews 
with  the  EU  officials  to  ensure  we  have  a  good  understanding  of  the  processes  in  place 
including any specificities that the study needs to consider. 

We understand that the different actions will vary in the proposal evaluation process (for 
example, in the number of stages in the evaluation process, the criteria against which the 
proposals are evaluated, and the weight given to the different criteria). We have provided 
you with a diagram of our understanding of the standard process for proposal evaluation 
in Horizon 2020, which we will use to explore the commonalities and differences across the 
different  actions.  We  are  particularly  interested  in  the  nuances  of  the  process  for  the 
different actions and the rationale behind these. 

We would like to reiterate that information shared in the interviews will be confidential and 
that the study is aimed at learning purposes. In particular, this interview will be used to 
develop a better understanding of the proposal evaluation system in H2020, as well as the 
challenges and benefits of the process. 

Introduction 

1.  Which part of the Horizon 2020 programme do you (your agency) cover? 

General questions 

2.  What are the broader challenges in the evaluation of proposals for [action]? 

3.  What are the benefits of the current proposal evaluation system [action]? 

4.  With a view towards Horizon Europe and beyond, are there areas for improvement 
in the proposal evaluation process for [action] and H2020 more broadly that you 
would like to see incorporated into Horizon Europe? 

Overview of the proposal evaluation system for [action] 

5.  Can you summarise the proposal evaluation system for [action]? 

6.  In your views, how does the proposal evaluation process reflect the objectives of 

the programme? 

7.  In case of two-stage submission: 

a.  What is the rationale behind using a two-stage submission format? 

b.  Can you describe the evaluation process for first stage submissions? 

c.  Why are only ‘excellence’ and ‘impact’ considered in the first stage? 

258 

 
 
8.  In case of a multi-step evaluation process: 

a.  What is the rationale behind using this format? 

b.  In your views, is the current process achieving its objectives? 

c.  What happens in each step of the process? 

9.  Has  the  proposal  evaluation  process  for  [action]  been  adapted  throughout  the 

course of H2020? If so, why and how? 

Final comments 

10. Are there any documents or data that you consider relevant for us to review for this 

study? Is there anyone else we should speak to? 

11. Is there anything else you would like to add that we have not already discussed? 

TIME PERMITTING ONLY AND IF NOT ALREADY ANSWERED BY INTERVIEWEE 

12. Current funding systems are faced with an increasing number of applicants, which 

adds burden to programme managers and to reviewers. 

a.  How have you dealt with the high number of applicants? 

b.  Have you had challenges in engaging external experts in the process? 

i.  If so, how have you overcome these challenges? 

13. One of the trends associated with an increased number of applicants and reduced 
resources  is  a  tendency  towards  conservatism  (i.e.  funding  less  innovative  and 
riskier  ideas).  How  do  you  ensure  that  innovative  research  is  funded  through 
[action]? 

14. How  do  you  promote  the  transparency  in  the  proposal  evaluation  process  for 

[action]? 

15. At what stages of the process do you think bias can be introduced? 

a.  What mechanisms do you have in place to ensure bias does not occur? 

16. Do  you  have  evidence  on  the  reliability  of  the  proposal  evaluation  system  for 

[action]? 

259 

 
 
ANNEX B. SURVEYS TO ASSESS BURDEN OF THE PROCESS 

1 

EU Proposal Evaluation Survey - Applicants 

1. Welcome Page 

RAND Europe is conducting a study for DG RTD evaluating the peer review processes 
involved in H2020 funding. This study aims to help ensure that processes in research 
funding systems are consistent with scientific values of rigour and objectivity, while 
delivering decisions efficiently and effectively. As part of this assessment, we are 
conducting a survey of key stakeholders across H2020’s funding processes. 

2. Privacy Information 

About the project 
RAND Europe is conducting a study for DG RTD evaluating the peer review processes 
involved in H2020 funding. This study aims to help ensure that processes in research 
funding systems are consistent with scientific values of rigour and objectivity, while 
delivering decisions efficiently and effectively. As part of this assessment, we are 
conducting a survey of key stakeholders across H2020’s funding processes. 
Who are we? 
RAND Europe is an independent not-for-profit research organisation 
(www.rand.org/randeurope). 
Why are we contacting you? 
Because of your knowledge and experience of H2020’s funding application and decision-
making processes. 
What personal data do we hold? 
From data provided by DG RTD we have collated information on stakeholders including 
name, title, organisation and email address so that we can invite you to participate. We 
are holding this data in compliance with Article 14 of the GDPR. 
Who is handling this information? 
DG RTD is the data controller and is responsible for personal information in relation to 
this project. RAND Europe is undertaking this work for the DG RTD and is acting as a 
data processor. The data we are collecting on behalf of DG RTD will be processed in line 
with the DG RTD privacy policy. 
What is the legal basis for processing your data? 
The personal data we are processing includes your contact details that have been shared 
with us by the DG RTD Team. We are using these solely to invite you to participate in 
this study and to manage data collection. We do not anticipate that your rights and 
freedoms will be affected as a result of our use of your data in this way. We are therefore 
processing your data on the basis that it is within the legitimate interests of DG RTD to 
undertake this task. 
Any responses to the survey will be confidential and your contact details will be 
separated from your responses prior to analysis. We will not ask you to enter your 
contact details as part of the survey process. 
How long will we keep this data? 
RAND Europe will retain your contact details for up to one year after the end of the 
research project, after which time your contact details will be deleted from the project 
files. 
What are your rights? 
You have the right to request access to and rectification or erasure of personal data. You 
can also request us to place restrictions on our processing of your data and have the 
right to object to our processing as well as the right to data portability. To exercise any 
of these rights or if you have any questions, please contact RAND Europe (as the data 
processor) and DG RTD (as the data controller). 
How do you contact us to exercise these rights? 
You may contact the RAND Europe data protection officer at redpo@randeurope.org, 
citing "20141 Study on the Proposal Evaluation System of the EU R&I Framework 
Programme" in the subject line. 

260 

 
 
If you have concerns that you feel we are unable to address about our use of personal 
data you are able to refer this to the Information Commissioner’s Office (ICO) at 
https://ico.org.uk// 
What happens if I agree to participate in the survey? 
We would like to ask you to complete a survey to help us understand the current 
proposal evaluation processes within H2020, with a specific focus on peer review. 
Responses to the survey will be confidential. We will not ask you to enter your contact 
details as part of the process. 
We hope you will be happy to inform this important study for DG RTD. 
If you would like any further information regarding the project please contact the study 
team at fundingsurveyEU@randeurope.org. 

3. Information about survey completion 

The survey should take no more than 15 minutes of your time. Your participation is 
entirely voluntary, and you may choose to change your mind about participating at any 
time before, during or after the survey. You do not have to give any reasons for 
withdrawing but we draw your attention to your rights to delete your data at the bottom 
of this form. 

As outlined in the privacy information notice, the data you provide will be kept until 12 
months after the project ends. The data you provide will be deleted at this point and the 
cross-analysed data will be kept indefinitely in an anonymised form. It is our legitimate 
interest to obtain and retain information on your name, organizational affiliation and role, 
as this is necessary to facilitate contact with you for the project and in doing so does not 
affect your rights or freedoms. Retention of your information is subject to your rights 
below. All records will be kept in compliance with the General Data Protection Regulation 
(GDPR) 2018. This survey will be conducted by RAND Europe staff so the data will be 
recorded and stored in accordance with RAND Europe's procedures. Further information 
on RAND Europe's privacy policy can be found here: 
https://www.rand.org/randeurope/privacy.html. 

If you have any queries about any aspect of this project please contact the study team 
on fundingsurveyEU@randeurope.org. 

Your legal rights 
RAND Europe operates in accordance with EU law including GDPR. You are provided with 
certain rights that you may have the right to exercise through us. In summary those 
rights are: 
(1) To access, correct or erase your data; 
(2) To object to the processing of your data; 
(3) To request that our processing of your data is restricted; 
(4) To request that your data be transferred; 
(5) To withdraw your consent for us to process your data. 
If you wish to exercise any of these rights please contact the RAND Europe Data 
Protection Officer by email at REdpo@rand.org or in writing to Data Protection Officer, 
RAND Europe, Westbrook Centre, Milton Road, Cambridge, CB4 1YG, UK. 

4. Consent form to participate in survey 

1. I confirm that I have read the information sheet on participation in the survey. * 

  Yes 

  No 

2. I understand that my participation is voluntary and that I am free to withdraw at any 
time without giving any reason, without being penalised in any way or my legal rights 
being affected. * 

261 

 
 
 
 
 
  Yes 

  No 

3. I agree to participate in this survey. * 

  Yes 

  No 

5. Survey on the EU R&I proposal evaluation system 

Instructions: Please answer the following questions with regard to your latest grant 
application to an H2020 scheme. Where a time estimation is asked for, this is the time 
spent on the activity in question. 

4. Thinking back to your last grant application to H2020, which funding stream did you 
apply to? Please select only one. * 

  Access to risk finance 

  Climate action, environment, resource efficiency and raw materials 

  Cross-cutting activities (Focus Areas) 

  ERC Grants 

  Europe in a changing world - Inclusive, innovative and reflective societies 

  Food security, sustainable agriculture and forestry, marine and maritime and inland 
water research 

  Future and Emerging Technologies (FET) 

  Health, demographic change and wellbeing Recognition Prize 

  Innovation in SMEs Inducement Prize 

  Joint Technology Initiatives (JTIs) 

  LEIT - Information and Communication Technologies 

  LEIT - Nanotechnologies, Advanced Materials, Biotechnology and Advanced 
Manufacturing and Processing 

  LEIT - Space Science with and for society 

  Marie Skłodowska-Curie actions (MSCA) 

  Research infrastructure (RI) 

262 

 
 
  Secure societies - Protecting freedom and security of Europe and its citizens 

  Secure, clean and efficient energy 

  Smart, green and integrated transport 

  Spreading excellence and widening participation 

  SME Instrument 

  EIC Pilot 

  Other (please specify): 

5. Was your last grant application to H2020 a single or multi-stage submission process? 
In a single-stage submission process applicants have one deadline to submit their full 
application, whereas for the multi-stage submission process, applicants have two 
deadlines, whereby they initially submit a short proposal, and then are invited to submit 
a full proposal if the short proposal is successful. * 

  Single-stage 

  Multi-stage 

6. Proposal submission process 

6. Thinking back to your last proposal submission to H2020, how much time did you 
personally spend (in days) writing the proposal for the first-stage submission? Please 
provide answer in numerics (e.g. 2). * 

7. To the best of your knowledge, how much time did your colleagues and co-applicants 
spend in total (in days) writing the proposal for the first-stage submission? Please 
provide answer in numerics (e.g. 2). * 

8. Were you invited to submit a full proposal? * 

  Yes 

  No 

7. Full proposal submission process 

9. Thinking back to your last proposal submission to H2020, how much time (in days) did 
you personally spend on writing the full proposal? Please provide answer in numerics 
(e.g. 2). * 

263 

 
 
  
 
 
  
  
 
10. To the best of your knowledge, how much time did your colleagues and co-applicants 
spend in total (in days) writing the full proposal? Please provide answer in numerics (e.g. 
2). * 

11. Did the funding call that you applied to have an interview stage? * 

  Yes 

  No 

8. Invitation to Interview 

12. Were you, and collaborators, invited for an interview? * 

  Yes 

  No 

9. Interview process 

13. How much time did you and collaborators spend (in hours) preparing for your 
interview? Please include any additional research, preparation or time spent developing 
or practicing a presentation. Please provide answer in numerics (e.g. 2). * 

14. How much time (in hours) did you and collaborators spend travelling to the 
interview? Please provide answer in numerics (e.g. 2). * 

15. How much time (in hours) did you and collaborators spend attending the interview? 
Please provide answer in numerics (e.g. 2). * 

10. Prior experience of H2020 or EU framework programmes 

264 

 
 
  
 
 
  
 
 
 
 
  
 
  
 
  
 
16. Was this your first application to H2020 (or previous framework programmes) for 
funding? * 

  Yes 

  No 

11. Prior experience of H2020 or EU framework programmes 

17. Approximately how many previous grant applications have you submitted to H2020 
(or previous framework programmes)? Please provide answer in numerics (e.g. 2). * 

18. Were these applications for the same programme or for different programmes? * 

  Same programme 

  Different programme 

  Both (same and different) 

  N/A 

12. H2020 application success 

19. Was your most recent H2020 application successful? * 

  Yes 

  No 

  Waiting to hear 

20. How many people were involved in the preparation of your most recent proposal? * 

21. Overall, how useful was the feedback you received during the process? * 

  Extremely useful 

  Very useful 

  Somewhat useful 

  Not so useful 

  Not at all useful 

265 

 
 
 
  
 
 
 
  N/A 

22. If applicable, please provide any reasons why the feedback was/was not useful. 

23. How user-friendly did you find the proposal submission tool? * 

  Extremely user-friendly 

  Very user-friendly 

  Somewhat user-friendly 

  Not so user-friendly 

  Not at all user-friendly 

  N/A 

13. Prior experience of grant applications 

24. On average, how many grant applications have you participated in within a single 
year overall (i.e. to different funding bodies)? Please provide answer in numerics (e.g. 2). 
* 

25. Approximately, how many years' experience do you have working in your current 
field? Please provide answer in numerics (e.g. 2). * 

26. Did you find the process of submitting the application particularly time-consuming 
compared to other funding schemes that you have applied to? * 

  Yes 

  No 

  Unsure 

27. Considering your own involvement, which aspects (if any) of the process were 
particularly time-consuming (tick all that apply)? 

  First-stage submission 

  Full submission 

266 

 
 
  
 
 
 
 
  
  
  Interview 

  Other (please specify): 

14. Future Participation 

28. Would you participate again as an applicant? * 

  Yes 

  No 

  Unsure 

15. Applicant Information 

29. In which organisation do you currently work, or did you work most recently? * 

  Higher Education Institution 

  Non-research International Organisation (Association of States) 

  Non-research Private Non-profit 

  Non-research Public Sector 

  Non-research commercial sector – including SME’s, among which consultancy firms 

  Private Non-profit Research Centres 

  Private/Commercial Research Centres 

  Public Research Centres (e.g. International Research Centres and Joint Research 
Centres) 

  Other organisation type 

30. What is your gross annual salary (in Euros)? * 

   <20000 

  20000 - 30000 

  30000 - 40000 

  40000 - 50000 

  50000 - 60000 

267 

 
 
  
 
   >60000 

  Prefer not to say 

31. In which country do you currently work? * 

  EU Member States or the UK 

  Associated Countries/partially associated countries (or accession formalities in 
process) 

  Other (please specify): 

16. EU Member States or the UK 

32. Please select EU Member State or the UK * 

  Austria 

  Belgium 

  Bulgaria 

  Cyprus 

  Croatia 

  Czechia 

  Denmark 

  Estonia 

  Finland 

  France 

  Germany 

  Greece 

  Hungary 

  Ireland 

  Italy 

  Latvia 

268 

 
 
  
 
  Lithuania 

  Luxembourg 

  Malta 

  Netherlands 

  Poland 

  Portugal 

  Romania 

  Slovakia 

  Slovenia 

  Spain 

  Sweden 

  United Kingdom 

17. Associated Countries/Partially Associated Countries 

33. Please select Associated Country or partially Associated Country * 

  Albania 

  Bosnia and Herzegovina 

  Faroe Islands 

  FYROM 

  Iceland 

  Israel 

  Moldova 

  Montenegro 

  Norway 

  Serbia 

  Switzerland 

269 

 
 
  Turkey 

18. Country of origin 

34. What is your country of origin? * 

  EU Member States or the UK 

  Associated Countries/partially associated countries (or accession formalities in 
process) 

  Other (please specify): 

19. EU Member States or the UK 

35. Please select EU Member State or the UK * 

  Austria 

  Belgium 

  Bulgaria 

  Cyprus 

  Croatia 

  Czechia 

  Denmark 

  Estonia 

  Finland 

  France 

  Germany 

  Greece 

  Hungary 

  Ireland 

  Italy 

  Latvia 

270 

 
 
  
 
  Lithuania 

  Luxembourg 

  Malta 

  Netherlands 

  Poland 

  Portugal 

  Romania 

  Slovakia 

  Slovenia 

  Spain 

  Sweden 

  United Kingdom 

20. Associated Countries/Partially Associated Countries 

36. Please select Associated Country or partially Associated Country * 

  Albania 

  Bosnia and Herzegovina 

  Faroe Islands 

  FYROM 

  Iceland 

  Israel 

  Moldova 

  Montenegro 

  Norway 

  Serbia 

  Switzerland 

271 

 
 
  Turkey 

21. Additional Applicant Information 

37. What is your age group? * 

  Under 35 

  35 - 49 

  50 - 59 

  60+ 

  Prefer not to say 

38. What is your gender? * 

  Female 

  Male 

  Prefer not to say 

  Prefer to self-describe as (non-binary, gender-fluid, agender, please specify) 

22. Any other comments 

39. Are there any other comments that you would like to add? 

2 

EU Proposal Evaluation Survey - Evaluators 

1. Welcome Page 

RAND Europe is conducting a study for DG RTD evaluating the peer review processes 
involved in H2020 funding. This study aims to help ensure that processes in research 
funding systems are consistent with scientific values of rigour and objectivity, while 
delivering decisions efficiently and effectively. As part of this assessment, we are 
conducting a survey of key stakeholders across H2020’s funding processes. 

2. Privacy Information 

About the project 
RAND Europe is conducting a study for DG RTD evaluating the peer review processes 
involved in H2020 funding. This study aims to help ensure that processes in research 
funding systems are consistent with scientific values of rigour and objectivity, while 
delivering decisions efficiently and effectively. As part of this assessment, we are 

272 

 
 
  
 
  
 
 
 
 
conducting a survey of key stakeholders across H2020’s funding processes. 
Who are we? 
RAND Europe is an independent not-for-profit research organisation 
(www.rand.org/randeurope). 
Why are we contacting you? 
Because of your knowledge and experience of H2020’s funding application and decision-
making processes. 
What personal data do we hold? 
From data provided by DG RTD we have collated information on stakeholders including 
name, title, organisation and email address so that we can invite you to participate. We 
are holding this data in compliance with Article 14 of the GDPR. 
Who is handling this information? 
DG RTD is the data controller and is responsible for personal information in relation to 
this project. RAND Europe is undertaking this work for the DG RTD and is acting as a 
data processor. The data we are collecting on behalf of DG RTD will be processed in line 
with the DG RTD privacy policy. 
What is the legal basis for processing your data? 
The personal data we are processing includes your contact details that have been shared 
with us by the DG RTD Team. We are using these solely to invite you to participate in 
this study and to manage data collection. We do not anticipate that your rights and 
freedoms will be affected as a result of our use of your data in this way. We are therefore 
processing your data on the basis that it is within the legitimate interests of DG RTD to 
undertake this task. 
Any responses to the survey will be confidential and your contact details will be 
separated from your responses prior to analysis. We will not ask you to enter your 
contact details as part of the survey process. 
How long will we keep this data? 
RAND Europe will retain your contact details for up to one year after the end of the 
research project, after which time your contact details will be deleted from the project 
files. 
What are your rights? 
You have the right to request access to and rectification or erasure of personal data. You 
can also request us to place restrictions on our processing of your data and have the 
right to object to our processing as well as the right to data portability. To exercise any 
of these rights or if you have any questions, please contact RAND Europe (as the data 
processor) and DG RTD (as the data controller). 
How do you contact us to exercise these rights? 
You may contact the RAND Europe data protection officer at redpo@randeurope.org, 
citing "20141 Study on the Proposal Evaluation System of the EU R&I Framework 
Programme" in the subject line. 
If you have concerns that you feel we are unable to address about our use of personal 
data you are able to refer this to the Information Commissioner’s Office (ICO) at 
https://ico.org.uk// 
What happens if I agree to participate in the survey? 
We would like to ask you to complete a survey to help us understand the current 
proposal evaluation processes within H2020, with a specific focus on peer review. 
Responses to the survey will be confidential. We will not ask you to enter your contact 
details as part of the process. 
We hope you will be happy to inform this important study for DG RTD. 
If you would like any further information regarding the project please contact the study 
team at fundingsurveyEU@randeurope.org. 

3. Information about survey completion 

The survey should take no more than 15 minutes of your time. Your participation is 
entirely voluntary, and you may choose to change your mind about participating at any 
time before, during or after the survey. You do not have to give any reasons for 
withdrawing but we draw your attention to your rights to delete your data at the bottom 

273 

 
 
 
of this form. 

As outlined in the privacy information notice, the data you provide will be kept until 12 
months after the project ends. The data you provide will be deleted at this point and the 
cross-analysed data will be kept indefinitely in an anonymised form. It is our legitimate 
interest to obtain and retain information on your name, organizational affiliation and role, 
as this is necessary to facilitate contact with you for the project and in doing so does not 
affect your rights or freedoms. Retention of your information is subject to your rights 
below. All records will be kept in compliance with the General Data Protection Regulation 
(GDPR) 2018. This survey will be conducted by RAND Europe staff so the data will be 
recorded and stored in accordance with RAND Europe's procedures. Further information 
on RAND Europe's privacy policy can be found here: 
https://www.rand.org/randeurope/privacy.html. 

If you have any queries about any aspect of this project please contact the study team 
on fundingsurveyEU@randeurope.org. 

Your legal rights 
RAND Europe operates in accordance with EU law including GDPR. You are provided with 
certain rights that you may have the right to exercise through us. In summary those 
rights are: 
(1) To access, correct or erase your data; 
(2) To object to the processing of your data; 
(3) To request that our processing of your data is restricted; 
(4) To request that your data be transferred; 
(5) To withdraw your consent for us to process your data. 
If you wish to exercise any of these rights please contact the RAND Europe Data 
Protection Officer by email at REdpo@rand.org or in writing to Data Protection Officer, 
RAND Europe, Westbrook Centre, Milton Road, Cambridge, CB4 1YG, UK. 

4. Consent form to participate in survey 

1. I confirm that I have read the information sheet on participation in the survey. * 

  Yes 

  No 

2. I understand that my participation is voluntary and that I am free to withdraw at any 
time without giving any reason, without being penalised in any way or my legal rights 
being affected. * 

  Yes 

  No 

3. I agree to participate in this survey * 

  Yes 

  No 

5. Survey on the EU R&I proposal evaluation system 

274 

 
 
 
 
 
 
Instructions: Please answer the following questions with regard to the last occasion when 
you were an evaluator for a H2020 scheme. Where a time estimation is asked for, this is 
the time spent on the activity in question. 

4. Thinking back to the last time you acted as an evaluator for H2020, which funding 
stream did you act as an evaluator for? Please select only one. * 

  Access to risk finance 

  Climate action, environment, resource efficiency and raw materials 

  Cross-cutting activities (Focus Areas) 

  ERC Grants 

  Europe in a changing world - Inclusive, innovative and reflective societies 

Food security, sustainable agriculture and forestry, marine and maritime and inland 
water research 

  Future and Emerging Technologies (FET) 

  Health, demographic change and wellbeing Recognition Prize 

  Innovation in SMEs Inducement Prize 

  Joint Technology Initiatives (JTIs) 

  LEIT - Information and Communication Technologies 

LEIT - Nanotechnologies, Advanced Materials, Biotechnology and Advanced 
Manufacturing and Processing 

  LEIT - Space Science with and for society 

  Marie Skłodowska-Curie actions (MSCA) 

  Research infrastructure (RI) 

  Secure societies - Protecting freedom and security of Europe and its citizens 

  Secure, clean and efficient energy 

  Smart, green and integrated transport 

  Spreading excellence and widening participation 

275 

 
 
  
  
  SME Instrument 

  EIC Pilot 

Other (please specify): 

6. Single/Multi-stage submission process 

5. Was this process a single-stage or multi-stage submission? In a single-stage 
submission process applicants have one deadline to submit their full application, whereas 
for the multi-stage submission process, applicants have two deadlines, whereby they 
initially submit a short proposal, and then are invited to submit a full proposal if the short 
proposal is successful. * 

  Single-stage 

  Multi-stage 

7. Proposal submission process 

6. On average, how much time (in hours) did you spend reviewing one first-stage 
proposal submission as an expert reviewer? Please provide answer in numerics (e.g. 2). 
* 

7. Approximately, how many applications did you review for the first proposal submission 
stage? Please provide answer in numerics (e.g. 2). * 

8. Full proposal submission process 

8. On average, how much time (in hours) did you spend reviewing one full proposal 
submission as an expert reviewer? Please provide answer in numerics (e.g. 2). * 

9. Approximately, how many applications did you review for the full proposal submission 
stage? Please provide answer in numerics (e.g. 2). * 

276 

 
 
  
  
 
  
 
 
 
  
 
 
 
  
 
 
 
 
 
9. Consensus group discussion 

10. When you last acted as an evaluator, were you involved in the consensus group 
discussion? * 

  Yes 

  No 

10. Consensus group discussion meeting 

11. When you last acted as an evaluator, how much additional time (in hours) did you 
spend on preparing for the consensus group discussion meeting? Please provide answer 
in numerics (e.g. 2). * 

12. When you last acted as an evaluator, how much time (in hours) did you spend 
travelling to the consensus group discussion meeting? Please provide answer in numerics 
(e.g. 2). * 

13. When you last acted as an evaluator, how much time (in hours) did you spend 
attending the consensus group discussion meeting? Please provide answer in numerics 
(e.g. 2). * 

14. When you last acted as an evaluator, approximately how many applications did you 
discuss in the consensus group discussion meeting? Please provide answer in numerics 
(e.g. 2). * 

15. When you last acted as an evaluator, approximately how many individuals attended 
the consensus group discussion meeting? Please provide answer in numerics (e.g. 2). * 

11. Panel review stage 

16. When you last acted as an evaluator, were you involved in the panel review stage? * 

277 

 
 
  
 
 
 
  
 
 
  
 
  
 
  
 
  
  Yes 

  No 

12. Panel review meeting 

17. When you last acted as an evaluator, approximately how much additional time (in 
hours) did you spend preparing for the panel review meeting? Please provide answer in 
numerics (e.g. 2) and refer to the total time spent (i.e. for all proposals discussed at the 
meeting). * 

18. When you last acted as an evaluator, approximately how much time (in hours) did 
you spend travelling to the panel review meeting? Please provide answer in numerics 
(e.g. 2). * 

19. When you last acted as an evaluator, how much time (in hours) did you spend 
attending the panel review meeting? Please provide answer in numerics (e.g. 2). * 

20. To the best of your knowledge, how many applications were discussed in the last 
panel review meeting you attended? Please provide answer in numerics (e.g. 2). * 

21. To the best of your knowledge, in the last panel review meeting you attended, how 
many other individuals attended? Please provide answer in numerics (e.g. 2). * 

278 

 
 
 
  
 
 
 
 
  
 
 
 
 
  
 
 
 
 
  
 
 
 
 
  
 
13. Interview 

22. When you last acted as an evaluator, were you involved in an interview process? * 

  Yes 

  No 

14. Interview process 

23. When you last acted as an evaluator, approximately how much time (in hours) did 
you spend preparing for the interviews? Please provide answer in numerics (e.g. 2). * 

24. When you last acted as an evaluator, approximately how much time (in hours) did 
you spend overall conducting the interviews? Please provide answer in numerics (e.g. 2). 
* 

25. When you last acted as an evaluator, approximately how much time (in hours) did 
you spend travelling to the interviews? Please provide answer in numerics (e.g. 2). * 

26. How many different applicants did you interview? Please provide answer in numerics 
(e.g. 2). * 

27. How many evaluators, on average, were evolved in conducting each interview? 
Please provide answer in numerics (e.g. 2). * 

279 

 
 
 
 
 
  
 
 
 
 
  
 
 
 
 
  
 
 
 
 
  
 
 
 
 
15. Prior experience of H2020 or EU framework programmes 

28. Was this the first time you were involved in the review process for H2020 funding (or 
previous framework programmes)? * 

  Yes 

  No 

16. Prior experience of H2020 or EU framework programmes 

29. Approximately how many times have you acted as an evaluator for a H2020 (or 
previous framework programme) proposal evaluation process? Please provide answer in 
numerics (e.g. 2). * 

17. Prior experience of grant application review 

30. On average, how many grant applications do you review in a single year as an 
external peer reviewer (i.e. to different funding bodies)? Please provide answer in 
numerics (e.g. 2). * 

31. Approximately, how many years' experience do you have working in your current 
field? Please provide answer in numerics (e.g. 2). * 

32. Did you find the process of reviewing applications particularly time-consuming 
compared to other funding schemes that you have reviewed for? * 

  Yes 

  No 

  Unsure 

280 

 
 
  
 
 
 
  
 
 
 
  
 
 
 
 
  
 
  N/A 

33. Which aspects (if any) of the process were particularly time-consuming (tick all that 
apply)? 

  Reviewing first-stage proposal submission 

  Reviewing full proposal submission 

  Interviews 

  Consensus group meeting 

  Panel review meeting 

Other (please specify): 

18. Future Participation 

34. Would you participate again as an evaluator? * 

  Yes 

  No 

  Unsure 

19. Reviewer Information 

35. In which organisation do you currently work, or did you work most recently? * 

  Higher Education Institution 

  Non-research International Organisations (Association of States) 

  Non-research Private Non-profit 

  Non-research Public Sector 

  Non-research commercial sector – including SME’s, among which consultancy firms 

  Private Non-profit Research Centres 

  Private/Commercial Research Centres 

281 

 
 
 
  
  
 
Public Research Centres (e.g. International Research Centres and Joint Research 
Centres) 

  Other organisation type 

36. What is your gross annual salary (in Euros)? * 

   <20000 

  20000 - 30000 

  30000 - 40000 

  40000 - 50000 

  50000 - 60000 

   >60000 

  Prefer not to say 

37. In which country do you currently work? * 

  EU Member States or the UK 

Associated Countries/partially associated countries (or accession formalities in 
process) 

Other (please specify): 

20. EU Member States or the UK 

38. Please select EU Member State or the UK. * 

  Austria 

  Belgium 

  Bulgaria 

  Cyprus 

  Croatia 

  Czechia 

282 

 
 
  
 
  
  
  
 
  Denmark 

  Estonia 

  Finland 

  France 

  Germany 

  Greece 

  Hungary 

  Ireland 

  Italy 

  Latvia 

  Lithuania 

  Luxembourg 

  Malta 

  Netherlands 

  Poland 

  Portugal 

  Romania 

  Slovakia 

  Slovenia 

  Spain 

  Sweden 

  United Kingdom 

21. Associated Countries/Partially Associated Countries 

283 

 
 
39. Please select Associated Country or partially Associated Country * 

  Albania 

  Bosnia and Herzegovina 

  Faroe Islands 

  FYROM 

  Iceland 

  Israel 

  Moldova 

  Montenegro 

  Norway 

  Serbia 

  Switzerland 

  Turkey 

22. Country of origin 

40. What is your country of origin? * 

  EU Member States or the UK 

Associated Countries/partially associated countries (or accession formalities in 
process) 

Other (please specify): 

23. EU Member States or the UK 

41. Please select EU Member State or the UK * 

  Austria 

  Belgium 

  Bulgaria 

284 

 
 
  
  
  
 
  Cyprus 

  Croatia 

  Czechia 

  Denmark 

  Estonia 

  Finland 

  France 

  Germany 

  Greece 

  Hungary 

  Ireland 

  Italy 

  Latvia 

  Lithuania 

  Luxembourg 

  Malta 

  Netherlands 

  Poland 

  Portugal 

  Romania 

  Slovakia 

  Slovenia 

285 

 
 
  Spain 

  Sweden 

  United Kingdom 

24. Associated Countries/Partially Associated Countries 

42. Please select Associated Country or partially Associated Country * 

  Albania 

  Bosnia and Herzegovina 

  Faroe Islands 

  FYROM 

  Iceland 

  Israel 

  Moldova 

  Montenegro 

  Norway 

  Serbia 

  Switzerland 

  Turkey 

25. Additional Reviewer Information 

43. What is your age group? * 

  Under 35 

  35 - 49 

  50 - 59 

286 

 
 
 
  60+ 

  Prefer not to say 

44. What is your gender? * 

  Female 

  Male 

  Prefer not to say 

Prefer to self-describe as (non-binary, gender-fluid, agender, please specify) 

26. Any other comments 

45. Are there any other comments you would like to add? 

287 

 
 
  
  
 
  
 
 
 
 
 
ANNEX C. INTERVIEW PROTOCOL FOR CASE STUDY DEVELOPMENT 

Interview Protocol 

RAND Europe has been commissioned by the  European Commission Directorate  General 
for  Research  and  Innovation  to  conduct  a  comprehensive  review  of  proposal  evaluation 
processes in EU R&I framework programmes The study will feed into the ex-post evaluation 
of Horizon 2020 and where relevant inform the development and adaptation of the future 
proposal evaluation approaches. In particular, we are seeking to assess (1) the relevance 
of the current proposal evaluation system to achieve the objectives of Horizon 2020; (2) 
the effectiveness and efficiency of the process; and what lessons can be learned in view of 
adapting the proposal evaluation system in the future. 

As part of this study, we are completing focussed case studies of other organisations to 
examine alternative approaches to proposal evaluation and learn from best practice. We 
are conducting a limited number of interviews with key stakeholders such as yourself to 
verify our understanding of proposal evaluation at [agency name], as well as to understand 
your  views  on  any  challenges  encountered,  and  how  these  were  resolved  at  [agency 
name]. 

We would like to reiterate that information shared in the interviews will be confidential and 
that the study is aimed at learning purposes. In particular, this interview will be used to 
develop a better understanding of the proposal evaluation system at your organisation. 

The interview will last between 45 and 60 minutes. 

Are you happy to proceed on this basis? 

Introduction 

1.  Can  you  briefly  describe  your  funding  programme  and  its  main  objectives?  (e.g. 

budget, who do you fund and why) 

2.  What is your role in the evaluation of proposals at [agency name]? 

Funding Processes 

3.  Could you provide me with an overview of proposal evaluation at [agency name]? 

a.  What are the different steps of the process? Prompts: Is there a two-stage 

process, are there interviews etc. 

i.  Where there are multiple stages - what differences are there between 

both stages (e.g. length of proposal, submitted materials, etc) 

b.  Is it a single or multi-step evaluation? 

i.  Are applications rejected after each step?   

ii.  Are  the  following  aspects  part  of  the  review  process:  reviewer 
training  prior  to  evaluation,  independent  review,  panel  review, 
consensus meeting, interviews? 

4.  To what extent do evaluation processes differ by funding stream at [agency name]? 

a.  Where are the most pronounced differences, and why do they exist? 

b.  Are  there  any  differences  in  process  between  research  and  innovation 

funding streams, for example? 

288 

 
 
c.  Are there differences in process between researcher and challenge led calls? 

d.  Are any aspects of the process done remotely? 

e.  What is your average time to grant? Do you think it could be shortened? 

5.  What is the rationale for this system? 

6.  Do you review your proposal evaluation processes? 

a.  If so, what does this consist of? How frequently? Formal or informal? 

7.  Have you made any changes to the evaluation process in recent years? 

a.  If yes, 

i.  What  have  these  been  and  what  was  the  rationale  behind  the 

changes? 

ii.  How have the changes been received? 

iii.  Have the changes had the impact you expected? 

b.  If no, have you considered introducing changes to the evaluation process? 

i.  If yes, why are these changes being considered? 

ii.  Why have you not introduced these changes? 

8.  Do you provide feedback to applicants? 

a.  What is the purpose of the feedback? 

b.  At what stage of the process do you offer feedback? Why is this the case? 

c.  What do you think ‘good’ feedback looks like? 

d.  Can applicants respond to feedback? 

e.  How  have  you  managed  the  need  for  good  feedback  and  evaluators’ 

workloads? 

9.   To what extent has a burden been placed on evaluators in reviewing proposals? 

a.  What steps have you taken to reduce the burden? 

10. What are your assessment criteria? 

a.  What was the rationale for these particular criteria? 

b.  What is the weighting of different criteria, are they all equal? 

11. What steps have you taken to fund innovative R&I? 

a.  How do you define innovation? 

b.  How  do  you  conceptualise  funding  risky  innovative  project?  What  does  it 

take to ‘produce’ innovation? 

289 

 
 
c.  (if relevant) How effectively does joint panel and consensus decision making 

support these objectives? 

12. Do you have evidence on the extent to which the proposal evaluation system you 

have in place funds the ‘best’ research and innovation projects? 

a.  How do you know that you are funding research and innovation that meets 

your objectives? 

13. How do you ensure proposal evaluation is a fair process without bias? 

a.  Have you put in place, or considered, blind evaluations for example? 

b.  Do you have any evidence on the equity of the process? 

Final comments 

14. Are there any documents or data that you consider relevant for us to review for this 

study? Is there anyone else we should speak to? 

15. Is there anything else you would like to add that we have not already discussed? 

290 

 
 
ANNEX D. INTERVIEWEE CODES 

Table 83: Interviewee codes 

Interviewee code 

Organisation 

INT01 

INT02 

INT03 

INT04 

INT05 

CFI 

CFI 

CFI 

NCN 

Innovate UK 

291 

 
 
 
 
 
 
 
Getting in touch with the EU 

IN PERSON 
All over the European Union there are hundreds of Europe Direct information centres. 
You can find the address of the centre nearest you at: https://europa.eu/european-union/contact_en 

ON THE PHONE OR BY EMAIL 
Europe Direct is a service that answers your questions about the European Union. 
You can contact this service: 
– by freephone: 00 800 6 7 8 9 10 11 (certain operators may charge for these calls), 
– at the following standard number: +32 22999696, or 
– by email via: https://europa.eu/european-union/contact_en 

Finding information about the EU 

ONLINE 
Information about the European Union in all the official languages of the EU is available on the Europa 
website at: https://europa.eu/european-union/index_en 

EU PUBLICATIONS 
You can download or order free and priced EU publications from: 
https://op.europa.eu/en/publications. Multiple copies of free publications may be obtained by 
contacting Europe Direct or your local information centre (see https://europa.eu/european-
union/contact_en). 

EU LAW AND RELATED DOCUMENTS 
For access to legal information from the EU, including all EU law since 1952 in all the official language 
versions, go to EUR-Lex at: http://eur-lex.europa.eu 

OPEN DATA FROM THE EU 
The EU Open Data Portal (http://data.europa.eu/euodp/en) provides access to datasets from the EU. 
Data can be downloaded and reused for free, for both commercial and non-commercial purposes. 

 
 
 
 
 
 
 
 
The interim evaluation of Horizon 2020, the 
report by the Lamy Group, and other 
stakeholder feedback, noted that the current 
system for evaluating proposals is in general 
appreciated and one of the reasons for 
widely acknowledged successful 
implementation of Horizon 2020. With a 
view of maintaining and further developing a 
modern, world-class system, and identifying 
further simplifications where possible, the 
study assessed the performance of the 
Horizon 2020 proposal submission and 
evaluation process, including by taking into 
account international practices.  

Studies and reports