COMMISSION STAFF WORKING DOCUMENT Evaluation of the European Research Council Executive Agency (ERCEA) Accompanying the document Report from the Commission to the European Parliament, the Council and the Court of Auditors Evaluation of the Consumers, Health, Agriculture and Food Executive Agency, the Executive Agency for Small and Medium-Size Enterprises, the Innovation and Networks Executive Agency, the Education, Audiovisual and Culture Executive Agency, the Research Executive Agency and the European Research Council Executive Agency
Date: 2020-04-29
Institutions: European Commission, Secretariat-General
Subjects: EU aid, EU financing, EU programme, executive agency, financial control, performance audit, research

EN 

  EN 

   EUROPEAN COMMISSION  Brussels, 29.4.2020  SWD(2020) 74 final  COMMISSION STAFF WORKING DOCUMENT Evaluation of the European Research Council Executive Agency (ERCEA) Accompanying the document Report from the Commission to the European Parliament, the Council and the Court of Auditors Evaluation of the Consumers, Health, Agriculture and Food Executive Agency, the Executive Agency for Small and Medium-Size Enterprises, the Innovation and Networks Executive Agency, the Education, Audiovisual and Culture Executive Agency, the Research Executive Agency and the European Research Council Executive Agency {COM(2020) 184 final} - {SWD(2020) 73 final} - {SWD(2020) 75 final} - {SWD(2020) 76 final} - {SWD(2020) 77 final} - {SWD(2020) 78 final}  
 
 
TABLE OF CONTENTS 

1. 

INTRODUCTION .............................................................................................................. 2 

2.  BACKGROUND ................................................................................................................ 3 

2.1 Description of the intervention and its objectives ............................................................ 3 

2.2 Baseline and points of comparison ................................................................................... 4 

3. 

IMPLEMENTATION / STATE OF PLAY ........................................................................ 5 

4.  METHOD ........................................................................................................................... 7 

4.1 Short description of methodology .................................................................................... 7 

4.2 Limitations and robustness of findings ............................................................................. 8 

5.  ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS .......................... 8 

5.1 Effectiveness ..................................................................................................................... 8 

5.2 Efficiency ........................................................................................................................ 12 

5.3 Coherence ....................................................................................................................... 16 

5.4 Retrospective Cost-benefit analysis ................................................................................ 17 

6.  CONCLUSIONS ............................................................................................................... 18 

ANNEX : EVALUATION QUESTIONS ................................................................................ 20 

1 

 
 
 
 
 
 
 
1. 

INTRODUCTION 

The purpose of this periodical (every three years) evaluation of the operation of the European 
Research Council Executive Agency (ERCEA) is to assess the ERCEA’s implementation of 
the  parts  of  the  EU  funding  programmes  which  have  been  entrusted  to  it.  The  ERCEA  is 
governed  by  the  Framework  Regulation  for  executive  agencies1,  its  Act  of  Establishment2, 
which sets out its mandate; the Act of Delegation3 which specifies the tasks to be carried out 
by, and the powers delegated to the Agency in order to perform its mandate; and the decision 
establishing the ERCEA Steering Committee. 

In  line  with  the  Commission’s  Better  Regulation  principles4,  the  evaluation  applies  several 
standard evaluation criteria. The evaluation assesses whether the Agency has fulfilled its tasks 
in  an  effective  and  efficient  way,  whether  there  are  overlaps  /  gaps  /  inconsistencies  in  the 
management  of  the  programme  portfolio  by  the  Agency,  and  whether  there  is  a  clear 
delineation  of  tasks  between  the  ERCEA  and  the  parent  DG  or  other  executive  agencies 
(coherence)5. 

The evaluation also assesses whether the functioning of the Agency has yielded the expected 
positive results as estimated in the ex-ante Cost-Benefit Analysis (CBA) for the delegation of 
tasks to the executive agencies6 and identify potential areas of improvement. To this end the 
estimations of the Cost-Benefit Analysis of 2013 have been tested to provide evidence on the 
validity of the assumptions made in the ex-ante scenario by considering the actual costs and 
benefits  of  programme  implementation  by  the  Agency  in  a  structured  way.  The  aspects 
covered  by  the  Cost-Benefit  Analysis  are  specified  in  Article  3(1)  of  the  Framework 
Regulation7 and the Guidelines on establishing and operating executive agencies8.  

The evaluation does not cover the achievements of the programmes managed by the ERCEA, 
which  are  subject  to  mid-term  and  ex-post  evaluations  themselves.  The  evaluation  of  the 
ERCEA nevertheless provides useful input for these programme evaluations, considering that 
the performance of the Agency affects the efficiency and effectiveness of the programmes it 
manages. 

1 http://data.europa.eu/eli/reg/2003/58/oj 
2 http://data.europa.eu/eli/dec_impl/2013/779/oj 
3http://ec.europa.eu/transparency/regdoc/index.cfm?fuseaction=list&coteId=3&year=2013&number=9428&lang
uage=EN 
4  Commission’s  Better  Regulation  Communication  COM  (2015) 215  final  and  its  accompanying  SWD  (2015) 
111 final and SWD (2015) 110 final. 
5  The  assessment  of  ‘EU  added  value’,  why  the  EU  should  act,  is  not  perceived  to  be  a  relevant  criterion,  as 
ERCEA  executes  tasks  which  the  European  Commission  has  transferred  to  it.  The  EU  added  value  of  the 
programmes that ERCEA manages is assessed as part of the programme evaluation. The previous needs which 
ERCEA were meant to address and whether they still are pertinent today (relevance) are presented in the context 
of the evaluation. 
6 Cost-Benefit analysis for the delegation of certain  tasks regarding the  implementation of Union Programmes 
2014-2020 to the executive agencies - Final report for the Commission of 19 August 2013. 
7  Identification  of  the  tasks  justifying  outsourcing,  a  cost-benefit  analysis  which  includes  the  costs  of 
coordination  and  checks,  the  impact  on  human  resources,  possible  savings  within  the  general  budgetary 
framework  of  the  European  Union,  efficiency  and  flexibility  in  the  implementation  of  outsourced  tasks, 
simplification of the procedures used, proximity of outsourced activities to  final beneficiaries, visibility of the 
EU as promoter of the EU programme concerned and the need to maintain an adequate level of know-how inside 
the Commission.  
8  Appendix  II  of  the  Guidelines  for  the  establishment  and  operation  of  executive  agencies  financed  from  the 
Union budget (C (2014) 9109 from 2 December 2014; pp. 64-72). 

2 

 
 
 
 
 
 
                                                           
The evaluation  examines the efficient  use of resources and the  effective achievement of the 
tasks  entrusted  to  it.  In  particular,  it  looks  at  whether  the  alignment  of  more  coherent 
programme  portfolios  with  the  Agency’s  core  competences  and  its  brand  identity  delivered 
the estimated qualitative benefits; whether the assembly of the management  of different  EU 
the  estimated  synergies, 
programmes  and  provision  of  support  services  delivered 
simplification  and  economies  of  scale;  whether  the  pooling  of  instruments  guaranteed 
consistent service delivery and whether there is scope for simplification and further efficiency 
gains. 

The evaluation covers all the tasks carried out by the Agency during the period 16 July 2015 
to 15 July 2018. The evaluation is supported by a study carried out by external contractors9. 

The results of this study are summarised hereafter and this evaluation will be presented to the 
European Parliament, to the Council and to the Court of Auditors in accordance with Article 
25(1) of the Framework Regulation. 

The results will feed into the reflection to assess the opportunity of expanding/ modifying the 
Agency’s  mandate  in  view  of  the  delegation  of  the  implementation  of  2021-2027  EU 
programmes. 

2.  BACKGROUND 

2.1 Description of the intervention and its objectives 

The  outsourcing  of  certain  management  tasks  to  the  Agency  according  to  the  Framework 
Regulation and the ERCEA’s Act of Establishment intends: 

  To allow the Commission to focus on its institutional tasks, i.e. tasks assigned to the 
institutions  by  the  Treaty  which  require  discretionary  powers  in  translating  political 
choices into action. Such institutional tasks should not be outsourced. 

  To enable the Commission to achieve the objectives of the delegated EU programmes 
more effectively and efficiently. According to the Cost-Benefit Analysis carried out in 
2013, delegating tasks to the ERCEA was estimated to be more cost-efficient than an 
in-house  scenario.  In  addition  the  CBA  showed  that  the  Agency  has  high  quality 
programme  management  and  service  delivery,  visibility  and  existing  communication 
outreach channels which have proved effective so far. Making use of the accumulated 
experience and expertise of the Agency would lead to efficiency gains. 

The original objective of the intervention is entrusting the Agency with the implementation of 
the following EU programmes in parts or fully: 

  Within the Horizon 2020 Framework Programme for Research & Innovation (Horizon 
2020), the specific objective ‘Strengthening frontier research through the activities of 
the European Research Council’ of Part I ‘Excellence Science’, 

  Within the 7th Framework Programme (FP7), the implementation of the legacy of the 

Specific Programme ‘Ideas’. 

The  ERCEA  has  one  parent-DG,  namely  Directorate-General  for  Research  and  Innovation 
(DG RTD). 

9 Study supporting the evaluation of the European Research Council Executive Agency (2015-2018), final report 
prepared by CSES and PPMI, March 2019. 

3 

 
 
 
 
 
 
 
 
                                                           
The  Agency  works  closely  with  the  Common  Support  Centre  (CSC),  now  named  Common 
Implementation  Centre  (CIC),  administratively  part  of  DG  RTD,  which  provides  the 
framework  for  harmonised  implementation  of  the  EU  Research  and  Innovation  (R&I) 
programmes for all R&I DGs family of implementing bodies, including Executive Agencies 
and Joint Undertakings. The CIC is responsible for defining common business processes for 
grant  management,  and  providing  IT,  legal,  audit  and  information  and  data  services  to  the 
Horizon 2020 implementing bodies. 

With  regards  to  the  tasks  delegated  in  the  framework  of  Horizon  2020  and  FP7  legacy,  the 
ERCEA performs the following programme implementation tasks: 

  managing programme implementation and specific projects on the basis of the relevant 
work  programmes  (WP)  established  by  the  Scientific  Council  of  the  European 
Research Council (ERC Scientific Council) and adopted by the Commission; 

  adopting  the  instruments  of  budget  execution  for  revenue  and  expenditure  and 

carrying out all the operations necessary for the management of the programme; 

  providing  support  in  programme  implementation  where  the  Commission  has 

empowered it to do so in the instrument of delegation; 

  providing support to the ERC Scientific Council in the conduct of all of its tasks. 

2.2 Baseline and points of comparison 

The current evaluation of the ERCEA operations during 2015-2018 assesses the actual costs 
and benefits of programme implementation by the ERCEA (executive agency scenario) when 
compared with the alternative scenario of management by the Commission services (in-house 
scenario). 

Accordingly, the reference point for the present ERCEA evaluation is the 2013 ex ante Cost-
Benefit Analysis, and the Specific Financial Statement (SFS) of the ERCEA. 

Wherever  possible,  the  analysis  of  the  ERCEA  performance  during  the  period  of  reference 
assesses  the  progress  compared  to  the  two  previous  evaluations,  covering  the  Agency’s 
operations  during  2009-2012  and  2012-2015.  The  current  evaluation  covers  the  period 
between  the  beginning  of  the  second  half  of  2015  and  the  end  of  the  first  half  of  2018. 
However,  wherever  possible  and  relevant,  and  provided  that  all  the  necessary  data  was 
available, the analysis was extended to include the whole of 2015 and, at times, 2014. 

Over the period 2009-2012 the savings resulting from the delegation of tasks to the ERCEA 
have  been  evaluated  at  around  EUR  45  million  compared  to  the  in-house  scenario,  and  at 
EUR 46.5 million over the period 2012-2015. The 2013 CBA estimated the efficiency gains 
over the period 2014-2024 at EUR 79 million and at EUR 41 million over the period 2015-
2018.  The  total  number  of  full  time  equivalents  (FTEs)  required  to  manage  the  relevant 
programmes in year 2020, the peak programming year in terms of workload, was estimated at 
529  for  ERCEA.  The  total  estimated  operational  budget  entrusted  to  ERCEA  in  2020  is 
around EUR 2 322.956 million in commitment appropriations and EUR 1 787.180 million in 
payment appropriations. Considerable efficiency gains were expected as compared to the in-
house scenario over the period, along with non-quantifiable benefits such as improved quality 
of programme management and service delivery, improved visibility of the EU programmes 
and proximity to beneficiaries. 

4 

 
 
 
 
 
 
 
 
3. 

IMPLEMENTATION/STATE OF PLAY 

Established  in  2007  and  autonomous  since  July  2009,  the  ERCEA  was  entrusted  with  the 
management  of  EU  actions  in  the  field  of  frontier  research  to  perform  the  task  of 
implementing  the  Specific  Programme  Ideas.  The  Agency’s  mandate  was  extended  by  the 
Commission  Implementing  Decision  No  2013/779/EU,  which  came 
into  effect  on 
1 January 2014.  The  ERCEA’s  renewed  mandate  under  Horizon  2020  concentrates  on  the 
implementation  of  the  ERC  programme  within  the  “Excellent  Science”  pillar,  thus  bringing 
continuity in relation to the FP7 period. 

The new mandate did not bring any changes in the governance of the Agency. A new written 
agreement  (WA)  between  ERCEA  and  DG  RTD  was  signed  in  April  2014  and  was  further 
amended  in  December  201510.  The  new  WA  defined  the  modalities  and  procedures  of 
interaction between ERCEA and its parent DG. The distribution of tasks and responsibilities, 
as presented in the WA, is included in sections 5.1 and 5.3 of this document. Compared to this 
reference, this evaluation assesses whether this distribution of roles has been respected for the 
2015-2018 period, and the effects of the intervention on the coherence and the efficiency of 
ERCEA’s action. 

What is specific to the ERCEA compared with other executive agencies is that the scientific 
funding strategy focuses solely on bottom-up, frontier research and scientific excellence, and 
methodologies  are  defined  by  the  European  Research  Council  (ERC)  via  its  independent 
Scientific Council11 (ScC) and then adopted by the Commission, while the executive agency 
serves as a dedicated implementation structure and is in charge of the day to day evaluation 
and  grant  management.  Another  specificity  relates  to  the  process  of  selection  of  experts  for 
the evaluation process: made by the ERC, it aims at ensuring the excellence of the evaluation 
and attracting top-class experts who may not otherwise be reached under procedures used for 
other research family programmes12. 

The  ERC  provides  funding  via  five  different  types  of  grants.  Four  of  them  were  launched 
under FP7 (Starting Grant (StG), Consolidator grant (CoG), Advanced grant (AdG) and Proof 
of  Concept  grant  (PoC)),  and  have  been  continued  under  Horizon  2020.  The  Synergy  grant 
(SyG)  was  launched  in  FP7  as  a  pilot,  and  relaunched  as  a  fully-fledged  grant  scheme  in 
201713  under  Horizon  2020.  Whereas  the  StG,  CoG,  AdG  and  PoC  are  all  single  grants 
awarded to  a principal investigator  (PI) who  can then recruit a team  of researchers to  assist 
with  the  research  project,  the  SyG  supports  small  teams  of  scientists  who  wish  to  jointly 
address  research problems  through  frontier  research. PoC  funding is  made available only to 
those  who  already  have  an  ERC  award  to  establish  proof  of  concept  of  an  idea  that  was 
generated in the course of their ERC-funded projects. 

10 Written  Agreement between  the European Research  Council Executive  Agency and the Directorate  General 
for Research and Innovation – Modalities and Procedures of Interaction, as last amended on 17 December 2015. 
11 The Scientific Council is the decision making body of the ERC and sets the ERC’s scientific funding strategy. 
The chair of the Scientific Council is also the President of the ERC. 
12 Under other research family programmes, candidates have to register in the Commission’s central database of 
experts via the Participant Portal. The experts are then selected and afterwards contracted and managed centrally 
by REA. The ERCEA’s independent experts do not have to register via the Participant Portal. The ERC selects 
experts for their peer review tasks on the basis of a proposal from the ERC’s ScC. 
13 Under the 2018 ERC WP. 

5 

 
 
 
 
 
 
                                                           
Overview of the ERC grants scheme14  
Grant scheme  
ERC WP  
of first 
implementation 

Support for 

Starting 
grant 

Consolidator 
grant 

Advanced 
grant 

Synergy 
grant 

Proof of 
Concept 
grant 

2007 

2013 

2008 

2012 & 2013 
(piloted) 

2018 
(relaunched) 

2011 

Early-career researchers to 
start their own 
independent research team 
or programme 

Researchers to consolidate 
their own research team or 
programme 

Well-established 
researchers to develop a 
new and outstanding 
project 

A group of 2-4 researchers 
to jointly address 
ambitious research 
problems 

ERC grantees to verify the 
innovation potential of 
ideas arising from ERC 
funded projects 

Qualifications 

Funding, € 

Duration, 
years 

PhD + 2-7 years’ 
experience 

 1 500 000  

 5  

PhD + 7-12 
years’ experience 

 2 000 000 

 5  

10-year excellent 
scientific track 
record 

Competitive 
scientific track 
record, 
commensurate to 
their career stage 

Holders of 
previous ERC 
grant 

 2 500 000 

 5  

 10 000 000 

 6  

 150 000 

1,5 

Overall,  the  actual  workload  of  the  ERCEA  during  the  evaluation  period  generally 
corresponded to the initial 2013 CBA estimates. The ERCEA’s workload is determined by the 
allocated  operational  budget,  the  number  of  proposals  received  and  the  number  of  grants 
concluded/projects managed. The operational budget managed by the ERCEA was significant 
in  scale  and  totalled  EUR  9,172  million  during  the  2014–2018  period.  It  ranged  from  EUR 
1,731 million in 2014 (19% more than estimated in the CBA and SFS) to EUR 1,995 million 
in 2018 (5% more than estimated). The actual number of new grants managed by the ERCEA 
under 2014-2018 calls was 13% higher than 2013 CBA forecast (5 450 versus 4 738 grants), 
with  a  substantial  increase  especially  in  2014  and  2015  (30%  and  23%  more  than  CBA 
estimates). On the other hand, the actual number of proposals received under the 2014-2018 
calls was substantially (32%) lower than the initial CBA calculations (40 532 versus 59 500 
proposals).  This  is  a  result  of  the  restriction  rules  on  re-submission  introduced  by  the  ERC 
Scientific  Council  in  the  ERC  Work  Programmes  to  contain  the  experts’  workload  at  a 
reasonable level and hence avoid compromising evaluation quality but also to keep the overall 
success rate at an appropriate level15 of around 15%. 

The ERCEA gradually grew in size in accordance with the multi-annual planning provided in 
the SFS, from 417 at the end of 2015 to 494 staff members16 at the end of 201817. The Agency 
is based in Brussels. 

14 Under ‘Funding’, up to an additional EUR 500 000, 750 000, 1 000 000 and 4 000 000 for StG, CoG, AdG and 
SyG respectively, could be requested in the proposal to cover the  following additional extraordinary costs: (a) 
eligible “start-up” costs for Principal Investigators moving to the EU or an Associated Country from elsewhere 
as a consequence of receiving the ERC  grant and/or (b) the  purchase of  major equipment and/or (c) access to 
large facilities. 
15 The CBA estimated that it would drop to 7.6%-8.7%. 
16 Total number of staff, financed from EU Budget and contributions from EFTA/EEA and/or third countries. 

6 

 
 
 
 
                                                           
A series of organisational and procedural changes took place during the implementation of the 
ERC programme under the multiannual financial framework (MFF) of 2014-2020: 

  Adoption  of  new  business  processes  and  use  of  the  Commissions  Horizon  2020  IT 
tools and systems for the grant management process and the  electronic submission of 
proposals (i.e. Compass and SyGMa); 

  The  adoption  of  the  multiannual  human  resource  (HR)  strategy  for  2017-2020  to 
address the findings of the 2016 staff opinion survey. Based on this new strategy and in 
combination with a dedicated action plan addressing attention points identified through 
the  survey,  the  Agency  implemented  specific  measures  related  to  staff  engagement, 
internal  communication,  career  development  opportunities,  work  and  private  life 
balance  and  training,  meant  to  improve  effective  HR  management  and  increase  the 
effectiveness and flexibility of the Agency’s operations. 

4.  METHOD 

4.1 Short description of methodology 

The  evaluation  was  supported  by  a  study  carried  out  by  external  contractors.  The 
methodology  of  the  ERCEA  evaluation  was  consistent  with  the  approaches  employed  for 
similar parallel evaluations of REA, EACEA, INEA, EASME and CHAFEA and built on the 
previous  evaluation  of  the  ERCEA  operations  during  2012-2015.  Whenever  possible,  data 
was collected and the analysis extended to cover 2014 as well as the whole of 2015 and 2018, 
even if formally outside the evaluation timeframe. Specific methodological approaches were 
used  for  each  of  the  evaluation  tasks  (Regulatory  framework,  the  ERCEA  mission  and 
governance; the ERCEA performance between 2015-2018, including effectiveness, efficiency 
and coherence; the Cost-Benefit Analysis and Conclusions and Policy Recommendations) to 
ensure  data  triangulation.  The  study  was  structured  around  a  series  of  evaluation  questions 
available in Annex. 

The evaluation relied on the evidence gathered and analysed using qualitative and quantitative 
methods and included: 

  an  extensive  documentary  review  and  desk  research  of  the  ERCEA  and 
Commission  documents,  including  Annual  Work  Programmes  and  Annual  Activity 
Reports,  audit  reports  and  related  documents,  CBA  studies  and  other  financial 
documents, the Written Agreement18, previous evaluations and action plans addressing 
the  ensuing  recommendations;  the  evaluation  also  relied  on  the  results  of  the  2016 
staff opinion survey; 

  38 semi-structured in-depth interviews with Commission staff and the ERCEA staff, 
including:  Directors  and  selected  Commission  staff  in  the  parent  DG  –  DG  RTD; 
heads of units and senior staff from the Common Support Centre; the Director of the 
ERCEA, heads of departments, heads of units and heads of sectors, other employees in 
the ERCEA; 

  6  interviews  with  senior  members  of  the  Scientific  Council  (including  the  President 

and all the Vice Presidents); 

17 For comparison, REA, the largest executive agency, had 735 staff members in 2018 while CHAFEA had 59 
staff members in 2016. 
18 Written  Agreement between  the European Research  Council Executive  Agency and the Directorate  General 
for Research and Innovation – Modalities and Procedures of Interaction, 17 December 2015. 

7 

 
 
 
 
 
                                                                                                                                                                                     
  13 interviews with the ERCEA beneficiaries, unsuccessful applicants and independent 

experts; 

  1  survey  addressing  the  ERCEA’s  independent  experts  (745  answers  or  25,4% 
response rate) and analysis of 2 surveys carried out by the ERCEA earlier in 2018: a 
survey of grantees (successful applicants) (964 answers or 31% response rate) and 
host  institutions  (319  answers  or  20%  response  rate)  and  a  survey  of  unsuccessful 
applicants (693 answers or 22% response rate); 

  a retrospective cost–benefit analysis; 
  benchmarking  of  the  ERCEA’s  performance  in  the  2012-2015  and  2015-2018 
periods  and  a  comparative  analysis  and  benchmarking  of  the  various  executive 
agencies was also carried out against a set of (qualitative and quantitative) indicators. 
This enabled the evaluation team to compare the performance of ERCEA effectively 
with that of other executive agencies. 

4.2 Limitations and robustness of findings 

All  the  relevant  stakeholders  within  the  Commission,  the  ERC  Scientific  Council  and  the 
ERCEA  were  duly  consulted  via  the  interviews  programme.  An  extensive  number  of  the 
ERCEA’s  independent  experts  were  covered  by  the  dedicated  survey  carried  out  during  the 
evaluation. The evaluation used the analysed and confirmed results of two surveys carried out 
in early 2018 by the ERCEA (of beneficiaries and unsuccessful applicants) complemented by 
a number of additional interviews. However, compared with the overall population reached, 
unsuccessful applicants participated less actively in the survey than programme beneficiaries. 

No  sampling  bias  was  observed  as  the  profile  of  the  respondents  to  the  surveys  was  very 
similar to the overall population, guaranteeing statistical representativeness. The non-response 
bias  (not  all  characteristics  of  the  group  that  did  not  reply  had  been  captured  in  full)  was 
mitigated  through  triangulation  with  the  results  of  follow  up  interviews.  The  triangulation 
approach,  using  multi-level  and  multi-stakeholder  dimension  in  the  data  collection,  ensured 
the robustness and reliability of the data and information used to draw up conclusions in the 
supporting study. 

Some views from Commission and Agency staff gathered from the interviews and mentioned 
in the study illustrate personal experience, although they may not be representative of overall 
relations between the Commission and the Agency. 

5.  ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS 

Analysis  and  answers  to  the  evaluation  questions  are  grouped  below  as  per  the  three 
evaluation criteria: effectiveness, efficiency and coherence. The quantitative results of the ex-
post cost benefit analysis are presented separately in section 5.4 below, while the qualitative 
aspects  are  embedded  into  the  answers  to  the  evaluation  questions  (sections  5.1-5.3).  The 
detailed evaluation questions and sub-questions are listed in annex. 

5.1 Effectiveness 

For  the  purpose  of  this  evaluation,  effectiveness  relates  to  how  successful  the  Agency  has 
been in achieving or progressing towards its objectives. 

8 

 
 
 
 
 
 
 
 
During 2015-2018, the ERCEA operated according to the 2013 legal framework that  set out 
its revised mandate and its delegated responsibilities. The ERCEA’s activities, as planned in 
its annual work programmes and reported in the Annual Activity Reports, corresponded to the 
tasks set out in its Acts of Establishment and Delegation. The ERCEA was also found to be 
fulfilling  its  delegated  management  responsibilities  to  provide  technical  support  and  the 
secretariat function to the ERC’s Scientific Council. From the other side and as acknowledged 
by the ERC ScC Members, the Commission duly fulfilled its obligations in guaranteeing the 
European  Research  Council’s  autonomy  and  integrity,  including  in  what  regards  the 
development  by  the  Scientific  Council  of  the  ERC  work  programme  setting  out  the  key 
scientific priorities for funding. 

The  ERCEA’s  governance  framework  is  characterised  by  a  dual  leadership,  unique  among 
executive  agencies.  The  ERCEA  is  accountable  both  to  the  ScC  and  to  its  parent  DG  (DG 
RTD),  which  has  supervisory  responsibilities  for  monitoring 
the  ERC  programme 
implementation,  and  in  overseeing  the  execution  of  the  Agency’s  administrative  and 
operational  budget.  Working  arrangements  and  procedures  for  interaction  between  the 
Agency and DG RTD, are set out in the Written Agreement (WA) of 2015. The delimitation 
of responsibilities between them was found to be generally clear, with a Written Agreement 
between the two providing operational guidance. 

In  2015-2018,  the  ERCEA  achieved  very  good  results  in  terms  of  most  key  performance 
indicators  (KPIs)  targets  as  set  out  in  its  Annual  Work  Programme19.  Compared  to  the 
previous years the Agency’s performance remained stable or even improved during the period 
of reference, particularly in regarding to the timely evaluation of proposals and the signing of 
grant agreements. 

The ERCEA’s indicator on Time-to-Inform (TTI)20 had various targets within the 2015-2018 
period also depending on the type of grant. During 2015-2018 the ERCEA was largely in line 
with the TTI targets with some deviations observed in the 2015 AdG and 2017 StG calls. It 
should  be  noted  that  the  average  TTI  of  successful  applicants  is  longer  compared  to  the 
average  TTI  for  unsuccessful  applicants  due  to  a  difference  in  the  procedures  involved  for 
successful and unsuccessful applicants. 

Despite  longer  TTI  than  those  of  Horizon  2020,  the  surveys  of  the  ERC  Horizon  2020 
principal investigators (grantees) and host institutions (HIs) revealed that 92% and 89% of the 
respective  respondents  were  “satisfied”  or  “very  satisfied”  with  the  overall  application  and 
submission  process.  The  satisfaction  level  was  lower,  however,  among  unsuccessful 
applicants (67% satisfied and 20% neutral). 

During the evaluation period, the ERCEA was largely in line with the Time-to-Sign (TTS)21 
target;  more  significant  deviations  from  the  target  concerned  the  2015  CoG  call  and  a  few 
PoC  calls  (mainly  in  2015).  The  ERCEA’s  performance  in  terms  of  TTS  significantly 
improved in 2017 thanks to further administrative simplifications in the granting phase. The 

19 Due to their specificity, the ERC grants have a derogation from the common Horizon 2020 targets as per Art. 
20(3) of the Horizon 2020 Rules for Participation and the TTI, TTS and TTG targets are set in the corresponding 
Annual Work Programmes of the ERCEA. 
20 The  time to inform all/  successful applicants on the  outcome  of  the evaluation of their  application from  the 
final date for submission of completed proposals. 
21 The time to sign grant agreements from the date of informing successful applicants. 

9 

 
 
 
 
 
 
 
                                                           
ERCEA was largely in line with the overall Time-to-Grant (TTG)22 target or close to it for all 
2014-2017 calls. 

The number of redress requests received by the ERCEA has been decreasing since 2014. The 
number  of  these  requests  that  lead  to  a  re-evaluation  of  proposals  is  very  low  and  dropped 
significantly  after  2013.  It  is  considerably  below23  the  Annual  Work  Programme  target  of 
0.1%. 

For payments to grants, the average ERCEA’s Time-to-Pay (TTP) stood significantly below 
the target for all types of payments (pre-financing, interim and final payments) in 2015-2018 
both for FP7 and Horizon 2020. The TTP for 2015 and 2016 for FP7 was close and within the 
target. 99% of payments were executed on time in 2016, 2017 and in the 1st semester of 2018. 

In payment to experts, average TTP for experts decreased during 2016-2017 to 13.3 and 10.7 
days respectively, or 60% below the TTP target of 30 days. Accordingly, the ratio of experts 
“strongly” or “rather” agreeing that the time to make payments was appropriate grew to 94% - 
compared to  84% in  the previous evaluation  period (2012-2015). A considerable number of 
experts (46%) expressed dissatisfaction with  the amounts received  for hotel/accommodation 
and subsistence allowance being insufficient, however these are set at corporate level. 

Similar to the previous years, the ERCEA managed to achieve full execution of its operational 
budget both in commitment and payment appropriations during 2015-2018. 

The  ERCEA  ethics  review  and  monitoring  procedures  generated  a  low  level  of  satisfaction 
among  both  grantees  and  Host  Institutions,  with  only  29%  of  grantees  and  35%  of  Host 
Institutions “very satisfied” or “satisfied” with the overall ERC ethics review process24. The 
process encompassed a two-stage approach – (1) an ethics review which takes place between 
the evaluation phase and grant signature and (2) ethics monitoring carried out during project 
implementation.  The  ERCEA  took  a  number  of  steps  to  simplify  the  ethics  procedure  by 
streamlining the two stages approach and adopted a new procedure in 2017. Accordingly, an 
initial  ethics  review  procedure  between  the  end  of  the  evaluation  stage  and  contract 
commencement stage scrutinises the ethical aspects of projects before the research begins; a 
further ethical assessment or ethics closure procedure signs the project off once the research 
has been completed. Also, a simplified risk-based approach to both ethics review and ethics 
monitoring  was  introduced  by  classifying  projects  into  categories  of  normal  or  high  ethics 
sensitivity where projects with normal ethics sensitivity will no longer be required to produce 
ethics deliverables. While the progress achieved is duly noted, it is concluded that the ERCEA 
should  further  review  and  adjust  its  ethics  supporting  guidance  document.  In  parallel,  it 
should  increase  awareness  among  grantees  and  Host  Institutions  on  the  need  for  particular 
ethics-related information for the ERC grants. 

With regard to the legality and regularity of transactions, the multiannual residual error rate 
supporting  the  ERCEA’s  declaration  of  assurance  decreased  during  evaluation  period  from 
1.31% in 2015 to 0.82% in 2017 and remained well below the 2% materiality threshold. For 

22  The  time  from  the  final  date  for  submission  of  complete  proposals  to  signature  of  grant  agreements  with 
applicants. 
23 Between 0.02 and 0.04% in 2015-2017. 
24  ERCEA  ethics  review  has  an  increased  complexity  as  the  frontier  research  is  often  carried  out  at  the 
boundaries of  knowledge.  Also, for some  areas of research (such as  human embryonic  stem cell research)  the 
ethics review had to involve the Commission and the Member States (comitology procedure), which prolonged 
the ethics review process. 

10 

 
 
 
 
 
 
 
 
                                                           
Horizon  2020,  there  were  not  enough  audits  to  draw  statistically  representative  conclusions 
for  the  ERCEA  specific  error  rate.  However,  since  the  ERCEA's  beneficiary  base  remained 
the  same  as  under  FP725  and  the  specificities  of  the  programme  have  not  been  changed  in 
Horizon  2020,  it  was  expected  that  the  error  rate  for  Horizon  2020  remains  below  the  2% 
materiality threshold. 

Overall  satisfaction  with  the  performance  of  the  ERCEA  is  one  of  the  highest  among 
executive agencies. 89% of grantees responding to the ERCEA survey of 2018 were satisfied 
(33%  of  them  –  very  satisfied)  with  the  services  provided  by  the  Agency.  This  result  is 
slightly  below  that  of  the  previous  evaluation  of  the  ERCEA  (95%  in  2014),  however  it  is 
higher than the overall level of satisfaction reported by the beneficiaries of CHAFEA (74%) 
and similar to what was reported by the beneficiaries of REA (86%) and EACEA (89%). A 
considerable part of the respondents to the surveys wished to work with the Agency again in 
the future (85% of beneficiaries, 94% of the ERCEA’s independent experts and around 39% 
of unsuccessful applicants). 

The survey of beneficiaries indicates  a high degree of satisfaction with the way the Agency 
communicated  with  them.  81%  of  the  respondents  to  the  beneficiaries’  survey  strongly  or 
rather  agreed  that  the  ERCEA  staff  assigned  to  their  project  were  easily  available  and 
responsive during the implementation. Most favourable results were received in relation to the 
ERCEA’s  availability  and  responsiveness  during  the  grant  amendment  (82%  strongly  or 
rather agreed) and in the grant finalisation phases (85% strongly or rather agreed). The areas 
with reported lower levels of satisfaction were related to the user-friendliness of the IT tools 
employed throughout the project management life-cycle. The Commission, which is leading 
the development of grant management IT tools at corporate level, is currently taking measures 
to address beneficiary satisfaction by improving user-friendliness of the IT tools. 

According  to  the  survey  of  the  ERCEA’s  independent  experts,  93%  of  respondents  agreed 
that  the  process  of  appointment  and  contracting  of  experts  was  smooth  and  clear,  and  were 
satisfied  with  various  aspects  related  to  the  appointment  and  contracting  process;  98%  of 
surveyed  experts  stated  that  the  invitation  was  issued  in  sufficient  time  to  organise  their 
schedule  or  work.  These  results  improved  compared  to  the  previous  survey  of  independent 
experts, where 87% of respondents agreed that the process of appointment and contracting of 
experts was smooth and clear. 

The  evaluation  identified  as  an  area  for  further  improvement  the  communication  with  the 
ERC independent experts serving in the evaluation panels. Some panellists surveyed indicated 
a possible mismatch between their area of expertise and their allocation to evaluation panels 
focusing  on  particular  scientific  disciplines.  However,  the  selection  procedure  is  expressly 
designed  so  that  panellists  evaluate  the  ERC  project  applications  outside  their  own  narrow 
subject specialisation. It was found that this is more a communications issue with members of 
the evaluation panels, and this could be further improved in the future. 

In  terms  of  proximity  to  the  addressees,  in  the  reference  period,  the  ERCEA  dedicated 
significant  efforts  in  engaging  with  potential  applicants  but  also  beneficiaries,  both  with 
Principal  Investigators  (PIs)  that  hold  the  ERC  grants  and  with  Host  Institutions  (HIs)  that 
accommodate  or  host  the  ERC  grant  holders,  through  information  days,  workshops  and 
conferences organised by the ERCEA in Brussels and across the Member States. In addition, 

25  i.e.  mostly  individual  researchers,  while  most  of  the  Horizon  2020  themes  support  collaborative  research 
projects where at least three organisations from different countries form a consortium. 

11 

 
 
 
 
 
 
 
                                                           
the ERCEA launched the “Top 20 Host Institutions initiative” (which account for ca. 70% of 
the total  ERC grant  holders), in  order to  promote a closer relationship  between the ERCEA 
and leading HIs aiming to provide a more consistent and uniform approach to administrative 
and financial queries. The ERCEA is also active on social media – its website attracts more 
than  half  a  million  visitors  yearly,  and  the  Agency  organises  information  stands  at  three  to 
five selected scientific conferences every year.  

Concerning  the  visibility  of  the  EU  and  the  particular  role  of  ERCEA  as  a  promoter  of  the 
programmes  entrusted  to  the  Agency,  the  feedback  from  beneficiaries  (via  interviews) 
indicate that they are to a very large extent aware that the ERCEA grants were funded from 
the EU budget. The redesigned logo of the ERC now specifically indicates that the ERC was 
established by the Commission. Moreover, the ERC grant label has gained in recognition over 
time,  commensurate  with  the  strong  bibliometric  track  record  associated  with  the  ERC 
grantees  (publications,  performance  in  terms  of  citations  and  in  the  top-cited  publications). 
Visibility  has  been  further  enhanced  through  the  awarding  of  prestigious  scientific  prizes, 
such  as  the  Nobel  and  Wolf  Prizes  and  the  Fields  Medal,  to  ERC  grantees.  In  2014,  an 
analysis of over 7000 leading researchers in Europe found that 30% had applied to the ERC’s 
calls and around one in six were ERC grant holders26. 

5.2 Efficiency 

This  section  considers  the  relationship  between  the  resources  used  by  the  Agency  and  the 
output.  The  analysis,  among  other  factors,  also  includes  analysis  of  administrative  and 
regulatory burden and looks at aspects of simplification. 

Similar  to  the  previous  evaluation  period,  during  2015-2017  the  ERCEA  proved  to  be  an 
efficient  and  cost-effective  structure  for  the  management  of  the  delegated  programmes.  Its 
administrative  budget  was  within  2.2-2.4%  of  the  operational  budget  based  on  commitment 
appropriations and within the range of 2.9-3.2% based on payment appropriations. 

Ratio between the ERCEA’s administrative and operational budget 

7,0%

6,0%

5,0%

4,0%

3,0%

2,0%

1,0%

0,0%

6,0% 

5,3% 

4,6% 

4,1% 

3,3% 

3,6% 

3,2% 

2,9% 

2,9% 

2,5% 

2,5% 

2,3% 

2,1% 

2,1% 

2,2% 

2,4% 

2,4% 

1,6% 

2009

2010

2011

2012

2013

2014

2015

2016

2017

Ratio (Commitment)

Ratio (Payment)

26  Interim  evaluation  of  Horizon  2020,  16  August  2017,  https://publications.europa.eu/en/publication-detail/-
/publication/fad8c173-7e42-11e7-b5c6-01aa75ed71a1/language-en/format-PDF/source-77918455. 

12 

 
 
 
 
 
 
 
 
                                                           
The ERCEA’s actual operational budget during 2014-2018 constituted EUR 9.172 billion, of 
which EUR 8.460 billion from EU budget and EUR 711.7 million related to EEA/EFTA27 and 
third  countries’  contributions  (not  considered  in  the  ERCEA’s  SFS).  The  ERCEA’s  actual 
administrative  budget  increased  from  EUR  39.34  million  in  2015  to  EUR  48.60  million  in 
2018 (in commitments, including EEA/EFTA and third countries’ contributions). 

A  detailed  analysis  of  the  ERCEA  cost-efficiency  and  results  of  the  cost-benefit  analysis 
(CBA) are presented under the point 5.4 below. 

As  previously  mentioned,  the  actual  workload  of  the  ERCEA  during  the  evaluation  period 
generally corresponded to the initial 2013 CBA estimates. The actual number of new grants 
managed by the ERCEA, which is the main workload driver for the Agency, under 2014-2018 
calls  was  15%  higher  than  CBA  forecast.  It  can  be  explained  by  the  fact  that  the  increased 
number of actual grants was primarily related to the additional contributions from EFTA/EEA 
and third countries that were not included in the CBA forecast. 

The actual ERCEA’s operational budget was higher than the SFS estimates for StG (148% of 
the initial estimates), CoG (123%) and PoC (131%). At the same time, it was lower for AdG 
(89%)  and  Synergy  Grants  (38%),  which  were  re-launched  only  in  2018,  following  the 
implementation of a 2012-2013 pilot Synergy Grants scheme. 

CBA estimated vs actual ERCEA’s operational budget 2014-2018 by grant scheme, EUR 
million 

Grant scheme 

StG 
CoG 
AdG 
PoC 
Synergy 

Total 

CBA 
1 934 
2 523 
3 195 
84 
673 
8 408 

Actual 
2 856 
3 113 
2 835 
110 
258 
9 172 

Actual 
vs CBA  
148% 
123% 
89% 
131% 
38% 
109% 

The  actual  number  of  proposals  (40  532)  received  under  the  2014-2018  calls  was 
substantially lower than the initial CBA calculations (59 500). This could be explained in part 
by the restriction rules on re-submission28. The lower number of applications determined that 
the success rate for 2014-2018 the ERC calls remained at a reasonable range29 of 12%-15%, 
whereas the CBA estimated that it would drop to 7.6%-8.7%. 

As  the  ERC  grant  schemes  under  Horizon  2020  remained  very  similar  to  FP7,  the  average 
grant size estimates for specific grant schemes in CBA were close to the actual average grant 
size in 2014-2018 calls: StG: EUR 1.427 million (CBA) vs EUR 1.482 million actual; CoG: 
EUR 1.871 million vs EUR 1.919 million; AdG: EUR 2.271 million vs EUR 2.367 million; 
PoC:  EUR  0.150  million  vs  EUR  0.149  million  and  Synergy  EUR  10.000  million  vs  EUR 
9.656 million. 

27 EEA stands for ‘The European Economic Area’; EFTA stands for ‘The European Free Trade Association’. 
28  Due  to  the  past  increasing  trend  in  resubmissions,  these  restrictions  were  introduced  by  the  ERC  Scientific 
Council  not  only  to  contain  the  experts’  workload  at  a  reasonable  level  and  hence  avoid  compromising 
evaluation quality but also to keep the overall success rate at an appropriate level around 15%. 
29 The average success rate for all calls under respective Annual Work Programme. 

13 

 
 
 
 
 
 
 
                                                           
Given the increased operational budget to grant schemes from one size, and the lower average 
grant  size  from  another  side,  the  actual  number  of  grants  managed  by  the  ERCEA  under 
2014-2018 calls was 15% higher than the CBA estimates. 

CBA estimated vs actual No of grants, by grant 
schemes, 2014-2018 calls 

Grant 
scheme 
StG 
CoG 
AdG 
PoC 
Synergy 
Total 

CBA  Actual 
1 355 
1925 
1 348 
1607 
1 407 
1191 
561 
700 
67 
27 
5450 
4 738 

Actual 
vs 
CBA  
142% 
119% 
85% 
125% 
40% 
115% 

The  number  of  running  projects  ‘per  head’  increased  from  10.49  in  2014  to  12.52  in  2017. 
Furthermore,  the  ratio  of  budget  ‘per  head’  decreased  from  EUR 4.45  million  to  EUR 4.04 
million (in terms of commitments) but increased from EUR 2.60 million to EUR 3.36 million 
(in terms of payments) in 2014-2017. 

Budget, staff and workload indicators,  ERCEA, 2014-2017, € MILLION OR % 

Operational budget, commitments 
Operational budget, payments 
Administrative budget, commitments 
Actual number of staff (at the end of the year)  388 
Programme management cost  at  the ERCEA, 
commitments 
Programme management cost  at  the ERCEA, 
payments 
Budget ‘per head’, commitments  
Budget ‘per head’, payments 
Proposals received 
Total running projects (at the end of the year) 
Running projects ‘per head’  

4.45 
2.60 
8 521 
4 070 
10.49 

2.1% 

3.6% 

2016 

2015 

2014 
1 725.97  1 792.29  1 767.46  1 924.73 
1 008.88  1 245.68  1 457.68  1 601.24 
36.04 

2017 

39.34 
417 

42.87 
461 

47.14 
477 

2.2% 

2.4% 

2.4% 

3.2% 

4.30 
2.99 
7 263 
4 864 
11.66 

2.9% 

3.83 
3.16 
8 077 
5 459 
11.84 

2.9% 

4.04 
3.36 
8 319 
5 843 
12.25 

One  of  the  key  challenges  for  the  ERCEA  during  the  reference  period  related  to  workload 
variation  across  the  Agency.  Between  2015  and  2018,  ERCEA’s  workload  increased  in 
respect to ethics reviews and number of grants under management. The Agency was able to 
cope with this additional workload thanks to appropriate allocation of staff (forward resource 
planning  and  adoption  of  workload  planning  models30),  administrative  simplifications  and 
efficiency gains. 

30 These models are still being used to estimate resource needs, and were viewed by interviewees at the ERCEA 
as having been an effective planning tool, one which has fed directly into resource and during annual budgetary 
and Annual Work Programme implementation. 

14 

 
 
 
 
 
  
 
                                                           
The  ERCEA  took  a  series  of  actions  to  cope  with  the  varying  levels  of  workload  across  its 
different  units.  The  ERCEA  was  also  effective  in  filling  vacancies  and  maintaining  a 
relatively low vacancy rate (3%) between 2015 and 2018. Staff turnover was relatively stable 
throughout  the  period  of  2015-2018  constituting  7%  on  average.  By  the  end  of  2017  the 
number of the ERCEA staff had risen to 477 (from 417 in 2015). 

The  staff  opinion  survey  carried  out  in  2016  revealed  that  the  ERCEA  demonstrated 
consistently  high  results  compared  to  the  averages  of  the  Commission  and  the  Executive 
Agencies  for  the  three  key  indicators,  i.e.  staff  engagement  index31,  overall  job  satisfaction 
and wellbeing. The ERCEA also ranked second among all the Commission services and the 
executive  agencies  in  terms  of  the  staff  valuing  the  Commission/executive  agency  as  a 
modern  and  attractive  workplace.  The  Agency  demonstrated  particularly  positive  results  for 
staff’s understanding of the Commission/executive agency purpose (95%), their willingness to 
make an extra effort when required (94%) and job clarity (92%). 

As  was  the  case  with  the  other  executive  agencies,  the  ERCEA  demonstrated  lower 
satisfaction  rates  in  the  overall  mobility  and  career  development,  the  links  between 
performance  and  career  progression  and  the  available  opportunities  to  move  to  another  job 
matching  their  skills  and  management  of  career  choices.  Those  results  were  still  somewhat 
higher  than  or  equal  to  the  levels  of  agreement  for  the  Commission  and  other  executive 
agencies. 

Responding  to  these  staff  opinion  survey  results,  the  Agency  adopted  in  2016  a  revised 
Human Resources (HR) Action Plan and established a new comprehensive multi-annual HR 
strategy.  This  was  complemented  by  actions  at  corporate  level,  notably  by  adopting,  in 
2017, new implementing provisions for promoting the inter-agency mobility of contract and 
temporary agent staff. 

The  evolution  of  key  indicators  for  staff  satisfaction  –  after  the  evaluated  period  –  is 
encouraging.  In  the  latest  staff  opinion  survey  (4th  quarter  of  2018,  i.e.  outside  of  the 
evaluation  period),  the  ERCEA  managed  to  increase  its  result  for  staff  feeling  that  the 
Agency cares about their well-being to 69%, up from 58% in 2016. Staff engagement rose to 
73% (from 70% in 2016). 

Regarding  ERCEA’s  organisational  structure,  the  Agency  is  composed  of  the  office  of  the 
Director,  the  office  of  the  Chief  accountant,  units  A1  and  A2  of  support  to  the  ScC  and 
external  communications,  respectively,  and  three  departments,  Department  B  (scientific 
management),  Department  C  (grants  management)  and  Department  D  (supporting  business 
functions e.g. IT, HR and legal).  

Overall, the size and structure of the ERCEA, and the level of resourcing in each department, 
were appropriate to its mandate and delegated tasks during the evaluation period. 

31 The staff engagement index is composed of 7 individual survey questions: 
• 
I have the appropriate and timely information to do my work well. 
•  My colleagues are committed to doing quality work. 
• 
• 
• 
•  My manager seems to care about me as a person. 
•  My line manager helps me to identify my training and development needs. 

I have a clear understanding of what is expected from me at work. 
I have recently received recognition or praise for good work. 
I feel that my opinion is valued. 

15 

 
 
 
 
 
 
 
 
 
                                                           
During 2015-2018, a number of simplifications were implemented by the ERCEA which led 
to  efficiency savings, some stemming from  changes across the whole of Horizon 2020 (e.g. 
electronic  grants  management  and  reporting),  whereas  others  were  instituted  internally  (e.g. 
simplification of ethics and granting procedures etc.) 

5.3 Coherence 

Coherence  looks  at  any  overlaps  and  complementarities  within  the  programme  portfolio 
managed by the ERCEA or delimitation of responsibilities between the ERCEA and its parent 
DG. 

For the period 2015-2018 the evaluation  did  not  find any  evidence of duplication, overlaps, 
gaps  or  inconsistencies  between  the  ERC  programme  and  other  Horizon  2020  programmes, 
which was attributed to  the ERC programme being the sole bottom-up research programme 
focused on promoting scientific excellence by supporting Principal  Investigators. There is  a 
strong  continuity in the  ERCEA mandate between the  FP7 and Horizon  2020 programming 
periods  –  the  ERCEA  is  operating  already  for  a  decade  and  its  institutional  maturity  has 
grown  progressively.  A  strong  factor  of  coherence  derives  from  the  fact  that  ERCEA  is 
responsible  for  implementing  the  ERC  programme  only,  i.e.  a  part  of  the  Research  and 
Innovation  framework  programmes  (Horizon  2020  and  legacy  of  FP7),  with  common  rules 
and procedures. 

The  transition  to  common  business  support  functions  provided  by  the  CSC  as  of  2014  and 
later CIC required new ways of working by the Agency to ensure the proper balance between 
the  specificities  of  the  ERC  programme  and  its  associated  grant  instruments,  and  the 
harmonisation of procedures and IT tools across the DGs and EA within the research family. 
While some difficulties were encountered in ensuring that the customisation of IT tools meets 
the  ERCEA’s  specific  needs  and  on  a  sufficiently  timely  basis,  this  is  being  addressed  and 
increasingly demonstrates a co-development approach. 

As explained in  section  5.1, the ERCEA has a specific  governance framework  compared to 
other executive agencies. It has only one parent DG, DG RTD, while it serves as a dedicated 
implementing structure of the independent European Research Council. This is also reflected 
in the composition of its Steering Committee, which comprises five members, including the 
Director-General  of DG  RTD, Director-level  representatives from DG RTD-A and  DG HR, 
and two members of the ScC. The parent DG has supervisory responsibilities for monitoring 
the  ERC  programme  implementation,  and  in  overseeing  the  execution  of  the  Agency’s 
administrative and operational budget. Delimitations of responsibilities and tasks between the 
ERCEA  and  its  parent  DG,  as  well  as  the  working  arrangements  and  procedures  for 
interaction, are set out in a clear and appropriate manner in the Written Agreement of 2015. 

Overall, there is  an  effective mechanism for monitoring, reporting and supervision  enabling 
the  Commission  to  benefit  from  the  know-how  created  within  the  Agency.  The  monitoring 
and reporting data and information  on the ERC  programme implementation  through regular 
reporting  mechanisms  (e.g.  Annual  Activity  Report,  quarterly  and  monthly  financial 
reporting, data analysis of programme implementation by country, theme, grant scheme etc.) 
are of high-quality, and sufficient for governance and scrutiny purposes. 

Also, there is an effective, both formal and informal, information flow between the ERCEA 
and the Commission services, and between the ERCEA and the Scientific Council to ensure 

16 

 
 
 
 
 
 
 
 
 
that both the ERC and the parent DG are kept informed about the Agency’s performance and 
implementation of the ERC programme. 

In the reference period the ERCEA stepped up its efforts in providing policy feedback to the 
ERC’s  ScC  to  facilitate  scientific  policy-making  on  frontier  research.  Measures  included 
policy briefings and data analyses of the ERC programme implementation data; presentation 
materials  for  external  presentations  (e.g.  to  national  policy  makers  and  scientific  audiences, 
but also initiatives like ‘Science behind the Project’, the ‘Qualitative Evaluation of Completed 
Projects’, and the ERIS database, that focus on the identification, categorisation, monitoring 
and assessment of ERC calls for proposals and projects. 

However, the evaluation identified the domain of policy feedback on frontier research as an 
area where the ERCEA’s role should be further defined and clarified. The parent DG should 
further define its needs in terms of policy feedback, including the format and frequency, and a 
corresponding  business  process  should  be  agreed  between  the  Agency  and  its  parent  DG. 
Currently  the  CIC  is  leading  the  efforts  of  strengthening  the  processes  and  procedures  for 
feedback  to  policy  at  the  R&I  family  level  as  part  of  the  Strategy  for  an  Effective 
Dissemination and Exploitation of Research Results in Horizon 2020. 

5.4 Retrospective Cost-benefit analysis 

The retrospective cost–benefit analysis for the 2015-2018 period was carried out based on the 
results  of  the  2013  ex-ante  CBA,  the  assumptions  laid  down  in  the  Specific  Financial 
Statements (SFS) and the actual costs of the ERCEA. 

As mentioned in section 2.2, the 2013 ex ante CBA estimated the savings resulting from the 
delegation of tasks to the ERCEA at around EUR 79.4 million over the whole period 2014-
2024  when  compared  to  the  management  by  the  Commission  services  (in-house  scenario), 
while  for  the  period  of  2015-2018  these  were  estimated  at  around  EUR  40.77  million.  The 
ERCEA SFS estimated the savings of the executive agency scenario at EUR 72.65 million for 
the  period  2015-2018.  The  ERCEA  SFS  differs  from  the  CBA  estimations  for  the 
administrative  budget  as  a  result  of  adjustment  of  the  ERCEA  staff  numbers  (reduction  by 
around 12%) and calculation of expenses without any indexation. 

The retrospective CBA revealed that the executive agency scenario actual costs in 2015-2018 
were much lower than the costs of the in-house scenario. The savings constituted EUR 76.97 
million32  and  were  89%  higher  compared  to  the  initial  CBA  estimates  and  6%  higher 
compared to the initial SFS estimates. 

32 These costs do not take into account the additional contribution to the ERCEA’s administrative budget of EUR 
7.65 million over 2015-2018 coming from EFTA/EEA and third country contributions to manage the additional 
operational budget of EUR 712 million coming from these countries. 

17 

 
 
 
 
 
 
 
 
 
                                                           
Estimated costs and savings of the executive agency scenario in 2015-2018, EUR 

300.000.000

250.000.000

200.000.000

150.000.000

100.000.000

50.000.000

0

236.555.225 

236.555.225 

245.557.775 

195.781.945 

163.903.000 

168.592.365 

40.773.280 

72.652.225 

76.965.410 

CBA estimations

SFS estimations

Actual

In-house scenario

Executive agency scenario

Estimated savings of EA scenario

As  forecasted  in  the  SFS  and  the  ex-ante  CBA,  savings  of  the  executive  agency  scenario 
primarily  resulted  from  a  higher  share  of  lower-cost  contract  staff  employed  within  the 
Agency and the lower overall number of staff. 

Significant  cost  savings  occurred  in  the  ERCEA’s  Title  II  ‘Infrastructure  and  operating 
expenditure’. The costs in Title I ‘Staff-related expenditure’ were higher than estimated in the 
SFS, as the latter did  not  take into account  increases to  staff costs due to salary indexation, 
promotions and increasing staff seniority.  

As  to  the  assumptions  laid  down  in  the  CBA,  it  should  be  noted  that  the  ERCEA  actual 
workload  during  the  evaluation  period  generally  corresponded  to  the  initial  estimates,  with 
variations across different grant schemes and years. The Agency duly took measures to ensure 
appropriate staffing levels across all departments, an appropriate balance between operational 
staff and administrative support staff and adequate flexibility to move resources around on the 
basis of workload fluctuations. 

The actual number of new grants managed by the ERCEA under 2014-2018 calls increased by 
13% compared to the 2013 CBA forecast. This is a result of the increased actual operational 
budget  of  the  ERCEA  by  9%  given  the  additional  contributions  from  EEA/EFTA  and  third 
countries. On the other hand, the average grant size of the ERC grant schemes under Horizon 
2020 remained similar to FP7. 

6.  CONCLUSIONS 

The results of the evaluation of the ERCEA operations during 2015-2018 period confirm that 
delegation  of  the  ERC  programmes  to  the  Agency  was  fully  justified  in  terms  of  the  cost-
savings achieved and value added generated. Higher than expected savings for the EU budget 
were made and these were further enhanced through administrative simplifications (e.g. from 
electronic  grants  management  and  reporting).  The  ERCEA  demonstrated  strong  value  for 
money  based  on  the  assessment  of  its  performance.  The  continued  delegation  of  the  ERC 
programme  implementation  remained  considerably  more  cost-effective  than  the  in-house 
scenario, generating substantial savings for the EU budget. 

18 

 
 
 
 
 
 
 
 
 
The  ex-post  cost  benefit  analysis  found  that  the  actual  savings  of  the  executive  agency 
scenario were of around EUR 77 million during the period 2015-2018 compared with the in-
house scenario. 

Given that the ERC was set up as an autonomous, scientifically-driven entity, the ERCEA has 
provided an effective dedicated implementation structure to the ERC. This would have been 
difficult  to  achieve  were  the  programme  implementation  functions  to  be  housed  within  the 
Commission. The ERCEA continued to contribute to the strong image and reputation of the 
ERC in the scientific world while the experts selected as the ERC Evaluation panel members 
were high-calibre. 

During this period, the ERCEA was effective in achieving its core objectives relating to both 
programme implementation and supporting the ERC’s ScC, and its beneficiaries were highly 
satisfied with the performance of the Agency. The ERCEA has consistently achieved its KPI 
targets. Beneficiaries’ satisfaction with the ERCEA performance increased to 89% compared 
with  the  previous  evaluation  (82%  in  2015).  The  ratio  of  budget  ‘per  operational  head’ 
increased  from  EUR 3.46  million  to  EUR 3.54  million  (in  terms  of  commitments)  or  from 
EUR 2.44 million to  EUR 2.99 million  (in terms of payments) in  2014-2017. The Agency’s 
administrative  budget  was  below  3%  of  the  operational  budget  based  both  on  commitment 
and payment appropriations between 2015 and 2017.  

The  coordination  and  communication  mechanisms  that  are  in  place  facilitate  information 
flows  between  the  ERCEA  and  the  Commission  services,  in  particular  with  regard  to  the 
content of projects supported and their results. No evidence of overlaps, duplication, gaps and 
inconsistencies within the programme portfolio were identified. 

The current evaluation identifies only three areas where further improvements are necessary 
from  either  the  ERC,  the  ERCEA,  Commission  or  the  ERCEA  and  Commission  together. 
These are in regard to: 

  The communications to ERC panel members on the rationale for their selection. 
  The ERCEA’s ethics supporting guidance document and the beneficiaries’ awareness 

on the need for particular ethics-related information for the ERC grants. 

  The  ERCEA’s  role  in  extracting  and  providing  policy-relevant  feedback  to  the 

Commission. 

The  parent  DG  and  the  Agency  are  preparing  an  action  plan  to  address  the  shortcomings 
identified in this evaluation. This action plan aims to increase the Agency’s performance and 
contribution to an efficient implementation of the delegated programmes. 

19 

 
 
 
 
 
 
 
 
ANNEX: EVALUATION QUESTIONS 

Effectiveness 

  To  what  extent  has  ERCEA  been  operating  according  to  the  legal  framework 

establishing it? 

  To  what  extent  has  ERCEA  achieved  its  objectives  with  special  focus  to  (a)  the 
implementation  of  the  delegated  programmes  Horizon  2020  and  FP7,  (b)  support  to 
the  Scientific  Council  and  (c)  the  implementation  of  the  internal  control  principles 
notably sound financial and human resource management? What, if anything, could be 
done to render ERCEA more effective in achieving these objectives? 

  To what extent has ERCEA contributed to an improved management of the delegated 
programmes  and  better  services  to  the  stakeholders  and  addressees9  in  terms  of  the 
elements  assessed  in  the  2013  CBA  and  as  compared  to  the  alternative  options 
mentioned in that CBA? 

  To what extent has ERCEA contributed to improved management of the programmes 

in terms of: 

−  Proximity  to  addressees  (e.g.  role  played  by  the  Agency  as  focal  point  for 
applicants  and beneficiaries of the programmes;  instruments and mechanisms 
put in place for communicating with the applicants and beneficiaries). 
−  Effective  implementation  of  the  programmes,  taking  into  account  the 
interests of the addressees and those of the EU: 
  rate of execution of commitment appropriations; 
  rate of execution of payment appropriations; 
  time to grant; 
  net time to pay; 
  residual multi-annual error rate identified at ex-post control. 
− Visibility of the EU as promoter of the programmes entrusted to the Agency 
(e.g.  compliance  with  the  Commission’s  guidelines  on  information  and 
visibility of programmes10, instruments put in place to ensure the visibility of 
the EU as promoter of the programmes). 

Efficiency 

  To  what  extent  have  the  actual  costs  (including  cost  of  coordination  and 
monitoring)  of  ERCEA  corresponded  to  the  estimates  of  the  2013  CBA?  If  not, 
what are the reasons behind? 

  To what extent have the actual benefits corresponded to the estimates of the 2013 

CBA? If not, what are the reasons behind? 

  To what extent has the management and execution of the programmes by ERCEA 

been cost-effective as compared to the alternative options11? 

  To  what  extent  has  ERCEA  contributed  to  improved  management  of  the 

programmes in terms of: 
o  Simplification  of  the  procedures  and  flexibility  in  the  implementation  of 

delegated tasks (e.g. capacity to adapt to periods of high workload)? 

o  Which further scope for simplification exists? 

  Which  aspects/means/actors  or  processes  render  ERCEA  more  or  less  efficient? 
What  could  be improved? What  is the quality of the services/advice provided by 
ERCEA to stakeholders and addressees? 

  To  what  extent  has  the  establishment  of  ERCEA  resulted  in  savings  to  the  EU 

20 

 
 
 
 
 
budget as compared to the alternative options (e.g. difference in costs between the 
Commission option and the Agency option)? 

  To  what  extent  have  ERCEA’s  internal  organisation  and  procedures  been 

conducive to its efficiency? 

  To  what  extent  is  ERCEA’s  internal  organisation  capable  and  flexible  to  rapidly 

respond to resource needs due to uncertainties related to volumes of work? 

  To  what  extent  does  ERCEA’s  human  resources  management  contribute  to  the 

achievement of the Agency’s objectives? 
o  Is the size and structure of the organisation appropriate? 
o  Is  the  balance  between  operational  staff  and  administrative  support  staff 

appropriate? 

o  Are the staff turnover and vacancy rate well managed? 
o  How  does  the  Agency  follow  up  on  the  findings  of  the  latest  staff  survey 

(2016)? 

Coherence 

  To what extent have there been overlaps/ gaps/ inconsistencies/ complementarities 
within  the  programme  portfolio  managed  by  ERCEA  and  how  are  these 
addressed? 
Is there a clear and appropriate delimitation of responsibilities and tasks between 
ERCEA  and  the  Parent  DG?  Are  there  overlaps  or  gaps?  Are  the  different 
responsibilities adequately communicated to the beneficiaries? 

 

  To what extent has ERCEA enabled the Commission to better focus on its policy 

related tasks? 
o  Are  there  any  governance  (financial  and  policy)  issues  in  relation  to  the 
implementation of the tasks that are delegated to the executive agencies/ issues 
as regards the Commission’s services being able to steer EU policy or budget 
implementation? 

o  Are  appropriate  mechanisms  and  instruments  in  place,  and  at  which  level,  to 
ensure  an  adequate  coordination  and  information  flow  between  ERCEA  and 
the Commission services, notably on the content of the projects supported and 
their results? 

o  Does  ERCEA  provide  useful  information  on  the  implementation  of  the 
delegated  programmes  and  their  progress  (in  terms  of  management  and 
content)  in  support  of  the  policy  process  (e.g.  information  required  for  the 
annual Management Plan of the Parent DG)? 

  To what extent have the activities of ERCEA resulted in unintended effects (both 

desirable and undesirable)? 

  To  what  extent  has  the  Commission,  in  the  presence  of  ERCEA,  been  able  to 
maintain an adequate level of know-how in relation to the programmes  entrusted 
to  the  Agency?  How  has  this  been  achieved?  What  are  the  feedback  channels, 
means  and  methods  used  for  this  purpose?  What  are  areas  for  improvement,  if 
any? 

  Have the monitoring, reporting and supervision arrangements in place enabled the 
Commission to benefit, in the short and medium term, from the know-how created 
within ERCEA? 

  How effective is the flow of information and communication between ERCEA and 
the Commission services (in particular Parent DG), and between ERCEA and the 
Scientific Council? 

21 

 
 
 
 
  To  what  extent  would  the  closing  down  of  ERCEA  result  in  losing  significant 
know-how in relation to the management of the programmes entrusted to ERCEA? 

22