COMMISSION STAFF WORKING DOCUMENT Evaluation of the European Research Council Executive Agency (ERCEA) Accompanying the document REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL AND THE COURT OF AUDITORS Evaluation of the Consumers, Health, Agriculture and Food Executive Agency, the Educational, Audiovisual and Culture Executive Agency, the Executive Agency for Small and Medium-Sized Enterprises, the European Research Council Executive Agency, the Innovation and Networks Executive Agency and the Research Executive Agency
Date: 2025-04-16
Institutions: Directorate-General for Research and Innovation, European Commission
Subjects: EU aid, EU financing, EU programme, executive agency, financial control, performance audit, research

EN 

  EN 

   EUROPEAN COMMISSION  Brussels, 16.4.2025  SWD(2025) 87 final  COMMISSION STAFF WORKING DOCUMENT Evaluation of the European Research Council Executive Agency (ERCEA) Accompanying the document REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL AND THE COURT OF AUDITORS Evaluation of the Consumers, Health, Agriculture and Food Executive Agency, the Educational, Audiovisual and Culture Executive Agency, the Executive Agency for Small and Medium-Sized Enterprises, the European Research Council Executive Agency, the Innovation and Networks Executive Agency and the Research Executive Agency                      {COM(2025) 171 final} - {SWD(2025) 88 final} - {SWD(2025) 89 final} - {SWD(2025) 90 final} - {SWD(2025) 91 final} - {SWD(2025) 92 final}  
 
 
Table of Contents 

1 

INTRODUCTION .................................................................................................................................................... 3 

2  WHAT WAS THE EXPECTED OUTCOME OF THE INTERVENTION? ..................................................... 5 

2.1 
2.2 

DESCRIPTION OF THE INTERVENTION AND ITS OBJECTIVES ................................................................................. 5 
BASELINE AND POINTS OF COMPARISON ............................................................................................................. 7 

3  HOW HAS THE SITUATION EVOLVED OVER THE EVALUATION PERIOD? ....................................... 7 

4 

EVALUATION FINDINGS (ANALYTICAL PART) .......................................................................................... 8 

4.1 TO WHAT EXTENT WAS THE INTERVENTION SUCCESSFUL AND WHY? ........................................................................ 8 
4.2. COST-BENEFIT ANALYSIS ....................................................................................................................................... 15 

5  WHAT ARE THE CONCLUSIONS AND LESSONS LEARNED? ................................................................. 16 

5.1 CONCLUSIONS ......................................................................................................................................................... 16 
5.2. LESSONS LEARNED ................................................................................................................................................. 16 

ANNEX I - PROCEDURAL INFORMATION ............................................................................................................ 18 

ANNEX II - METHODOLOGY AND ANALYTICAL MODELS USED ................................................................. 19 

ANNEX III - EVALUATION MATRIX AND ANSWERS TO THE EVALUATION QUESTIONS (BY 
CRITERION) .................................................................................................................................................................. 21 

ANNEX IV - OVERVIEW OF BENEFITS AND COSTS .......................................................................................... 22 

ANNEX V - COST-BENEFIT ANALYSIS .................................................................................................................. 23 

ANNEX VI – STAKEHOLDER CONSULTATION – SYNOPSIS REPORT .......................................................... 26 

1 

 
 
 
 
 
 
 
Term or acronym 

Act of delegation 

Act of establishment 

Agencies 
Commission 
EEA 
EFTA 
ERC 
ERCEA 
EU 
FP7 

Glossary 

Meaning or definition 

Commission Decision C(2013)9428 on delegating 
powers to the European Research Council Executive 
Agency with a view to performance of tasks linked to 
the implementation of Union programmes in the field 
of frontier research comprising, in particular, 
implementation of appropriations entered in the 
general budget of the Union 

Commission Implementing Decision 2013/779/EU of 
17 December 2013 establishing the European 
Research Council Executive Agency  
Executive agencies 
European Commission 
European Economic Area 
European Free Trade Association 
European Research Council 
European Research Council Executive Agency 
European Union 
the ‘Ideas’ specific programme implementing the 7th 
Framework  Programme  of  the  European  Community 
for 
and 
demonstration activities (2007-2013) 
Council Regulation (EC) No 58/2003 of 19 December 
2002 laying down the statute for executive agencies to 
be entrusted with certain tasks in the management of 
Community programmes 

technological 

development 

research, 

Framework  Regulation  for  executive 
agencies 

Horizon 2020 

Horizon 2020, the 8th EU Framework Programme for 
research and innovation (2014-2020) 

IT 

Information technology 

2 

 
 
 
 
 
 
 
 
 
 
 
1  INTRODUCTION 

Purpose and scope of the evaluation/fitness check 

The purpose of this evaluation is to assess the operations of the European Research Council Executive 
Agency  (ERCEA  or  the  Agency)  from  16 July  2018  to  31 March  2021.  Under  Article 25  of  the 
Framework Regulation for executive agencies1, the Commission must submit this evaluation to the 
European Parliament, the Council, the Court of Auditors and the Steering Committee of the Agency. 

This  evaluation  concerns  the  Agency’s  operations  according  to  its  mandate  under  the  2014-2020 
multiannual  financial  framework.  The  Commission  entrusted  the  ERCEA  with  implementing  the 
specific  objective  of  strengthening  frontier  research  by  supporting  the  activities  of  the  European 
Research Council for Part I – ‘Excellent Science’ of the specific programme implementing Horizon 
2020, the Framework Programme for research and innovation (Horizon 2020 specific programme) 
and the legacy of the ‘Ideas’ specific programme implementing the 7th Framework Programme of 
the European Community for research, technological development and demonstration activities (the 
‘Ideas’ specific programme, or FP7 ). 

The  ERCEA  is  governed  by  the  Framework  Regulation  for  executive  agencies.  An  ‘act  of 
establishment’2 sets out its mandate. An ‘act of delegation’3 specifies the tasks to be carried out and 
the powers delegated to the Agency to perform its mandate, as well as the decision establishing the 
ERCEA Steering Committee. 

The  European  Research  Council’s  remit  is  ‘to  encourage  the  highest-quality  research  in  Europe 
through competitive funding, and to support investigator-driven frontier research across all fields on 
the  basis  of  scientific  excellence’4. The  European  Research  Council  (ERC)  comprises  a  scientific 
council (the ERC Scientific Council), which sets out the overall scientific strategy, and the ERCEA 
as the dedicated implementation structure. The Agency implements the ERC’s actions established in 
the ERC work programme and the Agency’s annual work programme. 

The Agency ‘is dedicated to selecting and funding the excellent ideas that have not happened yet and 
the scientists that are dreaming them up’5. To this end, the ERC work programme offers five different 
types of grants: Starting, Consolidator, Advanced, Proof of Concept, and Synergy. 

The Agency manages all programme implementation phases over a project’s lifetime and its budget, 
concludes  and  manages  public  procurement  contracts,  and  provides  support  during  programme 
implementation,  such  as  communication  and  information  activities.  It  also  gathers  and  processes 
information for the Commission and the ERC Scientific Council. Additionally, the Agency supports 
the ERC Scientific Council in all its tasks under Article 8(1) of the Horizon 2020 specific programme. 

In  line  with  the  Commission’s  Better  Regulation  guidelines6,  the  evaluation  applies  the  standard 
evaluation  criteria,  by  assessing  whether  the  Agency  has  carried  out  its  tasks  effectively  and 
efficiently, whether there are any overlaps, gaps or inconsistencies in the Agency’s management of 

1  

2  
3  

4  
5  
6  

Council Regulation (EC) No 58/2003 of 19 December 2002 laying down the statute for executive agencies to be entrusted with certain tasks in 
the management of Community programmes, Official Journal L 11, 16 January 2003 p.1: http://data.europa.eu/eli/reg/2003/58/oj. 
Commission Implementing Decision 2013/779/EU of 17 December 2013 establishing the European Research Council Executive Agency 
Commission Decision C(2013)9428 on delegating powers to the European Research Council Executive Agency with a view to performance of 
tasks  linked  to  the  implementation  of  Union  programmes  in  the  field  of  frontier  research  comprising,  in  particular,  implementation  of 
appropriations entered in the general budget of the Union. 
ERC website https://erc.europa.eu/homepage 
ERCEA documents (annual activity reports and annual work programmes) 
Commission’s Better Regulation guidelines SWD (2021) 305 final and toolbox. 

3 

 
 
 
 
 
 
 
 
the programme portfolio, and whether tasks are delineated clearly between the ERCEA and the parent 
Directorate-General or other executive agencies (coherence)7. 

The  evaluation  also  assesses  whether  the  Agency’s  operations  have  yielded  the  expected  positive 
outcomes – as projected in the ex ante cost-benefit analysis and the specific financial statement for 
delegating tasks to executive agencies8 – and identifies potential areas of improvement. To this end, 
the 2013 cost-benefit analysis’s estimates and the related specific financial statement have been tested 
to provide evidence as to whether the assumptions made in them are valid by comparing them with 
the actual costs and benefits of the implementation of the programmes by the Agency. The aspects 
covered by the cost-benefit analysis are specified in Article 3(1) of the Framework Regulation9 and 
in the guidelines on establishing and operating of executive agencies10. 

The evaluation does not cover the achievements of the programmes managed by the ERCEA, which 
in turn are subject to mid-term and ex post evaluations. The evaluation of the ERCEA nevertheless 
provides  useful  input  for  these  programme  evaluations,  as  the  Agency’s  performance  affects  the 
efficiency and effectiveness of the programmes it manages. 

The evaluation of the ERCEA was supported by a study carried out by an external contractor11. The 
study  combined  a  retrospective  and  prospective  analysis.  The  retrospective  part  assessed  the 
Agency’s  operations  and  its  performance  during  the  evaluation  period,  while  the  prospective  part 
suggested recommendations for improving the Agency’s future performance. 

The  evaluation  of  the  Agency’s  performance  is  evidence-based,  using  a  mixed  methods 
complementary approach that combines: 

•  a  qualitative  approach,  based  on  open-ended  survey  questions,  interviews,  documentary 

reviews and desk research 

•  a qualitative cost-benefit analysis and 
•  a  quantitative  approach,  based  on  administrative  and  monitoring  data,  surveys  and  a 

quantitative cost-benefit analysis. 

Relevant  stakeholders  were  consulted  through  interviews  and  surveys  (the  Commission,  the  ERC 
Scientific  Council  and  ERCEA  staff,  beneficiaries,  unsuccessful  applicants,  and  experts).  Fewer 
unsuccessful applicants participated than beneficiaries. Overall, the response rate to the survey was 
lower than for the previous ERCEA evaluation12 (Survey A13 – 12.32% and Survey B14 – 15.92%, 
excluding partial responses), but sufficient data was available to analyse each target group. Newer 
grants were represented more strongly in surveys than longer-running projects, which generated fewer 
observations on the finalisation of the project implementation phase or final payments than on the 
application and selection phases. The interviewees were helpful for contextualising, complementing 

7  

8  

9  

The Better Regulation guidelines were applied in a proportionate way. There was no need to assess the evaluation criterions of ‘EU value added’ 
and of ‘relevance’ as ERCEA carries out tasks, which the Commission has transferred to it and these criterions are assessed when evaluating the 
programmes. 
Cost-benefit analysis for the delegation of certain tasks regarding the implementation of 2014 2020 Union programmes to the executive agencies 
– Final report for the Commission of 19 August 2013. 
The cost-benefit analysis should include the following: identification of the tasks justifying outsourcing, a cost-benefit analysis which includes 
the costs of coordination and checks, the impact on human resources, possible savings in the general budgetary framework of the EU, efficiency 
and flexibility in the implementation of outsourced tasks, simplification of the procedures used,  the proximity of outsourced activities to final 
beneficiaries, the visibility of the EU as promoter of the EU programme concerned, and the need to maintain an adequate level of know-how inside 
the Commission. 

10   Appendix II  of  the  Guidelines  for  the  establishment  and  operation  of  executive  agencies  financed  from  the  Union  budget  (C (2014)9109  of 

2 December 2014. 

11   PPMI  Consortium:  Public  Policy  Management  Institute  and  IDEA  Consult:  https://op.europa.eu/en/publication-detail/-/publication/a6202c1e-

e5bd-11ee-8b2b-01aa75ed71a1/language-en/format-PDF/source-310469595. 

12   Study supporting the evaluation of the European Research Council Executive Agency (ERCEA) 2015-2018. 
13   Survey A targeted individual recipients and their host institutions for the different types of grant based on applications for calls launched in 2018-
2019-2020, and unsuccessful applicants and their host institutions for the different types of grant based on applications for calls launched in 2018-
2019-2020-2021. 

14   Survey B targeted all external experts during the evaluation period (based on the date on which a claim for costs was created).  

4 

 
 
 
 
 
 
and deepening the quantitative findings and statistical trends, but not all of them provided exhaustive 
insights into the ERCEA’s performance in each reference year, focusing rather on specific processes. 

Overall, the evaluation of the Agency provides robust findings based on the triangulation of high-
quality data collected through documentary reviews, desk research, cost-benefit analysis, stakeholder 
consultations (survey programme and interviews) and analysis of the in-depth study areas. 

2  WHAT WAS THE EXPECTED OUTCOME OF THE INTERVENTION? 

2.1 

 Description of the intervention and its objectives 

According  to  the  act  of  delegation,  the  aim  of  delegating  certain  tasks  to  the  ERCEA  in  order  to 
implement programmes is twofold. 

•  To enable the Commission to focus on its core institutional tasks, which cannot be outsourced, 
namely tasks assigned by the Treaty on the Functioning of the European Union that require 
discretionary powers in translating political choices into action. 

•  To enable the Commission to achieve the objectives of the delegated EU programmes more 
effectively and efficiently. According to the 2013 cost-benefit analysis, delegating tasks to the 
ERCEA  was  estimated  to  be  more  cost-effective  than  keeping  them  ‘in  house’  in  the 
Commission.  In  addition,  harnessing  the  Agency’s  accumulated  experience  and  expertise 
would lead to efficiency gains. 

During the 2018-2021 period, the ERCEA helped to implement the following EU programmes: 

• 

the specific objective ‘Strengthening frontier research through the activities of the ERC’ of 
Part I ‘Excellence Science’ of the Horizon 2020 specific programme; 

• 

the legacy of the ‘Ideas’ specific programme implementing the 7th Framework Programme. 

In this respect, the ERCEA performed programme and project implementation tasks and supported 
the ERC Scientific Council in all its tasks on the basis of the act of delegation and the relevant work 
programmes established by the ERC Scientific Council and adopted by the Commission.  

The philosophy behind the intervention tallied with the Agency’s objectives, tasks and activities, as 
established  in  its  legal  basis.  It  also  considered  the  key  contextual  factors  and  events.  The  figure 
below summarises the intervention philosophy applied. 

5 

 
 
 
 
 
ERCEA’s intervention logic 

Source: Study supporting the triennial evaluation of the Agency 

6 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
           
 
The key inputs are human resources, financial resources, IT tools and communication channels. The 
key processes are derived from the ERCEA’s act of delegation, and the Agency’s main performance 
outputs  relate  to  how  it  manages  the  delegated  programmes.  The  achievement  of  these  outputs  is 
reflected in key performance indicators: standard indicators of financial management that describe 
key  metrics  of  the  proposal  and  project  management  life  cycle  (e.g.  number  of  calls,  number  of 
running  projects,  etc.)  and  support  for  the  ERC  Scientific  Council.  In  addition,  the  evaluation 
considered the key outputs on feedback to policy. 

Outcomes relate to the key results achieved during the evaluation period, from immediate outcomes 
to others in the medium to long term. Satisfaction among unsuccessful applicants, beneficiaries and 
experts  about  the  Agency’s  performance  was  one  of  the  key  measurements  of  results.  The  EU’s 
visibility  as  promoter  of  the  programmes  entrusted  to  the  Agency  was  assessed  by  analysing  the 
Agency’s communication function. Better-informed policy-making by the parent Directorate-General 
and  contributions  to  its  policy  priorities  and  to  the  ERC’s  remit  can  be  assigned  to  the  level  of 
outcomes. 

2.2 

 Baseline and points of comparison 

This  evaluation  of  ERCEA  operations  assesses  the  actual  costs  and  benefits  of  the  ERCEA’s 
programme implementation (executive agency scenario) compared with the alternative scenario of 
management  by  the  Commission  (in-house  scenario).  The  reference  points  for  this  evaluation  are 
therefore the 2013 ex ante cost-benefit analysis and the ERCEA’s specific financial statement. 

Compared with the in-house scenario, the estimated savings were worth EUR 66.6 million according 
to the specific financial statement, or EUR 42.2 million according to the 2013 cost-benefit analysis. 
Non-quantifiable benefits were also estimated, such as better-quality programme management and 
service delivery, improved visibility of EU programmes, and proximity to beneficiaries. 

3  HOW HAS THE SITUATION EVOLVED OVER THE EVALUATION PERIOD? 

Current state of play 

The ERCEA was established in 2008 to implement the ‘Ideas’ specific programme, and its mandate 
was extended in 2014 to implement the successor programme in the form of the ‘Excellent Science’ 
pillar under the Horizon 2020 specific programme.  

The  ERC’s  funding  strategy  focuses  exclusively  on  bottom-up  frontier  research  and  scientific 
excellence, based on the work programme established by the ERC Scientific Council and adopted by 
the Commission. This means that there are no predetermined priorities and that scientific excellence 
is  the  exclusive  goal.  Furthermore,  experts  are  selected  for  the  evaluation  process  with  a  view  to 
ensuring the excellence of the evaluation and attracting top-class experts in order to do this. 

The ERCEA exclusively implements the ERC work programme, which offers five different types of 
grant: 
• 

‘Starting’, for an early-career researcher with 2-7 years of experience since completing their 
PhD; 
‘Consolidator’, for a researcher with 7-12 years of experience since completing their PhD; 
‘Advanced’, for a well-established researcher to pursue a ground-breaking, ambitious project; 
‘Proof of Concept’, for an ERC grant recipient to prepare the proof of concept of a new idea 
to  facilitate  the  investigation  of  the  commercial  and  social  innovation  potential  of  ERC-
funded research; 

• 
• 
• 

7 

 
 
 
 
 
 
 
• 

‘Synergy’, for groups of two to four researchers to implement ambitious research projects, 
evaluated on the criteria of scientific excellence, which takes on the additional meaning of 
outstanding intrinsic synergetic effect. 

The bywords for the ERCEA’s programming mandate are continuity and stability. All types of grant 
(with  the  exception  of  ‘Synergy’)  were  launched  under  the  ‘Ideas’  specific  programme  and  have 
continued under the Horizon 2020 specific programme. The Synergy grant was re-launched in 2018 
after it had been piloted in 2012 and 2013. All grants are intended for single recipients, except for 
‘Synergy’. A call for each type of grant has been opened annually, except in 2021 when only three 
calls were opened (‘Starting’, ‘Consolidator’ and ‘Advanced’) due to the late adoption of Horizon 
Europe (the successor programme to Horizon 2020). 

In  2018,  2019  and  2021,  between  8 200  and  8 500  proposals  were  submitted  for  the  annual  calls 
launched. In 2020, there was a far higher number of proposals (9 428), up 14.3% compared to 2019. 
Between approximately 1 100 and 1 250 grant agreements were signed each year, with the exception 
of 2021, when only 653 grant agreements were signed under Horizon 2020. The number of running 
projects ranged from 6 000 to 6 600 at the end of each year. This number  increased from 2018 to 
2020 but decreased slightly in 2021. 

During the evaluation period, Agency staff numbers went up from 494 in 2018 to 515 in 2020. The 
average occupancy rate for posts was 98.3% between 2018 and 2020, but this fell to 94.2% in 2021, 
in  part  because  of  intensive  recruiting  by  new  or  expanding  agencies  under  the  2021-2027 
multiannual financial framework, and perceptions of limited career progression felt by many staff 
and mentioned during interviews. 

The  ERCEA  had  an  appropriate  organisational  structure  and  conducted  several  reorganisations  to 
strengthen  processes  further,  generate  efficiencies,  manage  increased  workloads  and  enhance 
synergies within the Agency (i.e. reorganisation of the Grant Management department and greater 
synergies  between  the Scientific Evaluation  and  Ethics Review Units, and  between  the  respective 
internal and external communication functions). The Agency continued to operate smoothly despite 
changes  in  leadership  (a  new  director  in  2020,  new  ERC  presidents  in  2020  and  2021).  Internal 
knowledge management process was identified by some interviewees as an area for improvement. 

The ERCEA rose to the challenges presented by the COVID-19 pandemic by switching to a remote 
working environment and applying staff teleworking. Several working groups were set up to respond 
to the changing circumstances and how they affected Agency staff. 

The Agency’s administrative budget increased from EUR 48 million in 2018 to EUR 53 million in 
2020. The operational budget also increased, from EUR 1 751 million in 2018 to EUR 2 088 million 
in 2020. 

4 

  EVALUATION FINDINGS (ANALYTICAL PART) 

4.1 To what extent was the intervention successful and why? 

 Effectiveness 

For  the  evaluation,  ‘effectiveness’  means  how  successfully  the  Agency  achieved  or  progressed 
towards its objectives to the satisfaction of its stakeholders, and to its ability to react and adapt to the 
challenges posed by the COVID-19 pandemic. 

8 

 
 
 
 
 
 
 
 
 
 
The  Agency  operated  within  its  legal  framework  in  terms  of  programme  implementation  and  the 
support it gave the ERC  Scientific Council, continuously  improved its operations,  and achieved a 
high level of overall effectiveness, as demonstrated by its key performance indicators15 and the high 
satisfaction ratings among stakeholders in surveys. 

The ERCEA responded flexibly to the COVID-19 pandemic by swiftly transitioning to teleworking, 
adjusting  processes  and  streamlining  procedures  while  at  the  same  time  maintaining  high 
performance standards. One such major adaptation took the form of remote evaluations (25 panels) 
and  applicant  interviews.  Most  stakeholders  agreed  that  the  transition  from  in-person  to  virtual 
activities had been managed smoothly by the Agency (90% of  principal investigators and 96% of 
their host institutions, 55% of unsuccessful applicants and 77% of their host institutions, and 95% of 
experts). Despite the suspension of in-person activities and events, the ERCEA still managed to give 
information and advice effectively to applicants and beneficiaries, as the favourable survey responses 
show  (88%  of  principal  investigators  and  100%  of  their  host  institutions,  59%  of  unsuccessful 
applicants and 77% of their host institutions, and 97% of experts). 

The COVID-19 pandemic was responsible for a slowdown in grant awards and, at the same time, for 
an increase in project amendments. The Agency stepped up its efforts to simplify the procedures for 
amendments  (2 466  in  2020;  1 141  in  2019).  The  ERCEA  was  highly  efficient  in  processing 
amendments,  approving  or  turning  down  requests  within  45 days  of  receipt  for  an  increasing 
proportion of grants (up from 98.3% in 2018 to 100% in 2021 for FP7 grants; up from 96.1% in 2018 
to 99.9% in 2021 for Horizon 2020 grants). At the same time, 87% of principal investigators and 99% 
of their host institutions appreciated the Agency’s flexibility in accommodating project amendments 
necessitated by the pandemic. 

The prompt management of grants was appreciated by the vast majority of principal investigators and 
their host institutions surveyed (between 83% and 90%), who felt that the timescales – from the call 
deadline  to  the  announcement  of  the  outcome  of  the  proposal  (time-to-inform),  from  the 
announcement of the proposal’s outcome to the signing of the contract (time-to-sign), and overall 
from the submission of the proposal to the signature of the grant agreement (time-to-grant) – were 
appropriate. 

For the 2018-2020 period, the target for time-to-sign indicator was consistently met for all calls, but 
did register an increase in the number of days needed to sign agreements. Similarly, the time-to-grant 
indicator recorded an upward trend but the target was met for most calls with the exception of the 
Synergy grant in 2018 (460 days instead of the targeted 430 days). 

The time-to-inform indicator had various targets for the 2018-2020 period for each type of grant and 
for each year: the longest was for Synergy grants (370 days) and the shortest for Proof of Concept 
grants (100 days). The indicator stayed within target, but with some variations in 2018 for the Synergy 
grant (342 days instead of 300) and the Consolidator grant (285 days instead of the targeted 280) due 
to a tight timetable for calls. 

The  Agency  was  able  to  execute  payments  much  faster  than  reach  targets  for  pre-financing  and 
interim  and final  payments  for both  FP7 and Horizon 2020 and for experts, demonstrating highly 
efficient  operations  to  the  satisfaction  of  principal  investigators  (91%)  and  their  host  institutions 
(100%).  93%  of  principal  investigators  and  100%  of  their  host  institutions  rated  the  process  as 
smooth. 96% of experts agreed that payment deadlines had been satisfactory, and 93% were satisfied 

15   The ERC has a derogation to set specific targets indicated in the ERCEA’s annual work programme, pursuant to Article 20 of Regulation (EU) 
No 1290/2013 of the European Parliament and of the Council of 11 December 2013 laying down the rules for participation and dissemination in 
‘Horizon 2020 – the Framework Programme for Research and Innovation (2014-2020)’ and repealing Regulation (EC) No 1906/2006. 

9 

 
 
 
 
 
 
 
 
with how payments were processed. However, experts agreed less (72%) that the payment received 
had been commensurate with the work done. The reimbursement of costs from visits to Brussels was 
felt to be sufficient by a higher proportion of experts (76%) than in the previous evaluation (46%). 
The remote referees (who provide individual expertise on individual proposals) in the survey saw it 
as prestigious to be asked to work for the ERC (92%) and as important to engage in peer reviews to 
strengthen contributions to scientific improvements (97%). 

With regard to budget execution, the Agency was able to implement its operational budget in terms 
of both commitment and payment appropriations, as it had done in the previous evaluation period. To 
this  end,  the  Agency  put  in  place  an  intensive  monitoring  system  and  was  quick  to  follow  up  on 
financial and scientific reports. 

The residual error rate remained within the 2% target for both FP7 grants (1.12% in 2018, 1.15% in 
2019, 1.10% in 2020) and Horizon 2020 grants (1.22%). The ERCEA’s programme risk profile is 
inherently  lower  than  other  programme  components  due  to  its  specificities  (e.g.  single  grant 
recipients, research institutes as  host institutions, flat-rate overheads). The Agency  had  also  taken 
steps to address the observations of the Court of Auditors in its opinion on the annual accounts. 

The  process  the  ERCEA  uses  to  evaluate  applications  has  been  carefully  designed  to  identify 
scientific excellence regardless of gender, age, nationality, institution, principal investigator or other 
potential  bias.  The  evaluation  process  was  perceived  differently  by  successful  applicants  and 
unsuccessful applicants and their host institutions, particularly in terms of the outcome and feedback 
received. In this respect, principal investigators (92%) and their host institutions (93%) felt that the 
process of selecting high-quality research for funding was fair and transparent. The comments from 
individual reviewers and panels were seen as useful by 94% of principal investigators and by 92% of 
their host institutions. However, the survey showed that the Agency could still make feedback clearer, 
as only 40% of unsuccessful applicants and 59% of their host institutions felt that they had been given 
a  clear  explanation  as  to  why  their  application  had  been  turned  down  and  helpful  comments  to 
improve proposals in view of re-submitting them to the ERC (25% of unsuccessful applicants and 
47% of their host institutions) or to a different funding body (23% of unsuccessful applicants and 
44%  of  their  host  institutions).  Ultimately,  45% of  unsuccessful  applicants  and  68%  of  their  host 
institutions felt that the overall evaluation process had been clear and transparent. 

The proportion of appeals upheld in the redress process was below 0.1% (0.03% in 2018, 0.04% in 
2019, 0.03% in 2020). 

To address the findings of the previous evaluation on ethics reviews and monitoring procedures, the 
Agency took a series of actions to simplify the process and improve the guidance documents made 
available to principal investigators and their host institutions, such as: 

•  decoupling  interim  payments  from  the  monitoring  of  ethical  clearance  for  grants  with  no 
highly  sensitive  ethics  issues,  while  preserving  the  link  between  the  final  payment  and  a 
project’s final ethical clearance; 

. 

• 

informing the principal investigators of projects with low or medium ethical sensitivity that 
some documents must be kept in the project file as requests for them could be forthcoming; 

•  pre-screening proposals made in-house before the official communication to candidates. This 
allowed the share of ‘fast’ ethical clearances at pre-screening to increase from 45% to more 
than 70%; only 30% of proposals had to be screened by the panel of ethical experts. This was 
first implemented under the 2021 call for Starting grants; 

10 

 
 
 
 
 
 
 
 
 
•  organising remote panels of ethical experts during the COVID-19 crisis. The screening panels 

continued to be held remotely while assessment panels took place in person; 

•  setting up a continuous improvement group on ethics in 2020 to find ways of simplifying the 

ethics processes to make further efficiency gains. 

Compared  to  the  previous  evaluation  (29%  of  principal  investigators  and  35%  of  their  host 
institutions), this evaluation’s survey showed greater satisfaction with the above procedures: overall, 
64% of principal investigators and 77% of their host institutions rated the ethics review process as 
smooth.  Most  principal  investigators  and  their  host  institutions  (between  66%  and  79%  of 
respondents) were positive about the clarity of the guidance material and information requested and 
the  process  timelines16.  Certain  respondents  saw  the  amount  of  information  as  extensive  and 
suggested specific  training courses, such as video  tutorials.  However, most stakeholders  (between 
59% and 68% of respondents) saw the ethics review as helping to comply with ethical guidelines 
(national, EU or international) and related legislation. 

Similarly,  65%  of  principal  investigators  and  78%  of  their  host  institutions  felt  that  the  ethics 
monitoring process had gone smoothly. Furthermore, between 67% and 82% agreed that the guidance 
materials and information requested had been clear. 

The Agency provided the ERC Scientific Council with support that was effectively high in quality as 
well as technically and scientifically competent for all its tasks. It did so by performing analysis and 
providing advice and information on programme monitoring and evaluation, design and review, by 
supporting standing committees and working groups, and through communication and dissemination 
activities. The ERC Scientific Council members interviewed were very satisfied with the quality and 
quantity of ERCEA support received. 

As  regards  stakeholder  proximity  during  the  application  and  submission  phases,  satisfaction  rates 
among  successful  applicants  in  the  surveys  were  generally  higher  than  those  among  unsuccessful 
applicants. 82% of the host institutions of principal investigators said that they knew who to contact 
with  questions  when  submitting  applications,  compared  with  58%  of  the  host  institutions  of 
unsuccessful applicants, 55% of principal investigators and 46% of unsuccessful applicants. Most of 
the host institutions of principal investigators (89%) felt that they had received prompt replies to their 
questions  about  the  application  or  selection  process,  compared  to  61%  of  the  host  institutions  of 
unsuccessful applicants, 59% of principal investigators and 46% of unsuccessful  applicants.  Most 
stakeholders regarded the information for applicants as user-friendly (70 to 93%) and easy to find (84 
to 95%) and felt that the eforms for applications and submissions were clear and easy to fill in (69 to 
85%). 

High  satisfaction  rates  were  registered  with  regard  to  communication  actions  during  the 
implementation phase. Principal investigators and their host institutions (83 to 99%) felt that they 
knew  who  to  contact  or  how  to  contact  them.  They  also  rated  ERCEA  staff  as  easy  to  reach  and 
responsive, and said that they answered promptly, clearly and accurately. 

The ERCEA demonstrated effective external  communication  through various channels such as its 
website, social media, press releases, publications, newsletters, awareness-raising activities on grant 
management,  e.g.  workshops  for  principal  investigators  and  their  host  institutions,  and  events 
organised  through  national  contact  points.  Overall,  most  stakeholders  felt  that  the  funding 
opportunities under the ERC programme were promoted effectively (88% of principal investigators 

16   The percentage range represents the smallest and the highest percentage of positive answers for all the categories of respondents as indicated in 

Annex 7 to the supporting study. 

11 

 
 
 
 
 
 
 
 
and 86% of their host institutions, 71% of unsuccessful applicants and 75% of their host institutions, 
and 91% of external experts). 

The  ERCEA  maintained  the  EU’s  visibility  as  programme  promoter.  Most  stakeholders  (89%  to 
100%) were aware that the ERC programme is funded from the EU budget, but significantly fewer 
(54%  to  86%)  were  aware  that  the  Commission  had  tasked  the  ERCEA  with  managing  the 
programmes. 

The surveys also showed high overall satisfaction with the services provided by the ERCEA (89% of 
principal investigators and 97% of their host institutions, 59% of unsuccessful applicants and 72% of 
their host institutions, 98% of experts). The stakeholders surveyed were very willing to work with the 
ERCEA again in the future (94% of principal investigators, 100% of their host institutions, 78% of 
unsuccessful applicants and 93% of their host institutions, 82% of experts), while the low success 
rate was cited as a reason for not re-applying by a majority of unsuccessful applicants (59%) and their 
host institutions (67%). 

Efficiency 

This section considers the relationship between the Agency’s resources and output. It covers factors 
such  as  the  management  and  execution  of  programmes  and  the  quality  of  financial  and  human 
resources management during the evaluation period. It also features an analysis of the administrative 
and  regulatory  workloads  and  examines  simplification  aspects.  These  areas  are  covered  by  key 
performance indicators and additional survey data. 

The ERCEA’s programme and grant portfolio was in line with the portfolio envisaged in the 2013 ex 
ante cost-benefit analysis and in the specific financial statement.  

Overall, the ERCEA’s actual workload during the evaluation period tallied with the initial estimates 
in the 2013 cost-benefit analysis and in the specific financial statement, as illustrated below by the 
main indicators (in EUR million or as a %): 

2018 

2019 

2020 

Operational budget, commitments  

Operational budget, payments  

Administrative budget, commitments  

Administrative budget, payments  

Number of staff17 (at the end of the year)  

Number of operational staff18  

Programme 
(commitments)  

management 

cost 

ratio 

Programme 
(payments)  

management 

cost 

ratio 

Budget ‘per operational head’ (commitments)  

Proposals received 

Total running projects  

Running projects ‘per operational head’  

1 999 

1 751 

48.60 

47.98 

494 

438 

2 103 

2 012 

51.51 

51.04 

508 

448 

2 218 

2 087 

52.03 

52.97 

515 

464 

2.41% 

2.39% 

2.34% 

2.74% 

2.54% 

2.54% 

6.41 

8 352 

6 077 

19.34 

6.57 

8 247 

6 151 

18.72 

6.16 

9 428 

6 553 

18.11 

17   Actually filled. 
18   Actually filled. 

12 

 
 
 
 
 
 
 
 
   
The  ERCEA  is  a  cost-effective  structure  in  terms  of  managing  delegated  programmes,  as 
demonstrated by the cost ratio between administrative and operational expenditure (2.7% in 2018; 
2.5% in 2019 and 2020).  

The operational budget allocated to the Agency for the 2014-2020 period from the EU contribution 
amounted to EUR 12.56 billion, in line with estimates in the specific financial statement. In addition, 
the Agency received contributions from the EEA/EFTA and third- countries (EUR 933 million) that 
were not included in the initial estimates. In total, the ERCEA’s operational budget for the 2014-2020 
period was EUR 13.5 billion, 7% higher than the specific financial statement estimates. 

Under the 2014-2020 calls, the actual number of proposals received (58 207) was much lower than 
initial cost-benefit analysis estimates (88 150). This might be due in part to the restrictions on re-
submitting  proposals.  Owing  to  the  previous  upward  trend  in  resubmissions,  the  ERC  Scientific 
Council had applied curbs in order to keep experts’ workloads at a reasonable level – and thus avoid 
compromising evaluation quality – and to maintain the overall success rate at a level of around 15%, 
whereas the cost-benefit analysis had forecast a drop to 8 to 9%. 

The actual number of new grants (7 832) managed by the ERCEA, which is the main workload driver 
for the Agency, was 10% higher than the projected figure in the 2013 cost-benefit analysis (7 137 
grants). The increased number of grants  related  primarily to  the additional  contributions  from  the 
EEA/EFTA and third countries and higher allocations for grant schemes with lower average grant 
size  (‘Proof  of  Concept’  and  ‘Starting’  grants).  The  ERCEA  successfully  maintained  the  initial 
projected productivity level in the 2013 cost-benefit analysis, as the actual ‘per head’ budget ratio19 
in 2020 (EUR 4.33 million) remained very close to estimates (EUR 4.39 million). 

During  the  evaluation  period,  the  Agency  adopted  several  measures  aimed  at  simplifying  its 
procedures. These involved simplifying grant management procedures and applying corporate and 
local  IT  tools.  The  ‘Continuous  Improvement’  programme  launched  in  2020  played  a  part  in  this 
simplification, as it aimed to increase productivity while easing pressures on staff. It was a bottom-
up exercise to collect ideas and identify opportunities for improvement. The initiative identified nine 
improvement  topics,  resulting  in  11 projects  that  yielded  positive  outcomes,  such  as  quicker  and 
simpler amendment processes and enlisting of remote referees more efficiently. 

The  use  of  IT  tools  has  reduced  the  overall  administrative  workload.  The  availability  of  new 
functionalities developed at corporate level for programming was evaluated and specific IT solutions 
developed locally were phased out after a careful assessment that took into account the specificities 
of the ERCEA’s business processes. Overall, progress was made towards greater alignment with the 
Commission’s systems. 

The Agency acted to further improve its IT tool efficiency and use of IT tools. It updated its guidance 
and procedures to ensure greater uptake of the enhanced monitoring tool20 in the SyGMa corporate 
system among scientific officers. The Agency began using the ‘Participant Portal Grant Management 
Service’ electronic exchange system to keep track of communication records; direct emails continued 
to be used for intermediate communication. The Agency also planned to improve the IT tools used in 
programming and evaluation procedures and aim for greater automation. 

19   The  calculation  of  the  ‘per  head’  budget  ratio  in  2020  was  based  on  the  following  estimates:  budget  –  executed  commitment  appropriations 

(EUR 2 227.56 million); number of staff – posts actually filled at the end of the year (515). 

20  

In the event of issues with projects, scientific officers can perform additional controls and activate the consolidated monitoring tool in the SyGMa 
risk module. 

13 

 
 
 
 
 
 
 
 
 
Most stakeholders felt that the following simplifications made their lives easier as regards the process 
of application and project implementation: the harmonisation of funding rules and procedures across 
programmes (53 to 78%), the simplification and clarification of call guidelines and application forms 
(66 to 80%), the simplified granting process (61 to 82%), reporting templates (66 to 87%) and the 
ethics review procedure and ethics workflow monitoring (60 to 72%), the establishment of a single 
point of contact for individual organisations/host institutions (64 to 85%), the use of simplified cost 
options  instead  of  real-cost  funding  (69  to  84%),  and  the  wider  use  of  common  IT  tools  across 
different programmes and programme strands (62 to 80%). 

In  terms  of  future  simplifications,  respondents  mainly  identified  more  user-friendly  IT  tools  for 
project management and reporting (86 to 92%) and for submitting applications (78 to 91%), further 
harmonisation of administrative requirements for projects financed under different programmes and 
actions (74 to 88%), and the further simplification of granting and reporting requirements (78 to 84%). 

During the evaluation period, the ERCEA provided an appropriate organisational structure, striking 
a balance between operational staff and administrative support staff (from 13% in 2018 to 11% in 
2020).  It  allocated  resources  flexibly  as  dictated  by  workload  fluctuations  (via  annual  workload 
planning  and  reorganisations).  It  also  adapted  promptly  and  effectively  to  the  challenges  of  the 
COVID-19 pandemic, thereby supporting efficient programme implementation. 

Coherence 

Coherence looks at any overlaps and complementarities within the Agency’s programme portfolio 
and the delimitation of responsibilities between the Agency and its parent Directorate-General. Other 
questions involved the extent to which the Agency was able to provide feedback to policy to its parent 
Directorate-General. 

In the ERCEA’s case there was strong coherence and continuity in terms of the programmes managed, 
as the ERCEA is the only structure to implement the EU programme dedicated to bottom-up frontier 
research focused on scientific excellence, starting with the ‘Ideas’ specific programme and continuing 
with  its  successor  programme,  the  ‘Horizon  2020  specific  programme’.  When  allocating  this 
portfolio,  there  were  no  identifiable  programming  synergies  with  other  parts  of  EU  framework 
programmes. 

The ERCEA’s mandate is also characterised by stability, since its governance structure and types of 
grant  have  stayed  the  same  (except  for  the  relaunch  of  ‘Synergy’  in  2018),  enabling  the  Agency 
progressively to build its know-how and expertise. 

The  Agency  is  governed  by  a  steering  committee,  comprising  representatives  of  the  parent 
Directorate-General  (Directorate-General  for  Research  and  Innovation)  and  the  ERC  Scientific 
Council. 

The delineation of responsibilities between the ERCEA and the parent Directorate-General is clear 
and appropriate and there are no overlaps, gaps or inconsistencies, in line with the legal framework 
(e.g.  act  of  delegation,  memorandum  of  understanding  with  the  parent  Directorate-General). 
However, the survey revealed little awareness among stakeholders about how tasks were delineated 
between the Agency, the ERC Scientific Council and the Commission (29% of principal investigators 
and 26% of unsuccessful applicants, 66% of the host institutions of principal investigators, 50% of 
the host institutions of unsuccessful applicants, and 53% of experts). 

The  parent  Directorate-General  is  responsible  for  supervising  the  programme  implementation  and 
overseeing the execution of the Agency’s budget. To this end, a monitoring and reporting mechanism 

14 

 
 
 
 
 
 
 
 
 
 
is in place, in particular through the annual work programme, the mid-year report and the Agency’s 
annual activity report. 

The Commission-led harmonisation of processes and tools yielded benefits and efficiencies, but also 
required  a  balance  between  harmonisation  and  customisation  to  meet  the  specific  needs  and 
characteristics of the ERC programme. This process of harmonisation necessitated system adaptations 
in close cooperation with the Commission. Overall, the Agency and the Commission were able to 
find  appropriate  solutions  to  the  specificities  of  the  ERC  programme  (such  as  adapting  reporting 
templates). To this end, the ERCEA was invited to contribute to the process of designing common 
procedures  and  processes  managed  by  the  Common  Support  Centre21,  which  establishes  the 
framework for harmonised implementation by the funding bodies implementing grants, in areas such 
as common business processes, IT, legal, audit, and information and data services. Trilateral meetings 
took place between the Agency, the Common Support Centre and the ERC Scientific Council with a 
focus on specific and technical topics. 

By delegating specific tasks to the Agency, the Commission was able to focus on policy-related tasks. 
To this end, the Agency provided feedback to policy to the parent Directorate-General on request. It 
also gave relevant feedback, analysis and data to the ERC Scientific Council. Broader initiatives to 
strengthen feedback to policy were the ‘Science behind the project initiative’, the ex post peer review-
based assessment of completed ERC projects, and the ERC research information system (ERIS)22.  

4.2. Cost-benefit analysis 

A  retrospective  cost-benefit  analysis  was  carried  out  for  the  2018-2020  period,  based  on  the 
assumptions laid down in the 2013 cost-benefit analysis and the specific financial statement, coupled 
with the Agency’s actual costs. 

Compared with the in-house scenario, actual savings during the 2018-2020 period were 23.4% higher 
compared  to  the  initial  specific  financial  statement  estimates  (EUR  82.2 million  compared  to 
EUR 66.6 million),  and  94.8%  higher  compared  to  the  initial  estimates  in  the  2013  cost-benefit 
analysis  (EUR  82.2 million  compared  to  EUR 42.2 million).  As  forecast,  the  savings  under  the 
executive agency scenario were primarily the result of higher numbers of lower-cost external staff 
(contract  agents  and  seconded  national  experts)  employed  by  the  Agency  and  lower  overall  staff 
numbers. 

The overall actual costs of the executive agency scenario amounted to EUR 155.1 million during the 
2018-2020 period, including contributions from EEA/EFTA and third countries (EUR 6.4 million in 
2018-2020).  Based  exclusively  on  the  EU  contribution,  the  actual  costs  were  EUR 148.7 million, 
which tallied with the initial estimates in the specific financial statement (EUR 149.2 million), with 
savings of EUR 0.5 million or 0.4% of the specific financial statement estimates). 

As  in  the  previous  evaluation,  the  actual  administrative  costs  under  ‘Staff-related  expenditure’ 
(Title I) were higher than the estimates in the specific financial statement, which did not take into 
account additional costs linked to salary indexation, promotions and the increasing seniority of staff. 
The average cost of temporary agents in the Agency  was higher than the average cost used when 

21   The  common  support centre is a  directorate  within  the Commission’s  DG  Research and  Innovation.  It provides  services  in  the  areas  of  legal 
support, ex-post audit, information technology systems and operations, business processes and programme information and data to the whole of 
the research DG family, executive agencies and joint undertakings implementing Horizon 2020. It was re-named the common implementation 
centre under the 2021-2027 multiannual financial framework. 

22   Developed by the ERCEA to present and manage information on ERC funding activities and projects. 

15 

 
 
 
 
 
 
 
making the initial estimates. The composition of staff (ratio between temporary staff and contract 
staff) tallied with estimates. 

Higher staff expenditure was offset by lower expenditure under Title II – Infrastructure and operating 
expenditure  and  Title III  –  Programme  support  expenditure  compared  with  specific  financial 
statement estimates (56% of estimates). For Title II, these savings related in part to the overestimated 
necessary average costs of the ERCEA’s infrastructure and operating expenditure (projected to be 20 
to 52% higher than for other executive agencies). 

5  WHAT ARE THE CONCLUSIONS AND LESSONS LEARNED? 

5.1 Conclusions 

The Agency achieved its core objectives on programme implementation effectively and efficiently, 
including executing the budget and providing quality support to the ERC Scientific Council in all its 
tasks, as demonstrated by the key performance indicators and the levels of satisfaction recorded in 
surveys. This in turn enabled the Commission to focus on its core institutional tasks. 

The  Agency’s  mandate  was  consistent  in  programming  terms,  as  it  implemented  a  research 
framework part related to frontier research focused on scientific excellence across a diverse thematic 
spectrum, specific for a bottom-up approach. Tasks were clearly delineated between the ERCEA and 
the parent Directorate-General. 

Delegating  programme  implementation  to  the  ERCEA  remained  considerably  more  cost-effective 
than the Commission  in-house scenario, generating  significant  cost  savings in  the EU budget  and 
good value for money, as shown by the findings of the retrospective cost-benefit analysis. 

5.2. Lessons learned 

The main lessons learned are ensuring channels for discussing the harmonisation of procedures and 
the optimisation of processes across agencies, continuing the ‘Continuous Improvement Programme‘ 
to identify potential efficiency gains, enhancing synergies and opportunities for joint communication 
activities, providing meaningful feedback to policy, continuing implementing and updating the HR 
strategy based on staff feedback, and further consolidating information in the reporting mechanisms. 

Cooperation and flows of information between the Agency and the Commission and other executive 
agencies will continue through various forums and working groups to build a common vision on the 
harmonisation  of  procedures  and  processes,  share  best  practices,  and  explore  various  synergies 
between agencies. Additionally, the ‘Continuous Improvement’ programme launched by the Agency 
in  2020  will  continue  to  identify  improvements  aimed  at  increasing  productivity  and  easing  staff 
workloads. 

Enhancing collaboration between the ERCEA and the parent Directorate-General on communication 
activities will contribute to firmly embedding joint communication planning, joint campaigns, sharing 
information, and re-using communication content.  

Since  its  foundation  in  2008,  the  Agency  has  developed  solid  knowledge  on  programme 
implementation,  which  is  needed  for  policy-making  by  the  parent  Directorate-General  and  by  the 
ERC Scientific Council when preparing work programmes. The Agency provided feedback to policy 
to  its  parent  Directorate-General  at  its  request;  from  2021,  it  switched  to  a  more  proactive  and 
structured  approach  when  a  Feedback-to-Policy  framework  was  established  under  2021-2017 
multiannual financial framework.  

16 

 
 
 
 
 
 
 
 
 
The ERCEA’s staff engagement index (measuring staff satisfaction) was high, reaching 73% in 2018. 
The Agency will continue adopting appropriate action plans with close staff involvement based on 
the feedback received from staff surveys. Close cooperation on HR matters between agencies will 
yield benefits in terms of knowledge sharing, enhancing synergies, and harmonised procedures and 
processes.  

Finally, in terms of reporting requirements, the Agency will constantly enhance the available data by 
providing qualitative information not captured by quantitative indicators, such as explanations on the 
changes to key indicators and trends and distinguish between structural changes and occasional or 
technical  factors.  Similarly,  clarifying  expectations  and  timelines  will  facilitate  preparations  for 
Steering Committee meetings. 

17 

 
 
 
 
 
ANNEX I - PROCEDURAL INFORMATION 

 The evaluation had a clearly established scope: 

•  The reference period of the evaluation ran from 16 July 2018 to 31 March 2021. 

•  The purpose of this evaluation was to assess the operation of ERCEA as per the criteria of 
effectiveness, efficiency and coherence. During the evaluation period, ERCEA implemented 
the ERC work programmes as set by the ERC Scientific Council and was in charge with day-
to-day grant management. 

•  The evaluation did not focus on the operational achievements of the ERC programme. 

•  The agencies established under 2021-2027 multiannual financial framework were outside the 

scope of this evaluation. 

The evaluation of ERCEA was part of the evaluation of all the six agencies, which was made in a 
coordinated  manner  based  on  a  supporting  study  carried  out  by  an  external  contractor.  The 
preparations for the evaluation  started in  2022, when an  interservice  group was  set  up by all  lead 
parent Directorates-General and the concerned central services (Directorate-General for Budget and 
the Secretariat-General). The group meetings (i.e. 6) were chaired by the Common Implementation 
Centre,  set  up  within  the  Directorate-General  for  Research  and  Innovation,  who  also  ensured  the 
secretariat. The evaluation of each agency was led by the respective lead parent Directorate-General 
(PLAN/2022/1912). The Commission also launched a ‘call for evidence’ on 10 March 2023 on its 
‘Have your say’ portal. The call was open for feedback on agencies activities until 7 April 2023. 

The methodology used for the evaluation was consistent across the agencies in accordance with the 
criteria provided in the Commission’s Better Regulation guidelines. The assessment of the evaluation 
criteria ‘EU added value’, i.e. why the EU should act, was not considered relevant as each agency 
carries out tasks which the Commission transferred to it. The EU added value of the programmes that 
each agency manages is assessed in the context of the programme evaluations. 

Evidence  was  taken  from  sources  such  as  the  Commission  databases,  annual  reporting  exercises, 
adopted decisions. The supporting study was prepared using a mixed-methods approach at the levels 
of  methodologies  and  methods  and  worked  on  a  qualitative  methodological  approach  (based  on 
documentary  review  and  desk  research,  interviews,  answers  to  open-ended  survey  questions  and 
qualitative cost-benefit analysis) combined with a quantitative methodological approach (based on 
administrative  and  monitoring  data,  surveys  and  quantitative  cost-benefit  analysis),  as  detailed  in 
Annex II. 

Knowledge from the present evaluation will inform internal decision-making in the Agency and its 
parent Directorate-General on possible improvements to the implementation of the programme. 

Furthermore,  the  results  of  the  evaluation  will  be  useful  for  accountability  purposes.  The  final 
evaluation reports regarding the performance of the six agencies will allow the Commission to report 
the  results  of  the  retrospective  cost-benefit  analysis  to  the  budgetary  authority,  while  informing 
various  European  Union  institutions  on  the  savings  achieved  and  identifying  potential  areas  for 
improvement. The results of this evaluation will be communicated to the Steering Committee of the 
Agency, to the European Parliament, the Council, and the European Court of Auditors. 

18 

 
 
 
 
 
ANNEX II - METHODOLOGY AND ANALYTICAL MODELS USED 

The evaluation was supported by a study carried out by an external contractor in accordance with the 
criteria provided in the Commission’s Better Regulation guidelines. The methodology for ERCEA 
evaluation was consistent with the approaches employed for the evaluations of the other agencies. 

The  evaluation  followed  an  evidence-based  and  mixed-methods  approach,  using  qualitative  and 
quantitative  data  to  answer  evaluation  questions  in  a  complementary  manner,  i.e.  the  results  of 
quantitative methods were used to enhance the understanding of the qualitative results, and vice versa. 
Triangulation  of  data  sources  strengthened  the  validity  of  the  findings.  The  study  was  structured 
around a series of evaluation questions outlined in the Annex to the study. 

The evaluation relied on the evidence gathered and analysed based on: 

•  Extensive  documentary  review  and  desk  research  on  legal  documents,  information  on 
ERCEA's  relations  with  its  parent  Directorate-General,  and  data  on  its  financial  and  non-
financial  performance,  including  the  memorandum  of  understanding,  the  annual  work 
programmes,  the  annual  activity  reports,  staff  surveys  and  workload,  IT,  procedures, 
continuous  improvement,  internal  and  external  communication,  minutes  of  the  Steering 
Committee meetings, results of audits performed by the Court of Auditors and the IAS, and 
information on actions taken after the previous evaluation of ERCEA. An in-depth analysis 
was conducted on IT developments and the Continuous Improvement Programme. 

• 

Interview programme: 40 interviews with 56 interviewees were conducted, representing the 
main stakeholders and exceeding the initial goal of 35 interviews. This was structured in three 
stages: 

-  The scoping phase consisted of exploratory and scoping interviews with Commission 
and Agency staff, targeting those who had a strategic outlook and were familiar with 
the  overall  key  developments  that  took  place  during  the  evaluation  period.  (2 
interviewees for parent Directorate-General; three interviewees for ERCEA). 

-  The  main  phase  consisted  of  semi-structured  interviews,  having  each  survey 
questionnaire  tailored  to  each  interview  target  group  to  ensure  that  all  evaluation 
questions and all aspects of the organisational model were covered (29 interviews with 
35 persons – Commission staff and ERCEA staff) 

-  The follow-up phase consisted of in-depth interviews to contextualise and deepen the 
understanding  of  the  data  emerging  during  the  previous  stages  of  the  interview 
programme and the data collection process overall. A small group of stakeholders was 
targeted, including Commission and Agency staff, but also beneficiaries, unsuccessful 
applicant, and experts. 

•  Survey programme: Two surveys were conducted as part of the evaluation: 

-  Survey A targeted the beneficiaries, unsuccessful applicants, and their host institutions 
for the different types of grants and based on applications for calls launched in 2018-
2019-2020  (and  2021  for  host  institutions  and  unsuccessful  applicants).  The  total 
population targeted amounted to 1233 and the overall response rate was of 16.22%, 
including  partial  responses  (17.95%  response  rate  for  principal  investigators  and 
12.61% for their host institutions, 20.58% for unsuccessful applicants and 10.90% for 
their  host  institutions).  Excluding  partial  responses,  the  overall  response  rate  was 
12.32%. 

19 

 
 
 
 
 
 
 
 
 
-  Survey  B  targeted  external  experts  involved  in  evaluation  panels,  remote  reviews, 
project monitoring, ethics review and monitoring in the evaluation period (based on 
cost  claim  creation  date).  The  total  number  of  experts  targeted  was  3261  and  the 
response  rate  was  18.49%  (including  partial  responses)  and  respectively  15.92% 
(excluding partial responses). 

•  Cost-Benefit Analysis included an analysis of workload and cost-effectiveness, by comparing 
the actual indicators with the estimates foreseen in the 2013 ex ante cost-benefit analysis and 
respectively the specific financial statement. 

Limitations and robustness of the findings 

Overall, the evaluation of ERCEA provides robust findings built on the triangulation of high-quality 
data  collected  through  a  documentary  review,  desk  research,  cost-benefit  analysis,  stakeholder 
consultation activities (the survey programme and interviews) and an in-depth analysis of some areas. 

All relevant stakeholders were consulted via interviews and surveys, however the response rate to the 
survey has been lower than the one registered in the previous evaluation of ERCEA23, which could 
be due to survey fatigue. Despite this, it was still largely sufficient to perform analyses per target 
group  (principal  investigators  and  their  host  institutions,  unsuccessful  applicants  and  their  host 
institutions).  In  cases  where  due  to  further  filtering,  the  number  of  observations  was  too  low,  the 
findings were not reported. 

Recently selected grants were more strongly represented than longer running projects. Information 
related to the finalisation of the project implementation phase or final payments for instance  were 
therefore based on less observations than those related to the application and selection phases. 

The  exploratory  and  in-depth  interviews  were  especially  useful  for  the  contextualisation, 
complementing and deepening the quantitative findings and statistical trends observed, however not 
all interviewees were able to provide exhaustive insights on the performance of ERCEA in all the 
years under consideration. For instance, some interviewees from the parent Directorate-General were 
interacting  with  the  Agency  only  since  (very)  recently  and  were  able  to  shed  light  on  recent 
interactions. Other interviews focused on very specific processes only, which was useful to deepen 
the understanding of these processes and thus contributed to a limited extent to the more overarching 
analysis. 

To the extent possible, the analysis of the evaluation questions was also informed by a comparison 
with  the  results  of  the  previous  evaluation  (2014-2017).  The  triangulation  approach  ensured  the 
robustness  and  reliability  of  the  data  and  information  used  to  draw  up  conclusions  and 
recommendations. However, due to their nature (notably on coherence), the evaluation largely relied 
on  desk  research  and  interview  data.  This  was  mitigated  by  the  extensiveness  of  the  interview 
programme, involving the Commission, ERC Scientific Council and ERCEA staff from various levels 
of management as well as experts, beneficiaries and unsuccessful applicants. 

23   Study Supporting the Evaluation of the European Research Council Executive Agency (ERCEA) 2015 – 2018.  

20 

 
 
 
 
 
 
ANNEX  III  -  EVALUATION  MATRIX  AND  ANSWERS  TO  THE  EVALUATION  QUESTIONS  (BY 
CRITERION) 

The core task of the evaluation was to respond to evaluation questions relating to the effectiveness, 
efficiency,  coherence,  as  defined  by  the  Better  Regulation  guidelines.  Evidence-based  answers, 
including the points of comparison and sources, can be found in Section 4 of the study supporting the 
evaluation performed by an external evaluator24. 

24 https://op.europa.eu/en/publication-detail/-/publication/a6202c1e-e5bd-11ee-8b2b-01aa75ed71a1/language-en/format-PDF/source-310469595 

21 

 
 
 
 
 
 
 
ANNEX IV - OVERVIEW OF BENEFITS AND COSTS 

The benefits and costs are detailed in the cost-benefit analysis in Section 4.2 and Annex V below. 

22 

 
 
 
 
ANNEX V - COST-BENEFIT ANALYSIS 

A cost-benefit analysis was performed to assess the actual situation when compared with the estimates 
in 2013 cost-benefit analysis and specific financial statement. The following approach was adopted: 

•  To use data on the actual performance of ERCEA for the executive agency scenario (actual 

execution of the administrative budget, actual staffing, etc.). 

•  To follow the assumptions laid down in the 2013 cost-benefit analysis and specific financial 
statement to ensure the comparability and validity of results, and to provide estimates of the 
comparable  ‘actual’  in-house  scenario  (the  comparator)  that  would  best  reflect  the  actual 
situation. 

•  To assess based on these estimates whether the conclusions of the ex ante assessment in cost-
benefit analysis and specific financial statement are still valid when compared with the actual 
situation, and what the overall possible savings are. 

To  deconstruct  the  ‘actual’  in-house  scenario  (‘the  comparator’),  estimates  were  based  on  the 
following cost-benefit analysis/specific financial statement assumptions: 

•  Number and composition of staff in the Commission under an in-house scenario correspond 
to  specific  financial  statement  estimations.  Additional  contract  agents  were  added  to  the 
estimated  Commission  staff  number  for  2018-2020  period  (14  contract  agents  in  2018,  9 
contract  agents  in  2019  and  2020)  to  reflect  the  additional  authorised  staff  in  ERCEA  to 
manage the additional budget from the contributions of EEA/EFTA that was not covered in 
the resource calculations in the 2013 cost-benefit analysis and specific financial statement. 

•  Commission staff costs and overheads correspond to the average costs set by Directorate-
General  for  Budget  for  the  estimates  of  human  resources  and  overheads  in  legislative 
financial statements for the respective year. 

•  Programme support expenditure (Title III) remains the same under both the in-house scenario 

and the executive agency scenario. 

The  table  below  presents  the  results  of  the  analysis  of  the  estimated  actual  costs  of  the  in-house 
scenario and the actual costs of the executive agency scenario. 

23 

 
 
 
 
 
 
 
Estimated actual costs of the in-house scenario and the actual costs of the executive agency 
scenario, EUR 

2018 

2019 

2020 

Total 2018-2020 

No. 

Cost 

No. 

Cost 

No. 

Cost 

In-house scenario  

Commission 

Title I. Staff-related expenditure 

590.4 

57 361 500  609.0 

61 488 000  641.5 

66 271 500 

185 121 000 

Temporary Agents 

Contract Agents 

Title II. Infrastructure and 
operating expenditure 

Title III. Programme support 
expenditure 

403.5 

48 016 500 

420 

51 660 000  442.7 

55 337 500 

155 014 000 

186.9 

9 345 000  189.0 

9 828 000  198.8 

10 934 000 

14 169 600 

15 225 000 

16 037 500 

30 107 000 

45 432 100 

2 672 790 

2 537 129 

1 456 300 

6 666 219 

TOTAL COST 

590.4 

74 203 890  609.0 

79 250 129  641.5 

83 765 300 

237 219 319 

Executive agency scenario  

ERCEA 

Title I. Staff-related expenditure 

Temporary Agents 

Contract Agents 

Seconded National Experts 

Interim supportive agents and 
trainees 

Professional development and 
recruitment costs 

Title II. Infrastructure and 
operating expenditure 

Title III. Programme support 
expenditure 

Total ERCEA cost: 

Commission 

Title I. Staff-related expenditure 

Temporary Agents 

Contract Agents 

Title II. Infrastructure and 
operating expenditure 

Total Commission cost: 

494 

120 

363 

11 

37 970 591 

15 375 191 

19 209 529 

475 502 

883 870 

508 

126 

371 

11 

40 559 573 

16 476 910 

20 711 076 

557 382 

577 352 

515 

128 

372 

15 

42 006 374 

120 536 537 

17 453 595 

21 099 003 

780 618 

553 670 

49 305 695 

61 019 608 

1 813 503 

2 014 893 

2 026 498 

2 236 852 

2 119 487 

6 382 838 

7 963 829 

8 416 138 

8 575 288 

24 955 255 

2 672 790 

2 537 129 

1 456 300 

6 666 219 

48 607 210 

51 512 840 

52 037 962 

152 158 012 

8.0 

5.0 

3.0 

745 000 

595 000 

150 000 

192 000 

8.0 

5.0 

3.0 

771 000 

615 000 

156 000 

200 000 

8.0 

5.0 

3.0 

790 000 

625 000 

165 000 

200 000 

2 306 000 

1 835 000 

471 000 

592 000 

937 000 

971 000 

990 000 

2 898 000 

TOTAL COST 

502.0 

49 544 210  516.0 

52 483 840  523.0 

53 027 962 

155 056 012 

ESTIMATED SAVINGS 

88 

24 659 680 

93 

26 766 290 

119 

30 737 338 

82 163 307 

Source: Study supporting the triennial evaluation of the Agency 

The analysis concluded that: 

•  The overall actual costs of the executive agency scenario25 constituted EUR 155.1 million 
during  2018-2020.  The  estimates  in  the  specific  financial  statement  (EUR  149.2  million 

25  

Including the cost of coordination and monitoring by the Commission and the costs covered from EEA/EFTA and third country contributions. 

24 

 
 
  
 
 
 
 
 
 
 
  
  
  
  
  
  
  
 
 
 
 
 
 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
during 2018-2020) were based on the EU contribution, but ERCEA’s administrative budget 
also included contributions from EEA/EFTA and third countries (EUR 6.4 million for 2018-
2020). Consequently, based on the EU contribution only, the actual costs of the executive 
agency  scenario  constituted  EUR  148.7  million,  which  means  that  the  actual  costs 
corresponded  to  the  initial  estimations  (with  savings  of  EUR  0.5  million or  0.4  %  of  the 
specific financial statement estimates). 

•  The costs of the executive agency scenario were much lower than the estimated costs of the 
in-house  scenario.  In  2018-2020,  the  actual  cost  savings  deriving  from  a  cost  difference 
between  the  executive  agency  scenario  and  the  in-house  scenario  constituted  EUR  82.2 
million (or 34.6 % of the estimated costs under the in-house scenario). 

•  Comparing the savings initially estimated in the specific financial statement and cost-benefit 
analysis with the actual savings from the delegation of tasks to ERCEA, the actual savings 
during  2018-2020  period  were  23.4%  higher  compared  to  the  initial  specific  financial 
statement estimates (EUR 82.2 million compared to EUR 66.6 million), and 94.8% higher 
compared to the initial cost-benefit analysis estimates (EUR 82.2 million compared to EUR 
42.2 million). As forecasted in the specific financial statement and the ex ante cost-benefit 
analysis, savings of the executive agency scenario resulted primarily from a higher share of 
lower-cost external personnel (contract agents and seconded national experts) and a lower 
overall number of staff26. 

26   The number of staff under the executive agency scenario was reduced; however, the in-house scenario was not amended in the specific financial 

statement. 

25 

 
 
 
 
 
 
ANNEX VI – STAKEHOLDER CONSULTATION – SYNOPSIS REPORT 

This annex summarises the stakeholder consultation conducted for the evaluation on the basis of the 
‘Better Regulation’ guidelines. A similar approach was adopted for the evaluation of all six agencies. 

1.  Outline of the consultation strategy 

The consultation strategy was designed as follows: 

Source: Study supporting the triennial evaluation of the Agency 

To  reflect  the  scope  of  the  Agency’s  activities,  the  consultation  included  various  categories  of 
stakeholders,  in  particular  Commission  staff  from  the  parent  Directorate-General,  Agency  staff, 
Steering  Committee  and  ERC  Scientific  Council  members,  ERC  programme  beneficiaries, 
unsuccessful applicants, and external experts. Two surveys, an extensive interview programme and 
the triangulation of various data sources were conducted. 

Consultation activity 

Target groups 

Date 

ERC programme beneficiaries and unsuccessful applicants: 
• 

individual  recipients  of  the  different  types  of  grant,  based  on 
applications for calls launched in 2018-2019-2020; 

Survey A 

Surveys 

• 

• 

host  institutions  for  the  different  types  of  grant,  based  on 
applications for calls launched in 2018-2019-2020-2021; 

unsuccessful applicants for the different types of grant, based on 
applications for calls launched in 2018-2019-2020-2021. 

29 November - 
12 December 2022 

Survey B  External experts in the evaluation period (based on the date on which 

a claim for costs was created). 

29 November - 
12 December 2022 

•  Commission staff from the parent Directorate-General 

•  ERCEA Directorate 

•  ERCEA heads and representatives of units 

Interview programme 

•  members  of  the  ERCEA  Steering  Committee  and  Scientific 

Council 

•  ERCEA programme beneficiaries and unsuccessful applicants 

• 

external experts  

Source: Study supporting the triennial evaluations of the Agency 

25 August 2022 - 
31 January 2023 

26 

 
 
 
 
These  different  types  of  stakeholder  consultation  complemented  each  other.  The  survey  of 
unsuccessful applicants, beneficiaries and external experts, and interviews with these stakeholders, 
provided information on effectiveness and efficiency. Interviews with Commission and ERCEA staff, 
as well as with Steering Committee and ERC Scientific Council members, provided information not 
only on these matters, but also made it possible to address coherence criteria.  

Consultation activity for each evaluation criterion 

Effectiveness 

Efficiency 

Coherence 

✓ 

✓ 

✓ 

Interviews 
Commission 

with 

the 

European 

Interviews with ERCEA representatives 

Steering 
Interviews  with  ERCEA 
Committee  and  ERC  Scientific  Council 
members 

Interviews  with  unsuccessful  applicants, 
beneficiaries and experts 

Survey  of  beneficiaries  and  unsuccessful 
applicants 

Survey of external experts  

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

Source: Study supporting the triennial evaluation of the Agency 

2.  Key results of consultation activities 

2.1.   Summarised results of the survey 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

Two surveys were performed: survey A for applicants (beneficiaries and unsuccessful applicants), 
and survey B for external experts. The questions in both surveys aimed to ensure comparability across 
groups of respondents, as well as complementarity with the other evaluation methods used, including 
interviews. The surveys ran for around 2 weeks. 

Survey A addressed individual recipients and their host institutions for the different types of grant 
based on applications for calls launched  in 2018-2019-2020, and unsuccessful applicants and host 
institutions for the different types of grant based on applications for calls launched in 2018-2019-
2020-2021. 450 replies were received from beneficiaries, 125 from the host institutions of principal 
investigators, 984 from unsuccessful applicants, and 442 from their host institutions. These represent 
a response rate of 18% for beneficiaries, 12.6% for  the host institutions of principal investigators, 
20.6% for unsuccessful applicants, and 10.9% for the host institutions of unsuccessful applicants. The 
overall response rate for survey A is 16.2%. 

Survey B targeted all external experts during the evaluation period (based on  the date on which a 
claim for costs was created). 603 replies were received, representing a response rate of 18.5%. 

27 

 
 
 
 
 
 
 
 
 
 
Main statistics related to the implementation of the surveys 

Target group 

Survey 
closure date 

Full launch of 
the survey 
(majority of all 
invitations sent 
on this date) 

No. of 
responses 
received 

Population 
targeted/ 
no. of 
invitations 
sent out 

Response rate* 

No. of 
responses 
included in the 
analysis after 
data cleaning 

Survey A: 
principal 
investigators  

Survey A: 
host 
institutions of 
principal 
investigators  

Survey A: 
unsuccessful 
applicants 

Survey A: 
host 
institutions of 
unsuccessful 
applicants 

Total survey 
A 

Survey B: 
experts 

29 November 

12 December  2 507 

Partial: 65 

Total: 17.95% 

Partial: 48 

Complete: 
385 

Total: 450 

Only including 
complete: 15.36% 

Complete: 385 

Total: 433 

991 

Partial: 46 

Total: 12.61% 

Partial: 14 

Complete: 
79 

Total: 125 

Only including 
complete: 7.97% 

Complete: 79 

Total: 93 

4 781 

Partial: 202 

Total: 20.58% 

Partial: 125 

Complete: 
782 

Total: 984 

Only including 
complete: 16.36% 

Complete: 777 

Total: 902 

4 054 

Partial: 169 

Total: 10.90% 

Partial: 56 

Complete: 
273 

Total: 442 

Only including 
complete: 6.73% 

Complete: 268 

Total: 324 

12 333 

Partial: 482 

Total: 16.22% 

Partial: 243 

Complete: 
1519 

Only including 
complete: 12.32% 

Complete: 
1 509 

Total: 2001 

Total: 1 752 

3 261 

Partial: 84 

Total: 18.49% 

Partial: 0 

Complete: 
519 

Total: 603 

Only including 
complete: 15.92% 

Complete: 511 

Total: 511 

Note: formula-based response rate calculation: 𝑟𝑒𝑠𝑝𝑜𝑛𝑠𝑒 𝑟𝑎𝑡𝑒 =

Source: Study supporting the triennial evaluation of the Agency 

𝑁𝑜 𝑜𝑓 𝑟𝑒𝑠𝑝𝑜𝑛𝑠𝑒𝑠 𝑟𝑒𝑐𝑒𝑖𝑣𝑒𝑑

𝑁𝑜 𝑜𝑓 𝑖𝑛𝑣𝑖𝑡𝑎𝑡𝑖𝑜𝑛𝑠 𝑠𝑒𝑛𝑡 𝑜𝑢𝑡

∗ 100% 

The  data  gathered  through  the  two  surveys  was  analysed  using  both  quantitative  and  qualitative 
methods. A summary of the findings from the survey data, grouped by the indicators set out in the 
evaluation framework, is presented below. 

Effectiveness 

Quality of services provided by the ERCEA 

The ERCEA’s beneficiaries and unsuccessful applicants were (very) satisfied overall with the quality 
of the services provided by the ERCEA during the application and implementation period. 

-  On communication during the application phase, host institutions knew in particular who to 
contact with questions when preparing or submitting applications, and for the vast majority 
the  time  taken  to  respond  to  questions  about  the  application  or  selection  process  was 
reasonable. 

28 

 
-  On  the  user-friendliness  of  the  application  process,  principal  investigators,  unsuccessful 
applicants and host institutions confirmed the clarity and user-friendliness of the information 
for  applicants,  the  eform  used  for  applications/submissions,  and  the  IT  tool  used  for  the 
application process. Most respondents found it easy to access the application results. 

-  On the finalisation phase of the grant agreement, the vast majority of principal investigators 
and  their  host  institutions  felt  that  ERCEA  staff  assigned  to  the  project  during  the  grant 
finalisation and contract negotiation phase had been readily available and responsive and the 
contract easy to understand. 

-  Principal investigators and their host institutions expressed their overall satisfaction with the 

communication and information provided in the implementation phase. 

-  Any principal investigators who were subject to monitoring (checks or reviews of financial 
and/or technical aspects), in relation to their project, or to audits of their organisation after a 
project has ended, were satisfied on the whole with the monitoring and audits carried out. 

-  When applicants were asked if they would consider re-applying for an ERC grant, 94% out 
of  the  370  principal  investigators  surveyed  and  100%  out  of  their  71  host  institutions 
responded  positively.  The  respective  shares  for  unsuccessful  applicants  and  their  host 
institutions were 78% out of 751 applicants and 93% out of 237 host institutions. 

Experts were also very satisfied with the quality of the services provided by the Agency during the 
process to register as an expert and when performing their expert duties. 

-  The  vast  majority  of  experts  were  satisfied  with  the  ERCEA’s  performance  in  terms  of 
managing  the  involvement  of  external  experts  in  evaluation  panels,  ethics  screening  and 
monitoring and/or project monitoring and ex post assessment through peer reviews. 

-  Almost  all  experts  were  satisfied  with  the  overall  quality  of  the  services  provided  by  the 
ERCEA  during  the  process  to  register  as  an  ERCEA  expert  and/or  when  performing  their 
external expert duties, as well as with the overall information provided by the ERCEA as they 
performed their tasks. 

-  On  the  ERCEA’s  facilities  in  Brussels  during  experts’  visits,  the  survey  revealed  high 
satisfaction with the overall service provided at the reception desk, the overall logistics and 
infrastructure,  IT  tools,  internet  availability  and  assistance  provided  by  IT  staff,  and  the 
overall levels of assistance and responsiveness of ERCEA staff during visits to Brussels. 79% 
(301  out  of  381)  were  satisfied  with  the  practical  information  they  got  about  hotels  and 
transport in Brussels. 

-  82% out of the 502 experts surveyed would definitely like to work again with the ERCEA as 

experts, and 15% (77 out of 502) said they would consider it (‘maybe’). 

Quality of communication by the ERCEA 

Unsuccessful  applicants  and  beneficiaries  on  the  whole  were  satisfied  with  the  various  external 
communication  channels,  the  accessibility  and  provision  of  information,  and  communication  in 
general. 

- 

In terms of overall performance, 86% of principal investigators (314 out of 364) and 97% of 
their  host  institutions  (70  out  of  72)  expressed  overall  satisfaction  with  the  ERCEA’s 
communication and interaction throughout the application and project implementation period; 
60% of unsuccessful applicants (379 out of 632) and 71% of their host institutions (150 out 

29 

 
of 212) were satisfied overall with the ERCEA’s communication and interaction throughout 
the application period. 

-  As regards external communication on the application process, most respondents agreed that 
the  overall  application  process  had  been  clear  and  transparent  (96%  out  of  400  principal 
investigators; 79% out of 824 unsuccessful applicants; 96% out of the 85 host institutions of 
principal investigators; 85% out of the 278 host institutions of unsuccessful applicants). 

-  During  the  implementation  phase  of  the  grant/project,  principal  investigators  and  host 
institutions were very satisfied with communication from/with the ERCEA: across all survey 
statements, more than 80% (286 out of 353) of principal investigators and more than 90% (62 
out  of  68)  of  their  host  institutions  were  satisfied  with  communication  during  the 
implementation phase. When asked about the communication channels used by the ERCEA 
during  the  grant/project  and  the  extent  to  which  these  provided  relevant  and  helpful 
information  when  needed,  principal  investigators  and  their  host  institutions  indicated  that 
email contact and the ERC website were their main sources of useful information. 

-  On the importance of different sets of factors related to communication when dealing with the 
ERCEA, all of the factors were considered important by more than 90% of the respondents, 
with the exception of the design and content of the ERCEA website and online communication 
tools (75% out of the 357 principal investigators and 77% out of their 73 host institutions) and 
the user-friendliness of IT tools used during project application/implementation (82%, or 295 
out of the 358 principal investigators). Shares among unsuccessful applicants and their host 
institutions were generally lower, with more neutral responses. 

-  Principal  investigators  and  their  host  institutions  primarily  regarded  the  knowledge  and 
courtesy of ERCEA staff, the ERCEA’s willingness to help and cooperate, and one-to-one 
contacts as relatively more important than unsuccessful applicants and their host institutions 
(between 94 and 98% compared to between 67 and 81%). 

Experts were also satisfied with the various external communication channels, the accessibility and 
provision of information, and overall communication. 

In  terms  of  how  frequently  the  experts  received  guidance  from  the  ERCEA  for  assessing 
proposals/project reports or for performing other tasks, 70%, or 359 out of the 511 experts, indicated 
that they had received initial guidance before conducting the first assessment; 54% (276 out of 511) 
had almost always received guidance before they started to assess a proposal/project or perform other 
tasks; 45%, or 229 out of 511 experts, stated that ad hoc guidance had been provided on request; and 
50%  (256  out  of  511)  indicated  that  they  had  received  continuous  support  throughout  the  project 
cycle. 

Almost all experts (97 to 98%, or 493 out of 507) were positive about  communication during the 
appointment process, i.e. from when the invitation was issued they had enough time to organise their 
schedule/work, and both the invitation letter and related explanations were clear. The usefulness of 
the information provided online was rated positively by experts, yet by far the most useful source for 
experts was direct support from ERCEA staff (96%, or 93 out of 97). 

The  vast  majority  of  experts  agreed  that  the  information  provided  by  the  ERCEA  during  the 
registration/appointment process had been clear and easy to understand (86%, or 244 out of 285). The 
assistance provided by the ERCEA during the registration/appointment process was a notable strong 
point: 86% (219 out of 254) felt it was sufficient and helpful, and 96 to 97% (250 out of 258) rated 
the ERCEA staff with whom they communicated as responsive and qualified to assist them. 

30 

 
In general, there was strong agreement as regards the phase prior to the performance of their tasks 
(between 91 and 97% of experts). However, only 63%, or 153 out of 244 experts, agreed that the 
elearning modules had been an effective learning tool prior to attending the online briefings. 

In terms of the evaluation/monitoring phase, 97%, or 481 out of 496 experts, agreed that they knew 
either  who  to  contact  or  where  to  get  help  on  any  questions  when  working  on  their  task;  all 
respondents  felt that  the  ERCEA staff with  whom they worked were responsive (e.g. by email or 
phone); and 99% (482 out of 485) felt that the answers to their questions were useful. 

Quality of feedback on the application/funding and evaluation/selection processes 

In terms of the processes related to the clarity and transparency of the evaluation/selection application 
processes, principal investigators and their host institutions strongly agreed (between 92 and 96%) 
that the process for selecting high-quality research for funding had been fair and transparent, that the 
individual reviews  and panel  comments  provided  had been  useful, and that  the  overall  evaluation 
process had been clear and transparent. 

However,  lower  scores  on  the  clarity  and  transparency  of  these  processes  were  recorded  among 
unsuccessful  applicants  and  their  host  institutions.  They  agreed  most  strongly  that  the  overall 
evaluation process had been clear and transparent (45% out of 817 unsuccessful applicants; 68% out 
of 258 host institutions of unsuccessful applicants), but only 36 to 40% of unsuccessful applicants 
and 59% of their host institutions agreed that the outcome of their application had been fair and that 
they had no complaints about it, and they had received clear explanations as to why the application 
had  been  turned  down.  The  lowest  scores  related  to  the  individual  reviews  and  panel  comments 
provided to help unsuccessful applicants improve and re-submit their proposals to another funding 
body (23% out of 269; 44% out of 178 respectively) or to the ERC (25% out of 694; 47% out of 183). 

Addressing the challenges posed by the COVID-19 pandemic 

Most respondents agreed that the Agency had reacted swiftly and flexibly to maintain the same level 
of service quality, that the transition from in-person to virtual activities had been managed smoothly 
by  the  Agency,  and  that  –  despite  the  suspension  of  in-person  activities  and  events  –  effective 
information and advice from the ERCEA had reached the applicants and beneficiaries. Beneficiaries 
and  experts  were  more  inclined  to  agree  with  the  statement  (with  shares  of  mostly  90%)  than 
unsuccessful applicants and their host institutions (with shares of 55 to 59% and 77% respectively). 
Beneficiaries  also  agreed  that  the  ERCEA  had  been  flexible  in  adjusting/extending  deadlines  and 
agreeing  to  other  project  changes  necessitated  by  the  COVID-19  pandemic  (87%  of  principal 
investigators (228 out of 262) and 99% of their host institutions (63 out of 64)). 

Efficiency 

Key performance indicators and payments 

83% (310 out of 375) of principal investigators and 85% (65 out of 75) of their host institutions felt 
that the timescale from the call deadline to the announcement of the outcome of the proposal (time-
to-inform), from the announcement of the proposal’s outcome to the signing of the contract (time-to-
sign) and, overall, from the submission of the proposal to the signature of the grant agreement (time-
to-grant) had been appropriate. 

The overall payment process was rated as smooth by 93% of principal investigators (246 out of 266) 
and by 100% of their host institutions (55 out of 55). Principal investigators and their host institutions 
also  strongly  agreed  that  the  Agency’s  deadlines  for  making  pre-payments,  processing  payment 
requests and making payments, had been appropriate (91 to 92% of principal investigators; 100% of 
their host institutions). Similarly, experts strongly agreed that the Agency’s time to process payment 

31 

 
requests or reimbursement requests and to make payments was satisfactory. However, experts agreed 
less that the remuneration received had been commensurate with the efforts involved (72%, or 366 
out of 504) or that sufficient costs had been reimbursed (76%, or 303 out of 399). Remote referees 
who answered the survey of experts agreed that being asked to work for the ERC was prestigious 
(92%, or 457 out of 495) and that peer reviews were important in order to strengthen contributions to 
scientific improvements (97%, or 484 out of 497). 

Granting process and project implementation 

Most principal investigators and their host institutions (between 80 and 97%) felt that the granting 
process and project implementation to date had been clear and transparent. In terms of ethics reviews 
and monitoring processes, between 64 and 78% agreed that these had gone smoothly. 

On the grant/project agreement finalisation phase, principal investigators and their host institutions 
strongly agreed that ERCEA staff assigned to the project during the grant finalisation and contract 
negotiation phase had been readily available and responsive (89%, or 348 out of 394; 95%, or 75 out 
of  79),  that  the  instructions  included  in  the  ERC  invitation  letter  received  at  the  beginning  of the 
granting process had been clear (84%, or 342 out of 405; 90%, or 75 out of 83), and that the same 
was  true  of  ERCEA  requests  on  aspects  such  as  changes  to  proposals  and  providing  missing 
information (84%, or 327 out of 390; 94%, or 74 out of 79). Overall, 80% of principal investigators 
(321 out of 403) and 92% of their host institutions (75 out of 82) agreed that the granting process had 
been clear and transparent. Slightly fewer (but still around two thirds) of respondents agreed that the 
process of validating beneficiaries had gone smoothly and that the degree of effort required had been 
appropriate, that the electronic tools used for the validation and assessment of beneficiaries had been 
user-friendly,  and  that  the  contract  had  been  easy  to  understand.  Ultimately,  49%  of  principal 
investigators  (183  out  of  376)  and  74%  of  their  host  institutions  (61  out  of  82)  felt  that  the  ERC 
granting process  had been  simpler and faster  than that of other national  and international  funding 
bodies. 

In  terms  of  the  ethics  review  process  (guidance  material,  information  requested,  experts,  etc.), 
principal investigators and their host institutions were less supportive of the statements in the survey, 
with  between  59%  and  79%  in  full  agreement.  The  least  supported  statement  was  that  the  ethics 
review  had  facilitated  compliance  with  ethical  guidelines  and  (national,  EU  or  international) 
legislation (59%, or 152 out of 259; 68%, or 37 out of 55). The most widely supported statement was 
that  ERCEA  staff  and  external  experts  assigned  to  the  ethics  review  process  had  clarified  what 
information should be provided (68%, or 184 out of 270; 79%, or 43 out of 54). Overall, 64% (173 
out of 269) of principal investigators and 77% (45 out of 59) of their host institutions agreed that the 
ethics review process had gone smoothly. 

Similarly, 65% of principal investigators (151 out of 234) and 78% of their host institutions (35 out 
of 45) agreed  that the  ethics  monitoring process  had  gone  smoothly.  Between 65%  and 82%  also 
agreed with statements on guidance material and information requested and – most of all – with the 
statement  that  ERCEA  staff  and  external  experts  assigned  to  the  ethics  monitoring  process  had 
clarified what information should be provided. 

On the grant amendment process, most principal investigators with experience of one or more grant 
amendment processes agreed that the ERCEA staff assigned to the process had been readily available 
and responsive, that the time to process grant amendment requests had been appropriate, and that the 
information and advice provided by the Agency during the process had been clear. Overall, 84% of 
principal investigators (110 out of 131) agreed that the grant amendment process had gone smoothly. 

On  the  scientific  and  financial  monitoring  and  reporting  processes,  there  was  strong  agreement 
(between  73%  and  97%  of  respondents)  with  each  of  the  statements  on  process  clarity  (73%  of 

32 

 
principal investigators (253 out of 344) and 90% of their host institutions (55 out of 61)) and in terms 
of  reporting  instructions  (73%,  or  233  out  of  320;  89%,  or  54  out  of  61),  the  positive  impact  of 
splitting financial and scientific reports (79%, or 261 out of 333; 83%, or 55 out of 66), and reasonable 
and  consistent  project  reporting  requirements  (83%  and  91%;  84%  and  97%).  Only  61%  of  the 
principal investigators (158 out of 261), but 88% of their host institutions (40 out of 45) felt that the 
monitoring  activities  carried  out  by  ERCEA  staff  and  external  experts  had  been  useful  for  the 
implementation of the grant/project. Overall, 85% of principal investigators (292 out of 344) and 97% 
of their host institutions (64 out of 66) felt that project implementation to date had gone smoothly. 

Principal investigators with experience of monitoring (checks or reviews of financial and/or technical 
aspects) in relation to their project, or audits of their organisation after the project had ended, were 
surveyed on the follow-up checks as part of their organisations’ grant/project and/or audit procedures. 
Although the number of observations was low, around 70 to 75% of principal investigators reported 
that the ERCEA staff and/or external experts assigned to the monitoring/audit procedure had clarified 
what information should be provided, that the scope of the information they had been asked to provide 
for monitoring/auditing had been  reasonable, and that the overall monitoring/auditing process had 
gone smoothly. 

Simplifications 

Results  from  the  survey  point  to  high  levels  of  satisfaction  among  beneficiaries  and  unsuccessful 
applicants with the simplifications recently adopted by the Agency. Nonetheless, respondents also 
agreed that additional efforts to simplify procedures and requirements would be helpful. 

On  the  other  hand,  many  respondents  did  not  express  any  firm  views,  which  suggests  that  many 
respondents found the question difficult to assess. This also applies to the question on potential further 
simplifications. 

to  compile 

templates  used 

Among principal investigators and their host institutions, most respondents agreed that the simplified 
reporting 
the  project 
application/implementation process. Among unsuccessful applicants, the strongest agreement related 
to the use of simplified cost options instead of real-cost funding, which had facilitated the project 
application/implementation process. The host institutions of unsuccessful applicants most strongly 
agreed with the statement that the wider use of common IT tools across different programmes and 
programme strands had facilitated the project application/implementation process. 

results  had 

facilitated 

structured 

In respect of future simplifications, principal investigators and their host institutions most strongly 
agreed that more user-friendly IT tools for project management and reporting would enable better, 
more efficient programme implementation. Among unsuccessful applicants, three options received 
equal support: 1) the wider use of simplified cost options (lump sums, unit costs and flat rates) instead 
of real-cost funding would allow better, more efficient programme implementation; 2) it would be 
helpful to have more user-friendly IT tools for submitting applications; and 3) further simplification 
of  the  granting  and  reporting  requirements  would  enable  programmes  to  be  implemented  more 
effectively  and  more  efficiently.  The  host  institutions  of  unsuccessful  applicants  most  strongly 
supported the statement  that  the further harmonisation of administrative requirements for projects 
financed under different programmes and actions would also allow better, more efficient programme 
implementation. 

Compared with the answers to the statements on simplification, the host institutions had more diverse 
views on the use of simplified cost options (lump sums, unit costs and flat rates) instead of real-cost 
funding. Although most of the host institutions (69%, or 40 out of 58, of principal investigators and 
74%,  or  163  out  of  221,  of  the  host  institutions  of  unsuccessful  applicants)  agreed  that  this  had 
facilitated the project application/implementation process, 19% (11 out of 40) and 14% (29 out of 

33 

 
221) disagreed with the statement – the largest shares recorded across all the options. On whether 
wider use of simplified cost options (lump sums, unit costs and flat rates) instead of real-cost funding 
would  enable  better,  more  efficient  programme  implementation  in  the  future,  59%  of  the  host 
institutions of principal investigators (40 out of 68) and 73% of the host institutions of unsuccessful 
applicants (178 out of 243) supported this option with all other options falling in the range of 81 to 
92%. At the same time, 25% of the host institutions of principal investigators (17 out of 68) and 13% 
of the host institutions of unsuccessful applicants (31 out of 243) disagreed or strongly disagreed with 
the statement. 

Coherence 

The survey results show very high general awareness among all groups that the ERC programme is 
funded from the EU budget: 99% of principal investigators (421 out of 426); 97% of unsuccessful 
applicants (860 out of 886); 100% of the host institutions of principal investigators (91 out of 91); 
98% of the host institutions of unsuccessful applicants (303 out of 310); and 89% of experts (440 out 
of 497).  

Most respondents were also aware (at the time of their application/invitation to register as an expert) 
that the ERCEA was entrusted by the Commission with managing the programme (between 54% and 
86%). Host institutions of principal investigators (66%) showed reasonable awareness of the roles of 
the  ERCEA,  the  ERC  Scientific  Council  and  the  Commission.  88%  of  experts  (446  out  of  505) 
indicated that they were aware of the ERC’s remit. Overall, 87% of principal investigators (372 out 
of 426), 71% of unsuccessful applicants (619 out of 866), 86% of the host institutions of principal 
investigators (80 out of 93), 75% of the host institutions of unsuccessful applicants (232 out of 308) 
and 91% of external experts (448 out of 494) stated that the funding opportunities under the ERC 
programme had been promoted effectively. 

2.2. 

Summarised results of the interview programme 

The  interview  programme  encompassed  scoping,  main-phase  and  follow-up  interviews  with  the 
representatives  of  the  Commission,  the  ERCEA,  the  Steering  Committee  and  the  ERC  Scientific 
Council, as well as with beneficiaries, unsuccessful applicants and external experts. The interview 
programme was designed to embrace a wide range of views, including from the Commission and the 
Agency as well as from other stakeholders. 

The interviews were conducted according to a standardised questionnaire, including questions on the 
evaluation criteria that were adjusted to the experiences of individual stakeholders. At each interview, 
notes were  compiled for further analysis. A total of 40 interviews – more than anticipated – were 
conducted. The interview phases and the participants are summarised in the table below. 

The interview data fed into responses to the relevant evaluation questions, in particular the aspects of 
these  questions  where  respondents’  opinions  were  of  primary  importance.  All  evidence  from  the 
interviews was incorporated into the evaluation’s final report. A summary of the findings from the 
interview data is presented below. 

General overview of the interview programme 

Interview phase 

Target group 

Exploratory/scoping  Directorate-General for Research and Innovation 

ERCEA 

Main phase 

Directorate-General  for  Research  and  Innovation,  including 
Steering Committee Chair and Vice-Chair 

No. of completed 
interviews  

1 (with 3 interviewees) 

1 (with 2 interviewees) 

10 (with 11 interviewees) 

34 

 
ERCEA 

ERC Scientific Council 

15 (with 19 interviewees) 

4  

Follow-up 

Directorate-General for Research and Innovation 

1 (with 2 interviewees) 

Beneficiaries 

Unsuccessful applicants 

Experts 

ERCEA 

Total:  

Source: Study supporting the triennial evaluation of the Agency 

Effectiveness 

2 

2 

2  

2 (with 9 interviewees) 

40 (with 56 interviewees) 

Interviews confirmed that ERCEA had achieved its objectives in implementing the ERC programme 
and its delegated tasks. Interviewees from the parent Directorate-General also confirmed high levels 
of  satisfaction  with  the  Agency’s  performance  and  its  interactions  with  the  parent  Directorate-
General.  The  ERCEA  had  been  excellent  in  terms  of  managing  the  programmes  and  grants, 
implementing  calls  in  line  with  scheduled  planning,  and  meeting  its  targets  for  key  performance 
indicators.  All  interviewees  from  the  parent  Directorate-General  and  the  ERC  Scientific  Council 
confirmed that the Agency had made a significant contribution to the implementation of the Horizon 
2020 ‘Excellent Science’ pillar in the 2018-2021 period. 

Interviewees also reported that the high levels of expertise and commitment on the part of Agency 
staff had been key in responding to the challenges faced during the evaluation period. All interviewees 
from the Agency, the ERC Scientific Council and the parent Directorate-General confirmed that the 
ERCEA had been highly adaptable and flexible, enabling it to maintain high performance standards. 
According  to  interviewees,  the  keywords  summarising  the  Agency’s  response  to  the  COVID-19 
pandemic  were  preparedness,  sense  of  commitment,  and  crisis  planning.  In  this  context,  several 
interviewees indicated that during the pandemic ERCEA staff had made additional efforts to simplify 
procedures  and  support  beneficiaries.  With  regard  to  changes  in  ERC leadership,  all  interviewees 
agreed that this had not affected the Agency’s performance. Respondents cited the experience and 
commitment of Agency staff in ensuring that the HR team’s processes and support for the switch to 
remote working went smoothly while developing mechanisms to facilitate team communication and 
promote well-being. 

Interviewees also confirmed that, during the evaluation period, the ERCEA had effectively provided 
high-quality and flexible technical support to the ERC Scientific Council. Interviewees indicated that 
the support provided by the ERCEA, in particular through units A1 and A2, to the ERC Scientific 
Council had been highly effective and professional as well as competent in technical and scientific 
terms. All ERC Scientific Council members involved in the interview programme were very satisfied 
with the quality and quantity of ERCEA support. Even during the pandemic, the support provided to 
the  ERC  Scientific  Council  was  of  high-quality  and  a  key  factor  in  the  correct  and  prompt 
administration of the ERC Scientific Council’s work. Interviewees from the ERC Scientific Council 
stressed the fact that the Agency often provided the ERC Scientific Council with information both on 
request as well as spontaneously and that the Agency and the ERC Scientific Council collaborated 
flexibly on the basis of mutual understanding and trust. In this regard, interviewees from the ERCEA 
and the ERC Scientific Council emphasised the importance of the Agency’s communication unit in 

35 

 
 
 
providing  tailored  communication  capable  of  building  on  expertise  from  the  Agency’s  other 
departments27. 

Interviews  with  the  Agency  confirmed  three  main  risks  to  its  effectiveness:  potential  cuts  in  the 
multiannual financial framework, in particular the consequence of a possible hard Brexit; the need to 
maintain  respect  for  the  ERCEA  as  the  ERC’s  dedicated  implementing  structure  and  potential 
negative  impacts  on  the  Agency’s  flexibility  and  agility  in  responding  to  future  programming 
priorities established by the ERC Scientific Council and in the quality of services delivered to the 
beneficiaries because of further imposed harmonisation. 

In terms of contacting applicants and principal investigators, interviewees stressed that, in general, 
collective efforts had been made to improve the Agency’s interaction with programme participants 
during the evaluation period (for instance, desk research directed at actions taken in response to the 
previous  evaluation,  such  as  raising  awareness  about  grant  management  workshops  for  principal 
investigators,  host  institutions/principal  investigator  events  and  increasing  the  visibility  of  these 
events through national contact points, communication events and the ERC website28). Interviewees 
pointed out that the ERCEA had also been active in helping beneficiaries and internal staff to develop 
skills for effectively communicating results. For example, a communication coach was hired in the 
event where a beneficiary unfamiliar with communication techniques had to give a presentation on 
their  research  findings.  Former  journalists  were  hired  to  provide  media  training  to  improve 
communication effectiveness. Interviewees also mentioned that interactions with beneficiaries and 
unsuccessful applicants alike showed that they had been very satisfied with the Agency’s services. 

On the other hand, the feedback from some interviews suggested that, while communication on results 
and the efforts made to promote the ERC’s visibility and its reputation for scientific excellence had 
been very effective, the involvement of the parent DG in some of the communication activities had 
been relatively limited. 

Efficiency 

All interviewees recognised that the ERCEA had been able to implement efficiency guidelines and 
streamline its processes over the years. The Agency feels that past efficiency gains cannot be repeated 
to the same extent in the future; however, the ‘Continuous Improvement’ programme does make it 
possible to identify opportunities that remain in this regard. 

In  the  area  of  IT,  several  interviewees  indicated  that  a  number  of  IT  tools  (e.g.  Bluebell  and 
Speedwell) developed and adopted by the Agency had helped to improve programme management in 
terms  of  simplifying  procedures.  Interviewees  also  confirmed  that  a  number  of  technical 
simplifications applied during the evaluation period had reduced the pressures on both beneficiaries 
and Agency staff.  

Overall, most interviewees confirmed that the IT tools had played a positive role in advancing process 
simplification. Corporate tools and internal developments alike were helpful in this regard. However, 
several  interviewees  suggested  that  the  corporate  tools  needed  to  maintain  sufficient  flexibility  to 
adapt efficiently to ERCEA specificities. Some matters needed to maintain direct, personal contact 
with  beneficiaries.  Although  the  use  of  IT  tools  was  seen  as  helpful,  as  they  reduced  the  overall 

27  

In 2021, the internal communication team within ERCEA was downsized and moved from the HR unit to Communication unit A2 in order to 
harness greater synergies between communication professionals (ERCEA, 2021 annual activity report). 

28   Action plan in response to prior evaluation, 7 December 2020. National contact points were set up across Europe by the national governments to 
provide information and personalised support to ERC applicants in their native language. Their mission was to raise awareness, inform and offer 
advice on ERC funding opportunities as well as support potential applicants in preparing, submitting and following up ERC grant applications. 

36 

 
 
administrative burden, more procedural simplification ran the risk of reducing the capacity to deal 
with any one-off demands from beneficiaries. 

With regard to processes, several interviewees indicated that ERCEA staff had worked extra hard 
during  the  pandemic  to  simplify  procedures  and  support  beneficiaries,  i.e.  in  relation  to  the 
amendment procedure. Interviewees also agreed that internal audits in general had been more than 
suitable and efficient and had always provided useful insights for improving processes. 

Interviews with the Agency’s HR staff also clarified several aspects in the HR strategy further. For 
instance,  the  main  reason  for  the  lower  rate  of  occupancy  for  posts  in  2021  was  said  to  relate  to 
intensive recruiting by new or expanding  executive agencies, resulting  in  the  higher turnover rate 
seen for the ERCEA and in more posts being filled by interim staff. 

Interviews with Agency staff confirmed that the ERCEA had maintained appropriate staffing levels 
for the Agency’s structure and appropriate flexibility to re-assign resources as dictated by workload 
fluctuations. It had also reacted promptly and effectively to the challenges posed by the COVID-19 
pandemic. Several interviewees felt that the Agency’s overall size and structure was appropriate, but 
others felt that specific departments should be consolidated. 

Interviewees in the Agency and the ERC Scientific Council felt that the workload facing ERCEA 
staff  was  high  and  had  increased  over  time.  The  COVID-19  pandemic  had  been  a  particularly 
challenging  period  because  it  became  necessary  to  organise  all  interviews  and  evaluation  panels 
remotely with people across different time zones. Several interviewees mentioned this as a time when 
assorted staff members had worked extra hard to manage these arrangements. It was also cited as an 
indication  that  ERCEA  staff  were  motivated  and  dedicated  in  terms  of  ensuring  that  the  Agency 
implemented high-quality grant management and evaluation processes. Alongside the greater number 
of proposals and running projects each year, interviewees cited other aspects behind this increase in 
workload: continuing to manage legacy projects in subsequent years; adapting to and customising 
processes to implement the corporate strategy; and – towards the end of the evaluation period – the 
implications of uncertainty as regards the UK and Switzerland (including ongoing evaluations for the 
UK). 

Interviewees also pointed to limited career progression prospects, in particular for contract staff, as 
one of the causes of staff turnover. Interviewees also mentioned limited access for contract staff or 
temporary staff to management positions and barriers to mobility. At the same time, they reported 
that inter-agency mobility was an effective way to improve staff motivation and development. 

Assorted  interviewees  felt  that  the  recruitment  process  took  too  long  and  provided  further 
explanations for this in relation to the availability of sufficiently filled reserve lists (with the relevant 
profiles) and the agility of the recruitment process when it came to screening CVs on the EPSO list. 

Coherence 

Interviews  confirmed  strong  continuity  in  the  Agency’s  programming  responsibilities  for 
implementing the ERC programme, as well as in the delegated tasks in terms of providing technical 
support to facilitate the work of the ERC’s Scientific Council. 

The  finding  in  the  previous  evaluation  on  the  harmonisation  approach  (namely  that  it  provided 
benefits, but sufficient flexibility had to be maintained to customise tools and meet the specific needs 
and characteristics of the ERC programme) was confirmed by interviewees in the current reference 
period. Specific concerns were raised about how long it took Agency staff to engage in this process. 
Moreover, the balance between  corporate harmonisation  and the ERCEA’s autonomy as the ERC 

37 

 
programme’s ‘dedicated implementation structure’ and the provider of support to the ERC Scientific 
Council in all its tasks, was questioned in this context. 

Overall, interviewees from both the parent Directorate-General and ERC Scientific Council stressed 
the importance of the quality and dedication of staff and highlighted the cooperation of staff working 
to  support  the  Scientific  Council  and/or  parent  Directorate-General  with  staff  from  the  scientific 
department to deepen analyses and inputs. According to the interviewees, this unique environment 
created significant added value with regard to the inputs provided by the ERCEA. 

Interviewees also indicated that the ERCEA provided highly effective, relevant, strategic and high-
quality feedback and inputs to inform the work of the ERC Scientific Council. Interviewees reported 
that the support from the ERCEA had been highly effective and professional as well as competent in 
technical and scientific terms. Even during the pandemic, the support provided to the ERC Scientific 
Council had been of high quality and a key factor in the work of the ERC Scientific Council being 
discharged correctly and promptly. Interviewees from the ERC Scientific Council stressed that the 
Agency often provided the ERC Scientific Council with information not only on request, but also 
spontaneously, and that the collaboration between the Agency and the ERC Scientific Council was a 
flexible one based on mutual understanding and trust. 

Interviewees further recognised that the ERCEA provided sufficient and relevant information on the 
programme’s operational implementation and that, in terms of supervision, the Commission’s role in 
the  Steering  Committee  allowed  it  to  closely  monitor  the  ERC  programme’s  implementation. 
Interviewees from the ERC Scientific Council confirmed that the Agency was dedicated to promptly 
providing information and inputs to prepare and inform Steering Committee meetings as and when 
needed.  The  representatives  of  the  parent  Directorate-General  would  prefer  to  receive  the  inputs 
earlier prior to Steering Committee meetings to give them more time to prepare. To further support 
monitoring in the Steering Committee and via the annual activity report, interviewees reported that 
changing  indicators  over  time  to  denote  progress  made  towards  achieving  the  objectives,  and 
explaining/contextualising them, would be helpful. 

Furthermore,  interviews  with  representatives  of  the  parent  Directorate-General  showed  that 
communications  on  preparing  and  adopting  processes  (including  with  Directorate-General  for 
Budget) had been  cooperative and respectful.  Finally, the interviews with  ERC Scientific Council 
members also revealed how the ERCEA supported effective information flows between the Scientific 
Council, the parent Directorate-General and itself. Interviewees underlined that other regular thematic 
meetings on specific topics (business processes, IT tools, etc.) had maintained an effective loop for 
feedback between the ERCEA and the parent Directorate-General. 

Several interviewees recognised the added value of seconding officials from the parent Directorate-
General to the ERCEA to strengthen the ties between the two: seconded officials maintained personal 
links with both the Commission and the Agency, helping to improve mutual understanding. 

2.3.  Comparison of the results of consultation activities 

The  table  below  presents  key  results  for  individual  consultation  activities,  arranged  according  to 
evaluation criteria and the extent to which results were consistent, complementary or contradictory 
across the consultation activities. Overall, as summarised in the table below, there was a high degree 
of convergence in the findings from the different consultation activities. 

38 

 
 
Key findings of the consultation activities 

Evaluation 
criterion 

Survey 

Interview programme 

Consistent 
results 
across 
consultation 
activities 

Complementary 
results across 
consultation 
activities 

Contradictory 
results across 
consultation 
activities 

High 

High 

Low 

All interviewees confirmed that the ERCEA had achieved its objectives in 
implementing the ERC programme and its delegated tasks. Interviewees also 
confirmed that the ERCEA had provided the ERC Scientific Council with 
effective technical support. 

All interviewees confirmed the ERCEA’s high degree of adaptability and 
flexibility in responding to challenges, in particular those caused by the COVID-
19 pandemic. Interviewees cited the high level of expertise and commitment of 
Agency staff as key in this regard. 

The main risks for to effectiveness that are confirmed by interviewees from the 
Agency are related to the consequences of a possible hard Brexit, the need to 
continue respecting the set-up of ERCEA as a dedicated implementation structure 
for the ERC, and the potential negative impacts of further imposed harmonisation. 

Communication on results and the efforts to promote the ERC’s visibility and 
reputation for scientific excellence were rated as effective by interviewees, who 
also mentioned that collective efforts had been made to improve the Agency’s 
interaction with programme participants during the evaluation period and to 
support beneficiaries and internal staff in developing skills to communicate the 
results achieved effectively. On the other hand, feedback from some interviewees 
suggested that the involvement of the parent Directorate-General in 
communication activities could be improved further. 

Effectiveness  Beneficiaries expressed high levels of satisfaction with the 
ERCEA’s performance throughout the project management 
process. External experts on the whole were satisfied with the 
services provided by the Agency when registering as experts and 
while performing their expert tasks. Most experts would like to 
work again with the ERCEA in the future. 

Most principal investigators and their host institutions rated the 
evaluation process as fair, transparent and clear. Unsuccessful 
applicants and their host institutions were less positive, 
particularly as regards the usefulness of feedback to help them 
improve and re-submit proposals to a different funding body. A 
significant proportion of beneficiaries would re-apply for an ERC 
grant. 

Most of the beneficiaries, unsuccessful applicants and external 
experts were satisfied with communication from the ERCEA and 
their interactions with the Agency. Most principal investigators 
and their host institutions rated the knowledge and courtesy of 
ERCEA staff, and the ERCEA’s willingness to help and 
cooperate as well as the individualised attention provided, as 
relatively more important than unsuccessful applicants and their 
host institutions. Experts considered direct support from ERCEA 
staff to be the most useful source during the appointment and 
registration process. 

Principal investigators and their host institutions agreed that the 
Agency had reacted swiftly and flexibly to address challenges 
posed by the COVID-19 pandemic. 

Most principal investigators and their host institutions rated the 
processes implemented by the ERCEA as suitably prompt, the 
granting process and project implementation as clear and 
transparent, and the grant amendment and payment processes as 
smooth. A lower number, but still an overwhelming majority, 
agreed that the ethics review and monitoring processes had gone 
smoothly. 

Experts too agreed strongly that the payment processes were 
sufficiently prompt (albeit while being less convinced that the 

Efficiency 

All interviewees recognised that the ERCEA had successfully implemented 
efficiency guidelines and streamlined its processes over the years. 

High 

High 

Low 

Most interviewees confirmed that the IT tools – both corporate tools and internal 
developments – had played a positive role in simplifying procedures. Several 
interviewees from the Agency suggested that corporate tools had to maintain 
sufficient flexibility to adapt efficiently to ERCEA specificities. 

39 

 
 
payment received was commensurate with the efforts involved) 
and that costs were adequately reimbursed. 

Most of the beneficiaries and unsuccessful applicants surveyed 
were satisfied with the simplifications recently adopted by the 
Agency, but agreed that there was still room for further 
improvement, particularly in terms of making the IT tools more 
user-friendly. Unsuccessful applicants also cited future 
simplifications involving simplified cost options and granting and 
reporting requirements. The host institutions of unsuccessful 
applicants cited further harmonisation of administrative 
requirements for projects financed under different programmes 
and actions. The host institutions’ views were more diverse on 
the use of simplified cost options (lump sums, unit costs and flat 
rates) instead of real-cost funding. 

Survey results showed very high general awareness among all 
groups that the ERC programme is funded from the EU budget 
and that the ERCEA is entrusted with managing the programme. 
Most respondents found that the funding opportunities under the 
ERC programme were promoted effectively. 

The host institutions and experts were reasonably aware of the 
role of the ERCEA, the ERC Scientific Council and the 
Commission.  

Coherence 

With regard to processes, interviewees indicated that during the pandemic 
ERCEA staff had worked extra hard to simplify procedures and support 
beneficiaries, i.e. in relation to the amendment procedure. 

Interviews with staff within the Agency confirmed that the ERCEA had 
maintained appropriate staffing levels for the organisation’s structure and been 
flexible enough to re-assign resources as dictated by workload fluctuations. 
Interviewees within the Agency and the ERC Scientific Council felt that the 
workload facing ERCEA staff was high and had increased over time. 

Interviewees also cited limited career progression prospects, particularly for 
contract staff, as one of the causes of staff turnover. 

Further improvements to the availability of sufficiently filled reserve lists and to 
the process for screening CVs on the EPSO list were mentioned as opportunities 
to improve and shorten the recruitment process.  

Interviewees confirmed strong continuity in the Agency’s programming 
responsibilities for implementing the ERC programme and in its delegated tasks. 
While the harmonisation approach had brought benefits, sufficient flexibility 
needed to be preserved to customise tools to meet the specific characteristics of 
the ERC programme, as confirmed by interviewees during the current reference 
period. 
Interviewees from the ERC Scientific Council indicated that the ERCEA had been 
very effective in providing relevant, strategic and high-quality feedback and 
inputs to inform the work of the ERC Scientific Council. Interviewees from the 
representatives of the parent Directorate-General and the ERC Scientific Council 
recognised that the ERCEA had provided sufficient and relevant information on 
the operational implementation of the programme, but representatives from the 
parent Directorate-General would prefer to receive the inputs earlier prior to 
Steering Committee meetings and had suggested more reporting on 
contextualisation and changes to indicators over time. 
The interviews with ERC Scientific Council members indicated further that the 
ERCEA had supported effective information flows between the ERC Scientific 
Council, the parent Directorate-General and the ERCEA. Interviewees underlined 
that other regular thematic meetings on specific topics (business processes, IT 
tools, etc.) had maintained an effective loop for feedback between the ERCEA and 
the parent Directorate-General. 

Several interviewees recognised the added value of seconding officials from the 
parent Directorate-General in the ERCEA in terms of strengthening ties and 
improving mutual understanding.  

40 

High 

High 

Low