COMMISSION STAFF WORKING DOCUMENT Evaluation of the Research Executive Agency (REA) Accompanying the document Report from the Commission to the European Parliament, the Council and the Court of Auditors Evaluation of the Consumers, Health, Agriculture and Food Executive Agency, the Executive Agency for Small and Medium-Size Enterprises, the Innovation and Networks Executive Agency, the Education, Audiovisual and Culture Executive Agency, the Research Executive Agency and the European Research Council Executive Agency
Date: 2020-04-29
Institutions: European Commission, Secretariat-General
Subjects: EU aid, EU financing, EU programme, executive agency, financial control, performance audit, research

EN 

  EN 

   EUROPEAN COMMISSION  Brussels, 29.4.2020  SWD(2020) 77 final  COMMISSION STAFF WORKING DOCUMENT Evaluation of the Research Executive Agency (REA) Accompanying the document Report from the Commission to the European Parliament, the Council and the Court of Auditors Evaluation of the Consumers, Health, Agriculture and Food Executive Agency, the Executive Agency for Small and Medium-Size Enterprises, the Innovation and Networks Executive Agency, the Education, Audiovisual and Culture Executive Agency, the Research Executive Agency and the European Research Council Executive Agency {COM(2020) 184 final} - {SWD(2020) 73 final} - {SWD(2020) 74 final} - {SWD(2020) 75 final} - {SWD(2020) 76 final} - {SWD(2020) 78 final}  
 
 
TABLE OF CONTENTS 

1. 

INTRODUCTION .............................................................................................................. 2 

2.  BACKGROUND ................................................................................................................ 3 

2.1 Description of the intervention and its objectives ............................................................ 3 

2.2 Baseline and points of comparison ................................................................................... 5 

3. 

IMPLEMENTATION / STATE OF PLAY ........................................................................ 6 

4.  METHOD ........................................................................................................................... 7 

4.1 Short description of methodology .................................................................................... 7 

4.2 Limitations and robustness of findings ............................................................................. 8 

5.  ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS .......................... 8 

5.1 Effectiveness ..................................................................................................................... 9 

5.2 Efficiency ........................................................................................................................ 12 

5.3 Coherence ....................................................................................................................... 14 

5.4 Retrospective Cost-Benefit Analysis .............................................................................. 16 

6.  CONCLUSIONS ............................................................................................................... 17 

ANNEX : EVALUATION QUESTIONS ................................................................................ 19 

1 

 
 
 
 
 
 
 
1. 

INTRODUCTION 

The purpose of this periodical (every three years) evaluation of the operation of the Research 
Executive Agency (REA) is to assess REA’s implementation of the parts of the EU funding 
programmes which have been entrusted to it and the provision of administrative and logistical 
support services. REA is governed by the Framework Regulation for Executive Agencies1, its 
Act of Establishment2, which sets out its mandate, the Act of Delegation3 which specifies the 
tasks  to  be  carried  out  by,  and  the  powers  delegated  to  the  Agency  in  order  to  perform  its 
mandate; and the decision establishing the REA Steering Committee. 

In  line  with  the  Commission’s  Better  Regulation  principles4,  the  evaluation  applies  several 
standard evaluation criteria. The evaluation assesses whether the Agency has fulfilled its tasks 
in  an  effective  and  efficient  way,  whether  there  are  overlaps  /  gaps  /  inconsistencies  in  the 
management  of  the  programme  portfolio  by  the  Agency,  and  whether  there  is  a  clear 
delineation  of  tasks  between  REA  and  the  parent  DGs  or  other  executive  agencies 
(coherence)5. 

The evaluation also assesses whether the functioning of the Agency has yielded the expected 
positive results as estimated in the ex ante Cost-Benefit Analysis (CBA) for the delegation of 
tasks to  the executive agencies6 and to  identify potential areas of improvement. To this end 
the  estimations  of  the  Cost-Benefit  Analysis  of  2013  have  been  tested  in  order  to  provide 
evidence on the validity of the assumptions made in the ex ante scenario by considering the 
actual  costs  and  benefits  of  programme  implementation  by  the  Agency  in  a  structured  way. 
The  aspects  covered  by  the  Cost-Benefit  Analysis  are  specified  in  Article  3(1)  of  the 
Framework Regulation7 and the Guidelines on establishing and operating executive agencies8. 

The evaluation does not cover the achievements of the programmes managed by REA, which 
are  subject  to  mid-term  and  ex  post  evaluations  themselves.  The  evaluation  of  REA 
nevertheless  provides  useful  input  for  these  programme  evaluations,  considering  that  the 

1 Council Regulation (EC) No 58/2003 of 19 December 2002 laying down the statute for executive agencies to 
be  entrusted  with  certain  tasks  in  the  management  of  Community  programmes,  Official  Journal  L  011,  16 
January 2003 pp.1-8  http://data.europa.eu/eli/reg/2003/58/oj 
2 Commission Implementing Decision 2013/778/EU of 13 December 2013 establishing the Research Executive 
Agency and repealing Decision 2008/46/EC, http://data.europa.eu/eli/dec_impl/2013/778/oj 
3  Commission  Decision  C(2013)9418  of  20.12.2013  on  delegating  powers  to  the  Research  Executive  Agency 
with a view to performance of tasks linked to the implementation of Union programmes in the field of research 
and innovation comprising, in particular, implementation of appropriations entered in the general budget of the 
Union, http://ec.europa.eu/transparency/regdoc/rep/3/2013/EN/3-2013-9418-EN-F1-1.PDF 
4 Commission’s Better Regulation  Communication COM (2015) 215 final and its accompanying  SWD (2015) 
111 final and SWD (2015) 110 final. 
5 The assessment of ‘EU added value’, why the EU should act, is not perceived to be a relevant criterion, as REA 
executes tasks, which the European Commission has transferred to it. The EU added value of the programmes 
that REA manages is assessed as part of the programme evaluation. The previous needs which REA were meant 
to address and whether they still are pertinent today (relevance) are presented in the context of the evaluation. 
6 Cost-Benefit analysis for the delegation of certain  tasks regarding the  implementation of Union Programmes 
2014-2020 to the executive agencies - Final report for the Commission of 19 August 2013. 
7  Identification  of  the  tasks  justifying  outsourcing,  a  Cost-Benefit  Analysis  which  includes  the  costs  of 
coordination  and  checks,  the  impact  on  human  resources,  possible  savings  within  the  general  budgetary 
framework  of  the  European  Union,  efficiency  and  flexibility  in  the  implementation  of  outsourced  tasks, 
simplification of the procedures used, proximity of outsourced activities to  final beneficiaries, visibility of the 
EU as promoter of the EU programme concerned and the need to maintain an adequate level of know-how inside 
the Commission.  
8  Appendix  II  of  the  Guidelines  for  the  establishment  and  operation  of  executive  agencies  financed  from  the 
Union budget (C (2014) 9109 from 2 December 2014; pp. 64-72). 

2 

 
 
 
  
 
                                                           
performance  of  the  Agency  affects  the  efficiency  and  effectiveness  of  the  programmes  it 
manages. 

The evaluation  examines the efficient  use of resources and the  effective achievement of the 
tasks  entrusted  to  the  Agency.  In  particular,  it  looks  at  whether  the  alignment  of  more 
coherent programme portfolios within the Agency’s core competences and its brand identity 
delivered  the  estimated  qualitative  benefits;  whether  the  assembly  of  the  management  of 
different EU programmes and provision of support services delivered the estimated synergies, 
simplification  and  economies  of  scale;  whether  the  pooling  of  instruments  guaranteed 
consistent service delivery and whether there is scope for simplification and further efficiency 
gains. 

The evaluation covers all the tasks carried out by the Agency during the period 16 July 2015 
to 15 July 2018. The evaluation is supported by a study carried out by an external contractor. 

The results of this study are summarised hereafter and this evaluation will be presented to the 
European Parliament, to the Council and to the Court of Auditors in accordance with Article 
25(1) of the Framework Regulation.  

The results will feed into the reflection to assess the opportunity of expanding/ modifying the 
Agency’s  mandate  in  view  of  the  delegation  of  the  implementation  of  2021-2027  EU 
programmes. 

2.  BACKGROUND 

2.1 Description of the intervention and its objectives 

The  outsourcing  of  certain  management  tasks  to  the  Agency  according  to  the  Framework 
Regulation and REA’s Act of Establishment intends: 

  To allow the Commission to focus on its institutional tasks, i.e. tasks assigned to the 
institutions  by  the  Treaty,  which  require  discretionary  powers  in  translating  political 
choices into action. Such institutional tasks should not be outsourced. 

  To enable the Commission to achieve the objectives of the delegated EU programmes 
more effectively and efficiently. According to the Cost-Benefit Analysis carried out in 
2013,  delegating  tasks  to  REA  was  estimated  to  be  more  cost-efficient  than  an  in-
house  scenario.  The  Act  of  Establishment  estimated  that  the  alignment  of  more 
coherent  programme  portfolios  with  the  Agency’s  core  competences  and  its  brand 
identity would bring qualitative benefits. In addition, it projected that assembling the 
management  of  different  EU  programmes  would  bring  synergies,  simplification  and 
economies of scale. 

The original objective of the intervention is entrusting the Agency with the implementation of 
several EU programmes in parts or fully: 

  Grants  to  Horizon  2020  framework  programme  for  research  and  innovation 
projects  (Horizon  2020).  REA  manages  scientific  projects  funded  under  the 
following parts of the Horizon 2020 framework programme (grouped by pillar): 

o  Excellent Science: 

  Marie Skłodowska-Curie Actions (MSCA); 
  Future and Emerging Technologies. 

o  Industrial leadership: 

3 

 
 
 
 
 
 
 
 
  Space Research. 

o  Societal challenges: 

  Food Security, Sustainable Agriculture and Forestry, Marine, Maritime 

and Inland Water Research and the Bioeconomy; 
Innovative, Inclusive and Reflective Societies; 

 
  Secure Societies (partially). 

o  Specific objectives: 

  Spreading Excellence and Widening Participation; 
  Science with and for Society. 

As of 2018, REA’s mandate was extended to the processing of grants to projects that 
may generate EU classified information (EUCI), including under the specific objective 
Secure Societies. 

  Research  projects  of  the  7th  framework  programme  for  research  and  innovation 
(FP7).  The  Agency  continues  to  manage  ongoing  FP7  research  projects  in  the 
following areas: 

o  Certain programme areas of the specific programme PEOPLE of FP7; 
o  The  actions  research  for  SMEs  and  research  for  SME  associations  of  the 

specific programme capacities of FP7; 

o  The theme space of the specific programme cooperation of FP7;  
o  The theme security of the specific programme cooperation of FP7 (partially). 

  In addition, REA was entrusted with administrative and logistical support services: 
o  The  Agency  runs  the  support  for  independent  experts  evaluating  project 
proposals, including the management of on-site evaluation facilities for most of 
Horizon  2020  activities  and  provides  services  for  the  legal  and  financial 
validation of participants in Horizon 2020 and certain other programmes; 
o  As  of  2018,  in  the  context  of  the  Commission’s  initiatives  for  synergies  and 
in  the 
efficiencies  between  programmes,  REA  plays  a  major  role 
development  and  roll-out  of  the  Single  Electronic  Data  Interchange  Area 
(SEDIA) project, building on REA’s participant validation activities supported 
by  the  Participant  Data  Management  tool.  This  entails  the  legal  validation  of 
third  parties  and  the  preparation  of  financial  capacity  assessments  for  all  the 
Commission  services  and  executive  agencies  (which  implement  grants  and 
procurements  under  direct  management)  and  the  first  level  of  transactions 
under indirect management9.  

The Agency has six parent Directorates-General: 

Internal Market, Industry, Entrepreneurship and SMEs (GROW);10 

  Research and Innovation (RTD) – Lead parent DG; 
 
  Education and Culture (EAC); 
  Communications Networks, Content and Technology (CNECT); 
  Agriculture and Rural Development (AGRI); 
  Migration and Home Affairs (HOME). 

9  Validation  services  to  Joint  Undertakings  are  also  provided  by  REA  not  under  SEDIA  but  within  the 
framework of Horizon 2020 support services. 
10 As of 1 January 2020 Directorate-General for Defence, Industry and Space (DEFIS) replaced DG GROW. 

4 

 
 
 
 
 
 
                                                           
The  Agency  also  works  closely  with  the  Common  Support  Centre  (CSC),  now  named  the 
Common Implementation Centre (CIC), administratively part of DG RTD, which provides the 
framework  for  harmonised  implementation  of  the  EU  Research  and  Innovation  (R&I) 
programmes  for  entire  DGs  in  the  R&I  family  of  implementing  bodies,  and  executive 
agencies  and  Joint  Undertakings.  The  CSC  is  responsible  for  defining  common  business 
processes  for  grant  management,  and  providing  IT,  legal,  audit  and  information  and  data 
services to the Horizon 2020 implementing bodies. 

With regard to the tasks delegated in the framework of Horizon 2020 and FP7 legacy, REA 
performs the following programme implementation tasks: 

a.  Manage  some  or  all  of  the  stages  in  the  lifetime  of  projects  of  the  delegated 
programmes.  In  this  regard,  it  is  responsible  for  launching  calls  for  proposals, 
evaluating  proposals  and  signing  grant  agreements,  monitoring  the  projects,  making 
the  necessary  checks  and  recovery  procedures,  and  for  performing  budget 
implementation  tasks  covering  revenue  and  expenditure  within  the  meaning  of 
Regulation (EU, Euratom) 966/2012; 

b.  Perform all the operations required to launch contests and award prizes in accordance 

with the Regulation (EU, Euratom) 966/2012; 

c.  Conclude public procurement procedures and manage the ensuing contracts, including 

the operations required to launch and conclude public procurement procedures; 

d.  Provide support in programme implementation.  

2.2 Baseline and points of comparison 

The  current  evaluation  of  REA  operations  during  2015-2018  assesses  the  actual  costs  and 
benefits of programme implementation by REA (executive agency scenario) when compared 
with the alternative scenario of management by the Commission services (in-house scenario). 

Accordingly,  the  reference  point  for  the  present  REA  evaluation  is  the  2013  ex  ante  Cost-
benefit analysis, and the specific financial statement11 (SFS) of REA. 

Wherever  possible,  the  analysis  of  the  performance  of  REA  during  the  period  of  reference 
assesses  the  progress  compared  to  the  two  previous  evaluations,  covering  the  Agency’s 
operations  during  2009-2012  and  2012-2015.  The  current  evaluation  covers  the  period 
between  the  beginning  of  the  second  half  of  2015  and  the  end  of  the  first  half  of  2018. 
However,  wherever  possible  and  relevant,  and  provided  that  all  the  necessary  data  was 
available, the analysis was extended to include the whole of 2015 and, at times, 2014. 

Over  the  period  2009-2013  the  savings  resulting  from  the  delegation  of  tasks  to  REA  have 
been evaluated at approximately EUR 106 million compared to the in-house scenario.  

According to the SFS of REA prepared in 2013 the estimated efficiency gains12 were of EUR 
80  million  over  the  period  2015-2018.  The  total  number  of  full  time  equivalents  (FTEs) 
required  to  manage  the  relevant  programmes  in  year  2020,  the  peak  programming  year  in 
terms  of  workload,  was  estimated  at  793  for  REA.  The  total  estimated  operational  budget 

11 REA’s SFS was updated in 2017 following the update of the REA’s mandate, linked primarily to delegation of 
classified projects from Societal Challenge 7 to REA and the expansion of centralised services for registration 
and validation of all beneficiaries and service providers (SEDIA). Consequently, the retrospective CBA is based 
on this latest SFS version of April 2017. 
12 The estimated efficiency gains over the period 2014-2024, covering full completion of the projects according 
to the SFS of 2013, were of EUR 158 million. 

5 

 
 
 
 
 
 
 
 
                                                           
entrusted  to  REA  in  2020  is  around  EUR  2  511,382  million  in  commitment  appropriations 
and  EUR  2  188,754  million  in  payment  appropriations.  Considerable  efficiency  gains  were 
expected  as  compared  to  the  in-house  scenario  over  the  period,  along  with  non-quantifiable 
benefits such as improved quality of programme management and service delivery, improved 
visibility of the EU programmes and proximity to beneficiaries. 

3. 

IMPLEMENTATION/STATE OF PLAY 

The  Agency’s  mandate  was  extended  by  the  Commission  Implementing  Decision  No 
2013/778/EU, which came into effect on 1 January 2014. Based on its renewed mandate, REA 
implemented large parts of the Horizon 2020 programme and the FP7 actions delegated to the 
Agency under its first mandate. In addition, REA was also mandated to provide administrative 
and logistical support services to all entities involved in Horizon 2020 management and the 
provision  of  participant  validation  services  for  a  variety  of  programmes.  As  of  1  January 
2018, REA plays an important role in the SEDIA project as detailed under point 2.1. 

The  governance  of  the  Agency  remained  the  same  at  the  start  of  the  new  mandate  and  was 
adjusted in 2016 when DG HOME joined the REA Steering Committee as a new parent DG. 
A  new  Memorandum  of  Understanding  (MoU)13  between  REA  and  its  parent  DGs  was 
concluded  in  2014  and  further  adjusted  in  2016.  The  MoU  defined  the  modalities  and 
procedures of interaction between REA and its parent DGs in view of the implementation of 
Horizon  2020  (and  legacy  of  FP7)  by  the  Agency.  The  distribution  of  tasks  and 
responsibilities, as presented in the MoU, is included in sections 5.1 and 5.3 of this document. 
Compared  to  this  reference,  this  evaluation  assessed  whether  this  distribution  of  roles  has 
been respected for the 2015-2018 period, and the effects of the intervention on the coherence 
and the efficiency of REA’s action. 

The  operational  budget  managed  by  REA  ranged  between  EUR 1  520  -  2 010  million  (in 
terms  of  commitments)  and  between  around  EUR 1  420  -  1 700  million  (in  terms  of 
payments)  per  year  during  the  period  2015-2017.  On  average,  REA’s  staff  managed  an 
increasing volume of the operational budget during this period, primarily because the growing 
volume  of  the  operational  budget  outpaced  the  increase  in  the  number  of  the  Agency’s 
operational staff (i.e. REA’s staff minus staff working in the central support services). 

The annual number of proposals received at REA fluctuated around 15 000 between 2015 and 
2017 and the total number of running projects decreased from 7 020 in 2015 to 6 420 in 2017. 
On  the  other  hand,  the  budget  per  operational  head  (in  commitments)  increased  from  EUR 
3.03  million  in  2015  to  EUR  3.54  million  in  2017.  During  the  reference  period,  the  actual 
workload of REA was higher than estimated in the 2013 CBA due to a higher than expected 
number of new Horizon 2020 grants. This also increased the workload of monitoring project 
implementation. 

REA gradually grew in  size  in accordance with  the multi-annual planning stated in the SFS 
and was the largest Executive Agency in 2018 with 735 actual staff members14. The Agency 
is based in Brussels. 

13 Memorandum of Understanding between the  Research Executive Agency and DG Research and Innovation, 
DG Education and Culture, DG Communication Networks, Content and Technology, DG Agriculture and Rural 
Development, DG Internal Market, Industry, Entrepreneurship and SMEs, DG Migration and Home Affairs of 5 
February 2016. 
14  Total number of staff, financed from EU Budget and contributions from EFTA/EEA and/or third countries.  

6 

 
 
 
 
 
 
 
                                                           
A series of organisational and procedural changes took place during the implementation of the 
Horizon 2020 programme under the multiannual financial framework (MFF) of 2014-2020: 

  Delegation of the management of additional activities and objectives, as detailed under 

point 2.1 above, and as a consequence new tasks for the Agency; 

  Further  simplification  and 

improvement  of  grant  management  processes  and 
procedures,  notably  electronic  submission  of  proposals,  due  to  the  adoption  of  new 
business processes and use of the Commissions Horizon 2020 IT tools and systems (i.e. 
Compass and SyGMa); 

  The  adoption  of  the  multiannual  human  resource  (HR)  strategy  for  2017-2020  to 
address the findings of the 2016 staff opinion survey. Based on this new strategy and in 
combination with a dedicated action plan addressing attention points identified through 
the  survey,  the  Agency  implemented  specific  measures  related  to  staff  engagement, 
internal  communication,  career  development  opportunities,  work  and  private  life 
balance  and  training,  meant  to  improve  effective  HR  management  and  increase  the 
effectiveness and flexibility of the Agency’s operations. 

4.  METHOD 

4.1 Short description of methodology 

The  evaluation  was  supported  by  a  study  carried  out  by  external  contractors15.  The 
methodology of the REA evaluation was consistent with the approaches employed for similar 
parallel  evaluations  of  ERCEA,  EACEA,  INEA,  EASME  and  CHAFEA,  and  built  on  the 
previous  evaluation  of  REA  operations  during  2012-2015.  Whenever  possible,  data  was 
collected and the analysis extended to also cover 2014, the whole of 2015 and the whole of 
2018, even if formally outside the evaluation timeframe. Specific methodological approaches 
were  used  for  each  of  the  evaluation  tasks  (Regulatory  framework,  REA  mission  and 
governance;  REA  performance  between  2015-2018,  including  effectiveness,  efficiency  and 
coherence;  the  Cost  Benefit  Analysis  and  Conclusions  and  Policy  Recommendations)  to 
ensure  data  triangulation  (confirmation  from  different  sources).  The  study  was  structured 
around a series of evaluation questions available in Annex. 

The evaluation relied on the evidence gathered and analysed using qualitative and quantitative 
methods and included: 

  an  extensive  documentary  review  and  desk  research  of  REA  and  Commission 
documents, including Annual  Work Programmes and Annual  Activity Reports, audit 
reports  and  related  documents,  CBA  studies  and  other  financial  documents,  the 
Memorandum  of  Understanding16  (MoU),  previous  evaluations  and  action  plans 
addressing the  ensuing  recommendations; the evaluation  relied also  on the results of 
the 2016 staff opinion survey; 

  31  semi-structured  in-depth  interviews  with  Commission  staff  and  REA  staff, 
including: members of the REA Steering Committee; selected Commission staff in the 
parent  DGs/mirror  units;  the  Director  of  REA  &  heads  of  department;  heads  of 
unit/representatives  of  the  operational  units;  heads  of  unit/representatives  of  units  of 

15  Study  supporting  the  evaluation  of  the  Research  Executive  Agency  (2015-2018),  final  report  prepared  by 
PPMI, CSES and IDEA Consult, February 2019. 
16 Memorandum of Understanding between the  Research Executive Agency and DG Research and Innovation, 
DG Education and Culture, DG Communication Networks, Content and Technology, DG Agriculture and Rural 
Development, DG Internal Market, Industry, Entrepreneurship and SMEs, DG Migration and Home Affairs of 5 
February 2016. 

7 

 
 
 
 
                                                           
the  Administrative  Department;  Representatives  of  the  REA  Staff  Committee,  other 
employees; 

  15 interviews with beneficiaries of different programmes (including MSCA, Societal 
Challenges, Industrial Leadership, and Specific Objectives), funding instruments, and 
organisation  types  (i.e.  higher  education  institutions,  public  research  organisations, 
small and medium enterprises); 

  3 surveys addressing the REA’s beneficiaries (652 answers or 10.5% response rate), 
unsuccessful applicants (306 answers or 6.1% response rate), experts (2470 answers or 
39.7% response rate) and of Commission staff (50 answers or 59.5% response rate)17;  

  a retrospective cost–benefit analysis; 
  a focus group meeting with representatives of REA and Commission staff (including 
the Common Support Centre), to validate the draft conclusions and recommendations 
of the evaluation; 

  a  comparative  analysis  and  benchmarking  of  the  various  executive  agencies  was 
also carried out against a set of (qualitative and quantitative) indicators. This enabled 
the evaluation team to compare the performance of REA effectively with that of other 
executive agencies. 

4.2 Limitations and robustness of findings 

All  the  relevant  stakeholders  within  the  Commission  and  REA  were  duly  consulted  via 
interviews  and  surveys.  An  extensive  number  of  REA  applicants,  beneficiaries  and  experts 
were  covered  by  the  surveys  and  the  response  rates  are  solid  and  comparable  with  similar 
evaluations. However, compared with the overall population reached, unsuccessful applicants 
participated less actively in the survey than programme beneficiaries.  
No  sampling  bias  was  observed  as  the  profile  of  the  respondents  to  the  surveys  was  very 
similar to the overall population, guaranteeing statistical representativeness. The non-response 
bias  (not  all  characteristics  of  the  group  that  did  not  reply  had  been  captured  in  full)  was 
mitigated  through  triangulation  with  the  results  of  follow  up  interviews.  The  triangulation 
approach,  using  multi-level  and  multi-stakeholder  dimension  in  the  data  collection,  ensured 
the robustness and reliability of the data and information used to draw up conclusions in the 
supporting study. 
Some views from Commission and Agency staff gathered from the interviews and mentioned 
in the study illustrate personal experience, although they may not be representative of overall 
relations between the Commission and the Agency. 

5.  ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS 

Analysis  and  answers  to  the  evaluation  questions  are  grouped  below  as  per  the  three 
evaluation criteria: effectiveness, efficiency and coherence. The quantitative results of the ex 
post cost benefit analysis are presented separately in section 5.4 below, while the qualitative 
aspects  are  embedded  in  the  answers  to  the  evaluation  questions  (sections  5.1-5.3).  The 
detailed evaluation questions and sub-questions are listed in annex. 

17  Relatively  low  response  rates  from  beneficiaries  and  unsuccessful  applicants  are  typical  for  this  type  of 
evaluations. 

8 

 
 
 
 
 
                                                           
5.1 Effectiveness 

For  the  purpose  of  this  evaluation,  effectiveness  relates  to  how  successful  the  Agency  has 
been in achieving or progressing towards its objectives. 

During 2015-2018, REA operated according to the legal framework that established it. REA’s 
activities,  as  planned  in  the  annual  work  programmes  and  reported  in  the  annual  activity 
reports,  corresponded  to  the  tasks  set  out  in  the  Commission  Decisions  and  the  Act  of 
Delegation. 

The modalities and procedures of interaction between the Agency and its parent DGs are set 
out in the Memorandum of Understanding (MoU) between REA and its parent DGs of 2016. 
The  MoU  provides  for  a  supervision  strategy  aimed  at  avoiding  gaps  and  duplication  of 
efforts resulting from crossover between monitoring and supervision tasks of the parent DGs 
and  the  execution  tasks  of  the  Agency,  while  ensuring  that  no  micro-management  was 
present.  In  2018,  three  separate  legal  documents18  were  adopted  in  the  context  of  REA’s 
extended mandate. These provisions have generally worked well and all parent DGs and REA 
appreciate  the  effectiveness  of  the  collaboration.  However,  several  processes  related  to  the 
selection  of  independent  experts,  validation  of  expert  lists,  and  the  participation  of 
Commission staff in project monitoring activities require further consolidation. In these areas, 
the  interviews  and  the  survey  revealed  that  a  number  of  Commission  staff  have  different 
expectations as regards REA’s role, possibly due to varying levels of awareness of the MoU 
provisions. Accordingly, the Commission and REA should work together to ensure that these 
provisions are clearly communicated to Commission staff. 

The Agency was flexible and effective in addressing the key changes that occurred during the 
evaluation period (delegation of a specific role in SEDIA; implementing projects generating 
EU  classified  information  for  Societal  Challenge  719),  and  continuously  improving  its 
operations, even though  it was already achieving a high level  of overall  effectiveness.  REA 
began  the  delivery  of  corporate  validation  services  on  1 January 2018.  During  2018,  the 
participant  validation  activity  grew  progressively  with  the  integration  of  new  clients  from 
both grant and procurement domains. 

In 2015-2018 REA achieved very good results in terms of most key performance indicators 
(KPI). Compared to previous years the Agency’s performance improved during the period of 
reference. The prompt conclusion of grant agreements (measured in ‘Time-To-Grant’ – TTG) 
improved, on average, from 222 days in 2014 to 203 in 2015 and became stable at 193 days 
during 2016-2018, 52 days (over 20%) below the Horizon 2020 target (245 days). The share 
of  grants  concluded  within  TTG  targets  reached  nearly  100%  for  the  2016-2017  calls. 
Regarding  payments  to  grants,  the  average  Time-to-Pay  (TTP)  stood  well  below  the 
contractual thresholds for all types of payments (pre-financing, interim and final payments) in 
2015-2018  both  for  FP7  and  Horizon  2020.  Concerning  the  share  of  payments  within 
contractual limits, nearly 100% of all pre-financing payments in 2015-2018 were executed on 
time. With regard to interim and final payments, REA maintained a similar performance level 
compared to the previous evaluation period for FP7 (94% of payments executed on time) and 
during 2017-2018 improved its performance for Horizon 2020 (98% of payments executed on 
time). 

18 Guiding REA’s operational work related to programme implementation, expert management and the provision 
of the validation support services to EU bodies managing grant and procurement procedures. 
19 ‘Secure societies – protecting freedom and security of Europe and its citizens.’ 

9 

 
 
 
 
 
 
 
                                                           
As was the case in previous years, REA achieved full execution of its operational budget both 
in commitment and payment appropriations during 2015-2018. 

The  REA  target  with  regards  to  the  share  of  projects  that  achieved  all  or  most  of  their 
objectives,  the  REA  Annual  Activity  Reports  showed  that  over  95%  of  FP7  projects  that 
closed in 2015-2018 achieved all or most of their objectives (against a target of 90%). 

With regard to the legality and regularity of transactions, the estimated residual multi-annual 
error  rate  remained  in  a  similar  range  over  2014-2018  compared  with  those  in  the  previous 
years for the FP7 programmes implemented by REA. The residual multi-annual error rate for 
the People Programme remained below 2% while for the SME actions, Space and Security it 
was above the materiality threshold of 2% that lead REA to enter reservations in its Annual 
Activity  Reports20.  In  addition,  it  should  be  noted  that  the  estimated  error  rate  can  be 
attributed, to a large extent, to the complexity of the funding schemes, which is generic to FP7 
as  a  whole;  therefore,  these  error  rates  remained  in  line  with  the  general  FP7  trend.  For 
Horizon  2020  financial  transactions,  REA  reports  the  residual  error  rate  common  to  the 
Research  family  as  a  whole  (2.24%  for  2017)  but  also  on  REA  specific  residual  error  rate 
(1.46% for 2017). Concerning MSCA, there were not enough audits to draw any conclusions 
for Horizon 2020 but it is expected21 that the error rate will remain below 2%. 

REA’s efforts to improve the quality of the evaluation process and the Evaluation Summary 
Reports (ESRs) were reflected in a reduced share of evaluation review/redress cases filed and 
(fully  or  partially)  upheld  compared  to  the  number  of  proposals  evaluated.  The  share  of 
evaluation review/redress cases filed decreased from 3.0% for the 2011 FP7 calls to 1.2% for 
the 2017 Horizon 2020  calls. Similarly, the share of evaluation review/redress  cases upheld 
decreased  from  0.8%  for  the  2011  calls  to  0.3%  for  the  2017  calls  and  for  the  2014-2017 
Horizon 2020 calls it stayed below the maximum target of 0.5%. 

of 

the  Research  Enquiry  Service 

As regards the provision of administrative and logistic support services, and in particular the 
(RES)22,  Financial  Capacity 
management 
Checks/Assessments  and  the  participant  Validation  Services,  the  overall  performance  of  the 
Agency  was  favourable  and  target  deadlines  for  validations  were  fully  respected  between 
2015  and  2018.  In  2017,  7  900  enquiries  were  received  of  which  99.4%  were  answered  or 
forwarded within 8 days, with an average of 3 working days. The REA target to perform 95% 
of validations within the 90 days from the “raise priority” date23 – for all direct management 
operations  at  the  Commission  –  was  fully  met  as  of  30  June  2018.  For  the  period  2014  to 
2016 REA handled from 1 600 to 2 000 Financial Capacity Assessments (FCA) requests per 
year. From 2017, the reporting on assessments provided to clients changed – only those cases 
in which financial data had been provided to REA clients were taken into account. 

Overall  satisfaction  with  the  performance  of  REA  is  one  of  the  highest  among  executive 
agencies. 86% of beneficiaries responding to the survey were satisfied (42% ‘very satisfied’) 

20  Due  to  the  small  proportion  of  the  financial  impact  on  the  overall  financial  portfolio  of  the  Agency,  these 
reservations were lifted by the Agency as of its 2019 Annual Activity Report. 
21  Based  on  historical  figures  for  FP7,  first  results  of  MSCA  audits  and  following  new  simplified  rules  under 
Horizon 2020. The last known results included in the 2019 Annual Activity Report, and based on a substantial 
sample of audited participations, confirmed this assumption. 
22 The Agency operates the Research Enquiry Service addressing questions raised by the public on Horizon 2020 
and other research matters. 
23  The  priority  setting  for  validations  is  automatically  set  by  the  Participant  Database  Management  (PDM) 
system. 

10 

 
 
 
 
 
 
 
                                                           
with the services provided by the Agency. This result is a further improvement compared to 
the  previous  evaluation  of  REA  (82%  in  2015)  and  is  higher  than  the  overall  level  of 
satisfaction  reported  by  the  beneficiaries  of  CHAFEA  (74%)  and  comparable  to  what  was 
reported by the beneficiaries of EACEA (89%) and ERCEA (89%). Almost all respondents to 
the  surveys  would  like  to  work  with  the  Agency  again  in  the  future  (98%  of  beneficiaries, 
99% of REA’s independent experts and around 90% of unsuccessful applicants). 

The  independent  experts  were  highly  satisfied  with  the  overall  processes  of  registration, 
selection and contracting and their level of satisfaction improved as compared to the previous 
2015 experts’ survey. However, the experts remained less positive about the online evaluation 
system used, the quality of the templates used for reports, and the time available for drafting 
them. 

In  terms  of  proximity  to  the  addressees,  REA  continued  to  serve  as  a  direct  ‘contact  point’ 
between  the  applicants/beneficiaries  of  EU  funding,  as  well  as  other  stakeholders  and  the 
Commission  during  the  evaluation  period.  Building  on  the  results  of  the  2015  beneficiary 
satisfaction  survey,  REA  made  sustained  efforts  to  boost  awareness  on  new  funding 
opportunities and opportunities for experts, to broaden the participants’ group, to consolidate 
a  service-oriented  communication,  and  to  support  parent  DGs  by  giving  visibility  to  EU 
research via success stories and providing input to the policy-feedback loop. 

The survey of beneficiaries indicates a broad satisfaction with the way in which the Agency 
communicated with them during various phases of the project life-cycle. As many as 81% of 
the respondents to the beneficiaries’ survey strongly or rather agreed that REA staff assigned 
to  their  project  were  easily  available  and  responsive  during  the  implementation.  Most 
favourable results were received in relation to REA’s availability and responsiveness during 
the  grant  amendment  (82%  strongly  or  rather  agreed)  and  in  the  grant  finalisation  phases 
(85%  strongly  or  rather  agreed).  About  74%  of  the  respondents  claimed  that  the  feedback 
received  on  the  scientific  and  technological  progress  in  their  project  was  useful.  The  areas 
with reported lower levels of satisfaction were related to the user-friendliness of the IT tools 
employed throughout the project management life-cycle. The Commission, which is leading 
the development of grant management IT tools at corporate level, is currently taking measures 
to address the beneficiaries’ lower satisfaction in this area. 

Concerning the visibility of the EU as a promoter of the programmes entrusted to REA, the 
survey shows that a large majority of beneficiaries were aware that REA grants were funded 
from  the  EU  budget  (95%)  and  that  the  European  Commission  entrusted  the  grant 
management to REA (84%). 

With regards to the lessons learned during the 2015-2018 period, the evaluation identified that 
the growing prevalence of actions with policy interest emerges as an area where REA and its 
parent DGs need to formalise the specific modalities and procedures for their implementation. 
Examples  are  the  Coordination  and  Support  Actions,  including  grants  to  identified 
beneficiary, which are managed by the Agency but are of particular interest for Commission 
policy  work.  Since  it  is  important  to  ensure  that  these  actions  can  respond  to  continuous 
developments  in  policy  needs,  a  closer  dialogue  between  REA  and  Commission  services 
should be maintained. 

11 

 
 
 
 
 
 
 
 
5.2 Efficiency  

This  section  considers  the  relationship  between  the  resources  used  by  the  Agency  and  the 
output.  The  analysis,  among  other  factors,  also  includes  analysis  of  administrative  and 
regulatory burden and looks at aspects of simplification. 

During  2015-2017  REA  proved  to  be  an  efficient  and  cost-effective  structure  for  the 
management of the delegated programmes. Its administrative budget (excluding the share of 
costs  of  the  central  administrative  and  logistical  support  services)  was  below  3%  of  the 
operational  budget  based  both  on  commitment  and  payment  appropriations  for  the  period 
2015-2017, while generally the cost-effectiveness of REA grew over 2010-2017. 

Ratio between REA’s administrative and operational budget 

Commitment  appropriations 

Payment appropriations 

6,0%

4,0%

2,6% 

2,0%

0,0%

3,5% 

3,1% 

2,9%  3,0%  2,6% 

3,3%  3,6%  3,6%  3,3% 

3,4% 

2,5%  2,7%  2,3%  2,1%  2,6%  2,5%  2,4% 

5,0% 

4,4% 

4,1%  3,7%  3,3% 

4,7% 

3,8%  3,6%  3,9% 

3,6%  3,3%  2,9%  3,0%  2,8%  2,6%  2,9% 

2009 2010 2011 2012 2013 2014 2015 2016 2017

2009 2010 2011 2012 2013 2014 2015 2016 2017

Including support services

Excluding support services

REA’s actual operational budget during 2014-2018 constituted EUR 9.109 billion, of which 
EUR 8.508  billion  from  EU  budget  and  EUR  602  million  related  to  EEA/EFTA24  and  third 
countries’  contributions  (not  considered  in  REA’s  SFS).  REA’s  administrative  budget 
increased from EUR 39.88 million in 2015 to EUR 43.92 million in 2017 (in commitments, 
without the central support services). 

A  detailed  analysis  of  the  REA  cost-efficiency  and  results  of  the  Cost-benefit  analysis  are 
presented under the point 5.4. 

The number of proposals managed by REA fluctuated at around 15 000 from 2015 to 2017. 
The number of total running projects decreased from 7 020 in 2015 to 6 420 in 2017. During 
the  evaluation  period,  the  actual  workload  of  REA  was  higher  than  estimated  in  the  2013 
CBA  due  to  factors  beyond  the  Agency’s  control  that  influenced  its  workload,  such  as  the 
average grant size and the number of proposals and grants. The actual number of new grants 
managed  by  REA  during  2014-2018  was  32%  higher  than  the  2013  CBA  forecast  (9  646 
versus 7 306). However, the increase in the number of grants managed by REA (20%), mostly 
relates to the smallest grants – MSCA Individual Fellowships, which are characterised by the 
lowest workload level per grant (number of person days required to manage a typical project 
life-cycle). The increased number of new projects also added to the workload of monitoring 
project implementation. 

Due  to  an  increase  in  the  number  of  operational  staff  during  this  period  and  the  closure  of 
many small FP7 projects (MSCA individual fellowships and grants), the number of running 
projects ‘per operational head’ decreased from 15.42 in 2014 to 11.32 in 2017. However, the 
ratio of budget ‘per operational head’ increased from EUR 3.46 million to EUR 3.54 million 

24 EEA stands for ‘The European Economic Area’; EFTA stands for ‘The European Free Trade Association’. 

12 

 
 
 
 
 
 
 
 
 
                                                           
  
(in  terms  of  commitments)  or  from  EUR 2.44  million  to  EUR 2.99  million  (in  terms  of 
payments) in 2014-2017. 

Budget, staff and workload indicators, REA, 2014-2017, € million or % 

Operational budget, commitments 
Operational budget, payments 
Administrative budget, commitments 
Administrative budget, payments 
Administrative  budget  without  the  central  support 
services, commitments 
Actual number of staff (at the end of the year) 
Actual  number  of  operational  staff  (without  the 
central support services) 
Programme  management  cost  at  REA  (excluding 
the central support services), commitments 
Programme  management  cost  at  REA  (excluding 
the central support services), payments 
Budget ‘per operational head’, commitments  
Budget ‘per operational head’, payments 
Proposals received 
Total running projects  
Running projects ‘per operational head’  

2014 
1 553.51 
1 096.88 
51.24 
49.68 

2015 
1 521.74 
1 418.48 
54.19 
54.10 

2016 
1 666.36 
1 642.94 
59.74 
59.23 

2017 
2 009.15 
1 697.18 
66.87 
66.12 

33.39 

39.88 

42.42 

43.92 

548 

449 

608 

503 

628 

516 

690 

567 

2.15% 

2.62% 

2.55% 

2.19% 

3.04% 

2.81% 

2.58% 

2.59% 

3.46 
2.44 
11 473 
6 925 
15.42 

3.03 
2.82 
15 415 
7 020 
13.96 

3.23 
3.18 
13 005 
6 658 
12.90 

3.54 
2.99 
14 927 
6 420 
11.32 

REA took a series of actions to cope with the varying levels of workload across its different 
units. The Agency introduced an annual workload measurement, which helped to re-balance 
its  staffing  levels  across  all  of  its  units  and  the  appropriate  ratio  of  administrative  and 
operational staff. REA was also effective in filling vacancies and maintaining a relatively low 
vacancy  rate  (2%)  between  2015  and  2018.  In  accordance  with  the  multi-annual  planning 
provided in the SFS, by the end of 2017 the number of REA staff25 had risen to 690 (from 608 
in 2015), of which 123 are dedicated for the support services. 

The staff opinion survey carried out in 2016 revealed that REA’s key strengths were related to 
overall  job  satisfaction  and  staff’s  satisfaction  with  the  Agency  as  a  modern  and  attractive 
workplace.  In  addition,  REA  achieved  positive  results  with  respect  to  the  willingness  of  its 
staff to give extra effort when required, the level of understanding of the Agency’s purpose as 
well as job clarity. 

At the same time, REA staff perceived less positively their career development and mobility 
opportunities,  work  recognition  as  well  as  equal  opportunities  and  internal  communication. 
Those  results  were  however  still  higher  than  or  equal  to  the  levels  of  agreement  for  the 
Commission and other executive agencies. 

The Agency has stepped up its HR management by the adoption of its 2016 HR Action Plan 
to  follow  up  these  staff  opinion  survey  results,  and  through  a  new  comprehensive  multi-
annual  HR  strategy.  The  evolution  of  key  indicators  for  staff  satisfaction  –  after  the 
evaluated  period  –  is  encouraging:  In  the  latest  staff  opinion  survey  (4th  quarter  of  2018), 

25 Total number of staff, financed from EU Budget and contributions from EFTA/EEA and/or third countries. 

13 

 
 
 
  
 
 
 
 
                                                           
REA managed to  increase its result for staff feeling that the Agency  cares  about  their well-
being to 63%, up from 49% in 2016. Staff engagement rose to 69% (from 67% in 2016). 

Overall, the size and structure of REA, and the  level  of resources in  each department,  were 
appropriate  to  its  mandate  and  delegated  tasks  during  the  evaluation  period.  The 
organisational  structure  of  REA  was  based  on  separate  organisational  units  for  different 
programmes and actions grouped into Department A ‘Excellent Science’26 and Department B 
‘Industrial  Leadership  and  Societal  Challenges’27  according  to  the  respective  pillars  of  the 
Horizon 2020. The units, responsible for the Agency’s horizontal tasks (Administration and 
Finance)  and  central  support  services  (Participant  Validation  and  Expert  Management  and 
Support  unit)  were  organised  into  the  Administration,  Finance  and  Support  Services 
Department.  REA’s  organisational  structure  was  reviewed  and  effectively  aligned  to  cope 
with changes, such as the evolution of the REA mandate. 

During 2015-2018, REA, in cooperation with the Commission, continued the optimisation of 
its  procedures  and  programme  management  functions  and  introduced  a  number  of 
simplifications.  The  improvements  and  simplifications  concerned  wider  use  of  remote 
evaluation  of  proposals,  remodelling  of  the  evaluation  building28,  improved  procedures  for 
allocating  proposals  to  the  most  suitable  experts,  measures  to  automate  and  improve  the 
detection of possible conflict of interest for experts, electronic workflows and wider use of IT 
tools,  etc.  REA’s  internal  staff  networks29  played  an  important  role  in  the  optimisation  and 
simplification  process.  They  assumed  a  key  role  in  the  development  of  improvements  to 
numerous  REA  operations  such  as  management  of  calls  (from  call  publication  to  grant 
signature)  and  grants  in  Horizon  2020.  In  addition,  they  contributed  to  the  development  of 
guidance, the sharing of good practices and to ensuring a harmonised implementation of the 
Horizon 2020 control system provided by the Common Support Centre.  

5.3 Coherence  

Coherence  looks  at  any  overlaps  and  complementarities  within  the  programme  portfolio 
managed by the Agency or delimitation of responsibilities between the Agency and its parent 
DGs. 

The  evaluation  did  not  find  any  evidence  of  overlaps,  duplications,  gaps  or  inconsistencies 
within the REA programme portfolio or the support services managed by REA. In the 2015-
2018 period, REA demonstrated flexibility in implementing a diverse thematic programming 
portfolio  and  being  answerable  to  six  different  parent  DGs.  There  is  a  strong  coherence 
between  the  FP7  and  Horizon  2020  programmes  delegated  to  REA,  because  of  strong 
continuity  in  a  number  of  programmes  (for  example,  implementation  of  MSCA  under  the 
Excellent  Science  pillar  in  both  framework  programmes  and  most  activities  within  FP7 
Security and the successor Secure Societies programme in Horizon 2020). 

26  Department  A  was  responsible  for  the  implementation  of  Marie  Skłodowska-Curie  and  FET-Open 
programmes. 
27  Department  B  was  responsible  for  the  implementation  of  Space  Research,  Sustainable  Resources  for  Food 
Security  and  Growth,  Inclusive,  Innovative  and  Reflective  Societies,  Safeguarding  Secure  Society,  Spreading 
Excellence, Widening participation, Science with and for Society programmes. 
28 Including by equipping a large number of new audio and/or videoconference rooms. 
29 The Networks of call coordinators (NCC), Legal Officers (NLO), Project Officers (NPO), Financial Officers 
(NFO) and Ethics Correspondents (NEC). 

14 

 
 
 
 
 
 
 
                                                           
Another factor of coherence derives from the fact that REA is responsible for implementing 
several different parts of a specific programme (Horizon 2020 and legacy of FP7) for multiple 
DGs and across a broad thematic spectrum. Accordingly, REA’s responsibilities are broadly 
coherent in that they are all part of a single programme, with common rules and procedures. 
Moreover, the business processes, as designed by the CSC, are harmonised for different sub-
programmes, which means that the overall grant management process is coherent, even if the 
thematic areas of intervention differ (with some specificities within the MSCA). 

The extension of REA’s mandate to provide administrative and logistical support services to 
all entities involved in Horizon 2020 management and also to a wider range of Commission 
DGs  has  strengthened  coherence,  not  least  due  to  the  introduction  of  a  common  approach 
across Horizon 2020. Overall, these developments indicate that REA is increasingly becoming 
a central provider of support services in the Commission. 

The  Memorandum  of  Understanding  of  2016  and  the  additional  working  arrangements  of 
2018  reflecting  REA’s  extended  mandate  provide  for  clear  and  appropriate  delimitation  of 
responsibilities and tasks between REA and its parent DGs. 

REA has continued to generate and retain know-how and knowledge, capitalising on the fact 
that  it  has  been  managing  some  of  the  programmes  for  a  decade.  While  programme 
implementation  know-how  is  embedded  within  the  Agency  itself,  significant  progress  has 
been  made  in  strengthening  REA’s  input  to  policymaking,  and  in  addressing  some  of  the 
weaknesses identified in the 2012-2015 period of operations, through the adoption of strategic 
and operational measures. For instance, REA organised thematic project cluster meetings and 
set  up  the  AGILE  Rapid  Reaction  Network30,  participated  in  coordination  meetings  and  in 
thematic  events  organised  by  the  Commission,  contributed  to  the  preparation  of  policy 
reports,  provided  important  input  for  the  Commission’s  communication  and  dissemination 
events, etc. 

However,  policy  feedback  is  still  a  domain  where  further  improvements  are  considered 
necessary, with  actions required from  both  the Commission  and the REA side to  strengthen 
the efficiency and effectiveness of this function. REA’s parent DGs need to formulate better 
their policy needs as these differ from one DG to another, due to the nature of ‘top-down’ or 
‘bottom-up’  research  programmes  for  which  they  are  responsible31.  Parent  DGs  should  also 
specify  in  which  form,  and  at  which  frequency,  they  would  prefer  policy  inputs  to  be 
produced by REA to allow it for a better customised policy feedback. The Catalogue of Policy 
Options developed by REA in 2018 serves as a basis for establishing specific policy feedback 
mechanisms and business processes most relevant to parent DGs. Currently the CIC is leading 
the efforts of developing procedures at the R&I family level for feedback onto policymaking 
as  part  of  the  strategy  for  an  effective  dissemination  and  exploitation  of  research  results  in 
Horizon 2020. 

30 The  AGILE  network  was  designed  to  provide a  flexible  tool  through  which  REA  could  provide  knowledge 
and  information  quickly  in  the  form  of  a  factsheet  overview  to  its  parent  DGs  (e.g.  when  aiming  to  offer 
qualitative answers to pressing policy questions from the parent/political DGs in a timely and efficient manner). 
31 In the case of top-down research programmes, where policymakers develop the Annual Work Plan, the link to 
policy  feedback  activities  is  more  direct  compared  with  bottom-up  research  programmes,  such  as  the  MSCA, 
where other types of feedback are required. 

15 

 
 
 
 
 
 
 
                                                           
5.4 Retrospective Cost-Benefit Analysis 

The retrospective Cost–Benefit Analysis for the 2015-2018 period was carried out  based on 
the assumptions laid down in the latest version of the Specific Financial Statements (SFS) and 
the  actual  costs  of  REA.  Since  REA’s  mandate  was  substantially  changed  compared  to  the 
2013 ex ante CBA, any comparison of the ex ante CBA estimates with the SFS estimates and 
the actual figures  must be considered carefully,  given the change in  the tasks  and workload 
level.  REA’s  SFS  was  updated  in  2017  following  the  update  of  the  REA’s  mandate,  linked 
primarily  to  delegation  of  classified  projects  from  Societal  Challenge  7  to  REA  and  the 
expansion  of  centralised  services  for  registration  and  validation  of  all  beneficiaries  and 
service providers (SEDIA). Consequently, the retrospective CBA is  based on the latest  SFS 
version of April 2017. 

The REA SFS estimated the savings resulting from the delegation of tasks to REA at around 
EUR 178 million over the whole period 2014-2024 when compared to the management by the 
Commission  services  (in-house  scenario),  while  for  the  period  of  2015-2018  these  were 
estimated at around EUR 80 million.  

The retrospective CBA revealed that the executive agency scenario actual costs in 2015-2018 
were much lower than the estimated costs of the in-house scenario and constituted savings of 
EUR 105 million (or 29% of the estimated costs under the in-house scenario)32. 

Estimated costs and savings of the executive agency scenario in 2015-2018, EUR 

344.651.804 

362.920.000 

264.750.000 

257.820.358 

79.901.804 

105.099.643 

400.000.000

350.000.000

300.000.000

250.000.000

200.000.000

150.000.000

100.000.000

50.000.000

0

SFS estimations

Actual

Costs of the in-house scenario

Costs of the Executive Agency (EA) scenario

Savings of the EA scenario

As forecasted in the SFS, savings of the executive agency scenario primarily resulted from a 
higher  share  of  lower-cost  contract  staff  (CAs)  employed  within  the  Agency  and  the  lower 
overall number of staff. 

Significant cost savings occurred in REA’s Title II ‘Infrastructure and operating expenditure’. 
The costs in Title I ‘Staff-related expenditure’ were higher than estimated in the SFS as the 
latter  did  not  take  into  account  increases  to  staff  costs  due  to  salary  indexation,  promotions 
and increasing staff seniority. 

32 These costs do not take into account the additional contribution to the REA’s administrative budget of EUR 
11.5 million over 2015-2018 coming from EFTA/EEA and third country contributions to manage the additional 
operational budget of EUR 602 million coming from these countries. 

16 

 
 
 
 
 
 
 
 
 
                                                           
Finally, it should be noted that the assumptions laid down in the SFS forecasted a workload 
level  significantly  lower  than  the  actual  one,  across  different  programmes.  Although  many 
parameters  influencing  the  workload  (operational  budget  of  delegated  Horizon  2020 
programmes  and  actions,  average  grant  size  and  corresponding  number  of  grants)  were 
beyond  the  control  of  the  Agency,  proper  organisation  and  management  helped  allocate 
human resources across REA in the most appropriate way and avoid shortcomings. 

6.  CONCLUSIONS 

The  results  of  the  evaluation  of  REA  operations  during  2015-2018  period  confirm  that  the 
delegation of programme implementation to REA remained considerably more cost effective 
than  the  in-house  scenario.  During  this  period  REA  continued  to  implement  the  delegated 
programmes in an effective, efficient and coherent manner, generating important cost-savings 
to the EU budget and high value for money. The ex-post cost benefit analysis found that the 
actual  savings  of  the  executive  agency  scenario  were  EUR 105  million  during  the  period 
2015-2018  compared  with  the  in-house  scenario  and  higher  than  the  EUR  80  million 
estimated in the SFS of 2013. 

During this period REA was effective in  achieving its objectives,  and its beneficiaries were 
highly  satisfied  with  the  performance  of  the  Agency.  REA  consistently  achieved  its  key 
performance  indicators.  Beneficiaries’  satisfaction  with  the  REA  performance  increased  to 
86%  compared  with  the  previous  evaluation  (82%  in  2015).  The  ratio  of  budget  ‘per 
operational  head’  increased  from  EUR 3.46  million  to  EUR 3.54  million  (in  terms  of 
commitments)  or  from  EUR 2.44  million  to  EUR 2.99  million  (in  terms  of  payments)  in 
2014-2017.  The  Agency’s  administrative  budget  was  below  3%  of  the  operational  budget 
based  both  on  commitment  and  payment  appropriations  between  2015  and  2017  (excluding 
the  costs  of  the  administrative  and  logistical  support  services).  REA  delivered  high  quality 
and  effective  administrative  and  logistical  support  services  to  its  clients  and  other 
stakeholders, achieving and exceeding the KPIs set. 

The  coordination  and  communication  mechanisms  that  are  in  place  facilitate  information 
flows between REA and the Commission services, in particular with regard to the content of 
projects  supported  and  their  results.  No  evidence  of  overlaps,  duplication,  gaps  and 
inconsistencies within the programme portfolio and support services managed by REA were 
identified; on the contrary, the extension of REA’s remit to include the provision of common 
administrative  and  logistical  support  services  for  the  research  and  innovation  family  has 
strengthened  coherence,  particularly  through  the  introduction  of  a  common  approach  across 
Horizon 2020. 

The current evaluation identifies only three areas where further improvements are necessary 
from both the Commission and the REA side and which aim to  further streamline or define 
the working arrangements between REA and the Commission. These relate to:  

 

 

Increasing  awareness  on  business  processes  related  to  the  selection  of  independent 
experts  for  evaluations,  validation  of  expert  lists,  as  well  as  the  participation  of 
Commission staff in project monitoring activities;  
formalising  modalities  and  procedures  for  the  implementation  of  actions  with  policy 
interest; 

  establishing a set of specific policy feedback mechanisms & business processes most 

relevant to parent DGs. 

17 

 
 
 
 
 
 
 
The  parent  DGs  and  the  Agency  are  preparing  an  action  plan  to  address  the  shortcomings 
identified  in  this  evaluation.  This  action  plan  aims  to  further  improve  the  Agency’s 
performance,  as  well as  the interaction and collaboration between Commission  services and 
REA and to contribute to an efficient implementation of the delegated programmes. 

18 

 
 
 
 
ANNEX: EVALUATION QUESTIONS 

Effectiveness 

  To what extent has REA been operating according to the legal framework establishing 

it? 

  To  what  extent  has  REA  achieved  its  objectives  with  special  focus  to  (a)  the 
implementation  of  the  delegated  programmes  Horizon  2020  and  FP7,  (b)  the 
implementation  of  the  support  services  and  (c)  the  implementation  of  the  internal 
control principles33, notably sound financial and human resource management? What, 
if  anything,  could  be  done  to  render  REA  more  effective  in  achieving  these 
objectives? 

  To  what  extent  and  how  has  REA  contributed  to  an  improved  management  and 
visibility  of  the  delegated  programmes  and  better  services  and  satisfaction  to  the 
stakeholders and addressees34 in terms of the elements assessed in the 2013 CBA and 
as compared to the alternative options mentioned in that CBA? 

  To  what  extent  does  REA’s  communication  function  support  the  mission  of  the 
Agency  and  does  REA  ensure  an  effective  feedback  loop  with  the  policy  making 
DGs? 

  To what extent has REA contributed to improved management of the programmes in 

terms of: 
−  Proximity  to  addressees  (e.g.  role  played  by  the  Agency  as  focal  point  for 
applicants and beneficiaries of the programmes; instruments and mechanisms put 
in place for communicating with the applicants beneficiaries and experts); 

−  Effective implementation of the programmes, taking into account  the interests of 

the addressees and those of the EU:  
  rate of execution of commitment appropriations; 
  rate of execution of payment appropriations; 
  time to grant; 
  net time to pay; 
  residual multi-annual error rate identified at ex-post control. 

−  Visibility of the EU as promoter of the programmes entrusted to the Agency (e.g. 
compliance  with  the  Commission’s  guidelines  on  information  and  visibility  of 
programmes35,  instruments  put  in  place  to  ensure  the  visibility  of  the  EU  as 
promoter of the programmes). 

Efficiency? 

  To what extent have the actual costs (including cost of coordination and monitoring) 
of REA corresponded to the estimates of the 2013 CBA? If not, what are the reasons 
behind? 

  To  what  extent  have  the  actual  benefits  corresponded  to  the  estimates  of  the  2013 

CBA? If not, what are the reasons behind? 

  To what extent has the management and execution of the programmes by REA been 

cost-effective as compared to the alternative options36? 

33 Replacing, as of 1 January 2018, the Internal Control Standards – see C(2017)2373 final. 
34 Including applicants, beneficiaries, experts. 
35 https://ec.europa.eu/info/resources-partners/european-commission-visual-identity_en  
36 Examples of alternatives to the Executive Agency: (i) management of the programme(s) by the Commission, 
(ii)  mixed Agency  - Commission management, (iii) partial management by the Commission while outsourcing 
some activities to the extent legally possible.  

19 

 
 
 
 
 
                                                           
  To what extent has REA contributed to improved management of the programmes in 

terms of:  
−  Simplification of the procedures and flexibility in the implementation of delegated 

tasks (e.g. capacity to adapt to periods of high workload)? 

−  Which further scope for simplification exists? 

  Which  aspects/means/actors  or  processes  render  REA  more  or  less  efficient?  What 
could  be  improved?  What  is  the  quality  of  the  services/advice  provided  by  REA  to 
stakeholders and addressees? 

  To what extent has the establishment of REA resulted in savings to the EU budget as 
compared to the alternative options (e.g. difference in costs between the Commission 
option and the Agency option)? 

  To what extent have REA’s internal organisation and procedures been conducive to its 

efficiency? 

  To what extent is REA’s internal organisation capable and flexible to rapidly respond 

to resource needs due to uncertainties related to volumes of work? 

  To  what  extent  does  REA’s  human  resources  management  contribute  to  the 

achievement of the Agency’s objectives?  
− 
− 

Is the size and structure of the organisation appropriate? 
Is  the  balance  between  operational  staff  and  administrative  support  staff 
appropriate? 

−  Are the staff turnover and vacancy rate well managed?  
−  How are provisions on overtime and teleworking implemented in the Agency? 
−  How does the Agency follow up on the findings of the latest staff survey (2016)? 

Coherence 

  To  what  extent  have  there  been  overlaps/  gaps/  inconsistencies/  complementarities 
within the programme portfolio and support services managed by REA  and how  are 
these addressed? 
Is there a clear and appropriate delimitation of responsibilities and tasks between REA 
and  the  Parent  DGs?  Are  there  overlaps  or  gaps?  Are  the  different  responsibilities 
adequately communicated to the beneficiaries? 

 

  To what extent has REA enabled the Commission to better focus on its policy related 

tasks? 
−  Are  there  any  governance  (financial  and  policy)  issues  in  relation  to  the 
implementation of the tasks that are delegated to the executive agencies/ issues as 
regards  the  Commission’s  services  being  able  to  steer  EU  policy  or  budget 
implementation? 

−  Are  appropriate  mechanisms  and  instruments  in  place,  and  at  which  level,  to 
ensure  an  adequate  coordination  and  information  flow  between  REA  and  the 
Commission  services,  notably  on  the  content  of  the  projects  supported  and  their 
results? 

−  Does  REA  provide  useful  information  on  the  implementation  of  the  delegated 
programmes and their progress (in terms of management and content) in support of 
the policy process  (e.g. information  required for  the annual  Management  Plan of 
the Parent DGs)? 

  To  what  extent  have  the  activities  of  REA  resulted  in  unintended  effects  (both 

desirable and undesirable)? 

  To what extent has the Commission, in the presence of REA, been able to maintain an 
adequate level  of know-how in  relation  to the programmes  entrusted to  the Agency? 

20 

 
 
 
 
How  has  this  been  achieved?  What  are  the  feedback  channels,  means  and  methods 
used for this purpose? What are areas for improvement, if any? 

  Have  the  monitoring,  reporting  and  supervision  arrangements  in  place  enabled  the 
Commission  to  benefit,  in  the  short  and  medium  term,  from  the  know-how  created 
within REA? 

  How  effective  is  the  flow  of  information  and  communication  between  REA  and  the 

Commission services (in particular Parent DGs)? 

  To what extent would the closing down of REA result in losing significant know-how 

in relation to the management of the programmes entrusted to REA? 

21