COMMISSION STAFF WORKING DOCUMENT Evaluation of the Research Executive Agency (REA) Accompanying the document REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL AND THE COURT OF AUDITORS Evaluation of the Consumers, Health, Agriculture and Food Executive Agency, the Educational, Audiovisual and Culture Executive Agency, the Executive Agency for Small and Medium-Sized Enterprises, the European Research Council Executive Agency, the Innovation and Networks Executive Agency and the Research Executive Agency
Date: 2025-04-16
Institutions: Directorate-General for Research and Innovation, European Commission
Subjects: EU aid, EU financing, EU programme, executive agency, financial control, performance audit, research

EN 

  EN 

   EUROPEAN COMMISSION  Brussels, 16.4.2025  SWD(2025) 92 final  COMMISSION STAFF WORKING DOCUMENT Evaluation of the Research Executive Agency (REA) Accompanying the document REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL AND THE COURT OF AUDITORS Evaluation of the Consumers, Health, Agriculture and Food Executive Agency, the Educational, Audiovisual and Culture Executive Agency, the Executive Agency for Small and Medium-Sized Enterprises, the European Research Council Executive Agency, the Innovation and Networks Executive Agency and the Research Executive Agency {COM(2025) 171 final} - {SWD(2025) 87 final} - {SWD(2025) 88 final} - {SWD(2025) 89 final} - {SWD(2025) 90 final} - {SWD(2025) 91 final}  
 
 
1 

INTRODUCTION 

Table of Contents 

2  WHAT WAS THE EXPECTED OUTCOME OF THE INTERVENTION? 

2.1 

2.2 

Description of the intervention and its objectives 

Baseline and points of comparison 

3  HOW HAS THE SITUATION EVOLVED OVER THE EVALUATION PERIOD? 

4 

EVALUATION FINDINGS (ANALYTICAL PART) 

4.1 

4.2 

To what extent was the intervention successful, and why? 

Cost-benefit analysis 

5  WHAT ARE THE CONCLUSIONS AND LESSONS LEARNED? 

5.1 

5.2 

Conclusions 

Lessons learned 

ANNEX I: PROCEDURAL INFORMATION 

ANNEX II: METHODOLOGY AND ANALYTICAL MODELS USED 

ANNEX III: EVALUATION MATRIX AND ANSWERS TO THE EVALUATION QUESTIONS (BY 
CRITERION) 

ANNEX IV: OVERVIEW OF BENEFITS AND COSTS 

ANNEX V: COST-BENEFIT ANALYSIS 

ANNEX VI: STAKEHOLDERS’ CONSULTATION – SYNOPSIS REPORT 

3 

5 

5 

8 

8 

10 

10 

16 

17 

17 

18 

20 

22 

24 

25 

26 

29 

1 

 
 
 
 
 
 
 
 
 
 
Glossary 

Term or acronym 
Act of delegation 

Act of establishment 

Agencies 
Commission 
CHAFEA 

EASME 

EEA 
EFTA 
EISMEA 

EU 
FET 
Framework  Regulation  for  executive 
agencies 

Seventh Framework Programme 

HaDEA 
Horizon 2020 

HR 
IAS 
IT 
REA 
SEDIA 

Implementing 

research  and 

innovation  comprising, 

Meaning or definition 
Commission Decision C(2013)9418 on delegating 
powers to the Research Executive Agency with a 
view  to  performance  of  tasks  linked  to  the 
implementation of Union programmes in the field 
of 
in 
particular, 
implementation  of  appropriations 
entered in the general budget of the Union 
Decision 
Commission 
2013/778/EU  of  13  December  2013  establishing 
the  Research  Executive  Agency  and  repealing 
Decision 2008/46/EC 
Executive agencies 
European Commission 
Consumers,  Health,  Agriculture  and  Food 
Executive Agency 
Executive  Agency  for  Small  and  Medium-Sized 
Enterprises 
European Economic Area 
European Free Trade Association 
European 
Innovation  Council 
Executive Agency 
European Union 
Future and emerging technologies 
Council  Regulation  (EC)  No  58/2003  of  19 
December  2002  laying  down  the  statute  for 
executive  agencies  to  be  entrusted  with  certain 
tasks 
the  management  of  Community 
programmes 
the 
The  seventh  framework  programme  of 
European Community for research, technological 
development  and  demonstration  activities  (2007-
2013) 
Health and Digital Executive Agency 
Horizon 2020, the 8th EU framework programme 
for research and innovation (2014-2020)  
Human resources 
Internal Audit Service 
Information technology 
Research Executive Agency 
Single Electronic Data Interchange Area 

and  SMEs 

in 

2 

 
 
 
 
 
 
 
 
 
 
 
 
1 

INTRODUCTION  

Purpose and scope of the evaluation/fitness check 

The  purpose  of  the  present  evaluation  is  to  assess  the  operations  of  the  Research  Executive 
Agency (‘REA’ or ‘the Agency’) between 16 July 2018 and 31 March 2021. In accordance with 
Article 25 of the Framework Regulation for executive agencies1, the Commission submits this 
evaluation  to  the  European  Parliament,  the  Council,  the  Court  of  Auditors  and  the  Steering 
Committee of the Agency. 

REA is governed by the Framework Regulation for executive agencies, its act of establishment2, 
which sets out its mandate, the act of delegation3, which specifies the tasks to be carried out and 
the powers delegated to the Agency, and the decision establishing the REA Steering Committee.  

This evaluation concerns the operations of the Agency in accordance with its mandate under the 
2014-2020  multiannual  financial  framework.  REA  was  entrusted  by  the  Commission  to 
implement  parts  of  the  Horizon  2020  Framework  Programme  for  research  and  innovation 
(‘Horizon  2020’)  and  the  legacy  under  the  Seventh  Framework  Programme  for  research, 
technological  development  and  demonstration  activities 
(‘the  Seventh  Framework 
Programme’), and to provide administrative and logistical support services. In this respect, the 
Agency managed some or all of the phases in the lifetime of projects, carrying out the necessary 
checks to that end, implemented the budget and provided relevant information to policymakers. 
The evaluation period of this report predates the establishment of REA’s new mandate under 
the 2021-2027 multiannual financial framework. 

In line with the Commission’s Better Regulation principles4, the evaluation assesses: (i) whether 
the Agency has fulfilled its tasks effectively and efficiently; (ii) whether there are overlaps, gaps 
or inconsistencies in the Agency’s management of the programme portfolio and (iii) whether 
there is a clear delineation of tasks between the  REA and the parent Directorates-General or 
other executive agencies (coherence)5.  

The evaluation  also  assesses whether the functioning  of the Agency has yielded the positive 
results  estimated  in  the  2013  ex  ante  cost-benefit  analysis  for  the  delegation  of  tasks  to  the 
executive agencies6 and the specific financial statement. The retrospective cost-benefit analysis 

1  

2  

3  

4  

5  

6  

Council Regulation (EC) No 58/2003 of 19 December 2002 laying down the statute for executive agencies to be entrusted with certain 
tasks in the management of Community programmes, OJ L11, 16.1. 2003, p.1 http://data.europa.eu/eli/reg/2003/58/oj. 

Commission  Implementing  Decision  2013/778/EU  of  13  December  2013  establishing  the  Research  Executive  Agency  and  repealing 
Decision 2008/46/EC, http://data.europa.eu/eli/dec_impl/2013/778/oj. 

Commission Decision C(2013)9418 on delegating powers to the Research Executive Agency with a view to performance of tasks linked 
to  the  implementation  of  Union  programmes  in  the  field  of  research  and  innovation  comprising,  in  particular,  implementation  of 
appropriations entered in the general budget of the Union. 

Commission’s Better Regulation guidelines SWD (2021) 305 final and toolbox. 

The Better Regulation guidelines were applied in a proportionate way. There was no need to assess the evaluation criterions of ‘EU value 
added’ and of ‘relevance’ as REA carries out tasks, which the Commission has transferred to it and these criterions are  assessed when 
evaluating the programmes. 

Cost-Benefit Analysis for the delegation of certain tasks regarding the implementation of Union Programmes 2014 2020 to the agencies - 
Final report for the Commission of 19 August 2013. 

3 

 
 
 
 
 
 
 
considered the updated estimates from the specific financial statement, as amended in 2017 to 
reflect the extended mandate of the Agency. These estimates have been tested in comparison 
with the actual costs for programme implementation. The aspects covered by the cost-benefit 
analysis are specified in Article 3(1) of the Framework Regulation for executive agencies7 and 
the Guidelines on establishing and operating executive agencies8. 

The evaluation of the Agency does not cover the achievements of the programmes managed by 
REA, which are subject to mid-term and ex post evaluations themselves. The evaluation of REA 
nevertheless  provides  useful  input  for  these  programme  evaluations,  considering  that  the 
performance  of  the  Agency  affects  the  efficiency  and  effectiveness  of  the  programmes  it 
manages. 

The evaluation of REA is supported by a study carried out by external contractors9. The study 
combined  a  retrospective  and  a  prospective  analysis.  The  retrospective  analysis  assessed  the 
operation of the Agency and its performance during the evaluation period, while the prospective 
analysis suggested recommendations for improving the Agency’s performance going forward. 

The  evaluation  of  the  Agency’s  performance  is  evidenced-based,  using  a  mixed-methods 
approach in a complementary manner, by connecting: 

•  a  qualitative  approach,  based  on  open-ended  survey  questions,  interviews,  a 

documentary review and desk research, a qualitative cost-benefit analysis; 

•  a  quantitative  approach,  based  on  administrative  and  monitoring  data,  surveys  and  a 

quantitative cost-benefit analysis.  

All  stakeholders  were  consulted  via  interviews  and  surveys  (Commission  staff,  REA  staff, 
beneficiaries,  unsuccessful  applicants,  experts).  Unsuccessful  applicants  participated  less 
actively compared with beneficiaries. There was a rather low response rate to the survey among 
beneficiaries and unsuccessful applicants (i.e. 10% on average), which could be due to survey 
fatigue.  

Overall, the evaluation of the Agency provides findings based on solid evidence. Triangulation 
of data sources improved the validity of the findings, while a retrospective cost-benefit analysis 
examined cost-effectiveness.  

7  

The cost-benefit analysis should include the following: identification of the tasks justifying outsourcing, a cost-benefit analysis which 
includes the costs of coordination and checks, the impact on human resources, possible savings within the general budgetary framework 
of  the  EU,  efficiency  and  flexibility  in  the  implementation  of  outsourced  tasks,  simplification  of  the  procedures  used,  proximity  of 
outsourced activities to final beneficiaries, visibility of the EU as promoter of the EU programme concerned and the need to maintain an 
adequate level of know-how inside the Commission. 

8   Appendix II of the Guidelines for the establishment and operation of executive agencies financed from the Union budget (C(2014) 9109 

from 2 December 2014.. 

9  

Study supporting the evaluation of REA by Public Policy Management Institute Consortium formed by Public Policy Management 
Institute and IDEA Consult.https://op.europa.eu/en/publication-detail/-/publication/c9645710-e5bd-11ee-8b2b-01aa75ed71a1/language-
en/format-PDF/source-310469329. 

4 

 
 
 
 
 
 
 
 
 
 
 
 
 
2  WHAT WAS THE EXPECTED OUTCOME OF THE INTERVENTION? 

2.1 Description of the intervention and its objectives 

According  to  the  Framework  Regulation  for  executive  agencies  and  REA’s  Act  of 
establishment, the purpose of entrusting the Agency with certain programme implementation 
tasks is as follows.  

• 

• 

It  enables  the  Commission  to  focus  on  its  core  institutional  tasks,  which  cannot  be 
outsourced, namely tasks assigned under the Treaty on the Functioning of the European 
Union that require discretionary powers in translating political choices into action. 

It enables the Commission to achieve the objectives of the delegated EU programmes 
more effectively and efficiently. According to the 2013 ex ante cost-benefit analysis, 
delegating tasks to REA was estimated to be more cost-efficient than dealing with them 
in-house. The provision of administrative and logistical support services will result in 
additional cost-efficiency gains and economies of scale. 

Between 2018 and 2021, REA implemented the following EU programmes in part or in full.  

1)  Horizon 2020 

a)  Excellent science (Pillar 1) 

-  Marie Skłodowska-Curie actions 
-  Future and emerging technologies 

b)  Industrial leadership (Pillar 2): Space research  
c)  Societal challenges (Pillar 3) 

-  Food  security,  sustainable  agriculture  and  forestry,  marine,  maritime  and  inland 

water research and bioeconomy 

-  Secure societies, including, as of 2018, the processing of grants that may generate 

EU classified information  
Inclusive, reflective and innovative societies 

- 

d)  Specific objectives (Pillars 4 and 5)  

-  Spreading excellence and widening participation 
-  Science with and for society 

2)  Legacy of the following parts of the Seventh Framework Programme: 

a)  The activities ‘research for SMEs’ and ‘research for SME associations’ of the capacities 

specific programme 

b)  The ‘space’ and ‘security’ themes of the cooperation specific programme 
c)  The people specific programme 

3)  REA also provided the following administrative and logistical support services: 

a)  planning  for  calls  for  proposals  and  contests  for  prizes  (including  support  for 
publication),  logistical  support  for  evaluation  of  project  proposals  (including  the 
management  of  the  evaluation  facilities),  contracting  and  payment  of  independent 
experts,  validation  of  legal  entities,  preparation  of  legal  entities’  financial  viability 
assessment, management of the Research Enquiry Service for Horizon 2020, the Seventh 
Framework Programme and other EU programmes, as the case may be; 

5 

 
 
 
 
 
  
 
 
b)  single-entry  point  for  participants  in  grants  and  procurement  activities  for  all  EU 
programmes,  for  the  benefit  of  the  EU  bodies  implementing  them  through  direct 
management10  within  the  framework  of  the  Single  Electronic  Data  Interchange  Area 
(‘SEDIA’)11;  this  mainly  entails  validation  of  legal  entities  and  preparation  of  legal 
entities’ financial capacity assessment. 

The intervention logic  analyses  the  Agency’s  objectives and  activities, as set  out  in  its legal 
basis.  It  was  adapted  considering  the  specific  context,  objectives,  inputs  and  processes  with 
which to measure the Agency’s key results and performance. The below figure summarises the 
approach used. 

10   Or the first level of transactions via indirect management. 

11   Validation services to joint undertakings are provided by REA within the framework of Horizon 2020 support services. 

6 

 
 
 
 
 
  
 
 
 
Figure 1. REA’s intervention logic: 

Source: Study supporting the triennial evaluation of the Agency 

7 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
The key inputs are human resources, financial resources, IT tools and communication channels. 
The  key  processes  derive  from  REA’s  Act  of  delegation  and  the  main  outputs  of  REA’s 
performance  relate  to  the  management  of  the  delegated  programmes.  Achieving  the  outputs 
depends on achieving key performance indicators, standard indicators of financial management, 
and key metrics related to the proposal and project management life-cycles (e.g. number of calls, 
number of running projects). The key feedback to policy related outputs were also considered. 
Outcomes relate to the key results achieved during the evaluation period and impacts (medium-
to-long term outcomes). The satisfaction of unsuccessful applicants, beneficiaries and experts 
with the performance of the Agency was one of the key elements that informed the measurement 
of these results. In addition, key outcomes are better-informed policymaking on the part of the 
parent  Directorates-General  and  the  Agency’s  contribution  to  policy  priorities  of  the  parent 
Directorates-General and the programmes implementation. 

The results of the quantitative cost-benefit analysis are also presented, including the workload 
analysis  and  evaluation  of  key  workload  drivers,  assumptions  and  productivity  indicators 
(operational budget delegated to the Agency, number of proposals, number of projects, average 
grant size, etc.). These results underpin the workload analysis and staffing estimates in the 2013 
cost-benefit analysis and specific financial statement, compared with the actual situation. 

2.2  Baseline and points of comparison 

The current evaluation of REA operations assesses the actual costs and benefits of programme 
implementation by REA (executive agency scenario) compared with the alternative scenario of 
management by the Commission (in-house scenario). To this end, the 2013 ex ante cost-benefit 
analysis and the specific financial statement of REA, as amended, are the reference points for 
comparison. 

Compared with the in-house scenario, the efficiency gains of the executive agency scenario were 
initially estimated at EUR 87.4 million in the specific financial statement, as amended. Non-
quantifiable benefits were also estimated, such as improved quality of programme management 
and service delivery, improved visibility of EU programmes and proximity to beneficiaries. 

3  HOW HAS THE SITUATION EVOLVED OVER THE EVALUATION PERIOD? 

Current state of play 

Established in 2008, REA’s mandate was renewed and extended in 2014 to implement parts of 
Horizon  2020  and  the  Seventh  Framework  Programme  and  to  provide  administrative  and 
logistical support services. 

The Agency’s mandate was further extended in 2018 when the Agency was entrusted with the 
legal  validation  of  third  parties  and  the  preparation  of  legal  entities'  financial  capacity 
assessment within the SEDIA framework for all EU programmes. REA was also entrusted with 
implementing projects generating EU  classified information under the Horizon 2020 societal 
challenge ‘Secure societies – protecting freedom and security of Europe and its citizens’.  

During the evaluation period, the number of Agency staff increased from 736 in 2018 to 781 by 
the  end  of  2020,  making  REA  the  largest  of  the  six  executive  agencies.  REA  achieved  an 
advanced level of organisational maturity and properly allocated and matched staff according 
to the diversity of its tasks and emerging needs. REA successfully adapted to the challenges of 

8 

 
 
 
 
 
 
 
 
 
the  COVID-19  pandemic  by  adapting  its  activities  to  a  remote  working  environment  and 
implementing teleworking for staff.  

REA’s  staff  were  highly  engaged  during  the  evaluation  period,  as  indicated  by  the  staff 
engagement index registered in staff survey (69% in 2018). Furthermore, the Agency analysed 
the findings of the staff surveys and adopted appropriate action plans with the close involvement 
of staff.  

Vacancy rates within REA increased during the evaluation period (from 3.29% in 2018 to 4.99% 
in 2020), partly due to the impact of the COVID-19 pandemic and other factors, such as limited 
reserve lists and staff career choices.  

A series of organisational and procedural changes were brought in during the evaluation period. 
REA carried out a major reorganisation in preparation for the new responsibilities delegated as 
of 1 April 2021 under the 2021-2027 multiannual financial framework. Between 2020 and 2021, 
REA created a Transition Steering Group and drew up an action plan to facilitate the transition 
in close  collaboration  with  the  Commission. The  Transition Steering Group  proposed a new 
organisational structure for REA, which was endorsed by its Steering Committee in December 
2020.  

The Agency’s administrative budget (commitments) increased from EUR 69.6 million in 2018 
to EUR 79.3 million in 2020. The operational budget (commitments) increased from EUR 2 282 
million in 2018 to EUR 2 731 million in 2020.  

The number of calls launched by the Agency increased from 20 in 2018 to 47 in 2021, while the 
number of grants signed remained relatively stable throughout the period (there were 2 030 in 
2018, increasing to 2 335 in 2020 and then falling to 2 050 in 2021). The number of running 
projects for both Horizon 2020 and the Seventh Framework Programme managed by REA fell 
from 6 248 projects in 2018 to 6 193 in 2019, and then increased again to 7 444 in 2021. 

The  number  of  key  administrative  and  logistical  support  services,  such  as  financial  capacity 
assessments, validation of entities and bank accounts, payments of experts, increased every year 
between 2018 and 2021, peaking in 2020 for several activities. The number of contracts signed 
for  expert  evaluators  rapidly  increased  from  16 781  contracts  in  2018  to  21 736  in  2020. 
Consequently, payments to expert evaluators increased to almost 27 000 payments in 2019. The 
number of financial capacity assessments handled by the REA also increased from 3 921 cases 
in 2018 to 6 010 in 2020. 

Furthermore, REA was responsible for legal entity and bank account validation from 2020 and 
validated 8 894 bank accounts in 2020 and 5 558 in 2021. The Agency also answered between 
8 000 and 11 500 enquiries every year through its Research Enquiry Service. 

The Agency’s extensive experience and knowledge of programme implementation meant that 
it  was  able  to  contribute  to  the  design  process  of  common  procedures  and  processes.  This 

9 

 
 
 
 
 
 
 
 
 
 
 
 
process was managed by the Common Support Centre12 and set the framework for harmonised 
implementation  by  all  implementing  bodies  with  respect  to  common  business  processes,  IT, 
legal, audit and information and data services. 

4  EVALUATION FINDINGS (ANALYTICAL PART) 

4.1  To what extent was the intervention successful, and why? 

Effectiveness 

For this evaluation, effectiveness relates to how successful the Agency has been in achieving or 
progressing towards its objectives to the satisfaction of its stakeholders. It also looked at the 
Agency’s ability to react and adapt to the challenges posed by the COVID-19 pandemic.  

The analysis of REA’s operations concluded that it operated within its legal framework during 
the evaluation period, continuously improved its operations and achieved a high level of overall 
effectiveness.  These  results are as demonstrated  by key performance indicators and the high 
satisfaction rate expressed in surveys by stakeholders. 

The  Agency  demonstrated  maturity,  stability  and  flexibility  in  executing  its  mandate  and 
addressing key matters that occurred during the evaluation period. These include implementing 
new activities in  line with the  extension of its mandate in 2018,  preparing  the new mandate 
under the 2021-2027 multiannual financial framework and addressing challenges related to the 
COVID-19 pandemic. 

REA achieved good overall results on programme implementation and achieved most of the key 
performance indicators, as mentioned below.  

In particular, REA was successful in preparing and signing grant agreements in good time, as 
measured  by  the  time-to-grant  indicator.  REA’s  average  time-to-grant  remained  below  the 
regulatory target of 245 days throughout the entire evaluation period (198 days in 2018, 215 
days in 2019, 205 days in 2020). The percentage of grants concluded within the time-to-grant 
target  reached  100%  in  2018  and  decreased  by  just  one  percentage  point  in  2019  and  two 
percentage points in 2020. This was mostly due to the COVID-19 pandemic, when applicants 
requested  additional  time  for  their  consortia  to  complete  the  necessary  administrative 
obligations before  grants  were signed. 87% of beneficiaries  surveyed agreed that the overall 
time  period  from  submission  of  the  proposal  to  the  signature  of  the  grant  agreement  was 
appropriate.  

The  Agency  has  improved  its  execution  of  payments  over  the  years,  as  the  percentage  of 
payments made within the contractual deadline increased from 98% in 2018 to 99% in 2019, 
and  to  100%  in  2020.  This  is  significant,  given  the  greater  number  of  projects  and  the 
complications  for  REA’s  operations  arising  from  the  COVID-19  pandemic.  High  levels  of 

12   The Common Support Centre is a directorate in the Commission’s Directorate-General for Research and Innovation. It provides services 
in the areas of legal support, ex post audit, IT systems and operations, business processes and programme information and data to  the 
whole research Directorates-General family, executive agencies and joint undertakings implementing Horizon 2020. It was renamed the 
Common Implementation Centre under the 2021-2027 multiannual financial framework. 

10 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
satisfaction  were  registered  among  the  beneficiaries  surveyed  with  the  timeliness  of  pre-
financing payments (94%), interim payments (91%) and final payments (89%).  

REA ensured high-quality procedures within the evaluation process, as evidenced by the small 
proportion of redress cases upheld: below 0.5% for Horizon 2020 calls (0.28% in 2018, 0.23% 
in 2019 and 0.31% in 2020). 

92%  of  experts  found  the  evaluation  process  to  be  smooth.  Overall,  remote  assessment  of 
proposals  was  rated  highly  by  experts,  since  89%  of  experts  considered  that  the  electronic 
evaluation system was easy to use and that the remote assessment of proposals was efficient. 
However,  some  experts,  who  experienced  both  in-person  and  remote  evaluations,  were  less 
satisfied with the efficiency of remote work, mainly due to the limitations of the IT tools and 
the absence of in-person networking. The Agency staff were responsive and provided guidance 
and  useful  answers  to  questions,  according  to  95%  of  experts  surveyed.  The  guidelines  and 
documents made available to experts were very useful and contributed to their understanding of 
the process and the related requirements before the start of evaluation, according to more than 
90% of experts surveyed. Experts had generally positive experiences but were less satisfied with 
the remuneration they received (68% of experts). The evaluation also noted the need for greater 
diversity among experts.  

The beneficiaries also rated the evaluation process positively (84% of beneficiaries). However, 
the survey suggested that the Agency’s feedback to applicants could be clearer, as only 48% of 
unsuccessful applicants felt satisfied with the transparency of the evaluation process.  

The Research Enquiry Service was considered helpful by the 97% of beneficiaries and 73% of 
unsuccessful applicants surveyed who had used it. However, awareness of this service could be 
improved since the survey showed that many beneficiaries (59%) and unsuccessful applicants 
(66%) were unaware of its existence.  

The REA Validation Service was also rated positively by 81% of beneficiaries surveyed, who 
agreed that the electronic tools for validation were user-friendly. The application process was 
considered  to  be  clear  and  transparent  by  92%  of  beneficiaries  and  77%  of  unsuccessful 
applicants surveyed.  

As mentioned above, the surveys showed high levels of overall satisfaction with the services 
provided  by  REA  (86%  of  beneficiaries,  95%  of  external  experts  and  57%  of  unsuccessful 
applicants). The stakeholders surveyed indicated a clear willingness to work with REA again in 
the future (94% of beneficiaries, 83% of unsuccessful applicants and 98% of experts). 

On  budget  execution,  the  Agency  fully  executed  its  operational  budget  in  terms  of  both 
commitment and payment appropriations, as in the previous evaluation period. 

To ensure sound financial management, the Agency improved its internal control framework in 
line with the Commission’s internal control framework implementation guide. This allows it to 
closely monitor progress against objectives. It also allows it to prevent and mitigate potential 
risks to its operations in a timely and effective way. The European Court of Auditors’ annual 
reports confirmed that REA’s operations, its cash flow and the changes in net assets were in line 
with  the  Financial  Regulation  and  with  the  accounting  rules  adopted  by  the  Commission’s 
accounting officer. The Agency also adopted remedial measures to address the observations and 
audit recommendations received. 

11 

 
 
 
 
 
 
 
 
 
 
 
 
 
The main indicator for the legality and regularity of transactions is the cumulative error rate, 
which  is  extrapolated  after  implementing  corrective  measures  following  audits.  As  in  the 
previous evaluation, under the Seventh Framework Programme, the cumulative residual error 
rate was below 2% for the people programme and above the 2% target for space, security and 
SME actions. This estimated error rate can be attributed largely to the complexity of the funding 
scheme under the Seventh Framework Programme. The lessons learned from the audits under 
the Seventh Framework Programme significantly contributed to developing an improved and 
simplified  control  framework  under  Horizon  2020.  Consequently,  the  Agency’s  cumulative 
residual error rate under Horizon 2020 ranged between 2.6% in 2018 and 1.75% in 2021 (target 
2-5%). For Marie Skłodowska-Curie actions under Horizon 2020, the cumulative residual error 
rate fell from 1.55% in 2018 to 0.5% in 2021 (target below 2%). 

As in the previous evaluation, REA ensured that staff were available to programme participants 
and  communicated  with  them  during  various  phases  of  the  project  lifecycle.  87%  of 
beneficiaries and 97% of experts surveyed knew whom to contact or how to contact REA for 
any question on implementation of their grants/projects. Furthermore, those REA staff assigned 
to projects were easily available and responsive, according to 86% of beneficiaries surveyed. 
REA also answered citizens’ questions on research framework programmes and the validation 
process via the Research Enquiry Service, which was appreciated by survey respondents. 

REA  intensified  its  efforts  around  external  communication  activities,  reaching  audiences 
through various channels. REA ensured the EU’s visibility as the promoter of the programmes 
through communication and media campaigns. Although 98% of beneficiaries were aware that 
their grants or contracts were funded from the EU budget, significantly fewer (67%) were aware 
that REA was entrusted by the Commission to manage the programmes. 

REA further sought to refine its approach to feedback to policy so as to provide an effective 
response in  collaboration with its parent  Directorates-General.  As a result,  specific plans for 
providing  project  results  relevant  for  policymaking,  agreed  between  REA  units  and  their 
Commission counterparts, were put in place for Horizon 2020 activities, starting in 2019.  

Efficiency 

This section considers the relationship between the resources used by the Agency and its output. 
Among other factors, it covers the management and execution of programmes delegated to the 
Agency and the extent to which financial and human resource management was sound during 
the  evaluation  period.  It  also  includes  an  analysis  of  and  ways  in  which  to  simplify 
administrative and regulatory burden. These aspects of the Agency’s performance are monitored 
through key indicators and with additional survey data.  

The Agency’s efficiency and workload are closely related to the budget managed, number of 
staff, number of proposals and number of grants. The main indicators for 2018-2020 (in EUR 
million or %) are provided below. 

12 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
  Indicator 

2018  

2019  

2020  

Operational budget, commitments (EUR) 

Operational budget, payments (EUR) 

Administrative budget, commitments (EUR) 

Administrative budget, payments (EUR) 

Number of staff (at the end of the year)13  

Number of operational staff 14 

Programme management cost ratio (commitments)  

Programme management cost ratio (payments)  

Budget ‘per operational head’ 
(commitments) (million EUR)  

Proposals received 

Total running projects  

Running projects ‘per operational head’  

2 282.5 

1 935.2 

69.6 

66.8 

736 

509 

3.05% 

3.45% 

2 505.3 

2 731.1 

2 140.7 

2 176.7 

75.8 

74.8 

753 

526 

79.3 

78.1 

781 

552 

3.02% 

3.49% 

2.91% 

3.59% 

4.84 

4.57 

4.8 

 14 610 

 14 749 

 16 719 

6 248 

13.25 

6 193 

11.31 

6 595 

11.58 

The Agency’s operational and administrative budget increased between 2017 and 2020. REA 
proved to be an efficient and cost-effective structure for managing the delegated programmes, 
as demonstrated by the programme management cost ratio between the administrative budget 
of the Agency and the operational budget it manages. This ratio of administrative to operational 
budgetary  commitments  improved  each  year  and  reached  2.91%  in  2020.  For  payments 
executed, the ratio was stable at almost 3.5% in 2018 and 2019 and increased slightly to almost 
3.6 % in 2020.  

In  comparison  with  the  2013  ex  ante  cost-benefit  analysis,  the  actual  number  of  new  grants 
managed by REA was approximately 27% higher between 2014 and 2020 (10 958 compared to 
13 911). This increase in number of new grants is mostly due to the grants awarded under Marie 
Skłodowska-Curie actions whose size was smaller than initially estimated and to the additional 
grants financed from EEA/EFTA and third-country contributions. The increased number of new 
projects consequently generated an increased project monitoring workload. 

On administrative support services15, the number of experts supported on site was lower than 
initially  estimated  due  to  the  use  of  remote  evaluations  during  the  COVID-19  pandemic. 
Between 2018 and 2020, the number of experts contracted was lower overall than the initial 
estimate  (58 770  contracts  compared  to  72 534).  However,  the  Agency’s  validation  services 
registered an increased  workload:  the  cumulative number of validations  of legal  entities and 

13   Currently filled. 

14   Currently filled. 

15   Estimates as per both the 2013 ex ante cost-benefit analysis and the 2015 cost-benefit analysis for the delegation of validation of legal 

entities and preparation of legal entities’ viability assessment. 

13 

 
 
 
 
 
 
 
Legal  Entity  Appointed  Representative  (LEAR)  extended  mandates  was  higher  than  initial 
estimates (55 754 compared to 33 429). There was also a higher number of financial capacity 
assessments (15 097 compared to 6 549).  

Expert management 

Number of experts supported on site 

Number of expert contracts 

Research enquiry service 

 2018 

 2019 

 2020 

6 418 

16 781 

7 119 

1 602 

20 253 

21 736 

Number  of  requests  received  by  the  Research 
Enquiry Service 

8 804 

9 826 

11 502 

Validation services 

Number of legal status of entities (FEL) validations  

Number of LEAR extended mandate validations 

Number of change requests / database maintenance 
on request of participants 

7 360 

9 767 

7 734 

8 769 

10 786 

11 338 

34 056 

47 527 

48 228 

Number of financial capacity assessments (FCA) 

3 921 

5 166 

6 010 

The  consolidation  of  validation  services  in  REA  constituted  a  major  efficiency  gain  for  the 
Commission  and  reduced  the  efforts  required  from  all  agencies,  the  Commission  and 
participants. REA became a ‘one-stop-shop’ for all applicants participating in procurement and 
grant processes and this reduced the administrative burden for applicants. This also allowed for 
a coherent and high-quality database of validated participants to be maintained, which would 
not have been possible had processing validations been distributed across a range of bodies. As 
regards the implementation of validation services, the Agency is looking into options to further 
simplify the process with the aim of reducing workload for its staff. 

Significant efficiency gains were achieved in REA’s management of Horizon 2020 following 
the introduction of simplified rules and procedures, streamlined IT systems and clearer guidance 
for  beneficiaries.  REA  showed  maturity  and  flexibility  as  it  adapted  to  the  changes  brought 
about  by  a  new  framework  programme  and  carried  out  all  necessary  simplifications.  It 
effectively used corporate IT tools to support the grant processes to become fully paperless and 
helped staff to work more quickly and reduce errors. The IT tools were appreciated by internal 
stakeholders but were rated less favourably by experts and applicants, who thought that they 
could be made more user-friendly. 

Overall, the Agency’s actual workload and productivity levels were higher than the assumptions 
made  in  the  ex  ante  cost-benefit  analysis.  Although  the  actual  workload  was  influenced  by 
various parameters beyond the Agency’s control, such as number of grants, average grant size 
or external factors such as the COVID-19 pandemic, the Agency’s organisational development 
contributed  to  these  good  results.  To  achieve  a  fit-for-purpose  organisation,  REA's 
organisational  structure  was  aligned  to  the  specific  programmes  it  manages,  ensuring 

14 

 
 
 
 
 
 
  
  
  
  
  
  
  
  
  
 
 
responsiveness to programme needs. The Agency carried out an annual workload assessment 
and  allocated  resources  according  to  each  unit’s  effective  workload.  This  contributed  to 
maintaining a balanced staff composition with around 70% operational staff, 20% neutral staff16, 
and 10% administrative staff.  

In  summary,  REA  was  ready  to  take  up  new  activities,  continuously  adapting  to  evolving 
challenges and improving the efficiency of its operations. REA also contributed to developing 
various implementing processes and guidelines for the whole community of agencies, shared 
best practices and actively participated in the various working groups with the Commission and 
other agencies. 

Coherence  

Coherence  concerns  the  overlaps  and  complementarities  within  the  programme  portfolio 
managed by the Agency, and the delimitation of responsibilities between the Agency and its 
parent Directorates-General. It also looked at the Agency’s ability to provide feedback to policy 
to its parent Directorates-General. 

The key findings revealed a clear and appropriate delimitation of roles between REA and its 
parent  Directorates-General  with  no  overlaps,  in  line  with  the  legal  framework.  The  Act  of 
delegation  clearly  sets  out  the  activities  delegated  to  the  Agency  and  those  reserved  for  the 
Commission.  The  memorandum  of  understanding,  signed  by  the  Agency  and  its  parent 
Directorates-General,  provides  for  the  interactions  within  the  overall  monitoring  and 
supervision framework. It  aims to avoid gaps, duplication of efforts, micro-management and 
sets  out  clear  responsibilities.  The  memorandum  of  understanding  ensures  a  constant  and 
smooth flow of know-how and information between the Agency and the parent Directorates-
General  via various mechanisms, strengthening cooperation and reporting at  different  levels, 
from the level of policy and project officers to the high-level Steering Committee meetings.  

During  the  evaluation  period,  the  Agency  had  a  coherent  programme  portfolio  since  REA 
implemented only parts of research framework programmes, for multiple parent Directorates-
General and across a diverse thematic spectrum. The delegated parts of the programme were 
governed by common rules and procedures under the respective research programme (Horizon 
2020 or the Seventh Framework Programme). There was strong coherence between the Seventh 
Framework Programme and its successor programme, Horizon 2020, in terms of continuity of 
activities. The harmonisation of business processes, led by the Common Support Centre, made 
the overall programme implementation across thematic areas coherent, although there were still 
some specific differences. However, some inconsistencies were observed in the tasks delegated 
to REA and were the result of sharing thematic areas between agencies.  

The Agency maintained good communication with the parent Directorates-General via a range 
of  well-functioning  mechanisms.  In  particular,  REA  made  significant  efforts  to  improve  its 
mechanisms for feedback to policy: (i) using the Catalogue of feedback to policy options, which 
provides  support  to  beneficiaries  to  enable  them  to  increase  the  dissemination  capacity  of 

16   Staff who were mainly responsible for financial transactions, audits and other types of support. 

15 

 
 
 
 
 
  
 
 
 
 
projects; and (ii) participating in the Rapid Reaction Network (i.e. AGILE)17. The Catalogue 
was followed in 2019 and 2020 by annual feedback to policy plans tailored to the needs of each 
part of the delegated programme.  

4.2  Cost-benefit analysis  

The retrospective cost-benefit analysis was carried out for 2018-2020 based on the assumptions 
laid down in the specific financial statement, as updated in 2017 following the extension of the 
REA’s mandate, and the actual costs of the Agency.  

Compared with the in-house scenario, the efficiency gains of the executive agency scenario were 
initially estimated at approximately EUR 87.4 million in the specific financial statement. The 
retrospective cost-benefit analysis determined that actual savings were 34.6% higher than initial 
estimates in the specific financial statement, amounting to EUR 117.6 million for 2018-2020. 

For 2018-2020, actual costs amounted to EUR 230.5 million, including EEA/EFTA and third-
country contributions (EUR 13.8 million during 2018-2020). Based only on the EU contribution, 
actual  costs  amounted  to  EUR  216.7  million.  Compared  to  the  specific  financial  statement 
estimates (EUR 218.5 million for 2018-2020), the actual cost based on the EU contribution was 
lower and savings of EUR 1.7 million (or 0.8 % of the specific financial statement estimates) 
were made, despite rising costs and the fact that the specific financial statement estimates were 
not adjusted for inflation. 

Estimated costs and savings of the executive agency scenario in 2018-2020, million EUR 

305,84

218,48

87,36

348,10

230,52

117,58

400,00

350,00

300,00

250,00

200,00

150,00

100,00

50,00

0,00

SFS estimations

Actual

Costs of the in-house scenario

Costs of the executive agency (EA) scenario

Savings of the EA scenario

Source: Study supporting the triennial evaluation of the Agency 

As in the previous evaluation (2015-2018), actual costs under Title I ‘Staff-related expenditure’ 
were higher than the estimates in the specific financial statement. This is because those estimates 
did not include costs arising from salary indexation, promotions and increasing staff seniority. 
The ratio of temporary staff to contract staff corresponded to the estimates. However, the share 
of higher-grade contract staff increased because of greater reliance on IT automation, enabling 
REA to concentrate its resources on more content-related tasks. REA also employed additional 
contract staff financed from EEA/EFTA and third-country contributions to manage this budget. 

17   AGILE  was  designed  as a  flexible  tool  through  which  the  REA  could provide  knowledge and  information to  its parent  Directorates-

General quickly in the form of a factsheet overview.  

16 

 
 
 
 
 
 
 
 
The  actual  costs  under  Title  II  ‘Infrastructure  and  operating  expenditure’  were  significantly 
lower than estimates in the specific financial statement due to lower building costs and other 
savings. The actual cost under Title III ‘Programme support expenditure’ was higher than the 
initial 2018 estimates and close to the 2019 and 2020 estimates.  

5  WHAT ARE THE CONCLUSIONS AND LESSONS LEARNED? 

5.1  Conclusions 

The  Agency  was  effective  and  efficient  in  achieving  its  objectives,  which  enabled  the 
Commission to focus on its core institutional tasks. The mandate of the Agency was coherent in 
terms of programmes: it implemented only research programmes but across a diverse thematic 
spectrum. 

The Agency achieved 100% execution of its operational budget in terms of both commitment 
and payment appropriations, while remaining cost-effective. This is clear from the programme 
management  cost  ratio  in  terms  of  commitments  between  the  administrative  budget  of  the 
Agency and the operational budget it manages, which improved each year to reach 2.91% in 
2020. Since this figure demonstrates extremely high efficiency, in the future the focus could be 
on  further  increasing  the  quality  of  output  (e.g.  as  measured  by  the  satisfaction  rate  of 
stakeholders and feedback to policy). 

Programmes were implemented in  a timely manner, with key indicators such as the  time-to-
inform and time-to-grant generally remaining below the set targets.  

The  Agency  almost  achieved  its  target  of  executing  100%  of  its  payments  on  time,  which 
represented a positive evolution from 98% in 2018 to 99% in 2019 and 100% in 2020.  

The use of the corporate eGrants suite, comprising all IT tools to manage grants, had a positive 
effect on the Agency’s operational efficiency and meant that business processes became fully 
paperless. Furthermore, it facilitated fully remote work during the COVID-19 pandemic. 

The results of the 2018-2020 retrospective cost-benefit analysis confirm that the delegation of 
programme  implementation  remained  considerably  more  cost-effective  than  the  in-house 
scenario, generating major savings for the EU budget and real value for money.  

There is a continuous exchange and flow of information and sharing of best practices between 
the agencies and the Common Support Centre on implementing common business processes, IT 
tools  and  further  efficiencies  and  improvements.  Strengthening  synergies  and  harmonising 
procedures and processes is at the core of this cooperation. All the agencies have been consulted, 
have provided feedback on their experience and contributed to developing harmonised business 
processes, IT tools, reporting templates and guidelines in working groups (steering committees, 
executive committees and key user groups per business process).  

Communication between the Agency and the Commission was facilitated via well-functioning 
mechanisms, which were formalised and improved over the evaluation period. No evidence of 
overlaps, gaps or significant inconsistencies (see also Section 5.2 below) in the management of 

17 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
the programme parts delegated to the Agency was identified. There was a clear delineation of 
tasks between REA and the parent Directorates-General. 

5.2  Lessons learned  

Some inconsistencies were observed in the tasks delegated to REA between 2018 and 2021 and 
were  the  result  of  sharing  thematic  areas  between  agencies.  These  inconsistencies  were 
addressed  in  the  new  mandates  of  the  agencies  for  the  2021-2027  programming  period  by 
delegating programme parts such as Space Research and FET-Open (previously managed by 
REA) to other agencies (HaDEA and EISMEA). Moreover, REA was entrusted with new parts 
of the research programme and other EU programmes with a thematic link.  

Delegation  of  similar  thematic  areas  and  programmes  to  one  agency  should  lead  to  more 
efficient  implementation,  while  fewer  parent  Directorates-General  should  simplify  the 
governance and supervision of REA by the parent Directorates-General.  

The main lessons identified relate to improving the set-up and cooperation with other agencies 
and  the  parent  Directorates-Generals  on  key  processes,  such  as  feedback  to  policy,  human 
resources  and  IT.  More  cooperation  between  agencies  would  be  beneficial  for  knowledge 
sharing, strengthening synergies and harmonising procedures and processes.  

Sharing best practices on expert management and improving collaboration between stakeholders 
(agencies, joint undertakings and parent Directorates-General) is key to attracting high-profile 
experts.  Under  the  new  mandate,  REA  ran  a  communication  campaign  to  attract  expert 
evaluators. This generated a 46% increase in new registrations in the expert database, a 13% 
increase in new registrations from women experts, and upward trends in new registrations from 
under-represented EU countries.  

Strong  collaboration  between  REA  and  the  parent  Directorates-General  on  communication 
activities mean that duplication of efforts would be avoided and strategic efforts to efficiently 
communicate on programmes would be harmonised.  

The corporate IT tools used throughout the project management process should aim to improve 
the user experience and increase user-friendliness. The improvements to IT tools should take 
user feedback into account. Opportunities to improve the user-friendliness of corporate IT tools, 
such as the development of the Funding & Tenders Portal (previously the Participants Portal), 
should be taken as part of a continuous process.  

It is important to consolidate the feedback to policy dialogue between REA and the Commission. 
The process and approach to ensure feedback to policy have been improved over time, taking 
into account the specific nature of the programmes implemented, the variety of the needs of the 
parent Directorates-General and the related business processes. There were many developments, 
starting with the catalogue of feedback to policy options developed by REA, which marked a 
turning point and was the basis for specific feedback to policy mechanisms relevant to parent 
Directorates-General. This led to an annual feedback to policy plan in 2019 and 2020 tailored 
to the needs specific to each part of programme delegated, in line with the policy priorities of 
the  parent  Directorates-General.  This  approach  brought  clarity  but  less  flexibility  for 
unpredictable and ad hoc feedback to policy needs. There was therefore a requirement to strike 
a balance between structured and ad hoc feedback to ensure a constant flow of information and 
flexibility in addressing evolving policy needs. 

18 

 
 
 
 
 
 
 
 
 
 
  
 
To this end, a feedback to policy framework was developed in 2021 by the Commission based 
on  best  practices  from  agencies  and  parent  Directorates-General.  This  framework  provides 
structure and direction, while offering sufficient flexibility to cater  for and adjust to specific 
policymaking  needs.  By  identifying  the  high-level  policy  priorities  and  needs  of  parent 
Directorates-General in  advance, feedback to policy activities are implemented following  an 
identified  plan.  The  need  for  ad  hoc  requests  is  thereby  reduced,  although  such  requests  are 
allowed and addressed by the Agency to fulfil specific policy needs of the parent Directorates-
General not provided for in the plan. 

Overall, ensuring efficient and operational feedback to policy remains a priority. To be effective, 
channels of communication between projects and policy must allow information to flow freely 
in both directions. Best practice in this area could be achieved in several ways, for example, 
through participation by Commission policy officers in project kick-off and review meetings 
and ad hoc clustering events. 

19 

 
 
 
 
 
 
 
 
 
ANNEX I: PROCEDURAL INFORMATION  

The evaluation had a clearly defined scope: 

•  The reference period of the evaluation was 16 July 2018 – 31 March 2021. 

•  The scope of the evaluation  covered REA’s mandate, which consisted of  programme 

implementation and administrative and providing logistical support services.  

With regard to programme implementation, REA managed the parts of Horizon 2020 
that had been delegated to the Agency since 2014 and legacy, namely ongoing projects, 
under the Seventh Framework Programme.  

In terms of support services, REA’s main tasks included planning and support for the 
publication  of  calls  for  proposal  and  contests  for  prizes;  logistical  support  for  the 
evaluation  of  proposals;  contracting  and  payment  of  independent  expert  evaluators; 
validating the legal status and financial capacity of participants within the framework of 
the Single Electronic Data Interchange Area, management of Research Enquiry Service.  

•  The current generation of agencies established under 2021-2027 multiannual financial 

framework was outside the scope of this evaluation. 

The evaluation of REA was part of the evaluation of all the six agencies, which was made in a 
coordinated  manner  based  on  a  supporting  study  carried  out  by  an  external  contractor.  The 
preparations for the evaluation started in 2022, when an interservice group was set up by all lead 
parent Directorates-General and the concerned central services (Directorate-General for Budget 
and the Secretariat-General). Six group meetings were chaired by the Common Implementation 
Centre, set up within the Directorate-General for Research and Innovation, who also ensured 
the secretariat. The evaluation of each agency was led by the respective lead parent Directorate-
General (PLAN/2022/1912). The Commission also launched a ‘call for evidence’ on 10 March 
2023 on its ‘Have your say’ portal. The call was open for feedback on agencies activities until 
7 April 2023.  

The methodology used for the evaluation was consistent across the agencies and in accordance 
with the criteria provided in the Commission’s Better Regulation guidelines. The assessment of 
the evaluation criteria ‘EU added value’, i.e. why the EU should act, was not considered relevant 
as each agency carries out tasks which the Commission transferred to it. The EU added value of 
the  programmes  that  each  agency  manages  is  assessed  in  the  context  of  the  programme 
evaluations. 

Evidence was taken from sources such as the Commission databases, annual reporting exercises, 
adopted decisions. The supporting study was prepared using a mixed-methods approach at the 
levels  of  methodologies  and  methods  and  worked  on  a  qualitative  methodological  approach 
(based on documentary review and desk  research, interviews, answers to  open-ended survey 
questions  and qualitative  cost-benefit analysis)  combined with  a quantitative methodological 
approach (based on administrative and monitoring data, surveys and quantitative cost-benefit 
analysis), as detailed in Annex II. 

Knowledge from this evaluation will inform on possible improvements in the implementation 
by the agencies of the programmes of the Commission.  

Furthermore, the results of this evaluation will be useful for accountability purposes. The final 
evaluation reports on the performance of the six agencies will allow the Commission to report 

20 

 
 
 
 
 
 
 
 
 
the results of the retrospective cost-benefit analysis to the budgetary authority while informing 
various European Union (EU) institutions on the value of the cost-savings achieved as a result 
of  the  executive  agency  scenario  compared  with  the  costs  of  in-house  management  by  the 
Commission,  as  well  as  identifying  potential  areas  for  improvement.  The  results  of  this 
evaluation will be communicated to the Steering Committee of the Agency, to the European 
Parliament, to the Council, and the European Court of Auditors. 

The Agency had six parent Directorates-General, namely: 

•  Lead Parent Directorate-General: Directorate-General for Research and Innovation  
•  Parent Directorates-General:  

-  Directorate-General for Education and Culture 
-  Directorate-General for Communication Networks, Content and Technology 
-  Directorate-General for Agriculture and Rural Development 
-  Directorate-General for Migration and Home Affairs 
-  Directorate-General for Defence, Industry and Space18 

18   Directorate-General for Defence, Industry and Space replaced Directorate-General for Internal Market, Industry, Entrepreneurship and 

SMEs as from 01 January 2020 

21 

 
 
 
 
 
 
 
 
 
 
 
ANNEX II: METHODOLOGY AND ANALYTICAL MODELS USED 

The evaluation was supported by a study carried out by an external contractor in accordance 
with the criteria provided in the Commission’s Better Regulation guidelines. The methodology 
for the evaluation of REA was consistent with the approaches employed for the evaluations of 
the other agencies. 

The evaluation followed an evidence-based and mixed-methods approach, using qualitative and 
quantitative data to answer evaluation questions. Triangulation of data sources strengthened the 
validity of the findings. The mixed-method approach was used based on the nature of evaluation 
questions and issues. The study was structured around a series of evaluation questions outlined 
in the Annex to the study19. 

The evaluation relied on the evidence gathered and analysed using qualitative and quantitative 
methods and included: 

•  Extensive documentary review and desk research concerning all relevant documentation 
relating to REA’s legal base, its relation with the parent Directorates-General, and data 
on  its  financial  and  non-financial  performance,  including  the  memorandum  of 
understanding, the annual work programmes, the annual activity reports, the European 
Court of Auditors (ECA) and Internal Audit Service (IAS) audit reports, the previous 
evaluations  and  cost-benefit  analysis;  the  results  of  2018  and  2021  staff  satisfaction 
survey.  As  part  of  the  desk  research,  an  in-depth  analysis  was  conducted  on 
communication and feedback to policy.  

• 

Interview program: 40 interviews in total with 51 interviewees, representing the main 
stakeholders, structured in three stages:  

-  The scoping  phase consisted of exploratory and scoping  interviews on REA’s 
latest  and  most  significant  developments  during  the  reference  period  (one 
Commission staff and one REA staff) 

-  The  main  phase  consisted  of  semi-structured  interviews,  having  each  tailored 

questionnaire to each target group (Commission staff, REA staff) 

-  The follow-up phase consisted of in-depth interviews with senior management 
in Commission and REA, beneficiaries, unsuccessful applicants, and experts. 

•  Survey programme: Two surveys were conducted as part of the evaluation:  

-  Survey A targeted REA beneficiaries and unsuccessful applicants who applied 
for  funding  in  the  reference  period.  702  responses  from  beneficiaries  and 
respectively  875  responses  from  unsuccessful  applicants  were  received.  This 
represents  a  response  rate  of  14%  for  beneficiaries,  and  9%  for  unsuccessful 
applicants, making an average response rate of 10% for Survey A  

19  

https://op.europa.eu/en/publication-detail/-/publication/c9645710-e5bd-11ee-8b2b-01aa75ed71a1/language-en/format-PDF/source-
310469329 

22 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
-  Survey B focused on external experts who worked with REA in the reference 
period. A total of 469 responses were received, representing a response rate of 
33%. 

All  eligible  respondents  were  approached  for  the  beneficiary  and  expert  surveys. 
Unsuccessful  applicants  were  initially  sampled  for  inclusion  at  5  000  potential 
respondents and supplemented later with another 5 000 potential respondents.  

•  Cost-benefit  analysis  included  a  workload  analysis  and  the  assessment  of  the  cost-
effectiveness of the delegation of programme implementation, by comparing the actual 
indicators with the estimates in the 2013 ex-ante cost-benefit analysis and respectively 
the specific financial statement. 

Limitations and robustness of the findings 

All relevant stakeholders were consulted via interviews and surveys (Commission staff, REA 
staff, beneficiaries, unsuccessful applicants, experts). Unsuccessful applicants participated less 
actively compared with  beneficiaries. There was a rather low response rate in the survey for 
beneficiaries and unsuccessful applicants (i.e. 10% on average), which could be due to survey 
fatigue. The absolute majority of respondents were associated with Marie Skłodowska-Curie 
actions as it represented 79% of the overall portfolio of grants of REA. 

No sampling bias was observed as the profile of the respondents to the surveys was very similar 
to the overall population, guaranteeing statistical representativeness. The non-response bias (not 
all characteristics of the group that did not reply had been captured in full) was mitigated through 
triangulation with the results of follow-up interviews. 

The  triangulation  approach,  using  multi-level  and  multi-stakeholder  dimension  in  the  data 
collection, ensured the robustness and reliability of the data and information used to draw up 
conclusions  in  the  supporting  study. It  used  diverse  data  sources,  including  documentary 
reviews, desk research, surveys, and interviews, ensuring a comprehensive assessment of REA's 
operations.  

23 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
ANNEX  III:  EVALUATION  MATRIX  AND  ANSWERS  TO  THE  EVALUATION 
QUESTIONS (BY CRITERION)  

The purpose of the evaluation was to respond to evaluation questions relating to the criteria of 
effectiveness,  efficiency,  coherence,  as  defined  by  the  Commission’s  Better  Regulation 
guidelines. Evidence-based answers, including  the  points  of comparison  and sources, can be 
found in Section 4 of the study supporting the evaluation20. 

20  

https://op.europa.eu/en/publication-detail/-/publication/c9645710-e5bd-11ee-8b2b-01aa75ed71a1/language-en/format-PDF/source-
310469329  

24 

 
 
 
 
 
 
 
 
 
 
ANNEX IV: OVERVIEW OF BENEFITS AND COSTS  

The benefits and costs are detailed in the Cost Benefit Analysis in section 4.2 and  Annex V 
below. 

25 

 
 
 
 
 
 
 
 
 
 
ANNEX V: COST-BENEFIT ANALYSIS 

A  cost-benefit  analysis  was  performed  to  assess  whether  the  conclusions  of  the  ex  ante 
assessments (the estimates of savings provided in the specific financial statement, as amended 
following  the  extension  of  REA’s  mandate  in  2018)  are  still  valid  when  compared  with  the 
actual situation, and what the overall actual savings are.  

The following approach was adopted: 

1.  To use data on the actual performance of REA (actual execution of the administrative 

budget, actual staffing, etc.) for the executive agency scenario. 

2.  To follow the assumptions laid down in the specific financial statement to ensure the 
comparability and validity of results, and to provide estimates of the comparable ‘actual’ 
in-house (Commission) scenario (the comparator), which would best reflect the actual 
situation. 

3.  To assess based on these estimates whether the conclusions of the ex ante assessments 
provided in the specific financial statement are still valid when compared with the actual 
situation, and what the overall possible savings are. 

To deconstruct the ‘actual’ in-house scenario (the ‘comparator’), estimates were based on the 
following assumptions: 

•  Number  and  composition  of  staff  in  the  Commission  under  the  in-house  scenario 
correspond to the estimates21 in specific financial statement. Additional contract staff 
were added to the estimated Commission staff number for 2018-2020 (31 contract staff 
in  2018, 34  contract staff  in  2019  and 37  contract  staff in  2020) to  reflect  additional 
authorised  staff  in  REA  to  manage  the  additional  operational  budget  from  the 
contributions  of  EEA/EFTA  and  the  participation  of  candidate  countries  and/or  third 
countries  that  was  not  covered  in  the  resource  calculations  in  the  specific  financial 
statement, thereby ensuring comparability of the ‘in-house’ and the ‘executive agency’ 
scenarios.  

•  Commission  staff  costs  and  overheads  correspond  to  the  average  costs  set  by 
Directorate-General for Budget for the estimates of human resources and overheads in 
legislative financial statements for the respective year.  

•  Programme support expenditure (Title  III)  remains the same under both  the in-house 

scenario and the executive agency scenario. 

The table below presents the results of the analysis of the estimated actual costs of the in-house 
scenario and the actual costs of the executive agency scenario. 

21   Additional resources allocated under amendment of the Specific Financial Statement are also considered. 

26 

 
 
 
 
 
 
 
 
 
 
 
 
Estimated actual costs of the in-house scenario and the actual costs of the executive agency 
scenario, EUR 

2018 

2019 

2020 

Total  

No. 

Cost 

No. 

Cost 

No. 

Cost 

2018-2020 

In-house scenario  

Commission 

Title 

I. 

Staff-related 

847.6  81 827 300 

866.6  86 449 100  910.6 

92 895 000 

261 171 400 

expenditure 

Temporary Agents 

571.7  68 032 300 

582.9  71 696 700  611.6 

76 450 000 

216 179 000 

Contract Agents 

275.9  13 795 000 

283.7  14 752 400  299.0 

16 445 000 

44 992 400 

Title  II.  Infrastructure 

   20 342 400 

   21 665 000 

22 765 000 

64 772 400 

and 

operating 

expenditure 

Title 

III.  Programme 

7 895 694 

7 118 710 

7 137 528 

22 151 932 

support expenditure 

TOTAL COST 

847.6  110065394 

866.6  115232810  910.6  122 797 528 

348 095 732 

Executive agency 

scenario  

REA 

Title 

I. 

Staff-related 

736  51 823 438 

753  57 142 945 

781 

60 490 293 

169 456 676 

expenditure 

Temporary Agents 

175  19 602 919 

182  21 733 801 

185 

22 900 071 

64 236 791 

Contract Agents 

561  28 605 176 

571  31 933 551 

596 

34 258 373 

94 797 100 

Interim  supportive  agents 

944 870 

858 668 

567 461 

2 370,999 

and trainees 

Professional  development 

2 670 473 

2 616 925 

2 764 388 

8 051 785 

and recruitment costs 

Title  II.  Infrastructure 

and 

operating 

expenditure 

9 841 834 

   11 504 336 

11 717 096 

33 063 265 

Title 

III.  Programme 

7 895 694 

7 118 710 

7 137 528 

22 151 932 

support expenditure 

Total REA cost: 

   69 560 966 

   75 765 991 

79 344 917 

224 671 873 

Commission 

Title 

I. 

Staff-related 

13.5 

1 578 900 

13.3 

1 628 800 

13.2 

1 650 000 

4 857 700 

expenditure 

27 

 
 
 
 
 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
 
2018 

2019 

2020 

Total  

No. 

Cost 

No. 

Cost 

No. 

Cost 

2018-2020 

Temporary Agents 

13.1 

1 558 900 

13.2 

1 623 600 

13.2 

1 650 000 

4 832 500 

Contract Agents 

0.4 

20 000 

0.1 

5 200 

0.0 

0 

25 200 

Title  II.  Infrastructure 

and 

operating 

expenditure 

324 000 

332 500 

330 000 

986 500 

Total Commission cost: 

1 902 900 

1 961 300 

1 980 000 

5 844 200 

TOTAL COST 

749.5  71 463 866 

766.3  77 727 291  794.2 

81 324 917 

230 516 073 

ESTIMATED SAVINGS 

98  38 601 528 

100  37 505 520 

116 

41 472 611 

117 579 659 

Source: Study supporting the triennial evaluation of the Agency 

The analysis concluded that: 
The overall actual costs of the executive agency scenario22 amounted EUR 230.5 million during 
the period 2018-2020. The estimate in the specific financial statement (EUR 218.5 million for 
the period 2018-2020) was based on the EU contribution, but REA’s administrative budget also 
included EEA/EFTA and third country contributions (EUR 13.8 million during the period 2018-
2020).  Consequently,  based  on  the  EU  contribution  only,  the  actual  costs  of  the  executive 
agency  scenario  amounted  to  EUR  216.7  million,  which  means  that  the  actual  costs 
corresponded to the initial estimations (with savings of EUR 1.7 million, or 0.8% of the estimate 
in the specific financial statement). This was achieved despite rising costs and the fact that the 
estimates in the specific financial statement were not adjusted for inflation. 

The costs of the executive agency scenario were much lower than the estimated costs of the in-
house scenario. In 2018-2020, the actual cost-savings deriving from a difference in cost between 
the executive agency scenario and the in-house scenario amounted to EUR 117.6 million (or 
33.8% of the estimated costs under the in-house scenario). 

When  comparing  the  savings  initially  estimated  in  the  specific  financial  statement  with  the 
actual savings from the delegation of tasks to REA, the actual savings during the 2018-2020 
period were 34.6% higher than the initial estimate in the specific financial statement (EUR 117.6 
million, compared with EUR 87.4 million). As forecast in the specific financial statement, the 
savings of the executive agency scenario resulted primarily from a higher share of lower-cost 
external personnel being employed within the Agency, and a lower overall number of staff23. 

22  

Including the cost of coordination and monitoring by the Commission and the costs of REA covered from EEA/EFTA and third-country 
contributions. 

23   The number of staff was reduced in the executive agency scenario; however, it was not modified in the in-house scenario in the specific 

financial statement. 

28 

 
 
 
 
 
  
  
  
  
  
  
  
 
 
 
 
 
 
 
ANNEX VI: STAKEHOLDERS’ CONSULTATION – SYNOPSIS REPORT  

This annex summarises the stakeholders’ consultation conducted for the evaluation, in line with 
the  Commission’s  Better  Regulation  guidelines.  A  similar  approach  was  adopted  for  the 
evaluation of all six agencies.  

1.  Outline of the consultation strategy 

The consultation strategy was designed as follows: 

 Source: Study supporting the triennial evaluation of the Agency 

REA’s stakeholders are grouped as follows:  

- 
- 
- 
- 

staff of the Agency; 
staff from the five parent Directorates-General;  
applicants (successful and unsuccessful) to the funding programmes that REA manages;  
experts that REA hires to assess funding proposals. 

Applicants and experts were surveyed and all four groups of stakeholders were interviewed.  

Consultation activities and target groups covered 

Consultation 
activity 

Surveys 

Survey 
A 

Survey 
B 

Interview 
programme 

Target groups 

Date 

Beneficiaries and unsuccessful applicants of the programmes 
managed by REA 

29 November – 16 
December 2022 

External experts contracted by REA 

•  Commission officials from parent Directorates-General 
•  REA senior managers 
•  REA heads and representatives of units 
•  Representatives of the REA Staff Committee 
•  Beneficiaries  of  and  unsuccessful  applicants  to  the 

programmes managed by REA 
•  External experts contracted by REA 

24 November – 7 
December 2022 

23 August 2022 –
March 2023 

Source: Study supporting the triennial evaluation of the Agency 

These  stakeholder  consultations  complemented  each  other  and  informed  the  answers  to  the 
evaluation  questions  on  effectiveness  and  efficiency.  Interviews  with  Commission  and  REA 
staff  provided  information  both  on  these  questions  and  on  coherence-related  evaluation 
questions. 

29 

 
 
 
 
 
 
 
 
 
Consultation activity and evaluation criteria covered 

Effectiveness 

Efficiency 

Coherence 

Interviews with the European 
Commission 

Interviews with REA 

Interviews with applicants and experts 

Survey of REA’s applicants 

Survey of REA’s expert evaluators 

✓ 

✓ 

✓ 

✓ 

✓ 

Source: Study supporting the triennial evaluation of the Agency 

2.  Key results of the consultation activities 

2.1 Summarised results of the survey 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

Two  surveys  were  carried  out:  Survey  A  for  applicants  (beneficiaries  and  unsuccessful 
applicants) and Survey B for experts. The survey questions aimed to ensure complementarity 
with the interview programme. Survey A was active for almost 3 weeks and Survey B for 2 
weeks.  

Survey A addressed all applicants from the calls for proposals launched under the 2017-2020 
annual  work  programmes.  702  responses  were  received  from  beneficiaries,  representing  a 
response rate of 14%. 875 responses were received from unsuccessful applicants, representing 
a response rate of 9%. The overall response rate was 10% for Survey A.  

Survey  B  targeted  all  external  experts  contracted  by  REA  between  2017  and  2020.  3 694 
responses were received (a response rate of 33%). 

Main statistics related to the implementation of the surveys 

Target group 

Full launch 
of the survey 
(most 
invitations 
sent on this 
date) 

Survey 
closure 
date 

Population 
targeted/ 
no. of 
invitations 
sent  

No. of responses 
received 

Response 
rate* 

Beneficiaries 

29 
November 
2022 

16 
December 
2022 

5 111 

Partial*: 153 
Complete: 549 
Total: 702 

Unsuccessful 
applicants 

29 
November 
2022 

16 
December 
2022 

Partial*: 251 

10 500 

Complete: 624 

Total: 875 

Experts 

24 
November 
2022 

7 
December 
2022 

11 610 

Partial*: 469 

Complete: 3 225 
Total: 3 694 

Total: 14% 
Only 
including 
complete: 11 
% 
Total: 9% 
Only 
including 
complete: 6% 
Total: 33% 
Only 
including 
complete: 
28% 

No. of 
responses 
included in the 
analysis after 
cleaning 

Partial*: 118 

Complete: 549 
Total: 667 

Partial*: 167 

Complete: 624 
Total: 791 

Partial*: 258 

Complete: 3 

221 
Total: 3 479 

Note: The response rate was calculated based on the formula: 𝑅𝑒𝑠𝑝𝑜𝑛𝑠𝑒 𝑟𝑎𝑡𝑒 =

𝑁𝑜.𝑜𝑓 𝑟𝑒𝑠𝑝𝑜𝑛𝑠𝑒𝑠 𝑟𝑒𝑐𝑒𝑖𝑣𝑒𝑑

𝑁𝑜.𝑜𝑓 𝑖𝑛𝑣𝑖𝑡𝑎𝑡𝑖𝑜𝑛𝑠 𝑠𝑒𝑛𝑡 𝑜𝑢𝑡

∗ 100 % 

Source: Study supporting the triennial evaluation of the Agency 

30 

 
 
 
 
 
 
 
 
 
 
 
 
Effectiveness  

86%  of  beneficiaries  (456  of  529)  and  95%  of  experts  (3 007  of  3 173)  were  satisfied  with 
REA’s services during the evaluation period. Unsuccessful applicants were less satisfied (57%, 
or 287 of 498), which is similar to the last evaluation period (55%). All three stakeholder groups 
surveyed indicated a high level of willingness to work with REA again: namely 94% (512 of 
543) of beneficiaries, 83% (516 of 620) of unsuccessful applicants and 98% (3 138 of 3 201) of 
experts. Those experts who were not willing to work with REA again indicated the imbalance 
between the total remuneration received and the amount of time spent on evaluation tasks (63%, 
or  40  of  63)  as  their  main  reason,  followed  by  low  remuneration  (35%,  or  22  of  63)  and 
personal/work reasons (32%, or 20 of 63). 

The  effectiveness  of  the  Agency’s  evaluation  and  selection  process  for  applications  was 
assessed  differently  depending  on  the  groups  surveyed.  Unsuccessful  applicants  expressed 
dissatisfaction  with  the  feedback  given  in  the  free  responses.  Only  48%  (324  of  668)  of 
unsuccessful applicants felt satisfied with the transparency of the evaluation process, while 84% 
of beneficiaries rated the evaluation process itself positively (504 of 597). 

As regards the way in which applicants learned about funding opportunities, the surveys showed 
that they relied on their own organisation or institution (24%, or 334 of 1 373, of beneficiaries 
and 52%, or 381 of 729, of unsuccessful applicants) and on the Europa website (21%, or 283 of 
1 373  of  beneficiaries,  and  43%,  or  311  of  729,  of  unsuccessful  applicants).  National 
agency/contact points were also a source of information for both beneficiaries (12%, or 161 of 
1 373) and unsuccessful applicants (19%, 138 of 729). In contrast, REA social media was used 
the least frequently (16 beneficiaries and 10 unsuccessful applicants, or 1% of each group). 

Experts learned about the opportunity to work with REA via direct invitation (41%, or 1 434 of 
3 479),  colleagues  at  their  institution  (28%,  or  933  of  3 479)  and/or  participation  in  another 
Commission/Agency project (26%, or 848 of 3 479). Info days and other EU promotional events 
were rarely cited by experts as the way they learned about the opportunity to work with REA 
(2%, or 79 of 3 479). REA social media and magazines were the least used channel.  

On the perception of the EU as the promoter of programmes entrusted to REA,  98% (612 of 
625) of beneficiaries agreed or strongly agreed that they were aware that the grant and tender 
funding came from the EU budget. However, significantly fewer (67%, or 413 of 616) agreed 
or strongly agreed that they were aware that REA was entrusted to manage the programme by 
the Commission. 

Communication activities were rated positively by both experts and applicants. 95% (2 721 of 
2 865)  of  experts  agreed  or  strongly  agreed  that  REA  staff  were  responsive  and  provided 
answers to their questions. 86% (413 of 481) of beneficiaries agreed that the REA staff assigned 
to  their  project  were  responsive  and  available.  59%  (990  of  1 678)  of  experts  consulted  the 
online information on the Funding & Tenders Portal (formerly the Participants Portal) before or 
during the registration process. Of the information provided there, the Horizon 2020 manual 
was rated the most helpful, with 91% (840 of 922) of experts agreeing or strongly agreeing that 
it was helpful. On the applicants’ side, 89% (538 of 608) of beneficiaries and 83% (557 of 675) 
of unsuccessful applicants agreed or strongly agreed that information for applicants was easy to 
find. There was greater disparity between unsuccessful applicants and beneficiaries as regards 
the  process,  with  83%  (504  of  509)  of  beneficiaries  agreeing  or  strongly  agreeing  that  the 
requirements were reasonable and proportionate compared to 69% (464 of 672) of unsuccessful 
applicants.  

31 

 
 
 
 
 
 
 
 
 
 
86% (113 of 131) of applicants who used the Research Enquiry Service found it helpful or very 
helpful.  The  validation  service  was  also  rated  positively,  with  81%  (45  of  56)  of  applicants 
agreeing or strongly agreeing that the electronic tools for validation were user-friendly. Overall, 
the level of satisfaction with the clarity of the application process was 92% among beneficiaries 
(558 of 605 agreed or strongly agreed that the application process was clear and transparent) 
and 77% (516 of 669) among unsuccessful applicants.  

Efficiency 

The help desk service was mostly rated positively, with 79% (544 of 692) of experts finding it 
useful or very useful. The FAQ, glossary and reference documents were all rated similarly, with 
88% (757 of 895), 80% (643 of 798) and 89% (817 of 925) of experts finding them useful or 
very  useful,  respectively.  As  regards  the  evaluation  of  proposals,  93%  (2 987  of  3 196)  of 
experts  agreed  or  strongly  agreed  that  the  guidelines  were  clear.  However,  this  number  was 
slightly lower (88%, or 2 447 of 2 818) when they were asked if they agreed that discussions 
between REA and experts created a common understanding of proposal assessment.  

The time taken to process payment requests was generally rated very positively by experts and 
applicants alike. 94% (381 of 404) of beneficiaries agreed or strongly agreed that the time taken 
by  REA  to  make  a  pre-payment  was  appropriate.  Satisfaction  levels  were  slightly  lower  for 
interim and final payments (91%, or 248 of 272, and 89%, or 80 of 90). As regards the experts, 
94% (2 996 of 3 161) agreed or strongly agreed that the time taken by REA to make payments 
was satisfactory. Reimbursement for in-person evaluation trips to Brussels was also rated very 
highly, with 95% (856 of 892) of experts agreeing or strongly agreeing that the process was 
straightforward.  

Remote  assessment  of  proposals  was  highly  rated  by  89%  of  experts  (2 831  of  3 172),  who 
agreed or strongly agreed that this process was efficient. However, most respondents (72%, or 
2 302 of 3 207) had only experienced remote evaluation and not face-to-face evaluation, and 
consequently did not have a frame of reference for comparison. Free responses revealed that 
some  experts  who  had  experienced  both  face-to-face  and  remote  evaluation  considered  the 
remote version to be less efficient.  

Beneficiaries were generally satisfied with reporting requirements. 82% (421 of 512) agreed or 
strongly  agreed  that  project  reporting  requirements  were  reasonable  given  the  scale  of  the 
project, and 88% (420 of 478) agreed or strongly agreed that reporting requirements remained 
stable over the course of the project. The clarity of reporting requirements was rated slightly 
lower,  with  only  78%  (392  of  499)  of  beneficiaries  agreeing  or  strongly  agreeing  that 
instructions  given  on  reporting  requirements  were  clear  and  user-friendly.  As  regards 
monitoring,  results  were  similar,  with  79%  (407  of  514)  of  beneficiaries  agreeing  that  the 
monitoring process  was clear. Overall, 85% (442 of 520) of beneficiaries agreed or strongly 
agreed that overall project implementation was straightforward. 

Beneficiaries indicated satisfaction with the time taken to evaluate their proposals, with 88% 
(485 of 549) agreeing or strongly agreeing that the time taken was appropriate. Time to grant 
was rated similarly, at 87% (477 of 550). However, only 76% (2 434 of 3 194) of experts agreed 
or strongly agreed that they had enough time to review proposals. This was reiterated by experts 
in their free responses.  

32 

 
 
 
 
 
 
 
 
 
 
 
 
 
Regarding  experts’  payments,  only  68%  (2 184  of  3 189)  agreed  or  strongly  agreed  that  the 
payment they received matched the effort spent on their task. Low levels of remuneration were 
mentioned over 160 times (out of 3 479 responses) in the experts’ free responses, which focused 
on three main problems with the rate: first, the effect of inflation was said to have eroded the 
real value of the remuneration over the last decade. Second, many experts mentioned that the 
effort required to evaluate a proposal well did not match the hours stated in the contract, and 
therefore, when the actual number of hours worked was taken into account, the total payment 
received was inadequate. Experts suggested that they spent between 50 and 100% more time 
evaluating proposals than was specified in the contract. This concern will be less relevant from 
2024, when new contracts will no longer mention time but only the monetary value of each task. 
Third, differences in the cost of living and tax requirements led to different perceptions of the 
value of the remuneration received. 

IT tools were the area rated by applicants as having the greatest scope for further simplification, 
with over 90% (448 of 495) of beneficiaries agreeing or strongly agreeing that IT tools could be 
further  improved.  However,  72%  (416  of  580)  of  beneficiaries  and  70%  (446  of  635)  of 
unsuccessful  applicants  also  agreed  or  strongly  agreed  that  they  found  the  IT  tool  for  the 
application process to be user-friendly.  

2.2  Summarised results of the interview programme 

The  interview  programme  was  designed  to  embrace  the  views  of  all  key  stakeholders 
(Commission,  Agency,  beneficiaries,  unsuccessful  applicants  and  external  experts).  It  was 
designed in three stages: scoping, main phase and follow-up interviews, as described in Annex 
II above. 

The main-phase interviews were conducted using a standardised questionnaire, which included 
questions  on  the  evaluation  criteria  and  was  adjusted  to  the  experience  of  individual 
stakeholders.  Follow-up  interviews  were  conducted  based  on  questionnaires  tailored  to  the 
candidate’s area of expertise in the case of the lead-parent Directorate-General and REA, and 
on the responses to the free-text questions in the survey in the case of applicants and experts. 
A total of 51 interviews was conducted, summarised below: 

-  18 interviews (23 interviewees) with representatives of the parent Directorates-General 
-  15 interviews (21 interviewees) with representatives of the Agency 
-  2 interviews (2 interviewees) with beneficiaries 
-  2 interviews (2 interviewees) with unsuccessful applicants 
-  3 interviews (3 interviewees) with experts. 

There was an overlap between experts and applicants. Two of the three experts had also been 
applicants and two of the four applicants had also been experts. A summary of the findings from 
the interview data are presented below.  

Effectiveness 

Overall, the key stakeholders interviewed considered that REA was highly effective in all its 
duties.  

Effective  project  implementation  was  considered  to  be  one  of  the  REA’s  strengths  by  the 
stakeholders interviewed. The Commission and REA staff were impressed by the number and 
quality  of  projects  implemented  during  the  previous  framework  programme,  as  well  as  the 

33 

 
 
 
 
 
 
 
 
 
transition to the new framework. There was no common view as regards the size of projects: 
Commission staff suggested that smaller projects would better suit their needs, while REA staff 
suggested  that  it  would  be  more  efficient  to  implement  larger  projects.  On  the  client  side, 
beneficiaries who had worked on multiple projects stated that effectiveness  depended on the 
individual project officer assigned to the project.  

As regards how the experts learned about the opportunity to work with the REA, all three experts 
interviewed had been invited directly. They suggested that the REA use universities or Info days 
for recruitment. Considering that these methods are already used, and that a limited number of 
experts  cited  universities  (28%)  and  Info  days  (2%)  in  the  survey,  there  is  potential  for 
broadening recruitment reach in this way. 

Within the evaluation process, remote evaluation was raised in the experts’ free responses. In 
this respect, two experts interviewed expressed difficulty with the blog interface specifically, 
saying that it was hard to keep track of new comments. Another expert interviewed expressed a 
sense of distance from fellow evaluators, which led to difficulty forming a consensus with them. 
It  was  noted  that,  whereas  in-person  consensus  meetings  offered  experts  an  opportunity  to 
network with researchers around Europe, remote consensus offered no such benefit, possibly 
reducing  the  incentive  for  some  to  become  experts.  It  was  suggested  that  video  meetings 
between experts and less anonymity in the profiles would help experts to understand one another 
better and thus increase both the quality of the evaluations and the added value for experts.  

As regards feedback on the applications and transparency, applicants who had also worked as 
experts felt satisfied with the transparency of the evaluation process. 

Applicants and experts interviewed found REA staff very cooperative.  

Commission and REA staff indicated mixed feelings about the formal and informal channels 
for  communication  between  them,  although  many  staff  (10)  expressed  high  levels  of 
satisfaction.  Job  shadowing  was  considered  a  successful  way  of  exchanging  knowledge  and 
should  be  further  used.  Fewer  interviewees  (7)  expressed  the  need  for  greater  flexibility  in 
informal communication channels. 

Efficiency 

The  Commission  staff  interviewed  were  unanimously  positive  about  REA’s  efficiency  in  its 
programme  management  duties.  REA  staff’s  enthusiasm  for  taking  on  new  challenges,  their 
flexibility and active communication were listed among the qualities that contributed to REA’s 
particularly high levels of efficiency. 

As confirmed by the survey, the applicants interviewed (4) were generally positive about the 
application procedure; however, all of them experienced difficulties when applying. Two relied 
heavily on outside guidance to complete the application process. Of those two, one mentioned 
that  consortia  able  to  hire  external  consultants  to  help  with  the  application  would  have  an 
advantage  due  to  the  complexity  of  the  process.  A  third  interviewee  mentioned  that  the 
complexity of the application process increased with the number of partners involved. The last 
interviewee also mentioned that adding new partners to the proposal could take time and might 
be an area for further simplification.  

34 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
Reporting requirements and deadlines were rated positively by the beneficiaries interviewed. 

As confirmed by the survey data, applicants  and experts  interviewed  were satisfied  with  the 
efficiency of the payment process. The interviewees corroborated the survey findings on some 
experts’ dissatisfaction with the amount of remuneration in comparison to the workload, which 
was perceived as higher than specified in the contract. One expert suggested that the workload 
was approximately double; this is reiterated by the free responses in the survey. One beneficiary 
who also worked as an expert (but had not taken the expert survey) confirmed that the workload 
was  much  higher  than  that  specified  in  the  contract.  Apart  from  improvements  to  IT  tools 
(discussed  below),  experts  did  not  see  any  scope  to  reduce  the  time  required  to  evaluate  an 
individual proposal, but instead suggested that the contracted hours be reassessed. 

Different IT tools are used by different stakeholders; therefore, perspectives from each group 
differ. REA staff were positive about the upgrades made or to be made. The speed at which IT 
tools were upgraded was noted as the main limitation on efficiency by both Commission and 
REA  staff.  Some  interviewees  noted  that  maintenance  focused  primarily  on  bug  fixes  rather 
than overall improvements to the system. Some staff also felt that the outward-facing tools were 
not as client-centric as they could be, a sentiment reflected by the clients themselves. 

In contrast, many applicants and experts expressed difficulty with IT tools. This corresponds to 
the survey finding that 90% of applicants believed that the IT tools could be further simplified. 
One beneficiary required a project officer to guide them through the entire process because it 
would otherwise have been too overwhelming. Other applicants said that it took time to acquaint 
themselves with the interface and that it was not intuitive. One beneficiary suggested the tool 
could be a barrier to new researchers, who may not have the time to learn to use the application 
tool.  Some  experts  found  that  the  remote  evaluation  tool  hampered  their  efficiency.  It  was 
difficult  to  find  the  newest  comments  on  the  blogs  and  notifications  of  new  activity  were 
difficult to track.  

Improvements suggested for IT tools: 

•  Both  applicants  and  experts  suggested  that  messages  in  their  IT  interfaces  could  be 

ordered from newest at the top to oldest at the bottom for ease of use. 

•  One applicant who had also worked as an expert suggested integrating the expert profile 
database  into  the  ORCHID  researcher  database24  to  reduce  the  time  needed  for  new 
experts to create their profiles. 

•  One interviewee mentioned that when reapplying, applicants had to re-enter all of the 
partners’  individual  details,  and  that  it  would  be  helpful  if  that  information  could  be 
automatically saved from the last application. Another interviewee also cited the initial 
part of the application, when applicants entered all the partners’ details, as a potential 
area for further streamlining. 

24   The ORCHID researcher database, also known as ORCID, is a unique identifier system that provides researchers with a persistent digital 

ID to distinguish their work and connect their contributions across institutions and publications. 

35 

 
 
 
 
 
 
 
 
 
 
 
 
 
Coherence  

Both REA and Commission staff interviewed were unanimously positive about the delineation 
of  responsibilities,  both  as  set  out  in  the  memorandum  of  understanding  and  as  understood 
through the day-to-day communications between them. REA’s ability to be flexible was cited 
by both Commission and REA staff as a factor enabling the Agency to function well in its role, 
and a feeling of mutual trust was often mentioned. 

Interviewees from the Commission praised REA staff’s proactive attitude, who tend to lead the 
way in developing improvements for the whole community.  

A  few  interviewees  from  the  Commission  mentioned  that  REA  sometimes  lacked  a  broad 
overview  of  relevant  issues,  particularly  for  feedback  to  policy  purposes  but  also  for  work 
programmes,  which  was  inevitable  due  to  the  nature  of  the  executive  agency  model.  Those 
Commission staff saw this as a challenge for them to tackle, rather than a failing on the part of 
REA. 

On horizonal services, REA does an excellent job of managing these tasks, and there were no 
complaints among the 44 REA and Commission  staff interviewed. However, questions were 
raised about the extent to which support services align with REA’s broader research-focused 
portfolio, as well as concern that an economy of scale may need to be re-evaluated in the future 
as programmes become larger. 

On feedback to policy, interviewees expressed a variety of reactions to the changes that feedback 
to policy underwent during the evaluation period. Some appreciated the increase in structure, 
while  others  found  that  they  missed  the  informal  nature  of  the  exchange.  Interviewees  were 
united in viewing policy as an ongoing process of improvement. 

2.3 Comparison of the results of consultation activities 

The table below presents the key results of each consultation activity, organised by evaluation 
criteria, as well as the level of consistency, complementarity and contradiction of results across 
consultation activities. Overall, the consultation activities were very consistent with one another 
and revealed many complementary insights.  

36 

 
 
 
 
 
 
 
 
 
 
Summary of key findings 

Evaluation 

criterion 

Survey 

Interview programme 

Consistency of 
results across 
consultation 
activities 

Complementarity of 
results across consultation 
activities 

Contradiction of 
results across 
consultation 
activities 

Effectiveness 

Applicants were very satisfied with 

All stakeholders were generally 

High 

High 

Low 

the  application  process.  Experts 

satisfied. There was a slight 

were  generally  satisfied.  Remote 

disagreement between REA and 

assessment of proposals was rated 

Commission staff about project 

highly  by  experts;  however,  some 

size. Some experts expressed 

experts who had experienced both 

dissatisfaction about the 

in-person  and  remote  evaluation 

effectiveness of remote evaluation. 

were 

less 

satisfied  with 

the 

Some experts felt that recruitment 

efficiency of the remote version.  

activities had further potential. 

Some external communication 

channels, for example Info days 

for expert recruitment, were not 

being used to the full extent. 

Efficiency 

Project reporting requirements, 

Programme management and 

High 

High 

Low 

payment processes and help 

reporting requirements were rated 

services were all rated highly. 

highly. Experts considered the 

IT tools, some aspects of remote 

workload to be higher than 

evaluations and the time taken to 

specified in the contract. IT tools 

carry out evaluations received less 

were rated less favourably by 

positive ratings. Some experts 

experts and applicants than by 

considered the remuneration to be 

internal stakeholders. 

too low compared to the effort. 

37 

 
 
 
 
 
Coherence 

NA 

Interviewees reported very high 

Medium 

High 

Medium 

satisfaction with REA’s cross-

agency leadership. The delineation 

of responsibilities was considered to 

be clear. There were mixed 

opinions about feedback to policy 

and the general broad overview. 

38