Evaluation of the operation of REA (2012-2015) : annexes.
Date: 2017-03-27
Institutions: Directorate-General for Research and Innovation
Subjects: EU programme, executive agency, legal status, management, research policy

EVALUATION  
OF THE OPERATION OF REA 
(2012-2015) 

Annexes 

 
 
  
 
 
 
 
 
 
 
 
 
 
EUROPEAN COMMISSION 

Directorate-General for Research and Innovation 
Directorate R — Resources 
Unit R4 — New Management Modes 

E-mail:  RTD-R4-evaluation@ec.europa.eu  

European Commission 
B-1049 Brussels 

2 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
EUROPEAN COMMISSION 

EVALUATION  
OF THE OPERATION OF REA 
(2012-2015) 

Annexes 

Prepared by: Public Policy and Management Institute 

Directorate-General for Research and Innovation 
2016

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
EUROPE DIRECT is a service to help you find answers  
to your questions about the European Union 
Freephone number (*): 
00 800 6 7 8 9 10 11 
(*) Certain mobile telephone operators do not allow access to 00 800 numbers  
or these calls may be billed 

LEGAL NOTICE 

Neither the European Commission nor any person acting on behalf of the Commission is  
responsible for the use which might be made of the following information. 

The views expressed in this publication are the sole responsibility of the author and do not  
necessarily reflect the views of the European Commission. 
More information on the European Union is available on the Internet (http://europa.eu). 

Luxembourg: Publications Office of the European Union, 2016. 

PDF  

ISBN 978-92-79-58797-9 

doi: 10.2777/679459 

KI-02-16-511-EN-N 

© European Union, 2016 
Reproduction is authorised provided the source is acknowledged. 

4 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
CONTENTS 

List of tables and figures ...................................................................................................................... 6 

Annexes ....................................................................................................................................... 8 

Annex 1. Detailed approach to key data collection and analysis methods ........................................ 8 

Documentary review and desk research ..................................................................................... 8 

Interview programme ............................................................................................................ 11 

Survey  12 

Cost-benefit analysis ............................................................................................................. 14 

Annex 2. REA interview questionnaire ...................................................................................... 16 

Annex 3. Questionnaires and aggregated data of REA beneficiaries surveys .................................. 30 

The questionnaire of the survey of REA grant beneficiaries: FP7 ............................................ 30 

The questionnaire of the survey of REA beneficiaries: Horizon 2020 ....................................... 38 

Aggregated data of REA beneficiaries surveys (FP7 and Horizon 2020) ................................... 46 

Annex 4. Questionnaires of REA independent experts surveys ..................................................... 55 

The questionnaire of the survey of REA independent experts: evaluators ............................... 55 

The questionnaire of the survey of REA independent experts: monitoring experts ................... 63 

5 

 
 
 
 
 
 
 
      
       
 
      
      
LIST OF TABLES AND FIGURES 

List of Tables 

Table 1. Interviews with REA staff and stakeholders conducted 17-21 August, 2-4 September and 16-18 
September. ............................................................................................................................... 11 
Table 2. Targeting and coverage of the survey programme ............................................................. 12 
Table 3. Summary of survey response and completion rates ........................................................... 14 
Table 4 Q3 Where did you first hear about the opportunity to become an independent expert of the 
Agency/Commission? ................................................................................................................. 71 
Table 5 Q4a Please assess the usefulness of the information provided online: ................................... 71 
Table 6 Have you sought for information or assistance from the Research Executive Agency (REA) during 
the registration process?............................................................................................................. 71 
Table 7 Q5a Please assess the quality of information/assistance provided by the REA: ....................... 71 
Table 8 Q7 Before you could be contracted as an expert, your identity and bank account details had to 
be verified (‘Legal Entity/Bank Account Validation’). This procedure is carried out separately from the 
registration process. Please assess the LE/BA validation procedure. ................................................. 72 
Table 9 Q9 Were you contacted with regard to your availability to work as an independent expert for a 
particular evaluation or project review before you received an expert contract to sign? ...................... 73 
Table 10 Q10. Please give your assessment of the following statements. .......................................... 73 
Table 11 Q12Experts are required to declare conflicts of interest (CoI), either at the time of signature of 
the contract or as soon as a CoI is identified. Please assess the process in relation to CoI. .................. 73 
Table 12 Q14 When an expert needs to travel for an assignment, travel costs are reimbursed to and 
from the expert's address as indicated in the contract (the ‘Point of Departure’ – POD). An expert who 
wishes to travel to and/or from a different address must request prior approval (‘request for change of 
point of departure’). Please assess the process in relation to requests to change the point of departure 
for assignments involving travel. ................................................................................................. 73 
Table 13 Q15 Have you sought information or assistance from the REA during the contracting process? 73 
Table 14 Q16 Regarding place of work for the assignment, did you have to carry out your tasks: ........ 74 
Table 15 Q16a Please assess the extent to which you are satisfied with the quality of REA evaluation 
facilities in Brussels: .................................................................................................................. 74 
Table 16 Q17 To what extent do you agree or disagree with the following statements concerning your 
work as an expert: ..................................................................................................................... 74 
Table 17 Q17 To what extent do you agree or disagree with the following statements concerning your 
work as an expert: ..................................................................................................................... 75 
Table 18 Q19 Did you visit the Reimbursement Help Desk during your evaluation in Brussels? ............ 76 
Table 19 Q19a Please assess the quality of information/assistance provided by the REA: .................... 76 
Table 20 Q18a Please assess the quality of information/assistance provided by the REA during the 
briefing: ................................................................................................................................... 76 
Table 21 Q19 Did you visit the Reimbursement Help Desk during your evaluation in Brussels? ............ 77 
Table 22 Q20 Did you use the REA-Evaluation@ec.europa.eu functional mailbox? .............................. 77 
Table 23 Q20a At what stage in the process did you use this service: ............................................... 78 
Table 24 Q20b Please assess the quality of information/assistance provided by the REA through the REA-
Evaluation@ec.europa.eu mailbox. ............................................................................................... 78 
Table 25 Q21 To what extent do you agree with the following statements concerning payments for your 
services? .................................................................................................................................. 79 
Table 26 Q23 Would you like to work with the REA as an independent expert again in the future? ....... 79 
Table 27 Q23a What are the main reasons for indicating that you are unlikely to work with the REA as an 
independent expert again in the future? ....................................................................................... 80 

6 

 
 
 
 
 
 
 
List of Figures 

Figure 46. Methodology for the surveys ........................................................................................ 13 
Figure 47 Q6 To what extent do you agree or disagree with the following statements concerning your 
registration process: .................................................................................................................. 72 
Figure 48 Q19 Did you visit the Reimbursement Help Desk during your evaluation in Brussels? ........... 77 
Figure 49 Q20 Did you use the REA-Evaluation@ec.europa.eu functional mailbox? ............................. 77 

7 

 
 
 
 
 
ANNEXES 

Annex 1. Detailed approach to key data collection and analysis methods 

Documentary review and desk research 

Literature  review is  essential in  mapping the  academic  and  analytical  field of  ideas  and  studies  that  can  be 
related to the context of the external evaluation. It  serves as a source of contextual information, additional 
evaluation questions, indicators for assessment, and data for comparison with other agencies. The literature 
review in this evaluation consisted of: 

primary sources of information (documents, empirical studies); 

 
  Secondary sources of information (generalisations). 

To this date we have reviewed primary and secondary sources that included: 

EU-level documents; 
Previous evaluations and Audit Reports of REA; 

 
 
  Studies/Action Plans/Reports. 

In addition, to conduct a thorough literature review as well as to collect all the necessary data for the cost-
benefit  analysis  the  evaluation  team  was  provided  with  all  the  documents  outlined  in  the  Tender 
Specifications.  Also,  we  extracted  important  documents  from  the  Agency’s  intranet,  including  monthly 
financial reports, bimonthly budget execution reports and Strategy on IT Governance. 

A list of reviewed sources is provided below. 

EU level documents: 

  Commission  Decision  2008/46/EC  of  14  December  2007  setting  up  the  ‘Research  Executive  Agency’ 
for  the  management  of  certain  areas  of  the  specific  Community  programmes  People,  Capacities  and 
Cooperation in the field of research in application of Council Regulation (EC) No 58/2003. 

  Commission Decision C(2007)2466 of 13 June 2007 on the adoption of the rules to ensure consistent 
verification of the existence and legal status of participants, as well as their operational and financial 
capacities,  in  indirect  actions  supported  through  the  form  of  a  grant  under  the  Seventh  Framework 
Programme  of  the  European  Community  for  research,  technological  development  and  demonstration 
activities (2007-2013) and under the Seventh Framework Programme of the European Atomic Energy 
Community (Euratom) for nuclear research and training activities (2007-2011). 

  Commission  Decision  C(2010)5184  of  30  July  2010  amending  Decision  C(2008)  3980  final  of 
31/VII/2008 delegating powers to the Research Executive Agency with a view to performance of tasks 
linked to implementation of the specific  Community  programmes People, Capacities and Cooperation 
in  the  field  of  research  comprising,  in  particular,  implementation  of  appropriations  entered  in  the 
Community budget. 

  Commission  Decision  C(2013)9418  of  20  December  2013  on  delegating  powers  to  the  Research 
Executive  Agency  with  a  view  to  performance  of  tasks  linked  to  the  implementation  of  Union 
programmes  in  the  field  of  frontier  research  comprising,  in  particular,  implementation  of 
appropriations entered in the general budget of the Union. 

  Commission  decision  C(2014)  7428  of  16  October  2014  approving  the  amended  Annual  Work 

Programme 2014 of the Research Executive Agency and repealing Decision C(2014) 5003. 

  Commission decision C(2014)9109 of 2 December 2014 establishing guidelines for the establishment 

and operation of executive agencies financed by the general budget of the Union. 

  Commission decision C(2015) 2297 of 10 April 2015 approving the Annual Work Programme 2015 of 

the Research Executive Agency. 

  Commission  Implementing  Decision  2013/778/EU  of  13  December  2013  establishing  the  Research 

Executive Agency and repealing Decision 2008/46/EC. 

  Communication  to  the  Commission  on  the  delegation  of  the  management  of  the  2014-2020 

programmes to executive agencies, Brussels, SEC(2013) 493/2. 

8 

 
 
 
 
 
 
  Council Decision 2006/971/EC of 19 December 2006 concerning the Specific Programme ‘Cooperation’ 
implementing  the  Seventh  Framework  Programme  of  the  European  Community  for  research, 
technological development and demonstration activities (2007 to 2013). 

  Council decision of 3 December 2013 establishing the specific programme implementing Horizon 2020 
-  the  Framework  Programme  for  Research  and  Innovation  (2014-2020)  and  repealing  Decisions 
2006/971/EC, 2006/972/EC, 2006/973/EC, 2006/974/EC and 2006/975/EC. 

  Council  Regulation  (EC)  No  58/2003  of  19  December  2002  laying  down  the  statute  for  executive 

agencies to be entrusted with certain tasks in the management of Community programmes. 

  Decision  N°  1982/2006/EC  of  18  December  2006  of  the  European  Parliament  and  the  Council 
concerning  the  Seventh  Framework  Programme  of  the  European  Community  for  research, 
technological development and demonstration activities (2007-2013). 
Legislative financial statement to the Commission Decision 2008/46/EC of 14 December 2007. 

 
  Memorandum  of  Understanding  between  the  Research  Executive  Agency  and  DG  Research  and 
Innovation,  DG  Education  and  Culture,  DG  Enterprise  and  Industry,  DG  Communication  Networks, 
Content  and  Technology,  DG  Agriculture  and  Rural  Development:  Modalities  and  Procedures  of 
Interaction. 

  Memorandum  of  understanding  of  ‘It  hosting  services  for  the  ABAC  system’  between  the  Research 

Executive Agency and the Directorate-General for Informatics (DIGIT). 

  Regulation  (EC)  N°  1906/2006  of  the  European  Parliament  and  the  Council  of  18  December  2006 
laying down the rules for the participation of undertakings, research centres and universities in actions 
under the Seventh Framework Programme and for the dissemination of research results (2007-2013). 
  Regulation  (EU)  N°  1290/2013  of  the  European  Parliament  and  the  Council  of  11  December  2013 
laying  down  the  rules  for  the  participation  and  dissemination  in  Horizon  2020  -  The  Framework 
Programme for Research and Innovation (2014-2020) and repealing Regulation (EC) No 1906/2006. 
  Regulation  (EU)  N°  1291/2013  of  11/12/2013  of  the  European  Parliament  and  of  the  Council 
establishing Horizon 2020 – The Framework Programme for Research and Innovation (2014-2020) 
  Regulation (EU, EURATOM) No 966/2012 of the European Parliament and of the Council of 25 October 
2012  on  the  financial  rules  applicable  to  the  general  budget  of  the  Union  and  repealing  Council 
Regulation (EC, Euratom) No 1605/2002 [Official Journal L 298, 26.10.2012]. 

  Service Level Agreement between the European Union and the Executive Agencies for the definition of 
the administrative  and financial terms governing the relations between an Executive Agency and DG 
Budget in relation to the implementation of the ABAC System. 

  Service  level  agreement  Concerning  the  collaboration  between  Directorate-General  for  Human 
Resources  and  Security  of  the  European  Commission  (DG  HR)  and  the  Research  Executive  Agency 
(REA). 

  Service  level  agreement  Concerning  the  collaboration  of  Research  Executive  Agency  (REA)  with  the 

Office for Administration and Payment of Individual Entitlements (PMO). 

  Specific  Financial  Statement  related  to  the  Decision  Establishing  the  European  Research  Council 

Executive Agency and repealing Decision 2008/37/EC. 

Previous evaluations and Audits Reports of REA: 

  CSES (2011), Evaluation of the Executive Agency for Competitiveness and Innovation, Final Report. 
  Deloitte (2013), External Evaluation of the ERCEA, Final report, June 2013. 
  Deloitte (2013), External Evaluation of the REA, Final report, June 2013. 
 

European Court of Auditors (2009), Delegating implementing tasks to executive agencies: a successful 
option?, Special Report No 13. 
ICF GFK (2013), Cost Benefit Analysis for the delegation of certain tasks regarding the implementation 
of Union Programmes 2014-2020 to the Executive Agencie.s 

 

  Monthly execution reports of REA Operational and Administrative Budgets 2012, 2013, 2014, 2015. 
  Monthly Overview report on resources, audit, internal control and key performance indicators (KPIs) in 

the REA 2013, 2014, 2015. 

  REA (2010), Financial circuits in the REA. 
  REA (2012), REA 2012 Staff Survey Report. 
  REA (2012), Report on the REA Supervision Campaign 2012: Review of project reports, Brussels. 
  REA  (2013),  Report  on  the  assessment  of  the  Internal  Control  Systems  (ICS  15)  outcome  of  2013 

ICAT survey. 

  REA (2014), REA Interim Report to the parent DGs and the Steering Committee: First Semester 2014. 

9 

 
 
 
  REA (2015), Report on the assessment of the Internal Control Systems (ICS 15) outcome of the 2014 

ICAT exercise and proposal of an action plan. 

  REA Administrative Budget 2012, 2013, 2014, 2015. 
  REA client satisfaction study and recommendations for stakeholder communication. 
  REA Risk Assessment Exercise 2012, 2013, 2014. 
 

Technopolis (2007), Cost benefit analysis for the Research Executive Agency, Revised report. 

Studies/Action Plans/Reports: 

  REA Work programmes 2012, 2013, 2014 and 2015. 
  REA Annual Activity Reports 2012, 2013 and 2014. 
  Minutes of the REA Steering Committee Meetings. 
  Minutes of the REA Management Meetings 2012, 2013, 2014, 2015. 
  REA Information Technology Master Plan 2010, 2011, 2012, 2013, 2014, 2015. 
  Minutes of the REA IT Management Board Meeting 2012, 2013, 2014. 
  Minutes of the REA Network of Legal Officers Meetings.  
  Minutes of the REA Network of Call Coordinators Meetings. 
  Minutes of the REA Network of Financial Officers Meetings.  
  Quarterly Reports of Activities of the Network of Project Officers. 
  Benzecri, J.P. (1973), L'analyse de données, Dunod Edition, Paris. 

  Bourdieu, P. (1984), Distinction: a Social Critique of the Judgment of Taste, Cambridge, MA: Harvard 

University Press. 

  CAF Resource  Centre of the European  Institute  of Public Administration,  CAF  2013: Improving Public 

Organisations through Self-Assessment. 

  DG BUDG (2013), Risk Management: Specific Guidance for Grants (direct management). 
  DG RTD (2007), Ex-post Audit Strategy of FP6 common to the Research DGs Period 2007-2010. 
  DG RTD (2009), Overview of the Internal Control Framework in the REA. 
  DG RTD (2015), H2020 General Model Grant Agreement — Mon.o 
  DG RTD (2015), H2020 General Model Grant Agreement — Multi. 
  Di Franco, G. (2001). EDS: esplorare, descrivere e sintetizzare i dati. Milan: Franco Angeli 
  Di  Franco,  G.,  (2006)  Corrispondenze  multiple  e  altre  tecniche  multivariate  per  variabili  categoriali, 
Franco  Angeli,  Milano;  Greenacre,  M.  and  Blasius,  J.,  (2006)  Multiple  Correspondence  Analysis  and 
Related  Methods]  Chapman-Hall,  Boca-Raton,  FL;  Holmes  S.,  (2007),  Multivariate  Analysis:  The 
French Way, Probability and Statistics: Essays in Honor of David A. Freedman, Volume 2, 219-233 IMS 
Lecture Notes - Monograph Series, IMS, Beachwood, OH. 

  DIGIT (2010), Infrastructure Services Provision Service Catalogue Oracle RDBMS Hosting. 
  Directorate  General  Energy  and  Transport,  Evaluation  of  the  first  three  years  of  operation  of  the 

Executive Agency for Competitiveness and Innovation, Final report, 2009. 

  Directorate  General  for  Health  and  Consumers,  1st  interim  evaluation  of  the  Public  Health  Executive 

 

Agency, Final Report, 2010 
European Commission (2010), Interim Evaluation of the Seventh Framework Programme, Final report, 
Brussels 
European Commission (2013), Risk Management in the Commission: Implementation Guide, Brussels 
European Commission (2014), ICS 8: Complementary Guidance Exception Reporting 

 
 
  REA (2008), Internal Communication Strategy 2009-2010 
  REA (2011), Developing an Internal Staff Mobility Policy for the REA: Initial Measures to offer Internal 

Mobility Opportunities to REA Staff, Brussels 

  REA (2011), Document Management in the Research Executive Agency: Internal Procedures 
  REA  (2012),  Decision  of  the  Steering  Committee  of  the  Research  Executive  Agency  concerning  the 
terms and conditions for internal investigations in relation to the prevention of fraud, corruption and 
illegal activity detrimental to the communities’ interest 

  REA (2012), IT governance in the REA 
  REA (2012), Learning and Development Framework 2012 
  REA (2012), Overview of the REA's financial circuits for the operational expenses 
  REA (2013), REA Business Continuity Plan 
  REA (2014), ICT Activities: 2014 - 2016 
  REA (2015), Overview of the REA's financial circuits for the administrative expenses 

10 

 
 
 
Interview programme 

A total of 29 interviews were carried out in August and September with various REA and EC officials outlined 
in  the  table  below.  These  interviews  helped  to  collect  new/internally  available  data  and  gain  a  deeper 
understanding of the internal processes of the organisation, its human resources policy, commitment of the 
staff  to  the  internal  quality  procedures,  advantages  and  drawbacks  of  the  governance  structure  and  other 
issues. The initial results were also used in  developing the survey questionnaires. All interviews were semi-
structured, adapted to the specific target group of the interview programme and were conducted face-to-face 
in  the  premises  of  the  interviewees.  A  table  outlining  interviewees  is  presented  below,  while  the  interview 
questionnaire can be found in Annex 2. 

Table 1. Interviews with REA staff and stakeholders conducted 17-21 August, 2-4 September and 
16-18 September. 

1.  W. De Meyere 

Person 

M. Carneiro 
S. Fumero 
F. De La Torre Francia 
E.G. Chira 

J. B. Veyret 

M. Spulber 

F. Willekens 
L. Byrne 
J. Hemmelskamp 

C. Amting 

A.Tristan 

E. Tsavalopoulos  

I. Reyes 
P. Kolar 

I. Marien-Dusak 

J. Methey 

2. 

3. 

4. 

5. 

6. 
7. 

8. 

9. 

10. 

T. Telemachou  
G. Ambroziewicz 

11.  A. Oram 

E. Pellizzari 

13. 

12.  K. Rosenow 
D. Bennink 
O. Pastre 
S. Goffin 
G. Valcarcel 
F. Olsson 
F. Marx 
K.-G. Barthel 
K. Hellevuo 
M.-S. Giannonni 
B. Mester 

14.  G. Gascard 

15.  W. Beurms 

16.  A. Longo 
A. Loncke 
P. Fernandez-Cañadas 

17. 

18.  W. Burtscher 

19.  B. Pyke 

G. Willmot 
20.  A. Luchetti 

Unit 

REA.C1 

REA.C3 

REA.A2 

REA.A2 

REA.A2 

REA.A4 

REA.B3 

REA.B3 

REA.B3 

REA.C1 

RTD.A6 
RTD.F3 

Interview type 

Group interview 

Group interview  

Group interview 

Group interview  

Individual interview 

Individual interview 
Individual interview 

CNECT.R5 

Individual interview 

RTD.A 

RTD.B5 

REA.C4 

Individual interview 

Group interview  

Group interview 

REA.B2 

Group interview 

REA.A1+A3 

Group interview 

REA 

REA 

Individual interview 

Individual interview 

DG AGRI 

Group interview 

DG RTD 

DG RTD 

DG Home 

Individual interview 

Individual interview 

Group interview 

REA, Department A 

Individual interview 

11 

 
 
 
21. 

T. Hallantie 
M. Lange 
M. Pettinarolli 

22.  R. Bultynck 

N. Giacomuzzi 
C. Strouzas 
23.  M. Curavic 

C. Bernot 
V. Puzzulo 
A. Marino 
T. Brefort 
P. M. Schmidt 
V. Bricola 
P. van der Zandt 
F. Roffi 
G. Suriano 
25.  M. Tachelet 

24. 

C. Amting 

26.  M. Tachelet 

27. 

W. de Meyere 
J. Hemmelskamp 
O. Pastre 
28.  V. Canetti 

29. 

E. Armani 

Survey 

REA.A5 

Group interview 

REA.C2 

Group interview 

REA.B1+B4 

Group interview 

REA.B5 

Group interview 

REA 

REA 

Group interview 

Group interview 

REA chairs of networks 

Group interview 

REA Staff Committee 

Individual interview 

DG HR 

Individual interview 

This  data  gathering  method  aimed  at  collecting  first-hand  information  (in  the  form  of  quantitative  and 
qualitative  data)  needed  to  assess  the  extent  to  which  various  organisational  factors  and  specificities  were 
making  a  positive/negative  impact  on  the  performance  of  the  agency.  In  total,  two  interlinked  surveys 
addressing the following target groups were carried out: 

  Survey A: an  online survey of experts contracted by the REA  to carry out evaluation and tasks 

in 2014 and 2015.  

  Survey  B  was  an  online  survey  targeting  beneficiaries  of  programmes  managed  by  the  REA 
which  were  financed  under  the  2011-2014  calls  for  proposals. Perceptions of  respondents to 
this survey were particularly relevant for assessing the extent to which the agency made a progress in 
improving  its  customer-oriented  results  and  the  degree  to  which  the  enablers  of  the  agency  were 
instrumental to achieving such results. 

Table 2. Targeting and coverage of the survey programme  

Target groups 

Survey A: REA’s 
independent experts 
Survey B: beneficiaries of 
programmes managed by 
REA 

Types of respondents 

Coverage 

Independent experts  

FP7 and H2020 beneficiaries 

All expert evaluators and monitors 
contracted by the REA in 2014 and 2015 
All projects managed by the REA from 
the 2011-2014 calls for proposals 

Overall methodology of the survey programme 

Our  survey  programme  involved  a  number  of  operational  activities  that  ensured  an  appropriate  scoping, 
implementation and analysis of the surveys. More specifically, the survey process involved the following steps: 

Production and programming of the surveys; 

  Scoping of the surveys; 
 
  Survey dissemination, data collection and management of the related activities (i.e. launching of the 
online  surveys,  monitoring  of  survey  results  and  undertaking  of  mitigation  actions  to  ensure  active 
participation in the survey programme); 

  Data treatment and analysis by using appropriate analytical instruments aimed at producing analytical 

reports and recommendations. 

12 

 
 
 
 
 
 
Figure 1. Methodology for the surveys 

Scoping  

Production and 
programming of 
surveys 

Survey dissemination, 
data collection and 
survey management 

Analysis of survey 
results 

Source: PPMI. 

Below we describe each step of the process in greater detail. 

Scoping and sampling 

Survey A 
This survey targeted REA’s independent experts contracted to carry out evaluation and monitoring tasks. The 
survey covered all independent experts contracted by the REA in 2014 and 2015. 

Survey B 
CORDA  was  the  main  source  of  information  for  surveying  FP  beneficiaries  whose  projects  were  managed  by 
the REA. The survey covered projects funded under the 2011-2014 calls for proposals.  

Production of surveys 

Annexes  3  and  4  present  the  questionnaires  for  the  beneficiary  and  experts  surveys,  respectively.  The 
Steering Group was invited to review, comment on and modify these questionnaires in order to prepare them 
for their launch in mid-September.  

It is important to note that survey  B was designed to avoid any possible duplications with the  simplification 
survey launched by the European Commission in late September 2015. The evaluation team was given access 
to the set of questions to be asked in this survey. Consequently, any information deemed useful for answering 
the evaluation questions that could be derived  from  this survey  were utilised by the evaluation team. These 
topics/questions were thus not be asked in the survey questionnaire designed for this evaluation. 

Data collection 

Following the launch of the surveys we monitored their progress and took risk mitigation (contingency) actions 
for the satisfactory response rates. In addition, reminders to participate in the survey were sent. Both online 
surveys were administered using the Fluid Surveys v.4.0 tool possessed by PPMI.  

Data treatment and analysis of survey results 

Once the surveys were closed, the collected data were categorised and analysed. To a large degree the initial 
steps  of  this  process  were  ‘automated’  by  the  pre-defined  structure  of  the  database.  Detailed  scrutiny  of 
responses was also carried out in order to detect any unforeseen data-collection/recording issues and to spot 
any unexpected trends that emerged from the collected data. The table below summarises the response and 
completion rates for surveys A and B. 

13 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Table 3. Summary of survey response and completion rates 

Title of survey 

Survey A: REA experts, evaluators  

Survey A: REA experts, monitors  

Survey B: REA beneficiaries, FP7  

Survey B: REA beneficiaries, H2020  

Number of possible 
participants 

10321 

574 

3070 

1629 

Responses 

5364 

321 

873 

542 

Response 
rate 

52.0 % 

55.9 % 

28.4 % 

33.3 % 

Completed 
questionnaires 

Completion 
rate 

5083 

305 

658 

480 

49.2 % 

53.1 % 

21.4 % 

29.5 % 

Our  team  applied  various  tools  and  techniques  in  order  to  analyse  the  survey  data,  both  quantitative  and 
qualitative.  Our  detailed  approach  to  processing  and  examining  statistics  is  presented  in  the  section  on 
statistical analysis of the survey data.  

On  the  basis  of  the  survey  results,  we  measured  the  expectations  of  the  Agency’s  clients,  their  perception 
concerning  the  Agency’s  performance  and  assess  the  gap  between  these  two  factors  (expectations  versus 
perceptions).  This  provided  information  as  to  which  aspects  of  organisational  performance  could  be  most 
strengthened in the future based on their importance and service quality in the eyes of the beneficiaries.  

Data protection 

The  research  team  fully  acknowledged  and  respected  the  privacy  rights  of  individuals  and  is  committed  to 
handling  any  personal  information  obtained  from  participants  in  the  survey  programme  or  from  the  REA 
and/or its parent DGs in accordance with the applicable law and regulations, as well as ensuring protection of 
such  data.  At  the  beginning  of  each  survey  a  clear  description  of  its  aims  was  provided  to  persons  and 
organisations invited to participate in the survey, so that they could be aware of the purpose of the collection 
of data. For analysis and reporting purposes we used only aggregated survey results. Possibilities of disclosing 
personal data to third parties were prevented. 

Cost-benefit analysis 

Cost  benefit  analysis  (CBA)  is  one  of  a  number  of  techniques  of  economic  evaluation,  a  type  of  evaluation 
which provides a comparative assessment of two or more courses of action, in this case the REA as opposed to 
other  organisational  arrangements,  in  terms  of  their  costs  and  consequences.  Costs  capture  the  resources 
(such as staff) which are  committed to an activity while the consequences  are  the  results of  those  activities 
which are of value to the funders of the activities concerned. The CBA is summarised by an assessment of: 
  effectiveness (in particular: the time to select, award and pay; client satisfaction globally):  
  efficiency  (human  resources  needed  at  the  REA  and  the  Commission,  administrative  cost  including 

the cost for coordination and checks; assess any possible duplication):  

  possible savings (overall).  

For this progress evaluation, the methodology and scope of the CBA followed the specific requirements set in 
Article 3(1) of the Council Regulation (EC) No 58/2003 of 19 December 2002, in the Commission Decision of 2 
December 2014 (C(2014)9109 final) establishing guidelines for the establishment and operation of  executive 
agencies financed by the  general budget of  the  Union (specifically  appendix  II  ‘Scope and  methodology of a 
Cost-Benefit-Analysis  of  the  delegation  of  tasks  to  an  executive  agency’)  and  in  the  ToR.  Following  the 
requirements  of  the  documents  indicated  above,  the  CBA  covered  both  quantitative  aspects  (which  were 
addressed in the retrospective CBA) and qualitative aspects (which were integrated into the overall evaluation 
framework and evaluation questions).  

In  many  CBAs,  costs  and  consequences  are  both  measured  in  monetary  terms.  In  the  case  of  the  REA,  the 
costs were monetised but the consequences (qualitative aspects) were measured in terms of the measures of 
organisational performance, or effectiveness, set out in the ToR.  

When assessing the effectiveness of the REA it is necessary to consider the options against which it is being 
compared.  The  relevant  comparison  could  be  an  alternative  way  of  organising  the  same  activities.  For 
example,  the  previous  CBAs  compared  the  costs  associated  with  REA  with  those  which  could  be  expected 
under the alternative of carrying out the programmes’ management functions within the Commission.  

14 

 
 
 
 
 
 
 
 
 
The data inputs for the analysis were obtained from a number of key sources which can be categorised either 
as  desk  research  reviewing  documents  relating  to  the  REA  and  stakeholder  consultation.  The  desk  research 
particularly considered the CBA model (background files) from the previous CBA carried out in 2013 and the 
necessary  quantitative  data  (CORDA  statistics  and  financial  figures).  Stakeholder  consultation  involved 
interviews with key officials in the REA and parent DGs, interviews with other relevant stakeholders and focus 
groups to test the initial findings. Surveys of beneficiaries were also an important source of evidence on the 
agency’s performance.  

15 

 
 
 
 
 
Annex 2. REA interview questionnaire 

Overview of REA and its evolution, including changes between the different programming periods 

1.1. REA’s governance system 

According  to  the  plan,  starting  from  September  2014  the  REA  was  gradually  delegated  activities:  Societal 
Challenge  2,  Societal  Challenge  6,  new  parts  of  Societal  Challenge  7  and  the  ‘FET  open’  action  from  the 
Excellent  Science  part  of  H2020.  The  transfer  of  activities  from  the  H2020  Specific  Objectives  ‘Spreading 
Excellence and Widening Participation’ and ‘Science with and for Society’ took place only in January 2015. 

  Was the handover of the abovementioned activities smooth? Some of the activities were delegated to 
the  REA  before  the  calls  were  launched,  while  others  –  after  the  calls.  If  any,  were  there  any 
‘teething’ problems at the beginning of Horizon 2020? 

The  new  mandate  triggered  changes  in  the  governance  of  the  REA:  new  parent  DGs  joined  the  Steering 
Committee under Horizon 2020. A Memorandum of Understanding was signed between the parent DGs in late 
2014. What were the key points of discussion and how did the decisions made affect REA’s governance and 
management systems? 

  The EC guidelines for the establishment and operation of executive agencies financed from the Union 
budget  emphasise  the  agencies’  autonomy  and  dependence.  If  at  all,  how  did  the  new  agreement 
between the parent DGs change this balance?  

In  one  of  the  AARs  the  possible  late  adoption  of  the  new  mandate  of  the  REA  was  considered  a  significant 
risk, as it would delay the agency's readiness to take up new tasks. This would impact on the operation of the 
agency as the recruitment of additional staff and programme implementation tasks would have to be delayed. 
To what extent has this risk materialised in the transitory period between FP7 and Horizon 2020? 

According to the EC guidelines for the establishment and operation of executive agencies financed from the 
Union  budget,  the  steering  committee  is  made  up  of  five  members.  In  circumstances  where  the  number  of 
parent DGs exceeds five, the steering committee and its chairmanship may function on the basis of a rotating 
representation of the parent DGs. In your opinion, is this arrangement optimal for the REA? 

In  addition  to  Steering  Committee  and  other  high-level  meetings,  what  mechanisms  ensure  that  the 
information prepared by the REA is communicated to and used by relevant units and officials in the EC?  

Coordination and monitoring of the REA in the parent DGs: which types of resources were committed to the 
activities? What kind of mechanisms, both formal and informal, facilitated the process? 

Overall, how would you assess REA’s governance system? Some of the more specific issues to be discussed 
include: 

  Did  the  REA  and  its  Director  effectively  support  the  Steering  Committee  (&  other  stakeholders, 
including  the  Programme  Committee)  by  providing  effective  monitoring  of  its  operations  and 
performance?  

  Did  the  REA  and  its  Director  effectively  cooperate  with  other  agencies  in  programmes  where  more 

than one institution is involved in the management activities? 

  Did  the  REA  and  its  Director  effectively  provide  direction  and  coordination  to  programme 

implementation in line with the Commission’s priorities? 

  Did the Steering Committee effectively coordinate the proper use of REA’s resources? 

1.2. Organisational structure 

The number of services managed by REA increased in H2020 in comparison with FP7, posing new challenges 
for  both  the  internal  organisation  of  the  REA  and  the  coordination  of  its  activities.  The  REA's  mission 
statement as of 2014 was updated taking into account the new mandate and units have adapted their unit's 
mission statements accordingly.  

16 

 
 
 
 
 
 
 
 
 
 
 
 
 

The  new  mandate  of  the  REA  was  supported  by  a  new  organisational  structure,  which  entered  into 
force  on  1  May  2014.  In  addition  to  the  increased  portfolio  of  programmes,  what  were  other  key 
reasons for introducing the changes? 

  A staff redeployment exercise was achieved in April 2014, while internal mobility of staff continued in 

2015. To what extent did this process influence the day-to-day operation of the agency?  

  Were alternative organisational structures considered? If yes, what were their comparative strengths 

and weaknesses? 

  What  are  the  major  advantages  of  the  new  organisational  structure?  Would  you  see  any  risks  and 
disadvantages  (e.g.  compartmentalisation  of  activities;  peak  workload  in  some  units  may  coincide 
with low workload in other units – how is this handled in the agency?)  

According to the EC guidelines for the establishment and operation of executive agencies financed from the 
Union budget, the organisation charts of the  agencies are  reflected in the Commission's organisation charts 
as  follows:  the  structure  of  the  agencies'  organisation  charts  is  reproduced  in  full  or  in  part  in  the 
organisation charts of the parent DGs (‘mirror’ chart). Is this arrangement optimal for both the REA and the 
EC? 

1.3. Cooperation  between  the  EC  and  REA  in  preparation  of  FP7  and  Horizon  2020  Annual  Work 
Programmes 

The  research  and  innovation  DGs,  under  the  lead  of  DG  RTD,  are  responsible  for  the  preparation  of  the 
Horizon  2020  work  programmes  which  are  adopted  by  the  Commission.  The  REA  contributes  to  the 
preparatory  work  of  the  Horizon  2020  work  programmes  on  issues  regarding  the  implementation  aspects 
(incl.  dissemination,  use  and  exploitation  of  results).  Overall,  would  you  assess  this  process  as  effective? 
Were there any lessons learned? 

According to the EC guidelines for the establishment and operation of executive agencies financed from the 
Union  budget,  the  director  must  ensure  that  the  agency  has  recruited  or  is  in  the  process  of  recruiting  the 
staff allocated to it as per the CBA. However, evidence shows that some of the assumptions in the CBA, for 
example,  overestimated  the  average  grant  size.  What  kind  of  corrective  measures  will  be  implemented  to 
narrow down the gap with the CBA assumptions?  

1.4. Key simplification measures introduced in 2012-2015: 

Examples  of  administrative  novelties  introduced  in  programmes  managed  by  the  REA  under  Horizon  2020 
include: 

  Simpler  management  of  projects:  a  single  funding  rate  for  all  beneficiaries  and  all  activities  in  the 
same grant; no time-sheets for personnel working full time on EU projects; only one certificate on the 
financial statements at the end of the project for beneficiary reaching triggering ceiling; the limitation 
of the mandatory ex ante financial capacity check to private coordinators only and in case of funding 
in excess of EUR 500 000; indirect costs covered by a single flat-rate applied to the direct costs; etc. 
  Acceptance of average personnel costs and beneficiaries’ usual accounting practices for direct costs. 
 
  Guidance became centrally provided via National Contact Points 
Evaluation criteria and sub-criteria were refined .  
 
  New budget flexibility measures were introduced. 

‘No negotiation’ approach. 

To  what  extent  did  these  simplifications  affect  REA’s  key  performance  indicators?  To  what  extent  did  these 
simplifications affect REA’s ability to monitor and control the projects, foreseen impact on error rates? Were 
there any other simplification measures introduced? In particular, what were the key simplification measures 
introduced between late 2012 and 2015? 

Client  feedback  and  satisfaction:  are  there  any  indications  which  imply  increased  satisfaction  from  the 
beneficiaries?  If any, did the measures produce  any unintended affects (e.g. the time-to-inform period of 5 
months in Horizon 2020 may be perceived as a negative development even though the whole time-to-grant 
period of max 8 months is an overall improvement compared to FP7)? 

17 

 
 
 
 
 
 
 
 
 
 
 
Evaluation  of  the  simplifications  introduced:  excluding  pre-financing,  substantive  spending  on  Horizon  2020 
will pick up in 2016 and reach a significant level of 2017, at which point the level of error and the real impact 
of simplification can be conclusively assessed. Considering the timing of this evaluation, it will be challenging 
to  assess  the  full  impact  of  the  simplification  measures  on  some  of  the  KPIs.  Do  you  agree  with  this 
assertion? 

2. Effectiveness 

2.1. REA’s operation in accordance to its legal framework 

The  overall  EU  legal  framework  governing  the  operation  of  executive  agencies  clearly  defines  the  activities 
that can be reserved to the executive agencies and the Commission: 

 

 

In accordance with Article 6 of Regulation (EC) No 58/2003, the executive agencies may be entrusted 
with the management of some or all of the stages in the lifetime of projects; provision of support in 
programme implementation; and provision of administrative and logistical support services.  
Pursuant to Article 58(7) of Regulation (EU, Euratom) No 966/2012, the following tasks are reserved 
to the Commission: tasks involving a large measure of discretion implying political choices; decisions 
submitted  to  comitology;  and  some  other  tasks,  including  inter-service  consultations  within  the 
Commission. 

In your opinion, have there been any activities entrusted to the REA which pose a risk in terms of crossing 
the line between the programme management/REA and policy development/EC sides? 

Decision  2008/46/EC  and  (starting  with  2014)  Decision  2013/778/EU  entrusted  the  REA  with  such  tasks  as 
(1) managing some stages of programme implementation and some phases in the lifetime of specific projects 
on the basis of the relevant work programmes; (2) adopting the instruments of budget execution for revenue 
and  expenditure  and  carrying  out  all  the  operations  necessary  for  the  management  of  the  programme;  (3) 
gathering,  analysing  and  passing  on  to  the  Commission  all  the  information  needed  to  guide  the 
implementation  of  the  relevant  Community  programme  parts;  and  (4)  provision  of  logistical  and 
administrative support in programme implementation. In your opinion, has the REA carried out these tasks to 
the highest extent possible (i.e. was fully coherent with the legal framework) or would you say that activities 
in some of these areas could be given more attention? 

In  2013/2014,  the  new  Commission  Decision  establishing  the  REA  and  the  new  Delegation  Act  came  into 
effect.  To  the  best  of  your  knowledge,  were  there  any  problems  related  to  this  transitional  period  and  in 
particular to adjusting REA’s operations to the updated legal framework? 

The parent DGs must define at the level of the Memorandum of Understanding a supervision strategy aimed 
at avoiding gaps or duplication of efforts resulting from crossover between their monitoring and supervision 
tasks and the execution tasks of the agency. How is this strategy implemented in practice? In your opinion, is 
it effective? Do you see any possibilities to improve this framework? 

Officials seconded to the executive agencies in the interests of the service maintain regular contact with their 
home  DG  throughout  their  secondment.  What  is  their  role  in  ensuring  that  the  REA  abides  by  the  policy 
guidelines set by the Commission? What formal and informal mechanisms and tools facilitate this process? 
How is it ensured that the existing legal framework/formal communication mechanisms do not involve ‘micro-
management’ by parent DGs and undermine the cost-efficiency of the delegation scenario? 

Since its creation the REA has undergone continuous fine-tuning of the internal procedures and organisational 
structures, streamlining its processes and integrating its activities in FP7 common tools. In your opinion, was 
the  operation  of  the  REA  flexible  enough  to  accommodate  the  key  changes,  while  at  the  same  time 
maintaining concordance with the legal framework establishing the agency? Are you aware of any challenges 
related to this transition faced by the agency? 

2.2. Extent to which the REA has achieved its objectives 

Generally, how would you assess the overall operation of the REA in terms of its: 

18 

 
 
 
 
 
 
 
 
 
 
 
 
 
 

strengths  (e.g.  generally  strong  KPI  results,  particularly  in  execution  on  commitment  and  payment 
appropriations, TTG, TTP, etc.); 

  weaknesses (e.g. staff turnover and its effects on the quality of the services); 
 
 

opportunities (e.g. further consolidation of the activities following years of change); 
threats  (e.g.  delays  in  payment  appropriations,  inherent  risks  in  the  SME  legacy  instrument, 
complexity  of  Security  Research  activities,  high  error  rate,  expert  payments;  the  extremely  high 
workload periods created by beneficiaries using REA’s central support services)? 

When  it  comes  to  performance  management  of  the  REA  (defining  and  estimating  KPIs),  is  this  process 
effective? Do you see any challenges/problems in terms of defining KPIs and estimating them? What could be 
improved in this regard? 

Preparation  of  FP7  &  H2020  AWPs:  the  research  and  innovation  DGs,  under  the  lead  of  DG  RTD,  are 
responsible for the preparation of the Horizon 2020 work programmes which are adopted by the Commission. 
The REA contributes to the preparatory work of the Horizon 2020 work programmes on issues regarding the 
implementation aspects (incl. dissemination, use and exploitation of results). Overall, would you assess this 
process as effective? Were there any lessons learned? 

Is  the  REA  equally  effective  in  all  stages  of  proposal  evaluation/project  management  (call/proposal  stage; 
contract  negotiations;  implementation  of  the  projects/project  management;  follow-up  and  audits)?  In  your 
opinion, which stages of those mentioned above could be further improved? 

Successfully launched calls and research topics are the key output of the REA. However, in some cases not all 
planned  research  topics  are  funded  due  to  a  variety  of  reasons,  including  lack  of  interest  from  potential 
applicants (i.e. cases where no proposals were received, or no proposal was of sufficient quality). What are 
the ways in which the REA aims to ensure the sufficient production of key outputs (calls, proposals received, 
grants  awarded,  and  amount  of  commitment  appropriations  managed)?  Do  you  see  any  potential  ways  for 
improvement in this regard? 

Overall, what evidence is there to suggest that the Executive Agency  has fully achieved its objectives in the 
2012-2015  period  (both  as  provided  in  the  legal  basis  and  the  annual  work  programmes)?  Considering the 
threats  and  possible  weaknesses  discussed  above,  are  there  any  areas  of  concern?  In  your  opinion,  which 
areas would be better addressed in case of an alternative scenario (i.e. the Commission-led services)? 

2.3. Extent  to  which  the  REA  led  to  an  improved  management  of  the  programme(s),  including  in 
terms of simplification measures, proximity to addressees and visibility of the EU 

Examples of alternatives to the current executive agency management model include (1) management of the 
programmes  by  the  Commission;  (2)  mixed  agency-Commission  management;  (3)  partial  management  by 
the  Commission  while  outsourcing  some  activities  to  the  extent  legally  possible.  In  your  opinion,  does  the 
current  model  ensure  the  best  management  of  the  programme  and  highest  quality  services  to  the 
stakeholders? Would you say that other alternatives could improve the management of the programmes? In 
which ways? 

Please  share  your  opinion  on  whether  the  current  model  is  optimal  in  terms  of  (1)  addressing  resource 
constraints (esp. during the periods of high workload); (2) capacity to provide specific skills. 
A number of simplification measures (e.g. simpler form of grants, streamlining and harmonisation of funding 
rules, etc.) were introduced during FP7.  If  you are  aware  of these  measures,  could  you mention those that 
seem most effective to you? 

Could  you  also  share  your  views  on  what  type  of  simplification  measures  (and  in  which  areas)  could  be 
introduced in the new 2014-2020 financial framework? Please focus on such measures that, in your opinion, 
could potentially contribute to the reduction of administrative burden both to the beneficiaries and the agency 
as well as enhance the capacity to adapt to periods of high workload. 

The  REA  has  an  objective  to  be  as  responsive  as  possible  to  the  needs  of  beneficiaries  both  quantitatively 
(e.g. in terms of TTG and TTP) and qualitatively (in terms of  the beneficiaries’ satisfaction with the services 

19 

 
 
 
 
 
 
 
 
 
 
 
 
provided  and  overall  involvement  of  the  scientific  community  in  the  management  of  the  programmes).  In 
your opinion, is the current extent to which resources and procedures are delegated to maintain contact with 
beneficiaries  appropriate?  What  could  be  changed?  What  impact  does  the  involvement  of  institutional 
stakeholders (esp. parent DGs) have on this process? 

The REA is expected to act in compliance with the Commission’s guidelines on information and visibility of the 
programmes,  as  well  as  the  instruments  put  in  place  to  ensure  the  visibility  of  the  Commission  as  the 
promoter  of  programmes  entrusted  to  the  REA.  To  the  best  of  your  knowledge,  are  the  appropriate 
funding/instruments in place to ensure the visibility of the Commission as a promoter of the programme? 
In  your  opinion,  is  the  REA  brand  well  known  among  the  beneficiaries?  What  would  you  say  are  the  key 
perceptions about the REA among beneficiaries and stakeholders? Would you say that these perceptions have 
an impact on the effectiveness of the agency? What kind of impact? 

2.4. REA as a learning and client-oriented organisation  

This  aspect  of  the  evaluation  will  be  primarily  assessed  via  beneficiary  and  staff  surveys.  Your  feedback, 
however, may help identify a number of good practices. To this end, please provide outstanding examples of:  
Improvements  of  processes  aimed  at  internal  (i.e.  REA’s  staff)  or  external  users  (i.e.  beneficiaries) 
that have been realised in the past years/months. 

 

  Arrangements  to  ensure  a  higher  level  of  flexibility  when  responding  to  the  needs  of  stakeholders 
(applicants  and  beneficiaries,  the  Commission,  other  stakeholders)  in  the  implementation  of  the 
delegated tasks. 
 
Effective learning at the organisation/take-up of lessons learned and recommendations suggested; 
  Streamlining  and  harmonisation  of  funding  rules  and  procedures  across  different  programmes  and 

units. 
Improvements in IT systems and communication with beneficiaries. 
Improvements in HR policy. 

 
 

3. Operational efficiency 

3.1. Detailed deconstruction of key steps of the call/application/project management life-cycle 

To be discussed:  

  Call stage: consultations with the Commission, managing calls publications, budgeting… 
  Application  stage:  information  for  applicants,  reception  of  proposals,  admissibility  and  eligibility 
check,  evaluation,  respective  indicators  (time-to-inform,  redress,  budget  execution/success  rates, 
etc.) 

  Conclusion of grant agreements: validation of beneficiaries, financial capacity and cross-checks, grant 

 

finalisation and contracting, respective indicators (TTG)  
Follow- up and monitoring: pre-financing, interim and final payments, reporting, amendment of grant 
agreements, respective indicators (TTP, TTA)  
 
Ex-post controls and recoveries: audit strategies and plans, ex-post audits, corrections/recoveries  
  Horizontal  question:  new  H2020  rules  and  procedures  throughout  programme/project  management 

cycle (clarity, stability, suitability, efficiency, etc.) 

3.2. Overall efficiency of REA’s activities 

General discussion on the achievement of REA’s key performance indicators and the progress made in 2012-
2015. Some of the more specific issues to be addressed during interviews include: 

  Budget  execution  of  commitment  and  payment  appropriations.  Reports  show  that  the 
operational  commitment  and  payment  appropriations  from  the  EU  general  budget  (excluding 
earmarked  revenue)  were  fully  executed,  while  the  execution  rate  for  the  (non-differentiated) 
administrative appropriations reached 97.5 % in 2013.  

  Time-to-grant  (TTG).  TTG figures show an overall improvement over 2012-2015, with H2020 calls 
significantly outperforming FP7 programmes.;  For  the first H2020 calls the TTG remained below the 
defined target of 8 months (245 days) with an average of 217 days;  

20 

 
 
 
 
 
 
 
 
 
 
o Cumulative TTG for FP7 is higher than that for H2020 projects  -> was this mainly achieved 
due to the simplification measures and maximum durations introduced under Horizon 2020 
(Total  TTG  up  to  max.  8  months  (245  days)  including  a  TTI  up  to  max.  5  months  (153 
days)); or also additional simplifications/other efforts made by the REA? 

o TTG  for  ITN  in  2014  (258  days)  was  above  the  target  of  245  days  –  what  were  the  key 

reasons behind this outcome?  

o In 2012/2013 TTG targets were not reached for COFUND and SME actions, also Security  
o Regarding  the  new  limit  of  5+3=8  months  for  TTG,  minutes  of  a  Steering  Committee 
meeting  suggest  that  there  has  been  some  negative  feedback  from  stakeholders  as 
evaluation results were now communicated later (after 5 months) than in the past (after 3-
4  months)  and  that  stakeholders  perceived  one  target  of  8  months  rather  than  two 
separate  targets  (for  5  months  and  for  3  months).  Has  this  aspect  been  addressed  and 
communicated to the beneficiaries? 

  Redress cases. The target for the agency is 0 % of redress cases upheld. Actual data show that less 
than 1 % of redress cases were upheld in late year’s FP7. What has been the trend in the number of 
complaints  received?  Is  there  an  increasing  trend  due  to  the  growing  number  of  proposals  or,  for 
example, introduction of two-stage calls? 

  Error rate. The target was set at 2 %, however data show that the residual error rate was frequently 
not met in Space and Security (error rates of about 3 %), SME programme (5 % residual error rate 
and over 10 % non-registration of RTD performer invoices in 2014), but just over 1 % in People.  

o What  are  the  inherent  risks  in  the  above-mentioned  programmes  (e.g.  subcontracting  in 

SME instrument, fraudulent behaviour)?  

o What are the most recurring errors in FP7 and H2020 programmes (e.g. wrong calculation 
of  hourly  rates,  absence  of  time  recording  systems  and  wrong  calculation  of  overhead 
rates…) 

o The  error  rate  for  some  instruments  increased  since  2012  (e.g.  for  SMEs)  –  was  this  the 

result of more intensive ex-post controls introduced in later years of FP7? 

o According to the 2014 AAR, the high rate of error for the SME actions justifies maintaining 
a  reservation  for  this  programme  while  the  lower  error  rate  for  the  People  Programme 
confirms  the  REA’s  assessment  about  reasonable  assurance  that  this  programme  is  not 
impacted  by  material  errors.  As  of  2015  the  REA  will  put  in  place  new  ex-ante  controls 
(requesting SMEs to provide evidence that the RTD performers’ invoices have been issued, 
registered and paid) to make sure that SME beneficiaries comply with the rules – what have 
been the experiences to date? 

  Time-to-pay  (TTP):  during 2014  97 % of project-related payments were made on time. Over FP7, 
90-95 %  of  interim  and  final  payments  were  made  on  time  and  around  97-99 %  of  pre-financing 
payments. There was a substantial improvement in 2013 and 2014  – what factors drove this result? 
TTP for H2020 first interim payments? Other issues to be discussed: 

o For  payments  to  experts,  the  REA  temporarily  experienced  difficulties  in  the  summer 
months  due  to  high  workload,  staff  changes  induced  by  the  reorganisations  and  certain 
delays in the recruitment of new staff. Review of the AAR suggests that the same situation 
occurred in both 2013 and 2014. 

o Some 89.7 % of the administrative  payments were made on time (as a rule within 30 
days from receipt of payment request), which is an improvement of 7.5 percentage points 
compared to 2012. 

  Average  grant  size  &  number  of  projects  per  officer.  The  AARs  suggest  that  for  some  of  the 
newly  delegated  actions,  the  significantly  lower  grant  size  was  only  partly  counterbalanced  by  a 
slightly  reduced  budget  delegated,  particularly  for  Space  and  Security  research  where  the  average 
grant  size  is  at  around  50 %  of  what  was  assumed  in  the  CBA.  What  measures  have  been/will  be 
taken to address this situation in future calls? 

  Achievement  of  objectives.  The KPI which indicates the share of closed projects that achieved all 
or most of their objectives reached about 97-98 % in 2012-2014. The target set by the REA is 90 %. 
Has there been discussion on this indicator with respect to its target value?  

  External  evaluations.  In  2013  the  REA  started  using  external  evaluations,  thus  considerably 
reducing  the  travel  and  accommodation  costs  of  experts  and  related  carbon emissions.  The  savings 
made on experts' fees and allowances can be estimated to reach up to  EUR 25 000-100 000 per call 

21 

 
 
depending  on  the  number  of  experts  attending  the  briefing.  What  are  the  key  risks  associated  with 
this move and how has the agency mitigated them? 

3.3. Operational budget 

Financial  management  of  the  operational  programmes  delegated  to  the  REA  is  largely  decentralised  to 
operational  units  who  have  full  control  over  initiation  and  authorisation  of  commitments,  payments  and 
recoveries. The central Finance Unit performs independent financial verification and provides assurance to the 
Director on the proper implementation of the operational budget throughout the REA. 

 

In  your  opinion,  what  are  the  main  advantages  and  disadvantages  of  a  decentralised  financial 
management  system?  Would  it  be  feasible  in  principle  to  adopt  a  more  centralised  system?  What 
would be the costs/disadvantages of such a system? 

The  lack  of  payment  appropriations  was  a  significant  risk  in  relation  to  the  REA's  ability  to  implement  the 
programmes smoothly and to reach its targets in terms of TTG and TTP. Various reports indicate that the risk 
materialised  in  2013  and  2014  as  the  increased  budget  had  not  been  fully  obtained  through  the  amending 
budget procedure and the global transfer. 

  Which areas/programmes were most affected by the lack of payment appropriations and how did this 

affect the agency’s KPIs?  

  What interventions were made to ensure sufficiency of payment appropriations and minimum delay of 

 

pre-financing, interim and final payments? 
To  what  extent  did  this  lack  affect  the  achievement  of  the  policy  objectives  for  FP7  as  well  as  for 
Horizon  2020?  What  mitigation  and  control  measures  can  be  adopted  in  in  the  future  to  avoid  a 
similar situation? 

3.4. Administrative budget and expenditures 
General  discussion  on  the  evolution  of  the  administrative  budget  and  its  main  budget  titles  during  2012-
2015: 

 Title I. Staff expenditure comprises the following cost items: 
o Remuneration, Allowances and Charges 
o Professional Development and Social expenditure 

 Title II. Infrastructure and operating expenditure includes: 

o Building expenditure 
o ICT expenditure 
o Movable property and Current operating expenditure 

 Title III. Programme support expenditure: 

o Programme management expenditure 
o Common support services expenditure  

According to the AARs the REA's administrative budget, as a share of the operational budget, remained in the 
range of 3 %  (just over 2.3 %  if central support services  are  excluded). This figure  was substantially lower 
than  the  6 %  ceiling  in  the  FP7  legal  base.  What  are  the  corresponding  figures  for  2014-2015  and  the 
expected future developments, especially considering the expansion of central support services?  

Future  developments  in  the  agency’s  operational  costs:  the  main  changes  from  2015  to  2016  concern  a 
decrease in the expenditure related to the evaluation platform, an increase in building-related expenditure, a 
further decrease of ex-post audit expenses and an increase in missions. Have the necessary provisions been 
sufficient and made on time?  

Staff costs (discussion on their evolution for the different types of staff): according to Article 18 of Regulation 
(EC) No 58/2003, the staff of the executive agencies consists of: 

 

 

 

officials seconded in the interests of the service and engaged by the agency as temporary staff within 
the  meaning  of  Article  2(a)  of  the  Conditions  of  Employment  of  Other  Servants  (CEOS)  (AT2a)  in 
positions of responsibility; 
temporary staff within the meaning of Article 2(f) of the CEOS (AT2f). Officials seconded at their own 
request by an Institution are engaged as AT2f; 
contract staff within the meaning of Article 3a of the CEOS (AT3a). The number of contract staff may 
not exceed 75 % of the total number of staff employed in an agency; 

22 

 
 
 
 
 
 
 
 
 

the  agency  may  use  seconded  national  experts  (SNEs)  and  other  types  of  staff  under  private-law 
contracts. 

3.5. Internal control systems & audits: 

Overview of the REA’s internal and external control systems, particularly: 

  REA’s Internal Audit Capability (IAC); 
 
 

the Commission’s Internal Audit Service (IAS); 
the European Court of Auditors (ECA). 

+ discussion on key mechanisms designed to monitor the functioning of the internal control systems: 

 

 
 
 
 
 
 

bi-annual  reports  submitted  by  the  Heads  of  Unit  in  their  capacity  of  Authorising  Officer  by  Sub-
Delegation (AOSD); 
opinion on the state of control of the REA's Internal Control Coordinator (ICC);  
outcomes of activities of the ex-post audit function and fraud prevention measures;  
independent opinion of the REA's Internal Audit Capability (IAC) on the state of internal controls;  
observations and recommendations reported by the Internal Audit Service (IAS);  
observations and recommendations reported by the European Court of Auditors (ECA);  
observations and recommendations reported by DG BUDG (in the context of the validation of the local 
accounting systems by the Commission's Accounting officer).  

Multiple reports suggest that the agency’s management has reasonable assurance and that, overall, suitable 
controls are in place and working as intended. The risks are being appropriately monitored and mitigated, and 
necessary improvements and reinforcements are  being implemented. However, the Director, in his capacity 
as  Authorising  Officer  (AO)  for  the  administrative  budget  and  Authorising  Officer  by  Delegation  (AOD) 
expressed  reservations  for  the  ABB  activities  Space  and  Security  (Cooperation  Specific  Programme)  and 
Research for the Benefit of SMEs (Capacities Specific Programme).  

  What  have  been  the  key  intrinsic  risks  in  Space  and  Security  research  and  the  SME  instrument  (i.e. 
transactions do not reflect normal market conditions; SME bankruptcies; non-recording the RTD performance 
invoices in the SMEs accounts)?  
To  what  extent  and  how  have  these  risks  been  mitigated  (e.g.  through  additional  ex-ante  and  ex-post 
controls)? 

 

Key  stakeholders,  however,  have  been  concerned  about  the  administrative  burden  imposed  by  the  FP7 
managing services and have requested a re-balancing of trust and control. For this reason the REA and the 
Commission  services  have  reduced  their  level  of  ex-ante  control  (whilst  respecting  the  requirements  of  the 
Financial Regulation) and have decided to obtain most of the assurance from ex-post controls. 

  What are the possible implications and risks of the strategic move from ex-ante to ex-post controls?  
  What  implications  does  this  strategic  change,  as  well  as  the  higher  SME  participation  rate in  Horizon  2020, 

have for the error rate? 

One of the reports states that the target of a maximum residual error rate of 2 % cannot be reasonably met 
without a massive increase in the number of audits or in the administrative burden imposed on participants 
through widespread ex-ante controls.  

  Are  there  any  alternative  approaches  that  could  improve  the  residual  error  rate  without  excessive  controls 

(e.g. more extensive use of flat rates, unit costs and lump sums)?  

  Would additional automated tools aimed at detecting errors/fraud help reduce the agency’s exposure to risk? 

A discussion on the changes and simplifications introduced between 2012 and 2015: 
The Common Representative Audit Sample (CRaS) was introduced in 2012 across the research family.  
In H2020 ex-post audits became in the remit of the Common Support Centre. 

 
 

In  2014  the  REA  performed  an  assessment  on  the  effectiveness  of  the  ICS  according  to  ICS  15  with  the 
support  of  the  IT  tool  (iCAT,  internal  Control  Assessment  Tool)  provided  by  DG  BUDG  in  order  to  obtain  a 
diagnostic of strengths and weaknesses in the REA's internal control framework and to identify further actions 
for improvement. What were the results of this assessment? 

23 

 
 
 
 
 
 
 
 
 
 
 
3.6. IT resources 

Following the implementation of the IT Master Plan in 2014, completely new IT tools and workflows are used 
for  H2020.  Following  this  change  local  IT  developments  were  reduced  to  a  minimum  and  focused  only  on 
those  processes  for  which  no  corporate tools  were  available.  More  specifically,  the  REA  maintained  its  local 
project  support  tool  PROMIS,  which  complements  the  central  FP7  grant  management.  For  H2020,  the 
corporate workflow tool Compass and the Sygma replaced the PROMIS functionalities and Coreflow was used 
for FP7 legacy. 

  How would you assess the appropriateness/functionality of the aforementioned IT tools? What are the 

relative advantages/disadvantages of Compass and the Sygma compared to their predecessor?  

  How would you assess the availability/appropriateness/functionality of the common H2020 dedicated 

IT tools  

Staff training:  the AARs and other  documents imply  that the REA  staff  have  been intensively trained in the 
new  IT  systems  (Sygma,  COMPASS)  and  continued  attending  training  courses  about  the  new  rules  and 
procedures  for  the  new  Framework  Programme.  Has  the  Agency  collected  feedback  on  these  training 
activities and monitored the progress? 

Future plans and developments with IT becoming a common service: in cooperation with the CSC, the REA is 
active in the definition of the common procedures and the further development of the corporate IT tools. In 
particular the REA will further invest in fine tuning the business process and the related IT tools:  

 

for  the  centralised  contracting  and  payment  of  expert  evaluators  for  all  H2020  calls  and  for  the 
validation of participants; 
an automated tool for internal control systems was also being assessed; 

 
  Other tools (to be discussed; e.g. in 2015 the focus has been on grant management IT tools). 

3.7. Internal and external communication, visibility, proximity to beneficiaries 

In  cooperation  with  the  parent  DGs  and  the  H2020  Common  Support  Centre,  the  REA  implemented  the 
dissemination strategy for results of Horizon 2020. What are the main principles of this strategy and can the 
evaluation team access the related document?  

Reports  suggest  that  during  2012  the  REA  launched  a  number  of  information  campaigns  aimed  at  the 
beneficiaries and the wider society. Those activities were at least of the following types: 

 Various  conferences,  workshops  and  events  promoting  the  programmes/calls  and  disseminating  the 

research results 

 Information campaigns highlighting key management and control issues, such as the mass information 

campaign highlighting actions to correct the errors in the SME instrument.  

Which  of  these  activities  prove  to  be  most  effective  and  most  valued  by  the  beneficiaries?  To  what  extent 
could the beneficiaries make impact on the programme/set the agenda for these communication events? How 
were the results or even impacts, as well as the quality of these activities evaluated?  

Various  activities  raising  staff  awareness  on  the  REA’s  external  communication  strategy  we  implemented 
during 2012-2015. What evidence is currently available which can be used to assess their effectiveness? 

Internal  communication:  what  key  mechanisms  and  tools  ensure  the  effectiveness  of 
communications? What is the feedback received via the staff survey? 

internal 

Balancing  external  communication:  the  AARs  suggest  that  the  REA  will  revise  its  strategy  for  internal 
communication and continue to deliver on its external  communication strategy, with particular focus on  the 
communication on project results and success stories. 

3.8. HR policy 

Discussion on the main HR activities and policies implemented during 2012-2015: 

24 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
 

 

 

 

 

finalisation  of  the  internal  redeployment  exercise  linked  to  the  new  mandate  and  to  implement  the 
new REA organisation chart by the beginning of the second quarter; 
staff establishment plan for 2014-2020; 
staff employed within the REA expressed in FTEs throughout 2013-2015 
soordination and implementation of an effective recruitment process to fill vacant posts  
staff appraisals for the REA staff (for 2013), including objective setting for 2014; 
to  adapt  the  REA  Implementing  Provisions  to  the  changes  in  the  Staff  Regulations  and  the 
Commission's General Implementing Provisions 
to work together with DG HR and the other executive agencies to achieve the implementation of the 
new system for staff appraisal in executive agencies in good time for the appraisal exercise that is to 
be conducted in early 2015; 
to  support  the  activities  of  the  REA  Staff  Committee  and  moderate  the  social  dialogue  between  the 
Committee and the REA management; 
to  review  and  implement  the  Agency's  Learning  and  Development  Framework  for  2014  taking  into 
account, in particular, the training needs arising from the new actions delegated to the REA, the new 
H2020 rules and business processes and the new corporate IT tools (in cooperation with the Common 
Support Centre in DG RTD); 
to  continue  implementing  the  action  plan  developed  following  the  2012  staff  satisfaction  survey 
(including  actions  concerning  staff  mobility,  well-being,  teleworking  and  equal  opportunities)  and 
conduct a new survey by end-2014. 

In the context of changes to the mandate of the Agency, how did you ensure a clear delineation of tasks and 
prevent functions from being duplicated?  

How  were  the  changes  to  the  mandate  of  the  REA  addressed  in  the  training  strategy,  as  well  as  in 
organisational and individual development plans of the REA staff?  

What are the key practices to identify the current competences of people at the individual and organisational 
levels in terms of skills and expertise? How are the impacts of training and development programmes on the 
workplace and transfer of knowledge to colleagues being monitored? 

Discussion on the evolution of staff in the REA (number of staff by category, salary levels, etc.): according to 
Article 18 of Regulation (EC) No 58/2003, the staff of the executive agencies consists of: 

  Officials seconded in the interests of the service and engaged by the agency as temporary staff within 
the  meaning  of  Article  2(a)  of  the  Conditions  of  Employment  of  Other  Servants  (CEOS)  (AT2a)  in 
positions of responsibility; 
Temporary staff within the meaning of Article 2(f) of the CEOS (AT2f). Officials seconded at their own 
request by an Institution are engaged as AT2f; 

 

  Contract staff within the meaning of Article 3a of the CEOS (AT3a). The number of contract staff may 

 

not exceed 75 % of the total number of staff employed in an agency. 
The  agency  may  use  seconded  national  experts  (SNEs)  and  other  types  of  staff  under  private-law 
contracts. 

  Average staff costs per category (TA/CA) in 2013-2015 and expected future development. 

To what extent does the HR policy support  the development of a performance  culture in  the  REA?  Can you 
provide any examples of concrete schemes, measures to encourage better individual and team results? 

Planning  of  human  resources:  the  agency’s  work  programme  must  include  a  clear  presentation  of  the 
operating budget of the agency, as well as the staff figures with an indicative breakdown per programme and, 
within  each  programme,  per  activity  and  with  an  indicative  breakdown  for  the  different  types  of  agents. 
Based on your experience, have these plans accurately predicted the actual demand for human resources in 
the  different  programmes  and  activities?  How  would  you  assess  the  adequacy  of  the  staffing  level  and 
workload, possibilities to adapt to changing workload? 
High  staff  turnover  is  a  well-documented  structural  weakness  of  the  executive  agencies  which  negatively 
affects  their  performance  and  results  in  higher  programme  continuity  costs,  opportunity  costs  of  staff 
replacement and general loss of know-how in the organisation. 

  Staff turnover in 2012-2015? 

25 

 
 
 
 
 
 
 
 
To what extent and in which areas has staff turnover affect the agency’s performance? 

 
  Degree  of  difficulty  to  find  the  right  profiles  (time  spent  to  recruit  specific  profiles  and  number  of 

candidates for each job advertised)/ 

  What are the main reasons for staff turnover (e.g. relatively low salary level of CAs, other)? 
 

To this end, how can HR policy help mitigate these risks (e.g. specific policies whose goal is to retain 
the agency’s top performers)?  

Overall, to what extent have the following objectives been reached? 
effective recruitment of staff and equal opportunities policy; 

 
  maintenance of a constructive social dialogue, especially with the Staff Committee; 
 
 
 

staff training;  
career development and motivation; 
overall staff satisfaction. 

4. Utility 

4.1. Extent  to  which  the  Executive  Agency  enabled  the  Commission  to  focus  on  its  policy-related 
tasks 

The parent DGs define a supervision strategy aimed at avoiding gaps or duplication of  efforts resulting from 
crossover  between  their  monitoring  and  supervision  tasks  and  the  execution  tasks  of  the  agency.  The 
supervision  strategy  and  cooperation  modalities  defined  in  the  memorandum  of  understanding  should  be 
regularly reviewed and updated if need be to take into account any new developments and ensure that they 
are  fit  for  purpose.  Specifically,  such  mechanisms  as  organisational  set-up  of  the  agency  and  process  of 
approval of annual work programmes, the AAR, the agency’s budget and accounts by the steering committee 
ensure that supervision is implemented.  

  Overall,  does  clear  delimitation  of  responsibilities  and  tasks  between  the  REA  and  the  parent  DGs 

exist?  

  Has the creation of the EA enabled the Commission staff to focus better on the institutional tasks as 

compared to the alternative options?  

  Are  there  any  examples  of  formal  and  informal  activities  undertaken  by  the  Agency  that  could  be 

classified as policy-making? 

The  appropriate  monitoring  of  the  agency’s  activities  is  primarily  achieved  by  means  of  participation  of  the 
parent  DGs  in  the  steering  committee  by  regular  coordination  meetings  at  management  level  and  further 
contacts at working level. In the steering committee the representatives of the parent DGs verify whether the 
provisions of the memorandum of understanding regarding reporting and supervision of the EA are duly and 
timely fulfilled and how the agency has progressed with regard to the operational objectives and performance 
targets  of  the  work  programme.  In  addition,  the  chairman  of  the  agency’s  steering  committee  informs  the 
Directors-General of the parent  DGs about the  decisions adopted by the committee and about any relevant 
subjects regarding the agency’s work.  

 To  which  extent  do  these  mechanism  ensure  that  there  is  an  adequate  flow  of  information  and 

collaboration between the REA and the parent DGs?  

 Do  the  relevant  stakeholders  agree  that  the  mechanisms  and  instruments  are  appropriate  in  ensuring 

adequate coordination and information flow between the REA and the Commission? 
 Extent to which the stakeholders agree that the information provided by the REA is  

o relevant, up to-date and of high-quality; 
o provided in a user friendly form, well structured, well communicated and easily accessible;  
o delivered and published on time; 

 What  other  possible  mechanisms  exist/are  needed  to  enable  the  flow  of  information  between  the 

Agency, and the Commission? 

4.2. Extent  to  which  the  activities  of  the  Executive  Agency  resulted  in  unintended  effects  (both 
desirable and undesirable) 

The main unintended effect that was found in the previous evaluation of the REA was the higher than initially 
expected staff turnover. In addition, in relation to the start of H2020 and a new mandate for the REA, there is 

26 

 
 
 
 
 
 
 
 
 
the risk of a drop in performance  due to the impact of  the  change  management  process. New programmes 
were delegated to the REA with a high impact on its organisational structure, including internal redeployment 
and  transfer  of  staff  from  the  Commission.  At  the  same  time,  the  new  rules  and  modalities  of  H2020  will 
bring about significant changes in the administration of the programmes. In particular, the roll-out of the new 
corporate IT tools poses a risk of reduced efficiency due to the need for bug-fixing and fine-tuning.  

 Have these changes and novelties produced any undesirable effects?  
 Is there any evidence of simplification measures being perceived negatively by the beneficiaries?  
 Is there any evidence of good practice examples of agency’s activities emerging that were identified by 

the stakeholders of the REA in the period between 2012 and 2015? 

5. Maintaining sufficient know-how in the Commission 

5.1. Extent  to  which  the  Commission,  in  the  presence  of  the  REA,  been  able  to  maintain  the 
adequate level of know-how in relation to programme(s) entrusted to the agency  

Officials  seconded  by  the  Commission  to  the  EAs  play  a  particular  part  in  the  organisational  setting  of  the 
Agencies. They are entrusted with positions of responsibility and are characterised by their twofold statutory 
link to both the Commission (as seconded officials) and the agency (as temporary agents). As a result, they 
contribute to the transfer of know-how between the Commission and the agency.  

 To  which  extent  does  the  seconded  staff  ensure  the  flow  of  know-how  between  the  REA  and  the 

Commission?  

 To what extent is this process adequate and efficient? 
 Is the know-how provided by the seconded officials of high quality and up-to date?  
 What barriers exist in the flow of know-how?  
 What other mechanisms exist or are needed to foster the flow of know-how between the REA and the 

Commission? 

 Examples of possible improvements that could be implemented in the process. 

5.2. Extent to which the monitoring and reporting arrangements in place enabled the Commission 
to benefit, in the short and medium term, from the know-how created within the REA 

The monitoring of REA’s activities is ensured through participation of the representatives of parent DGs in the 
steering  committee.  They  participate  in  the  steering  committee  meetings  and  review  and  scrutinise  all  the 
information  provided  by  the  director  of  the  Agency.  Their  role  also  involves  verifying  as  far  as  feasible 
whether the information provided by the REA is reliable and sufficient to draw management conclusions at DG 
level (e.g. DG’s annual activity reports). 

 To what extent is the monitoring information provided by the REA to the Commission clear, relevant, of 

high quality and timely?  

 Has the Commission benefited via monitoring and reporting arrangement in short or medium term?  
 To which extent the know-how helped to serve the needs of policy units of the Commission? 
 Are  there  any  examples  of  innovations  adopted  by  the  Commission  as  a  results  of  knowledge  sharing 

with REA? 

5.3. Adequacy  of  information  flow  and  communication  between  the  REA  and  the  Commission 
services (in particular the parent DGs) exist? 

The  annual  planning  and  reporting  cycle  form  the  basis  of  the  monitoring  of  the  agency’s  activities,  mainly 
through  the  agency’s  annual  work  programme,  annual  activity  report  and  interim  reporting.  The  content, 
format and frequency of reporting is defined in the Memorandum of Understanding between the Commission 
and  the  Agency.  Through  the  agency’s  reporting  obligations  the  DGs  are  always  kept  informed  of  the 
agency’s work and any management difficulties. 

 To what extent are the monitoring and reporting arrangements adequate?  
 What  other  arrangements  (apart  from  those  mentioned  above)  exist  between  the  REA  and  the 

Commission?  

 Are quality, clarity, relevancy  and  timeliness of  the  information provided to the Commission up to the 

highest standards?  

27 

 
 
 
 
 
 
 
 
 
 
 What checks are put in place for ensuring that the right information is communicated to the right person 

at the right time? 

5.4. Extent to which closure of the agency would result in losing significant know-how within the 
EC in relation to the management of programmes(s) entrusted to the REA 

Under certain circumstances the EA could be closed and the management of the programme returned to the 
Commission (e.g. the management would prove to be cost-inefficient).  

 Under which risks/scenarios would the agency be closed? How likely are these risks/scenarios? 
 Would this result in a loss of know-how within the Commission?  
 What types of know how would be lost?  

6. Management and provision of central support services 

Under Horizon 2020 the REA’s mandate was further extended to provide administrative and logistical support 
services to all entities involved in Horizon 2020 management; and also expanded to certain programmes for 
health/consumer  protection,  competitiveness  and  innovation,  as  well  as  education,  culture  and  citizenship. 
The REA was in charge of the following support services: 

 call planning 
 support for call publication 
 general logistical support for the evaluations 
 contracting and payment for independent  experts (expert evaluators and expert monitors (‘reviewers’, 

only for actions managed by the REA)) 

 validation of legal entities (legal validation and SME validation) 
 preparation of legal entities financial viability assessment 
 management of the Research Enquiry Service 
 centralised handling of evaluation review requests 
 maintenance of the National Contact Points database 

What have been the main challenges associated with the execution of the above-mentioned services (e.g. the 
validation of LEAR extended mandate remain a challenge in 2014 and 2015, in particular for calls with a lot of 
newcomers,  private  sector  participants  and  non-EU  participants;  also  expert  payments)?  What  corrective 
measures were taken to address them? What measures are taken to ensure the quality of external experts? 

Simplifications  (to  be)  introduced:  through  review  of  the  AAR’s,  AWPs  and  other  reports  we  identified  a 
number of simplifications introduced, including: 

 In H2020 and further to the introduction of the SME wizard and SME self-declaration, while systematic 

ex-ante validation of the SME status is no longer necessary. 

 Since the end of February 2014, the SME status has been based on a SME  self-declaration  through a 

web-based questionnaire.  

 Following the correction of some bugs after the release of the tool, developed by DG DIGIT and DG RTD, 
experts  can  now  sign  their  contracts  electronically.  Further  efficiency  gains  are  expected  once  the 
electronic workflow for the payment of experts is put in place in 2015. 

 The  Participant  Portal  is  the  single  entry  point  for  any  information  on  a  call,  proposal  submission  and 
evaluation, grant management, eligibility of costs, etc. Compared to FP7 the emphasis has been put on 
providing complete and detailed information for beneficiaries through the portal. 

 As  regards  grant  management,  each  process  and  procedure  is  explained  into  detail  in  the  annotated 

grant agreement. 

Have  any  other  simplification  measures  been  introduced?  What  effect  did  these  simplifications  have  for  the 
effectiveness  and  efficiency  of  REA’s  central  support  services?  Have  there  been  any  plans  to  introduce 
additional simplification measures? 

Key operational risks: the AARs and other reports mention some risks related to support services, such as the 
delayed  payment  of  experts  or  the  reputational  risk  linked  to  the  validation  process.  What  corrective 
measures were adopted to mitigate these risks, and were those measures successful? More specifically: 

28 

 
 
 
 
 
 
 
 
 
 
 The REA’s new mandate gave the agency full responsibility for all expert evaluators for H2020. Phasing 
in this service smoothly for all clients was a priority for the REA, in particular in the first half of 2014. 
The  timely  payment  of  experts  remained  a  challenge  in  2015.  Is  the  automation  of  the  workflow 
(COMPASS/EMI) expected to improve the handling of expert payments?  

 A  high  number  of  calls  took  place  during  the  second  half  of  2014  and  a  considerable  number  of  new 
participants had to be validated by the support services in 2015. In addition, following the H2020 Rules 
for  Participation  which  introduced  the  extended  mandate  for  the  LEAR,  validation  of  the  LEAR  is  also 
necessary  for  all  organisations  with  whom  the  research  services  wish  to  sign  a  grant  within  the  set 
target of 3 months.  

7. Lessons learned and previous recommendations 

The previous evaluation report made the following recommendations: 

 update REA’s mandate to improve efficiency of the FP7 administrative and logistical support services in 

view of preparing the organisation for further growth; 

 continue efforts to simplify procedures; 
 improve communication with stakeholders; 
 further improve efficiency of internal procedures and the use of IT tools; 
 improve staff motivation and career prospects; 
 improve forecasting for the administrative budget; 
 define and monitor performance against a limited number of measurable key performance indicators; 
 enhance the visibility of programmes managed by the REA. 

According to the 2013 AAR, the REA developed an action plan in response to the recommendations made in 
the  context  of  the  evaluation,  which  had  been  fully  implemented  by  the  end  of  the  year.  What  concrete 
measures  and  actions  were  taken  to  address  these  recommendations?  Is  it  possible  to  get  access  to  this 
action plan? 

29 

 
 
 
 
 
 
 
 
Annex 3. Questionnaires and aggregated data of REA beneficiaries surveys 

The questionnaire of the survey of REA grant beneficiaries: FP7 

Background information about the survey 

The purpose of this survey, which is being conducted as part of the ongoing evaluation of the Research 
Executive Agency, is to measure your satisfaction with the services provided by the Agency and identify areas 
for its performance improvement. 

The survey questionnaire consists of the following parts: the introductory section, specific sections dedicated 
to the application process, the grant finalisation phase, the REA validation services, REA’s communication and 
interaction with you and a general section on the overall performance of the Agency. 

No personal information will be revealed publicly, in full compliance with the Specific Privacy Statement on 
Personal Data Protection. 

Thank you in advance for your cooperation! 

I. 

 Introduction 

1. The organisation to which your research team belongs is a… 
We  define  a  research  team  as  a  group  of  researchers  involved  in  implementing  a  joint  research 
project  (including  FP  projects)  or  other  collaborative  research  activity. If  you are  involved in many 
different  research  teams  related  to  various  research  projects  and  activities,  please  refer  to  your  research 
team  from  your  department,  division  or  centre  whose  researchers  became  involved  in  the 
implementation  of  FP  project  […].  Research  teams  are  different  from  research  networks,  which  comprise  a 
number of research organisations. 

University or higher education institution 
Public or government sector, e.g. research performing 
organisation 
Private, not-for-profit sector, e.g. research foundation 
Private industry (including SMEs) 
Other (Please specify) 

2. What is the core scientific discipline or area of your research team?  (more than one answer option 
available for interdisciplinary teams) 

Physical sciences 
Engineering 
Life sciences 
Social sciences 
Humanities 
Other (please specify): 

3. Apart from the project [...], are/were you or your research team involved in any of the following 
programmes?  
Please  indicate  the  number  of  projects  (both  ongoing  and  completed)  and  applications  that  you  or  your 
research team were involved in. 

Name of the programme 

FP7 programmes  
Horizon 2020 programmes 

Number of projects 
(completed and 
ongoing) 

Number of 
applications 

30 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4.  To  the  best  of  your  knowledge,  please  describe  your  organisation’s  funding  sources  and  the 
administrative agreements of grant management using the scales (from 1 to 5) below.  
Values 1 and 5 are explained in the table; if you wish to choose an intermediate option, please select among 
2, 3 or 4. 

1 

1 

Relies  primarily  on  the 
organisation’s 
own 
funds 
Has  a  clearly  established 
and 
centralised 
institutional entity dealing 
with 
administrative 
aspects of EU grants  

2 

2 

3 

3 

4 

4 

5 

5 

Relies  primarily  on  third-party  funding 
(including project-based funding) 

Grant  management  is  decentralised  in  my 
organisation;  i.e.  each  department/research 
team  deals  with  administrative  aspects  of  EU 
grants independently 

5.  To  what  extent  were  you  involved  in  the  following  administrative  procedures  and  activities 
during [project name]? 

To a 
large 
extent  

To a 
moderate 
extent  

To some 
extent 

To a little 
extent 

Not at all 

Do not 
know/canno
t answer 

Application and conclusion of grant 
agreement  phase:  preparation  of 
proposal, preparation of supporting 
proposal 
documentation, 
submission, 
agreement 
negotiation and finalisation  

grant 

Grant  management  phase  – 
administrative and financial issues: 
preparation  of  interim  and  final 
including  cost  claims; 
reports, 
financial 
general 
administrative issues. 
Grant  management  phase  – 
scientific 
scientific 
issues: 
management and reporting. 

reporting; 

6. What are your main channels of information on EU research grants? (multiple answers are possible)  

National Contact Point  
Another national source (e.g. research ministry, an EU 
liaison office at my university) 
Recommendation by colleagues, superiors, etc. 
A European Commission website (e.g. FP7/H2020 portal, 
REA website or CORDIS) 
EC social media feed 
Online advertising (e.g. ads in search engines or social 
networking sites) 
EU event or promotional material (e.g. an info day or an 
EU info stand at a conference) 
Research Enquiry Service 
Media (e.g. a specialised magazine) 
My work or invitation as an expert evaluator 
Do not know 
Other (please specify)  

II. Application process 

8. Please assess the following aspects of the application process for [project name]. 

Strongly 
agree  

Rather 
agree  

Neither 
agree nor 
disagree 

Rather 
disagree 

Strongly 
disagree 

Do not 
know/cannot 
answer 

Information  for  applicants  (e.g. 

31 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Strongly 
agree  

Rather 
agree  

Neither 
agree nor 
disagree 

Rather 
disagree 

Strongly 
disagree 

Do not 
know/cannot 
answer 

the 

the 

about 
call  objectives, 
eligibility  and  selection  criteria 
or documentation needed) was 
easy to find  
Information  for  applicants  (e.g. 
about 
call  objectives, 
eligibility  and  selection  criteria 
or documentation needed) was 
clear 
I  knew  who  to  contact  for  any 
question(s)  I  had  or  where  to 
get  help  when  preparing  my 
application 
I  knew  who  to  contact  for  any 
question(s)  I  had  or  where  to 
get  help  when  submitting  my 
application 
The 
the 
application  process  (e.g.  the 
volume  of 
the  proposal  or 
for  supporting 
requirements 
documents)  were  reasonable 
and proportionate  
The  evaluation  process  was 
clear and transparent 
The  electronic  tool  used  for 
submitting  the  application  was 
user-friendly 
Other 
specify) 

requirements 

practicalities 

(please 

for 

9. How  would  you  assess  the  overall  timeliness  of  the  following  processes  during  the 

application stage of [project name]? 

Strongly 
agree  

Rather 
agree  

Neither 
agree 
nor 
disagree 

Rather 
disagree 

Strongly 
disagree 

Do not 
know/cannot 
answer 

from 

The  time  period  from  the  call 
deadline to the time the outcome of 
the proposal was announced to you 
(i.e. 
was 
time-to-inform) 
appropriate 
The 
the 
period 
time 
announcement  of  your  proposal’s 
outcome to the time you signed the 
time-to-
(i.e. 
grant  agreement 
contract) was appropriate 
from 
The  overall 
submission  of 
to 
signature  of  the  grant  agreement 
(i.e. time-to-grant) was appropriate 

the  proposal 

time  period 

10. Before  successfully  applying  for  [project  name],  had  you  unsuccessfully  applied  for  a 

similar grant in the same FP7 programme? If yes, how many times? 

More than 3 times 
2 to 3 times 
Once 
Never 

32 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
To what extent was a clear explanation provided for your application(s) not being selected? 
This question will show up only for those who replied ‘More than 3 times’, ‘2 to 3 times’ or ‘Once’ in the 
question above. 

Very clear 
Clear 
Neither clear nor unclear 
Not very clear 
Not clear at all 
Do not know/cannot answer 

10. Would you have any comments regarding the application phase for REA grants? In particular, 
what simplification and service improvement measures would you suggest? 

III. Grant finalisation phase  

11. To what extent do you agree with the following statements about the granting process of 
[project name]? 

Strongly 
agree  

Rather 
agree  

Neither 
agree nor 
disagree 

Rather 
disagree 

Strongly 
disagree 

Do not 
know/ 
cannot 
answer 

The  REA  staff  assigned  to  my 
project  in  the  grant  finalisation  and 
negotiation  phase  were  easily 
available and responsive 
Requests  from  the  REA  (e.g.  for 
proposal  modification  or  providing 
missing information) were clear 
The  electronic  tools  used  in  the 
process 
negotiation/contracting 
were user-friendly  
The  electronic  tools  used  for  the 
validation 
of 
(i.e. 
beneficiaries were user-friendly 
The  process  of  validating 
beneficiaries  was 
required reasonable effort 
Other practicalities (please specify) 

the 
smooth  and 

LEAR) 

12.  Would  you  have  any  comments  regarding  the  grant  finalisation  phase  of  REA  grants?  In 
particular, what simplification and service improvement measures would you suggest? 

IV. Project implementation  

13. Please assess the following aspects of [project name] project management life-cycle.  

33 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Strongly 
agree  

Rather 
agree  

Neither 
agree nor 
disagree 

Rather 
disagree 

Strongly 
disagree 

Do not 
know/ 
cannot 
answer 

from 

information 

the  REA 
administrative 

The 
concerning 
requirements was clear 
The  REA  staff  assigned  to  my 
project  during  the  implementation 
were easily available and responsive 
The  feedback  I  received  from  the 
REA  staff  on  the  progress  of  my 
project  (e.g.  in  a  mid-term  review) 
was useful 
Project reporting requirements were 
clear 
Project reporting requirements were 
reasonable and proportionate 
for 
The  electronic 
managing  my  grant  were  user-
friendly 
The  process  of  monitoring  our 
project  by  the  REA  was  clear  and 
transparent 
Other practicalities (please specify) 

tools  used 

14. To what extent do you agree with the following statements? 

Strongly 
agree  

Rather 
agree  

Neither 
agree 
nor 
disagree 

Rather 
disagree 

Strongly 
disagree 

Do not 
know/can
not 
answer 

the  agency 
requests 

For  interim  payments:  the  time  it 
to  process 
took 
payment 
and  make 
payments was appropriate 
For  grant  amendments:  the  time  it 
took  the  agency  to  process  grant 
amendment 
was 
appropriate 
For  the  final  payment:  the  time  it 
took  the  agency  to  process  the 
payment 
and  make 
payment was appropriate 

requests 

request 

15.  Would  you  have  any  comments  regarding  the  grant  implementation  phase  of  REA  grants?  In 
particular, what simplification and service improvement measures would you suggest? 

V. REA Validation Services 

16. Have you used the REA Validation Services? 
Yes 
No 
Do not know/cannot answer 

Was it: 
This question will show up only for those who replied ‘Yes’. 

34 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Yes  

No  

Do not 
know/cann
ot answer 

the 

financial  viability 

For the validation of a participant? 
For 
assessment? 
For 
the 
Representative’s 
mandate validation? 

Legal  Entity  Appointed 
extended 

(LEAR) 

check 

17. Before contacting the REA Validation Services, did you consult the information available on the 
Participant Portal for the validation of participants? 

No 
Do not know/cannot answer 

Yes 

Did you find the information: 
This question will show up only for those who replied ‘Yes’. 

Strongly 
agree  

Rather 
agree  

Neither 
agree nor 
disagree 

Rather 
disagree 

Strongly 
disagree 

Do not 
know/ 
cannot 
answer 

Easy to access? 
Clear and concise? 

18. Overall, how would you rate the support received from the Validation Services? 

Very useful 
Fairly useful 
Neutral 
Not very useful 
Not at all useful 
Do not know/cannot answer 

VI. REA’s communication and interaction with you 

19. To what extent have the following communication channels used by the REA provided you with 
relevant, helpful information when you needed it? 

To a large 
extent 

To a moderate 
extent 

To some 
extent 

To a little 
extent 

Not at 
all 

Does not 
apply to 
me 

Email contact 
Telephone contact 
Face-to-face contacts 
(meetings, events) 
Recorded video 
briefings 
Live web briefings 
(with a chat function) 
REA’s Facebook profile 
REA website 

20. Which of the following new means of communication that the REA could potentially use in the 
future would be important for you? 

Very 
important 

Rather 
important 

Rather 
unimportant 

Very 
unimportant 

Do not 
know/ 
cannot 
answer 

Neither 
important 
nor 
unimportant 

35 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Live 
videoconferencing 
Live teleconferencing 
REA email news 
alerts 
REA’s own presence 
in the social media 
(e.g. Twitter, 
Facebook) 
REA’s own presence 
in specialised 
networking platforms 
(e.g. LinkedIn, 
ResearchGate) 

21.  To  what  extent  do  you  agree  with  the  following  statements  regarding  the  Participant  Portal 
and the REA website? 

Strongly 
agree  

Rather 
agree  

Neither 
agree nor 
disagree 

Rather 
disagree 

Strongly 
disagree 

Do not 
know/ 
cannot 
answer 

The  e-services  via  the  Participant 
Portal are user-friendly 
The  REA  website  is  user-friendly, 
relevant documents are easy to find 
The 
usefully 
complements  other  EU  research 
sites (e.g. the Participant Portal) 

website 

REA 

22. When preparing your application for [project name] or thereafter, did you contact the Research 
Enquiry Service or National Contact Points? 

Yes, once  

Yes, more 
than once  

Never, 
even 
though I 
was aware 
of this 
service 

Never, 
because I 
was not 
aware of 
this service 

Other/do 
not know/ 
cannot 
answer 

Research Enquiry Service 
National Contact Points 

To what extent was the response from these services helpful? 
This question will show up only for those who replied ‘Yes, once’ or ‘Yes, more than once’ in the question 
above. 

Very helpful  

Helpful 

Neither 
helpful 
nor 
unhelpful 

Not very 
helpful 

Not 
helpful at 
all 

Do not 
know/ 
cannot 
answer 

Research Enquiry Service 
National Contact Points 

23. Would  you  agree  or  disagree  with  the  following  statements  regarding  your  awareness  on  the 
REA managed calls/grants? 

36 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Strongly 
agree 

Rather 
agree 

Rather 
disagree 

Strongly 
disagree 

Neither 
agree 
nor 
disagree 

Do not 
know/ 
cannot 
answer 

Overall,  funding  opportunities  for  the  REA-
managed programmes are well advertised 
When  applying  for  this  grant,  I  was  aware  that 
REA grants were funded from the EU budget  
When  applying  for  this  grant  I  was  aware  that 
the REA was entrusted to manage its grants by 
the European Commission 

VII. Overall performance of the Agency 

24. Overall, how satisfied are you with the Agency’s services?  

Very satisfied 
Satisfied 
Neither satisfied nor dissatisfied 
Dissatisfied 
Very dissatisfied 
Do not know/cannot answer 

25. How important are the following sets of factors to you when you deal with the Agency?  

Very 
important 

Rather 
important 

Neither 
important 
no 
unimportant 

Rather 
unimportant 

Very 
unimportant 

Do not 
know/cannot 
answer 

The appearance of the 
Agency’s staff, its website 
and communication materials 
The knowledge and courtesy 
of the Agency’s employees, 
their cooperation with you 
Accessibility and clarity of 
information provided by the 
Agency 
The Agency’s ability to 
perform the service 
promptly, accurately and 
transparently 
The Agency’s willingness to 
help you and provide 
individualised attention 

26. Based on your experience, how do you agree with the following statements concerning the 
Agency’s actual performance and the services it provides? 

Strongly 
agree 

Rather 
agree 

Neither 
agree nor 
disagree 

Rather 
disagree 

Strongly 
disagree 

Do not 
know/cannot 
answer 

The  Agency  strives  to  provide 
excellent 
programme 
management  and  high  quality 
service 
The  Agency’s  employees  are 
committed to doing quality work 
The  Agency’s  website 
and 
information materials are visually 
appealing and user-friendly 
Employees  in  the  Agency  are 
knowledgeable and competent 
Employees  in  the  Agency  are 
consistently courteous with you 
Employees  in  the  Agency  closely 

37 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
the 

information 

cooperate with you 
Information  provided  by 
Agency is easily accessible  
The  Agency’s  communications 
provide 
relevant  and  useful 
information  
Events (information days, project 
meetings, 
visits, 
etc.)  organised  by  the  Agency 
are useful 
Employees  in  the  Agency  give 
you prompt service 
The  Agency‘s  documents  are 
error free 
The  Agency’s  procedures  are 
transparent 
Employees  in  the  Agency  are 
always willing to help you 
When  you  have  a  problem,  the 
Agency  shows  a  sincere  interest 
in solving it 
The  Agency  has  employees  who 
give you personal attention 

27. Would you consider applying for EU research funding again? 

Yes, certainly 
Yes, maybe 
No, probably not 
No, certainly not 
Do not know/cannot answer 

What are the main reasons for indicating that you are unlikely to apply for EU-managed funding in 
the future? (multiple answers are possible) 
This question will show up only for those who replied ‘No, probably not’ or ‘No, certainly not’ in the question 
above. 

The eligibility requirements for proposals are too high 
The success rate for REA-managed applications is too low 
The application procedure is too complex 
The administrative requirements for managing REA grants 
are too heavy 
Other reasons (please specify) 
Do not know/cannot answer 

28. Do you have any final comments and suggestions for improvements in the quality of the REA 
services which have not been addressed in this questionnaire? 

Thank you for your answers! 

The questionnaire of the survey of REA beneficiaries: Horizon 2020 

Background information about the survey 

The purpose of this survey, which is being conducted as part of the ongoing evaluation of the Research 
Executive Agency, is to measure your satisfaction with the services provided by the Agency and identify areas 
for its performance improvement. 

38 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
The survey questionnaire consists of the following parts: the introductory section, specific sections dedicated 
to the application process, the grant finalisation phase, the REA validation services, REA’s communication and 
interaction with you and a general section on the overall performance of the Agency. 

No personal information will be revealed publicly, in full compliance with the Specific Privacy Statement on 
Personal Data Protection. 

Thank you in advance for your cooperation! 

I. Survey questionnaire: REA beneficiaries 

1. The organisation to which your research team belongs is a… 
We  define  a  research  team  as  a  group  of  researchers  involved  in  implementing  a  joint  research 
project  (including  FP  projects)  or  other  collaborative  research  activity. If  you are  involved in many 
different  research  teams  related  to  various  research  projects  and  activities,  please  refer  to  your  research 
team  from  your  department,  division  or  centre  whose  researchers  became  involved  in  the 
implementation  of  FP  project  […].  Research  teams  are  different  from  research  networks,  which  comprise  a 
number of research organisations. 

University or higher education institution 
Public or government sector, e.g. research performing 
organisation 
Private, not-for-profit sector, e.g. research foundation 
Private industry (including SMEs) 
Other (Please specify) 

2. What is the core scientific discipline or area of your research team?  (More than one answer option 
available for interdisciplinary teams) 

Physical sciences 
Engineering 
Life sciences 
Social sciences 
Humanities 
Other (please specify): 

3. Apart from the project [...], are/were you or your research team involved in any of the following 
programmes?  
Please  indicate  the  number  of  projects  (both  ongoing  and  completed)  and  applications  that  you  or  your 
research team were involved in. 

Name of the programme 

FP7 programmes  
Horizon 2020 programmes 

Number of projects 
(completed and 
ongoing) 

Number of 
applications 

4.  To  the  best  of  your  knowledge,  please  describe  your  organisation’s  funding  sources  and  the 
administrative management of grant agreements using the scales (from 1 to 5) below. 
Values 1 and 5 are explained in the table; if you wish to choose an intermediate option, please select among 
2, 3 or 4. 
Relies  primarily  on  the 
organisation’s 
own 
funds 
Has  a  clearly  established 
centralised 
and 
entity 
institutional 
dealing 
with 
administrative  aspects  of 
EU grants  

Grant  management  is  decentralised  in  my 
organisation;  i.e.  each  department/research 
team  deals  with  administrative  aspects  of  EU 
grants independently 

Relies  primarily  on  third-party  funding 
(including project-based funding) 

4 

5 

1 

4 

1 

5 

3 

2 

2 

3 

5.  To  what  extent  were  you  involved  in  the  following  administrative  procedures  and  activities 
during [project name]? 

39 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
To a large 
extent  

To a 
moderate 
extent  

To some 
extent 

To a little 
extent 

Not at all 

Do not 
know/cannot 
answer 

preparation 

Application and conclusion of grant 
agreement  phase:  preparation  of 
proposal, 
of 
supporting 
documentation, 
proposal submission,  conclusion of 
grant agreement 
Grant  management  phase  – 
administrative 
financial 
issues:  preparation  of  interim  and 
final reports, including cost claims; 
general 
reporting; 
financial 
administrative issues 
Grant  management  phase  – 
scientific 
scientific 
issues: 
management and reporting 

and 

6. What are your main channels of information on EU research grants? (Multiple answers are possible)  

National Contact Point  
Another national source (e.g. research ministry, an EU 
liaison office at my university) 
Recommendation by colleagues, superiors etc. 
A European Commission website (e.g. FP7/H2020 portal, 
REA website, CORDIS…) 
EC social media feed 
Online advertising (e.g. ads in search engines or social 
networking sites) 
EU event or promotional material (e.g. an info day, an EU 
info stand at a conference etc.) 
Research Enquiry Service 
Media (e.g. a specialised magazine) 
My work or invitation as an expert evaluator 
Do not know 
Other (please specify)  

II. Application process 

7. Please assess the following aspects of [project name] application process: 

Strongly 
agree  

Rather 
agree  

Neither 
agree 
nor 
disagree 

Rather 
disagree 

Strongly 
disagree 

Do not 
know/cannot 
answer 

selection 

selection 

Information 
for  applicants  (e.g. 
about  the  call  objectives,  eligibility 
and 
criteria, 
documentation  needed,  etc.)  was 
easy to find  
Information 
for  applicants  (e.g. 
about  the  call  objectives,  eligibility 
and 
criteria, 
documentation  needed,  etc.)  was 
clear 
I  knew  who  to  contact  for  any 
question(s)  I  had  or  where  to  get 
preparing  my 
help 
application 
I  knew  who  to  contact  for  any 
question(s)  I  had  or  where  to  get 
help  when 
submitting  my 
application 

when 

40 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Strongly 
agree  

Rather 
agree  

Neither 
agree 
nor 
disagree 

Rather 
disagree 

Strongly 
disagree 

Do not 
know/cannot 
answer 

(e.g. 

requirements 

The  requirements  for  application 
the  volume  of 
process 
proposal, 
for 
supporting  documents,  etc.)  were 
reasonable and proportionate  
The  evaluation  process  was  clear 
and transparent 
The  electronic 
submitting 
user-friendly 
Other practicalities (please specify) 

tool  used 
for 
the  application  was 

8.  How  would  you  assess  the  overall  timeliness  of  the  following  processes  during  the  application 
stage of [project name]? 

Strongly 
agree  

Rather 
agree  

Neither 
agree 
nor 
disagree 

Rather 
disagree 

Strongly 
disagree 

Do not 
know/cannot 
answer 

from 

The  time  period 
from  the  call 
deadline to the time the outcome of 
the proposal was announced to you 
(i.e. 
was 
time-to-inform) 
appropriate 
the 
period 
time 
The 
announcement  of  your  proposal’s 
outcome to the time you signed the 
grant  agreement 
time-to-
(i.e. 
contract) was appropriate 
from 
The  overall 
submission  of 
to 
signature  of  the  grant  agreement 
(i.e. time-to-grant) was appropriate 

the  proposal 

time  period 

9.  Before  successfully  applying  for  [project  name],  had  you  unsuccessfully  applied  for  a  similar 
grant in the predecessor FP7 programme? If yes, how many times? 

More than 3 times 
2 to 3 times 
Once 
Never 

To what extent was a clear explanation provided for your application not being selected? 
This question will show up only for those who replied ‘More than 3 times’, ‘2 to 3 times’ or ‘Once’ in the 
question above. 

Very clear 
Clear 
Neither clear nor unclear 
Not very clear 
Not clear at all 
Do not know/cannot answer 

10.  Would  you  have  any  comments  regarding  the  application  phase  of  REA  grants?  In  particular, 
what simplification and service improvement measures would you suggest? 

III. Grant finalisation phase  

41 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
11. To what extent do you agree with the following statements about the granting process of 
[project name]? 

Strongly 
agree  

Rather 
agree  

Neither 
agree nor 
disagree 

Rather 
disagree 

Strongly 
disagree 

Do not 
know/ 
cannot 
answer 

The  REA  staff  assigned  to  my 
project  in  the  grant  finalisation  and 
negotiation  phase  were  easily 
available and responsive 
Requests  from  the  REA  (e.g.  for 
proposal  modification,  providing 
missing 
information,  etc.)  were 
clear 
The  electronic  tools  used  in  the 
contracting  process  were  user-
friendly  
The  electronic  tools  used  for  the 
of 
(i.e. 
validation 
beneficiaries were user-friendly 
The  process  of  validating 
beneficiaries  was 
required reasonable effort 
Other practicalities (please specify) 

the 
smooth  and 

LEAR) 

12.  Would  you  have  any  comments  regarding  the  grant  finalisation  phase  of  REA  grants?  In 
particular, what simplification and service improvement measures would you suggest? 

IV. REA Validation Services 

13. Have you used the REA Validation Services? 

Yes 
No 
Do not know/cannot answer 

Was it: 
This question will show up only for those who replied ‘Yes’ in the question above. 

Yes  

No  

Do not 
know/cannot 
answer 

the 

financial  viability 

For the validation of a participant? 
For 
assessment? 
For 
the 
Representative’s 
mandate validation? 

Legal  Entity  Appointed 
extended 

(LEAR) 

check 

14. Before contacting the REA Validation Services, did you consult the information available on the 
Participant Portal for the validation of participants? 

Yes 
No 
Do not know/cannot answer 

Did you find the information: 

42 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
This question will show up only for those who replied ‘Yes’ in the question above. 

Strongly 
agree  

Rather 
agree  

Neither 
agree nor 
disagree 

Rather 
agree 

Strongly 
agree 

Do not 
know/ 
cannot 
answer 

Easy to access? 
Clear and concise? 

15. Overall, how would you rate the support received from the Validation Services? 
Very useful 
Fairly useful 
Neutral 
Not very useful 
Not at all useful 
Do not know/cannot answer 

V. REA’s communication and interaction with you 

16. To what extent have the following communication channels used by the REA provided you with 
relevant, helpful information when you needed it? 
To a large 
extent 

To a moderate 
extent 

To a little 
extent 

To some 
extent 

Not at 
all 

Does not 
apply to 
me 

Email contact 
Telephone contact 
Face-to-face contacts 
(meetings, events) 
Recorded video 
briefings 
Live web briefings (with 
a chat function) 
REA’s Facebook profile 
REA’s website 

17. Which of the following new means of communication that the REA could potentially use in the 
future would be important for you? 

Very 
important 

Rather 
important 

Neither 
important 
nor 
unimportant 

Rather 
unimportant 

Very 
unimportant 

Do 
not/cannot 
answer 
know 

Live 
videoconferencing 
Live teleconferencing 
REA email news 
alerts 
REA’s own presence 
in the social media 
(e.g. Twitter, 
Facebook) 
REA’s own presence 
in specialised 
networking 
platforms (e.g. 
LinkedIn, 
ResearchGate) 

18.  To  what  extent  do  you  agree  with  the  following  statements  regarding  the  Participant  Portal 
and the REA website? 

Strongly 
agree  

Rather 
agree  

Neither 
agree nor 
disagree 

Rather 
disagree 

Strongly 
disagree 

Do not 
know/ 
cannot 
answer 

The  e-services  via  the  Participant 

43 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Strongly 
agree  

Rather 
agree  

Neither 
agree nor 
disagree 

Rather 
disagree 

Strongly 
disagree 

Do not 
know/ 
cannot 
answer 

Portal are user-friendly 
The  REA  website  is  user-friendly, 
relevant documents are easy to find 
The 
usefully 
complements  other  EU  research 
sites (e.g. the Participant Portal) 

website 

REA 

19. When preparing your application for [project name] or thereafter, did you contact the Research 
Enquiry Service or National Contact Points? 

Yes, once  

Yes, more 
than once  

Never, 
even 
though I 
was aware 
of this 
service 

Never, 
because I 
was not 
aware of 
this service 

Other/do 
not know/ 
cannot 
answer 

Research Enquiry Service 
National Contact Points 

To what extent was the response from these services helpful? 
This question will show up only for those who replied ‘Yes, once’ or ‘Yes, more than once’ in the question 
above. 

Very helpful  

Helpful 

Neither 
helpful 
nor 
unhelpful 

Not very 
helpful 

Not 
helpful at 
all 

Do not 
know/ 
cannot 
answer 

Research Enquiry Service 
National Contact Points 

20. Would  you  agree  or  disagree  with  the  following  statements  regarding  your  awareness  on  the 
REA-managed calls/grants? 

Strongly 
agree 

Rather 
agree 

Rather 
disagree 

Strongly 
disagree 

Neither 
agree 
nor 
disagree 

Do not 
know/ 
cannot 
answer 

Overall,  funding  opportunities  for  the  REA-
managed programmes are well advertised 
When  applying  for  this  grant,  I  was  aware  that 
the REA grants were funded from EU budget  
When  applying  for  this  grant  I  was  aware  that 
the REA was entrusted to manage its grants by 
the European Commission 

VI. Overall performance of the Agency 

21. Overall, how satisfied are you with the Agency’s services?  

Very satisfied 
Satisfied 
Neither satisfied nor dissatisfied 
Dissatisfied 
Very dissatisfied 
Do not know/cannot answer 

22. How important are the following sets of factors to you when you deal with the Agency?  

44 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Very 
important 

Rather 
important 

Neither 
important 
no 
unimporta
nt 

Rather 
unimpor
tant 

Very 
unimporta
nt 

Do not 
know/ca
nnot 
answer 

The appearance of the 
Agency’s staff, its website 
and communication 
materials 
The knowledge and 
courtesy of the Agency’s 
employees, their 
cooperation with you 
Accessibility and clarity of 
information provided by 
the Agency 
The Agency’s ability to 
perform the service 
promptly, accurately and 
transparently 
The Agency’s willingness to 
help you and provide 
individualised attention 

23. Based on your experience, how do you agree with the following statements concerning the 
Agency’s actual performance and the services it provides? 

Strongly 
agree 

Rather 
agree 

Neither 
agree 
nor 
disagree 

Rather 
disagree 

Strongly 
disagree 

Do not 
know/cannot 
answer 

in 

The  Agency  strives  to  provide 
excellent 
programme 
management  and  high  quality 
service 
The  Agency’s  employees  are 
committed 
to  doing  quality 
work 
The  Agency’s  website  and 
information  materials 
are 
visually  appealing  and  user-
friendly 
Employees  in  the  Agency  are 
knowledgeable and competent 
Employees  in  the  Agency  are 
consistently courteous with you 
Employees 
the  Agency 
closely cooperate with you 
Information  provided  by  the 
Agency is easily accessible  
The  Agency’s  communications 
provide  relevant  and  useful 
information  
Events 
days, 
(information 
project  meetings,  information 
visits,  etc.)  organised  by  the 
Agency are useful 
Employees  in  the  Agency  give 
you prompt service 
The  Agency‘s  documents  are 
error free 
The  Agency’s  procedures  are 
transparent 
Employees  in  the  Agency  are 
always willing to help you 
When  you  have  a  problem,  the 
Agency shows a sincere interest 
in solving it 
The Agency has employees who 
give you personal attention 

45 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
24. Would you consider applying for EU research funding again? 

Yes, certainly 
Yes, maybe 
No, probably not 
No, certainly not 
 Do not know/cannot answer 

What are the main reasons for indicating that you are unlikely to apply for EU managed funding in 
the future? (Multiple answers are possible) 
This question will show up only for those who replied ‘No, probably not’ or ‘No, certainly not’ in the question 
above. 

The eligibility requirements for proposals are too high 
The success rate for REA managed applications is too low 
The application procedure is too complex 
The administrative requirements for managing REA grants 
are too heavy 
Other reasons (please specify) 
 DO not know/cannot answer 

25. Do you have any final comments and suggestions for improvements in the quality of the REA 
services which have not been addressed in this questionnaire? 

Thank you for your answers! 

Aggregated data of REA beneficiaries surveys (FP7 and Horizon 2020) 

I. 

 Introduction 

1. The organisation to which your research team belongs is a… 
We  define  a  research  team  as  a  group  of  researchers  involved  in  implementing  a  joint  research 
project  (including  FP  projects)  or  other  collaborative  research  activity. If  you are  involved in many 
different  research  teams  related  to  various  research  projects  and  activities,  please  refer  to  your  research 
team  from  your  department,  division  or  centre  whose  researchers  became  involved  in  the 
implementation  of  FP  project  […].  Research  teams  are  different  from  research  networks,  which  comprise  a 
number of research organisations. 

University or higher education institution 
Public or government sector, e.g. research performing 
organisation 
Private, not-for-profit sector, e.g. research foundation 
Private industry (including SMEs) 
Other (Please specify) 

703 

195 

124 
111 
5 

2. What is the core scientific discipline or area of your research team?  (more than one answer option 
available for interdisciplinary teams) 

Physical sciences 
Engineering 
Life sciences 
Social sciences 
Humanities 
Other (please specify): 

323 
339 
412 
178 
108 
11 

3. Apart from the project [...], are/were you or your research team involved in any of the following 
programmes?  

46 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Please  indicate  the  number  of  projects  (both  ongoing  and  completed)  and  applications  that  you  or  your 
research team were involved in. 

Name of the programme 

FP7 programmes  
Horizon 2020 programmes 

Number of projects 
(completed and 
ongoing) 
1906 
619 

Number of 
applications 

3212 
2029 

4.1.  To  the  best  of  your  knowledge,  please  describe  your  organisation’s  funding  sources  and  the 
administrative agreements of grant management using five point scale, where 1 stands for  ‘Relies 
primarily on the organisation’s own funds’ and 5 stands for ‘Relies primarily on third-party funding 
(including project-based funding)’. 

1312 

1085 

876 

1400

1200

1000

800

600

400

200

0

378 

112 

1

2

3

4

5

4.2.  To  the  best  of  your  knowledge,  please  describe  your  organisation’s  funding  sources  and  the 
administrative agreements of grant management using five point scale, where 1 stands for  ‘Has a 
clearly  established  and  centralised  institutional  entity  dealing  with  administrative  aspects  of  EU 
grants’  and  5  stands  for  ‘Grant  management  is  decentralised  in  my  organisation;  i.e.  each 
department/research team deals with administrative aspects of EU grants independently ‘. 

538 

576 

548 

615 

417 

700

600

500

400

300

200

100

0

1

2

3

4

5

5.  To  what  extent  were  you  involved  in  the  following  administrative  procedures  and  activities 
during [project name]? 

47 

 
 
 
 
 
 
 
 
 
 
 
To a large 
extent  

To a 
moderate 
extent  

To some 
extent 

To a little 
extent 

Not at all 

Do not 
know/ca
nnot 
answer 

704 

198 

109 

82 

40 

5 

438 

270 

187 

142 

43 

58 

656 

189 

98 

89 

53 

53 

Application and conclusion of grant 
agreement  phase:  preparation  of 
proposal, preparation of supporting 
proposal 
documentation, 
submission, 
agreement 
negotiation and finalisation  

grant 

Grant  management  phase  – 
administrative and financial issues: 
preparation  of  interim  and  final 
including  cost  claims; 
reports, 
financial 
general 
administrative issues. 
Grant  management  phase  – 
scientific 
scientific 
issues: 
management and reporting. 

reporting; 

6. What are your main channels of information on EU research grants? (multiple answers are possible) 

National Contact Point  
Another national source (e.g. research ministry, an EU 
liaison office at my university) 
Recommendation by colleagues, superiors, etc. 
A European Commission website (e.g. FP7/H2020 portal, 
REA website or CORDIS) 
EC social media feed 
Online advertising (e.g. ads in search engines or social 
networking sites) 
EU event or promotional material (e.g. an info day or an 
EU info stand at a conference) 
Research Enquiry Service 
Media (e.g. a specialised magazine) 
My work or invitation as an expert evaluator 
Do not know 
Other (please specify)  

II. Application process 

525 

434 

524 

800 

48 

58 

204 

44 
22 
138 
7 
5 

7. Please assess the following aspects of the application process for [project name]. 

criteria 

selection 

Information 
for  applicants  (e.g. 
about  the  call  objectives,  eligibility 
or 
and 
documentation  needed)  was  easy 
to find  
Information 
for  applicants  (e.g. 
about  the  call  objectives,  eligibility 
or 
and 
documentation needed) was clear 
I  knew  who  to  contact  for  any 
question(s)  I  had  or  where  to  get 
preparing  my 
help 
application 
I  knew  who  to  contact  for  any 
question(s)  I  had  or  where  to  get 
submitting  my 
help  when 

selection 

criteria 

when 

Strongly 
agree  

Rather 
agree  

Neither 
agree nor 
disagree 

Rather 
disagree 

Strongly 
disagree 

Do not 
know/cannot 
answer 

403 

517 

109 

43 

10 

56 

358 

554 

109 

52 

13 

52 

332 

404 

196 

101 

33 

355 

408 

198 

81 

24 

72 

72 

48 

 
 
 
 
  
 
 
 
 
 
 
documents) 

(e.g. 
or 

requirements 

application 
The  requirements  for  application 
the  volume  of 
process 
for 
proposal 
supporting 
were 
reasonable and proportionate  
The  evaluation  process  was  clear 
and transparent 
The  electronic 
submitting 
user-friendly 

tool  used 
for 
the  application  was 

Strongly 
agree  

Rather 
agree  

Neither 
agree nor 
disagree 

Rather 
disagree 

Strongly 
disagree 

Do not 
know/cannot 
answer 

262 

485 

199 

100 

34 

320 

448 

200 

305 

442 

168 

67 

95 

29 

50 

58 

74 

78 

8.  How  would  you  assess  the  overall  timeliness  of  the  following  processes  during  the  application 
stage of [project name]? 

The  time  period 
from  the  call 
deadline to the time the outcome of 
the proposal was announced to you 
(i.e. 
was 
time-to-inform) 
appropriate 
The 
the 
period 
time 
announcement  of  your  proposal’s 
outcome to the time you signed the 
grant  agreement 
time-to-
(i.e. 
contract) was appropriate 
The  overall 
from 
to 
submission  of 
signature  of  the  grant  agreement 
(i.e. time-to-grant) was appropriate 

the  proposal 

time  period 

from 

Strongly 
agree  

Rather 
agree  

Neither 
agree nor 
disagree 

Rather 
disagree 

Strongly 
disagree 

Do not 
know/cannot 
answer 

307 

527 

144 

93 

9 

58 

338 

494 

136 

93 

30 

47 

285 

501 

162 

102 

37 

51 

9.  Before  successfully  applying  for  [project  name],  had  you  unsuccessfully  applied  for  a  similar 
grant in the same FP7 programme? If yes, how many times? 

More than 3 times 
2 to 3 times 
Once 
Never 

70 
173 
272 
623 

To what extent was a clear explanation provided for your application(s) not being selected? 
This question showed up only for those who replied ‘More than 3 times’, ‘2 to 3 times’ or ‘Once’ in the question 
above. 

Very clear 
Clear 
Neither clear nor unclear 
Not very clear 
Not clear at all 
Do not know/cannot answer 

III. Grant finalisation phase  

33 
231 
109 
90 
25 
21 

11.  To  what  extent  do  you  agree  with  the  following  statements  about  the  granting  process  of 
[project name]? 

49 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Strongly 
agree  

Rather 
agree  

Neither 
agree nor 
disagree 

Rather 
disagree 

Strongly 
disagree 

Do not 
know/ 
cannot 
answer 

460 

413 

110 

44 

15 

96 

398 

408 

142 

57 

240 

420 

206 

110 

202 

381 

221 

103 

212 

405 

209 

87 

11 

43 

42 

36 

122 

119 

189 

189 

the  grant 

The REA staff assigned to my project 
finalisation  and 
in 
negotiation  phase  were 
easily 
available and responsive 
Requests  from  the  REA  (e.g.  for 
proposal  modification  or  providing 
missing information) were clear 
The  electronic  tools  used  in  the 
negotiation/contracting process were 
user-friendly  
The  electronic  tools  used  for  the 
validation (i.e. LEAR) of beneficiaries 
were user-friendly 
The  process  of  validating 
smooth 
beneficiaries  was 
required reasonable effort 

the 
and 

IV. Project implementation  

13. Please assess the following aspects of [project name] project management life-cycle.  

Strongly 
agree  

Rather 
agree  

Neither 
agree nor 
disagree 

Rather 
disagree 

Strongly 
disagree 

Do not 
know/ 
cannot 
answer 

182 

305 

246 

212 

90 

95 

202 

215 

109 

201 

205 

280 

276 

87 

98 

152 

256 

121 

206 

212 

110 

38 

36 

27 

46 

32 

57 

52 

15 

21 

20 

16 

13 

28 

14 

28 

48 

85 

28 

34 

44 

64 

the 

from 

information 

the  REA 
administrative 

The 
concerning 
requirements was clear 
The REA staff assigned to my project 
during 
implementation  were 
easily available and responsive 
The  feedback  I  received  from  the 
REA  staff  on  the  progress  of  my 
project  (e.g.  in  a  mid-term  review) 
was useful 
Project  reporting  requirements  were 
clear 
Project  reporting  requirements  were 
reasonable and proportionate 
The  electronic 
for 
managing  my  grant  were  user-
friendly 
The  process  of  monitoring  our 
project  by  the  REA  was  clear  and 
transparent 

tools  used 

14. To what extent do you agree with the following statements? 

the  agency 

For  interim  payments:  the  time  it 
took 
to  process 
payment 
requests  and  make 
payments was appropriate 

Strongly 
agree  

Rather 
agree  

Neither 
agree nor 
disagree 

Rather 
disagree 

Strongly 
disagree 

Do not 
know/ca
nnot 
answer 

189 

226 

69 

32 

13 

129 

50 

 
 
 
 
 
 
 
 
 
 
 
 
 
Strongly 
agree  

Rather 
agree  

Neither 
agree nor 
disagree 

Rather 
disagree 

Strongly 
disagree 

Do not 
know/ca
nnot 
answer 

110 

176 

104 

33 

26 

209 

89 

123 

81 

28 

12 

325 

requests 

For grant amendments: the time it 
took  the  agency  to  process  grant 
amendment 
was 
appropriate 
For  the  final  payment:  the  time  it 
took  the  agency  to  process  the 
payment 
and  make 
payment was appropriate 

request 

V. REA Validation Services 

16. Have you used the REA Validation Services? 

Yes 
No 
Do not know/cannot answer 

Was it: 
This question showed up only for those who replied ‘Yes’. 

Yes  

No  

For the validation of a participant? 
For the financial viability check assessment? 
For the Legal Entity Appointed Representative’s 
(LEAR) extended mandate validation? 

148 
106 

142 

34 
60 

32 

183 
294 
661 

Do not 
know/cannot 
answer 
11 
26 

16 

17. Before contacting the REA Validation Services, did you consult the information available on the 
Participant Portal for the validation of participants? 

Yes 
No 
Do not know/cannot answer 

Did you find the information: 
This question showed up only for those who replied ‘Yes’. 

Strongly 
agree  

Rather 
agree  

Neither 
agree nor 
disagree 

Rather 
disagree 

Strongly 
disagree 

Easy to access? 
Clear and concise? 

35 
25 

60 
61 

22 
29 

22 
24 

7 
7 

18. Overall, how would you rate the support received from the Validation Services? 

Very useful 

Fairly useful 
Neutral 
Not very useful 
Not at all useful 
Do not know/cannot answer 

VI. REA’s communication and interaction with you 

51 

149 
22 
14 

Do not 
know/ 
cannot 
answer 
3 
3 

50 
72 
42 
11 
3 
5 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
19. To what extent have the following communication channels used by the REA provided you with 
relevant, helpful information when you needed it? 

To a large 
extent 

To a moderate 
extent 

To some 
extent 

To a little 
extent 

Not at 
all 

Email contact 
Telephone contact 
Face-to-face contacts 
(meetings, events) 
Recorded video 
briefings 
Live web briefings 
(with a chat function) 
REA’s Facebook profile 
REA website 

705 
147 

104 

8 

10 

3 
123 

210 
168 

120 

53 

42 

13 
211 

98 
133 

116 

64 

53 

26 
209 

35 
121 

109 

50 

39 

34 
103 

8 
223 

281 

411 

421 

473 
202 

Does not 
apply to 
me 
82 
346 

408 

552 

573 

589 
290 

20. Which of the following new means of communication that the REA could potentially use in the 
future would be important for you? 

Very 
important 

Rather 
important 

Neither 
important 
nor 
unimportant 

Rather 
unimportant 

Very 
unimportant 

129 

118 

261 

28 

343 

301 

435 

96 

224 

257 

185 

239 

145 

158 

73 

220 

Do not 
know/ 
cannot 
answer 

189 

198 

121 

108 

106 

63 

341 

214 

61 

171 

236 

209 

263 

198 

Live 
videoconferencing 
Live teleconferencing 
REA email news 
alerts 
REA’s own presence 
in the social media 
(e.g. Twitter, 
Facebook) 
REA’s own presence 
in specialised 
networking platforms 
(e.g. LinkedIn, 
ResearchGate) 

21.  To  what  extent  do  you  agree  with  the  following  statements  regarding  the  Participant  Portal 
and the REA website? 

The  e-services  via  the  Participant 
Portal are user-friendly 
The  REA  website  is  user-friendly, 
relevant documents are easy to find 
The 
usefully 
complements  other  EU  research 
sites (e.g. the Participant Portal) 

website 

REA 

Strongly 
agree  

Rather 
agree  

Neither 
agree nor 
disagree 

Rather 
disagree 

Strongly 
disagree 

190 

130 

130 

512 

408 

181 

267 

118 

135 

374 

308 

76 

52 

48 

32 

Do not 
know/ 
cannot 
answer 

85 

150 

218 

22. When preparing your application for [project name] or thereafter, did you contact the Research 
Enquiry Service or National Contact Points? 

Yes, once  

Yes, more 
than once  

Never, even 
though I was 
aware of this 
service 

Research Enquiry Service 
National Contact Points 

54 
166 

120 
328 

313 
365 

To what extent was the response from these services helpful? 

52 

Never, 
because I 
was not 
aware of this 
service 
383 
114 

Other/do not 
know/ 
cannot 
answer 

268 
165 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
This  question  will  show  up  only  for  those  who  replied  ‘Yes,  once’  or  ‘Yes,  more  than  once’  in  the  question 
above. 

Very 
helpful  

Helpful 

Research Enquiry Service 
National Contact Points 

61 
210 

99 
230 

Neither 
helpful 
nor 
unhelpful 
13 
33 

Not very 
helpful 

Not 
helpful at 
all 

1 
15 

2 
6 

Do not 
know/ 
cannot 
answer 
0 
0 

23. Would  you  agree  or  disagree  with  the  following  statements  regarding  your  awareness  on  the 
REA managed calls/grants? 

Overall,  funding opportunities  for  the  REA-
managed programmes are well advertised 
When  applying  for  this  grant,  I  was  aware 
that  REA  grants  were  funded  from  the  EU 
budget  
When  applying  for  this  grant  I  was  aware 
that  REA  was  entrusted  to  manage  its 
grants by the European Commission 

Strongly 
agree 

Rather 
agree 

Rather 
disagree 

Strongly 
disagree 

Neither 
agree 
nor 
disagree 

208 

506 

218 

694 

256 

60 

96 

18 

14 

17 

Do not 
know/ 
cannot 
answer 

96 

93 

580 

262 

95 

49 

36 

116 

VII. Overall performance of the Agency 

24. Overall, how satisfied are you with the Agency’s services?  

Very satisfied 
Satisfied 
Neither satisfied nor dissatisfied 
Dissatisfied 
Very dissatisfied 
Do not know/cannot answer 

257 
650 
161 
31 
9 
30 

25. How important are the following sets of factors to you when you deal with the Agency?  

The appearance of the 
Agency’s staff, its website 
and communication 
materials 
The knowledge and 
courtesy of the Agency’s 
employees, their 
cooperation with you 
Accessibility and clarity of 
information provided by the 
Agency 
The Agency’s ability to 
perform the service 
promptly, accurately and 
transparently 
The Agency’s willingness to 
help you and provide 
individualised attention 

Very 
important 

Rather 
important 

Neither 
important 
no 
unimporta
nt 

Rather 
unimporta
nt 

Very 
unimporta
nt 

Do not 
know/can
not answer 

271 

533 

193 

48 

26 

67 

671 

353 

821 

237 

796 

254 

738 

297 

46 

35 

38 

46 

53 

8 

2 

1 

3 

1 

1 

1 

1 

59 

42 

48 

53 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
26. Based on your experience, how do you agree with the following statements concerning the 
Agency’s actual performance and the services it provides? 

Strongly 
agree 

Rather 
agree 

Neither 
agree 
nor 
disagree 

Rather 
disagree 

Strongly 
disagree 

Do not 
know/cannot 
answer 

The  Agency  strives  to  provide 
excellent 
programme 
management  and  high  quality 
service 
The  Agency’s  employees  are 
committed to doing quality work 
The  Agency’s  website  and 
information  materials 
are 
visually  appealing  and  user-
friendly 
Employees  in  the  Agency  are 
knowledgeable and competent 
Employees  in  the  Agency  are 
consistently courteous with you 
Employees in the Agency closely 
cooperate with you 
Information  provided  by  the 
Agency is easily accessible  
The  Agency’s  communications 
relevant  and  useful 
provide 
information  
days, 
Events 
project  meetings, 
information 
visits,  etc.)  organised  by  the 
Agency are useful 
Employees  in  the  Agency  give 
you prompt service 
The  Agency‘s  documents  are 
error free 
The  Agency’s  procedures  are 
transparent 
Employees  in  the  Agency  are 
always willing to help you 
When  you  have  a  problem,  the 
Agency shows a sincere interest 
in solving it 
The  Agency  has employees  who 
give you personal attention 

(information 

311 

536 

154 

394 

447 

152 

150 

432 

291 

354 

457 

326 

185 

215 

466 

406 

375 

471 

151 

114 

231 

260 

498 

252 

201 

321 

205 

282 

213 

226 

342 

323 

303 

450 

370 

409 

441 

190 

241 

241 

165 

395 

187 

368 

222 

27 

17 

98 

24 

21 

45 

93 

50 

53 

60 

43 

75 

33 

42 

42 

27. Would you consider applying for EU research funding again? 

Yes, certainly 
Yes, maybe 
No, probably not 
No, certainly not 
Do not know/cannot answer 

4 

7 

26 

7 

7 

19 

29 

14 

22 

25 

11 

26 

10 

14 

20 

106 

121 

141 

136 

133 

142 

100 

109 

336 

131 

260 

161 

147 

177 

183 

983 
123 
13 
0 
19 

What are the main reasons for indicating that you are unlikely to apply for EU managed funding in 
the future? (multiple answers are possible) 
This question showed up only for those who replied ‘No, probably not’ or ‘No, certainly not’ in the question 
above. 

The eligibility requirements for proposals are too high 
The success rate for REA managed applications is too low 
The application procedure is too complex 
The administrative requirements for managing REA 
grants are too heavy 
Other reasons (please specify) 

4 
4 
7 

10 

4 

54 

 
 
 
 
 
 
 
 
 
 
 
Annex 4. Questionnaires of REA independent experts surveys 

The questionnaire of the survey of REA independent experts: evaluators 

Background information about the survey 

The purpose of this survey, which is being conducted as part of the ongoing evaluation of the Research 
Executive Agency, is to measure your satisfaction with the services provided by the Agency and identify areas 
for its performance improvement. 

The survey questionnaire consists of the following parts: the introductory section, specific sections dedicated 
to your registration and the creation of your profile, the Legal Entity/Bank Account Validation process, the 
selection and contracting process, the execution of the tasks assigned, as well as the payments made. 

No personal information will be revealed publicly, in full compliance with the Specific Privacy Statement on 
Personal Data Protection. 

Thank you in advance for your cooperation! 

I. Introduction 

1. What is your country of citizenship? 
If  you  are  a  national  of  multiple  countries,  please  indicate  your  first/birthright  citizenship.  Here  all  world 
countries  will  be  indicated  as  possible  options.  Experts  can  come  from  any  country,  not  only  EU  Member 
States or H2020 associated countries. 

2. What is your current country of residence?  
Please  specify.  Here  all  world  countries  will  be  indicated  as  possible  options.  Experts  can  come  from  any 
country, not only EU Member States or H2020 associated countries. 

II. Registration/Creation of expert profile 

3. Where did you first hear about the opportunity to become an independent expert of the 
Agency/Commission?  

Please select one option. 
By participating in the research project supported by FP6/FP7/H2020 
Relevant national source (research ministry, an EU liaison office at my university, etc.) 
Recommendation by colleagues, superiors in my institution 
European Commission website 
REA website 
European Commission social media feed 
Online advertising (e.g. ads in search engines or social networking sites) 
EU event or promotional material (e.g. an info day, an EU info stand at a conference) 
Media (e.g. a specialised magazine) 
Do not know/cannot remember 
Other (please specify): 

4.  Have  you  consulted/used  any  of  the  online  information  which  is  available  on  the  Participant 
Portal before starting the registration process? 
Yes 

No 

55 

 
 
 
 
  
  
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4a. Please assess the usefulness of the information provided online: 

This question will show up only for those who replied ‘YES’ in the question above. 

Very 
useful 

Fairly 
useful 

Neithe
r 

Not 
useful 

very 

Not 
useful 

at 

all 

not 

Do 
know/cannot 
answer/not 
relevant 

H2020 online manual 
FAQ for experts 
Reference Documents 
H2020  Helpdesk  –  the 
Research 
Enquiry 
Service (RES) 
IT Helpdesk 

5.  Have  you  sought  information  or  assistance  from  the  Research  Executive  Agency  (REA)  during 
the registration process? 

Yes 

No 

5a. Please assess the quality of information/assistance provided by the REA:  

This question will show up only for those who replied ‘YES’ in the question above. 

Strongly 
agree 

Rather 
agree  

Neither 
agree 
nor 
disagree 

Rather 
disagree 

Strongly 
disagree  

not 

Do 
know/cannot 
answer/not 
relevant 

Information  provided  by  the 
REA  during 
registration 
the 
process  was  clear  and  easy  to 
understand  
Overall, the registration process 
was smooth and clear 

6.  To  what  extent  do  you  agree  or  disagree  with  the  following  statements  concerning  your 
registration process: 

Please select one option in each row. 

Strongly 
agree 

Rather 
agree 

Neither 
agree 
disagree 

Rather 
disagree 

Strongly 
disagree 

nor 

Do 
not 
know/cannot 
answer/not 
relevant 

is 

transparent 

to 
independent 

Information  on  how 
become  an 
expert is easy to find 
Eligibility  requirements  to 
become  an  expert  are 
balanced 
It 
how 
independent  experts  are 
selected 
registration 
Overall, 
process  was  smooth  and 
clear 
Please 
comments 
registration 
any): 

related 
process 

your 
to 
(if 

provide 

the 

56 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
III. Legal Entity/Bank Account Validation 

7.  Before  you  could  be  contracted  as  an  expert,  your  identity  and  bank  account  details  had  to  be 
verified (‘Legal Entity/Bank Account Validation’). This procedure is carried out separately from the 
registration process. Please assess the LE/BA validation procedure. 

Strongly 
agree 

Rather 
agree  

Neither 
agree 
nor 
disagree 

Rather 
disagree 

Strongly 
disagree  

not 

Do 
know/cannot 
answer/not 
relevant 

and 

details 

It  was  clear  to  me  why  my 
identity 
bank 
account  information  needed  to 
be checked 
I  am  aware  that  several  bank 
accounts  can  be  linked  to  my 
expert  profile  and  that  I  can 
chose  which  one  I  want  to  be 
paid  into  when  I  submit  my 
reimbursement claim 
The  email  notification  that  I 
received  asking  me  to  provide 
my  identity  and  bank  account 
details was clear 
The  IT  tool  in  the  Participants 
LE/BA 
Portal 
validation is easy to use 
I  had  no  difficulties  in  providing 
supporting 
the 
documentation 
in  relation  to 
LE/BA validation 
I  am 
the 
information  that  I  provided  will 
be  protected  from  unauthorised 
access and use 
I  am  satisfied  with  the  support 
provided  by  the  people  who 
dealt with my file 

requested 

confident 

related 

that 

to 

8.  Please  provide  your  comments  related  to  the  Legal  Entity/Bank  Account  Validation  process  (if 
any): 
Please insert your comments in the box below. 

IV. Selection and contracting 

9.  Were  you  contacted  with  regard  to  your  availability  to  work  as  an  independent  expert  for  a 
particular evaluation or project review before you received an expert contract to sign?  
Do not know/cannot answer/not relevant 
Yes 

No 

10. Please give your assessment of the following statements.  
Rather 
Strongly 
disagree 
agree 

Rather 
agree  

Neither 
agree 
nor 
disagree 

The  contract  was 
sufficient 
time 
organise your schedule/work 

issued 
for  you 

in 
to 

57 

Strongly 
disagree  

Do  not  know/cannot 
answer/not relevant 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Overall, 
contracting process was clear 

selection  and 

the 

11.  How  far  in  advance  of  the  start  of  the  assignment  do  you  think  an  expert  contract  should  be 
issued? 

The  question  will  only  show  up  for  those  answering  ‘rather  disagree’  and  ‘strongly  disagree’  under  the  first 
statement of the question above. 

12. Experts are required to declare conflicts of interest (CoI), either at the time of signature of the 
contract or as soon as a CoI is identified. Please assess the process in relation to CoI.  

Please select one option in each row.  

Strongly 
agree 

Rather 
agree  

Neither 
agree 
nor 
disagree 

Rather 
disagree 

Strongly 
disagree  

not 

Do 
know/cannot 
answer/not 
relevant 

I 

It is clear to me  under what 
circumstances 
should 
declare a CoI  
The  consequences  of  a  CoI 
are clear 
I  understand  when  and  how 
to declare a CoI 

13. Have you ever declared a conflict of interest (CoI)? 

Yes 

No 

14. When an expert needs to travel for an assignment, travel costs are reimbursed to and from the 
expert's  address  as  indicated  in  the  contract  (the  ‘Point  of  Departure’  –  POD).  An  expert  who 
wishes  to  travel  to  and/or  from  a  different  address  must  request  prior  approval  (‘request  for 
change  of  point  of  departure’).  Please  assess  the  process  in  relation  to  requests  to  change  the 
point of departure for assignments involving travel.  

Please select one option in each row.  

Strongly 
agree 

Rather 
agree  

Neither 
agree 
nor 
disagree 

Rather 
disagree 

Strongly 
disagree  

not 

Do 
know/cannot 
answer/not 
relevant 

I  understand  when  and  how  to 
request a change of POD  
Decisions  on  a  change  of  POD 
are made quickly 

15. Have you sought information or assistance from the REA during the contracting process? 
Yes 

Cannot remember 

No 

V. Performance of the assignment/task 

16. Regarding place of work for the assignment, did you have to carry out your tasks: 
You can select more than one option. 

58 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
At your own premises (remotely) 
At the Commission/Agency premises (centrally) 
At the Commission/Agency premises and at your own premises and a (remote AND central) 
At the premises of the project beneficiaries (for project monitors) 
Do not know/cannot answer 

16a. Please assess the extent to which you are satisfied with the quality of REA evaluation 
facilities in Brussels:  

Please select one option in each row.  

This question will show up only for those who selected option 2 or 3 in the question above. 

Very 
satisfied 

Fairly 
satisfied  

Neither 

Not 
very 
satisfied 

at 
Not 
satisfied 

all 

not 

Do 
know/cannot 
answer/not 
relevant 

the 

reception 

logistics  and 
(meeting 
spaces, 
during 

The  overall  service  provided 
at 
desk 
(access  to  registration  desk, 
badge,  indication  of  meeting 
room) 
The  overall 
infrastructure 
rooms,  working 
refreshments 
meetings) 
IT  tools,  internet  availability 
and  assistance  provided  by 
IT staff. 
Hotel 
availability 
Catering Facilities (cafeteria) 
Overall 
responsiveness  of 
REA support services staff 

List/accommodation 

17. To what extent do you agree or disagree with the following statements concerning your work 
as an expert: 

Please select one option in each row. 

Strongly 
agree 

Rather 
agree  

Neither 
agree 
nor 
disagree 

Rather 
disagree 

Strongly 
disagree  

not 
Do 
know/cannot 
answer/not 
relevant 

by 

the 

provided 

call 
Information 
coordinator/project 
officer/project 
assistant  during  my  work  as  an  expert 
was clear and sufficient 
I  knew  either  who  to  contact  or  where  to 
get  help  regarding  any  questions  I  had 
when working on my tasks 
I  was  regularly  and  appropriately  briefed 
on the requirements for my work 
I  have  received  individual  attention  on 
logistical issues from the REA staff during 
my work 
Tasks  I  had  to  carry  out  were  clearly 
stated  in  the  contract,  ToRs,  and  other 
relevant documents 
The  EC  online  evaluation  system  is  easy 
to access 
The  online  evaluation  system  is  easy  to 
use 
The  templates  for  the  reports  I  had  to 
complete are fit for purpose 

59 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
call 

call 

The 
coordinators/project 
officers/project  assistants  who  I  have 
worked with were responsive 
The 
coordinators/project 
officers/project  assistants  who  I  have 
worked with were qualified 
I  was 
the  whole  workflow 
Overall, 
involved  in  as  an  expert  went  smoothly 
and did not cause unnecessary burden 
Please  provide  comments  related  to  your 
work as an expert (if any): 

VI. Payment 

18.  Did  you  attend  a  Briefing  on  Reimbursement  at  the  beginning  of  the  evaluation  week  in 

Brussels?  

Yes 

No 

Do  not  know/cannot  answer/not 
relevant 

18a. Please assess the quality of information/assistance provided by the REA during the briefing:  
Please select one option in each row.  

This question will show up only for those who replied ‘YES’ in the question above. 

Strongly 
agree 

Rather 
agree  

Neither 
agree 
nor 
disagree 

Rather 
disagree 

Strongly 
disagree  

not 

Do 
know/cannot 
answer/not 
relevant 

on 

presentation 

The 
reimbursement was clear  
The presentation was useful 
The  person  who  gave 
the 
presentation  was  responsive  to 
my questions 

19. Did you visit the Reimbursement Help Desk during your evaluation in Brussels? 

Yes 

No 

Do 
not 
answer/not relevant 

know/cannot 

19a. Please assess the quality of information/assistance provided by the REA:  

Please select one option in each row.  

This question will show up only for those who replied ‘YES’ in the question above. 

Strongly 
agree 

Rather 
agree  

Neither 
agree 
nor 
disagree 

Rather 
disagree 

Strongly 
disagree  

not 

Do 
know/cannot 
answer/not 
relevant 

that 

the 
I 

I 
received 
information/answer 
needed 
I am satisfied with the follow-up 
of my query 
The  person  who  I  spoke  to  was 
responsive  
The  person  who  I  spoke  to  was 
qualified to assist me 

60 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
20. Did you use the REA-Evaluation@ec.europa.eu functional mailbox?  

Yes 

No 

Do  not  know/cannot  answer/not 
relevant 

20a. At what stage in the process did you use this service:  

You can select more than one option. 

This question will show up only for those who replied ‘YES’ in the question above. 

Before signing my contract 
Before organising my travel arrangements 
Before submitting my reimbursement request 
After submitting my reimbursement request 

20b. Please assess the quality of information/assistance provided by the REA through the REA-
Evaluation@ec.europa.eu mailbox 
Please select one option in each row.  

This question will show up only for those who replied ‘YES’ in the question above. 

Strongly 
agree 

Rather 
agree  

Neither 
agree 
nor 
disagree 

Rather 
disagree 

Strongly 
disagree  

not 

Do 
know/cannot 
answer/not 
relevant 

response 

time  was 

The 
appropriate  
The 
response 
provided  me 
information I needed 
Overall,  I  am  satisfied  with  the 
service 
this 
mailbox 

received 
the 

received  via 

I 
with 

I 

21.  To  what  extent  do  you  agree  with  the  following  statements  concerning  payments  for  your 
services?  

Strongly 
agree  

Rathe
r 
agree  

Neither 
agree nor 
disagree 

Rather 
disagre
e 

Strongly 
disagree 

Do not 
know/cann
ot 
answer/not 
relevant 

The email notification informing 
me that I could prepare and 
submit my payment request 
was clear 
The email notification 
requesting me to provide 
further 
information/documentation 
supporting my payment request 
was clear 
The email notification informing 
me that I had been paid was 
clear 
The time it took the REA to 
process my payment requests 
and make payments was 
satisfactory 
I understood how to calculate 
the number of days for which I 

61 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Strongly 
agree  

Rathe
r 
agree  

Neither 
agree nor 
disagree 

Rather 
disagre
e 

Strongly 
disagree 

Do not 
know/cann
ot 
answer/not 
relevant 

would receive payment for my 
remote work 
The number of days I was paid 
for my remote work matched 
the effort I actually spent on 
my tasks 
It is clear which travel related 
expenses are not eligible for 
reimbursement 
The reimbursement of my 
travel expenses was 
appropriate 
I was able to understand the 
breakdown of the payment that 
I received 
The REA staff who processed 
my payment were responsive 
The way in which complaints 
about payments are handled by 
the REA is satisfactory 
Overall I am satisfied with the 
way that my payment was 
handled 

22. Please provide your comments related to the payment process (if any): 

Please insert your comments in the box below. 

VII. Final questions 

23. Would you like to work with the REA as an independent expert again in the future? 

Please select one option. 

Yes, certainly 
Yes, maybe 
No, probably not 
No, certainly not 
Do not know/cannot answer 

23a.  What  are  the  main  reasons  for  indicating  that  you  are  unlikely  to  work  with  the  REA  as  an 
independent expert again in the future? 
This question will show up only for those who replied ‘No, probably not' or 'No, certainly not' in the question 
above. 
The professional fee rate (currently set at EUR 450) is too low 
The  number  of  daily  fees  that  I  receive  for  my  remote  work  does  not  reflect  the  amount  of 
time I actually spend on the tasks.  
Registration procedures to become an expert are too complex 
Administrative procedures related to working as an expert are too complex 
The  amount  of  the  daily  and/or  accommodation  allowance  is  too  low  (currently  EUR  92/day 

62 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
and EUR 100/night) 
The reimbursement of my travel expenses was unreasonably low 
Personal/work reasons (e.g. changed area of work, too high workload) 
Other reasons (please specify): 
Do not know/cannot answer 

24. Is there anything else you would like to add regarding your experience when contracted by the 
REA to work as an independent expert for the Commission and its executive agencies?  
Please provide your comments in the box below. 

Thank you! 

The questionnaire of the survey of REA independent experts: monitoring experts 

Background information about the survey 

The  purpose  of  this  survey,  which  is  being  conducted  as  part  of  the  ongoing  evaluation  of  the  Research 
Executive Agency, is to measure your satisfaction with the services provided by the Agency and identify areas 
for its performance improvement. 

The survey questionnaire consists of the following parts: the introductory section, specific sections dedicated 
to  your  registration  and  the  creation  of  your  profile,  the  Legal  Entity/Bank  Account  Validation  process,  the 
selection and contracting process, the execution of the tasks assigned, as well as the payments made. 

No  personal  information  will  be  revealed  publicly,  in  full  compliance  with  the  Specific  Privacy  Statement  on 
Personal Data Protection. 

Thank you in advance for your cooperation! 

I. Introduction 

1. What is your country of citizenship? 
If  you  are  a  national  of  multiple  countries,  please  indicate  your  first/birthright  citizenship.  Here  all  world 
countries  will  be  indicated  as  possible  options.  Experts  can  come  from  any  country,  not  only  EU  Member 
States or H2020 associated countries. 

2. What is your current country of residence?  
Please  specify.  Here  all  world  countries  will  be  indicated  as  possible  options.  Experts  can  come  from  any 
country, not only EU Member States or H2020 associated countries. 

II. Registration/Creation of expert profile 

3. Where did you first hear about the opportunity to become an independent expert of the 
Agency/Commission?  

Please select one option. 
By participating in the research project supported by FP6/FP7/H2020 
Relevant national source (research ministry, an EU liaison office at my university, etc.) 
Recommendation by colleagues, superiors etc. 
European Commission website 
REA website 

63 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
European Commission social media feed 
Online advertising (e.g. ads in search engines or social networking sites) 
EU event or promotional material (e.g. an info day, an EU info stand at a conference) 
Media (e.g. a specialised magazine) 
Do not know/cannot remember 
Other (please specify): 

4.  Have  you  consulted/used  any  of  the  online  information  which  is  available  on  the  Participant 
Portal before starting the registration process? 
Yes 

No 

4a. Please assess the usefulness of the information provided online: 
This question will show up only for those who replied ‘YES’ in the question above. 

Very 
useful 

Fairly 
useful 

Neithe
r 

Not 
useful 

very 

Not 
useful 

at 

all 

not 

Do 
know/cannot 
answer/not 
relevant 

H2020 online manual 
FAQ for experts 
Reference Documents 
H2020  Helpdesk  –  the 
Research 
Enquiry 
Service (RES) 
IT Helpdesk 

5. Have you sought for information or assistance from the Research Executive Agency (REA) during 
the registration process? 
Yes 

No 

5a. Please assess the quality of information/assistance provided by the REA:  
This question will show up only for those who replied ‘YES’ in the question above. 
Strongly 
disagree  

Rather 
agree  

Rather 
disagree 

Strongly 
agree 

Neither 
agree 
nor 
disagree 

not 

Do 
know/cannot 
answer/not 
relevant 

Information  provided  by  the 
registration 
the 
REA  during 
process  was  clear  and  easy  to 
understand  
Overall, the registration process 
was smooth and clear 

6.  To  what  extent  do  you  agree  or  disagree  with  the  following  statements  concerning  your 
registration process: 
Please select one option in each row. 

Strongly 
agree 

Rather 
agree 

Neither 
agree 
disagree 

Rather 
disagree 

Strongly 
disagree 

nor 

not 
Do 
know/cannot 
answer/not 
relevant 

to 
independent 

Information  on  how 
become  an 
expert is easy to find 
Eligibility  requirements  to 
become  an  expert  are 
balanced 
It 
transparent  how 
is 
independent  experts  are 
selected 
Overall, 
registration 
process  was  smooth  and 
clear 
Please 
comments 
registration 
any): 

related 
process 

your 
to 
(if 

provide 

the 

64 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
III. Legal Entity/Bank Account Validation 

7.  Before  you  could  be  contracted  as  an  expert,  your  identity  and  bank  account  details  had  to  be 
verified (‘Legal Entity/Bank Account Validation’). This procedure is carried out separately from the 
registration process. Please assess the LE/BA validation procedure. 

Strongly 
agree 

Rather 
agree  

Neither 
agree 
nor 
disagree 

Rather 
disagree 

Strongly 
disagree  

not 

Do 
know/cannot 
answer/not 
relevant 

and 

details 

It  was  clear  to  me  why  my 
identity 
bank 
account  information  needed  to 
be checked 
I  am  aware  that  several  bank 
accounts  can  be  linked  to  my 
expert  profile  and  that  I  can 
chose  which  one  I  want  to  be 
paid  into  when  I  submit  my 
reimbursement claim 
The  email  notification  that  I 
received  asking  me  to  provide 
my  identity  and  bank  account 
details was clear 
The  IT  tool  in  the  Participants 
LE/BA 
Portal 
validation is easy to use 
I  had  no  difficulties  in  providing 
supporting 
the 
documentation 
in  relation  to 
LE/BA validation 
I  am 
the 
information  that  I  provided  will 
be  protected  from  unauthorised 
access and use 
I  am  satisfied  with  the  support 
provided  by  the  people  who 
dealt with my file 

requested 

confident 

related 

that 

to 

8.  Please  provide  your  comments  related  to  the  Legal  Entity/Bank  Account  Validation  process  (if 
any): 

Please insert your comments in the box below. 

IV. Selection and contracting 

9.  Were  you  contacted  with  regard  to  your  availability  to  work  as  an  independent  expert  for  a 
particular evaluation or project review before you received an expert contract to sign?  
Do not know/cannot answer/not relevant 
Yes 

No 

10. Please give your assessment of the following statements.  
Rather 
Strongly 
disagree 
agree 

Rather 
agree  

Neither 
agree 
nor 

Strongly 
disagree  

Do 
know/cannot 
answer/not 

not 

65 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
The  contract  was  issued  in 
sufficient 
to 
time 
organise your schedule/work 
Overall, 
contracting process was clear 

the  selection  and 

for  you 

disagree 

relevant 

11.  How  far  in  advance  of  the  start  of  the  assignment  do  you  think  an  expert  contract  should  be 
issued? 

The  question  will  only  show  up  for  those  answering  ‘rather  disagree’  and  ‘strongly  disagree’  under  the  first 
statement of the question above. 

12. Experts are required to declare conflicts of interest (CoI), either at the time of signature of the 
contract or as soon as a CoI is identified. Please assess the process in relation to CoI.  

Please select one option in each row.  

Strongly 
agree 

Rather 
agree  

Neither 
agree 
nor 
disagree 

Rather 
disagree 

Strongly 
disagree  

not 

Do 
know/cannot 
answer/not 
relevant 

It  is  clear  to  me  under  what 
circumstances  I  should  declare 
a CoI  
The  consequences  of  a  CoI  are 
clear 
I  understand  when  and  how  to 
declare a CoI 

13. Have you ever declared a conflict of interest (CoI)? 
Yes 

No 

14. When an expert needs to travel for an assignment, travel costs are reimbursed to and from the 
expert's  address  as  indicated  in  the  contract  (the  ‘Point  of  Departure’  –  POD).  An  expert  who 
wishes  to  travel  to  and/or  from  a  different  address  must  request  prior  approval  (‘request  for 
change  of  point  of  departure’).  Please  assess  the  process  in  relation  to  requests  to  change  the 
point of departure for assignments involving travel.  

Please select one option in each row.  

Strongly 
agree 

Rather 
agree  

Neither 
agree 
nor 
disagree 

Rather 
disagree 

Strongly 
disagree  

not 

Do 
know/cannot 
answer/not 
relevant 

I  understand  when  and  how  to 
request a change of POD  
Decisions  on  a  change  of  POD 
are made quickly 

15. Have you sought information or assistance from the REA during the contracting process? 
Yes 

Cannot remember 

No 

V. Performance of the assignment/task 

16. Regarding place of work for the assignment, did you have to carry out your tasks: 
You can select more than one option. 

66 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
At your own premises (remotely) 
At the Commission/Agency premises (centrally) 
At the Commission/Agency premises and at your own premises and a (remote AND central) 
At the premises of the project beneficiaries (for project monitors) 
Do not know/cannot answer 

16a. Please assess the extent to which you are satisfied with the quality of REA evaluation 
facilities in Brussels:  

Please select one option in each row.  
This question will show up only for those who selected option 2 or 3 in the question above. 
Do 
know/cannot 
answer/not 
relevant 

Not 
very 
satisfied 

Not 
at 
satisfied 

Very 
satisfied 

Fairly 
satisfied  

Neither 

all 

not 

the 

reception 

logistics  and 
(meeting 
spaces, 
during 

The  overall  service  provided 
at 
desk 
(access  to  registration  desk, 
badge,  indication  of  meeting 
room) 
The  overall 
infrastructure 
rooms,  working 
refreshments 
meetings) 
IT  tools,  internet  availability 
and  assistance  provided  by 
IT staff. 
Hotel 
availability 
Catering Facilities (cafeteria) 
Overall 
responsiveness  of 
REA support services staff 

List/accommodation 

17. To what extent do you agree or disagree with the following statements concerning your work 
as an expert: 

Please select one option in each row. 

Strongly 
agree 

Rather 
agree  

Neither 
agree 
nor 
disagree 

Rather 
disagree 

Strongly 
disagree  

not 
Do 
know/cannot 
answer/not 
relevant 

by 

the 

provided 

call 
Information 
coordinator/project 
officer/project 
assistant  during  my  work  as  an  expert 
was clear and sufficient 
I  knew  either  who  to  contact  or  where  to 
get  help  regarding  any  questions  I  had 
when working on my tasks 
I  was  continuously  and  appropriately 
briefed on the requirements for my work 
I  have  received  individual  attention  on 
logistical issues from the REA staff during 
my work 
Tasks  I  had  to  carry  out  were  clearly 
stated  in  the  contract,  ToRs,  and  other 
relevant documents 
The  templates  for  the  reports  I  had  to 
complete are fit for purpose 
The 
coordinators/project 
call 
officers/project  assistants  who  I  have 
worked with were responsive 
coordinators/project 
The 
officers/project  assistants  who  I  have 
worked with were qualified 
Overall, 
I  was 
the  whole  workflow 
involved  in  as  an  expert  went  smoothly 
and did not cause unnecessary burden 

call 

67 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Please  provide  comments  related  to  your 
work as an expert (if any): 

VI. Payment 

18.  Did  you  attend  a  Briefing  on  Reimbursement  at  the  beginning  of  the  evaluation  week  in 

Brussels?  

Yes 

No 

Do 
not 
answer/not relevant 

know/cannot 

18a. Please assess the quality of information/assistance provided by the REA during the briefing:  
Please select one option in each row.  

This question will show up only for those who replied ‘YES’ in the question above. 

Strongly 
agree 

Rather 
agree  

Neither 
agree 
nor 
disagree 

Rather 
disagree 

Strongly 
disagree  

not 

Do 
know/cannot 
answer/not 
relevant 

on 

presentation 

The 
reimbursement was clear  
The presentation was useful 
The  person  who  gave 
the 
presentation  was  responsive  to 
my questions 

19. Did you visit the Reimbursement Help Desk during your evaluation in Brussels? 

Yes 

No 

Do 
not 
answer/not relevant 

know/cannot 

19a. Please assess the quality of information/assistance provided by the REA:  
Please select one option in each row.  
This question will show up only for those who replied ‘YES’ in the question above. 

Strongly 
agree 

Rather 
agree  

Neither 
agree 
nor 
disagree 

Rather 
disagree 

Strongly 
disagree  

not 

Do 
know/cannot 
answer/not 
relevant 

that 

the 
I 

I 
received 
information/answer 
needed 
I am satisfied with the follow-up 
of my query 
The  person  who  I  spoke  to  was 
responsive  
The  person  who  I  spoke  to  was 
qualified to assist me 

20. Did you use the REA-Evaluation@ec.europa.eu functional mailbox?  
Yes 

No 

Do  not  know/cannot  answer/not 
relevant 

20a. At what stage in the process did you use this service:  

You can select more than one option. 

This question will show up only for those who replied ‘YES’ in the question above. 

68 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Before signing my contract 
Before organising my travel arrangements 
Before submitting my reimbursement request 
After submitting my reimbursement request 

20b. Please assess the quality of information/assistance provided by the REA through the REA-
Evaluation@ec.europa.eu mailbox 
Please select one option in each row.  

This question will show up only for those who replied ‘YES’ in the question above. 

Strongly 
agree 

Rather 
agree  

Neither 
agree 
nor 
disagree 

Rather 
disagree 

Strongly 
disagree  

not 

Do 
know/cannot 
answer/not 
relevant 

response 

time  was 

The 
appropriate  
The 
response 
provided  me 
information I needed 
Overall,  I  am  satisfied  with  the 
service 
this 
mailbox 

received 
the 

received  via 

I 
with 

I 

21.  To  what  extent  do  you  agree  with  the  following  statements  concerning  payments  for  your 
services?  

Strongly 
agree  

Rathe
r 
agree  

Neither 
agree nor 
disagree 

Rather 
disagre
e 

Strongly 
disagree 

Do not 
know/cann
ot 
answer/not 
relevant 

The email notification informing 
me that I could prepare and 
submit my payment request 
was clear 
The email notification 
requesting me to provide 
further 
information/documentation 
supporting my payment request 
was clear 
The email notification informing 
me that I had been paid was 
clear 
The time it took the REA to 
process my payment requests 
and make payments was 
satisfactory 
I understood how to calculate 
the number of days for which I 
would receive payment for my 
remote work 
The number of days I was paid 
for my remote work matched 
the effort I actually spent on 
my tasks 
It is clear which travel related 
expenses are not eligible for 
reimbursement 
The reimbursement of my 
travel expenses was 
appropriate 
I was able to understand the 
breakdown of the payment that 

69 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Strongly 
agree  

Rathe
r 
agree  

Neither 
agree nor 
disagree 

Rather 
disagre
e 

Strongly 
disagree 

Do not 
know/cann
ot 
answer/not 
relevant 

I received 
The REA staff who processed 
my payment were responsive 
The way in which complaints 
about payments are handled by 
the REA is satisfactory 
Overall I am satisfied with the 
way that my payment was 
handled 

22. Please provide your comments related to the payment process (if any): 

Please insert your comments in the box below. 

VII. Final questions 

23. Would you like to work with the REA as an independent expert again in the future? 

Please select one option. 
Yes, certainly 
Yes, maybe 
No, probably not 
No, certainly not 
Do not know/cannot answer 

23a.  What  are  the  main  reasons  for  indicating  that  you  are  unlikely  to  work  with  the  REA  as  an 
independent expert again in the future? 

This question will show up only for those who replied ‘No, probably not' or 'No, certainly not' in the question 
above. 
The professional fee rate (currently set at EUR 450) is too low 
The  number  of  daily  fees  that  I  receive  for  my  remote  work  does  not  reflect  the  amount  of 
time I actually spend on the tasks.  
Registration procedures to become an expert are too complex 
Administrative procedures related to working as an expert are too complex 
The  amount  of  the  daily  and/or  accommodation  allowance  is  too  low  (currently  EUR  92/day 
and EUR 100/night) 
The reimbursement of my travel expenses was unreasonably low 
Personal/work reasons (e.g. changed area of work, too high workload) 
Other reasons (please specify): 
Do not know/cannot answer 

24. Is there anything else you would like to add regarding your experience when contracted by the 
REA to work as an independent expert for the Commission and its executive agencies?  

Please provide your comments in the box below. 

Thank you! 

70 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Aggregated survey results: 

Table 4 Q3 Where did you first hear about the opportunity to become an independent expert of 
the Agency/Commission? 

By participating in a research project supported by FP6/FP7/H2020 

Recommendation by colleagues, superiors in my institution 

European Commission website 

Relevant national source (research ministry, an EU liaison office at my university, etc.) 

Other (please specify): 

Do not know/cannot remember 

EU event or promotional material (e.g. an info day, an EU info stand at a conference) 

REA website 

European Commission social media feed 

Online advertising (e.g. ads in search engines or social networking sites) 

Media (e.g. a specialised magazine) 

1 367 

1 236 

813 

717 

496 

323 

198 

122 

61 

35 

20 

Table 5 Q4a Please assess the usefulness of the information provided online: 

Very 
useful 

Fairly 
useful 

Nei
the
r 

Not 
very 
useful 

Not at all 
useful 

IT Helpdesk 

666 

785 

368 

118 

FAQ for experts 

1475 

1489 

181 

60 

H2020 online manual 

1350 

1603 

148 

64 

Reference Documents 

1067 

1510 

307 

68 

H2020 Helpdesk – the 
Research Enquiry 
Service (RES) 

612 

763 

381 

84 

34 

10 

9 

10 

36 

Do not 
know/cannot 
answer/not 
relevant 

1 500 

267 

316 

513 

1 599 

Table 6 Have you sought for information or assistance from the Research Executive Agency 
(REA) during the registration process? 

Yes 

818 

No 

4570 

Table 7 Q5a Please assess the quality of information/assistance provided by the REA: 

Strongly 
agree 

Rather 
agree 

Overall, the registration 
process was smooth and clear 

370 

314 

Neither 
agree 
nor 
disagree 
75 

Information provided by the 
REA during the registration 
process was clear and easy to 
understand 

433 

299 

61 

Rather 
disagr
ee 

51 

28 

Strong
ly 
disagr
ee 
16 

Do not 
know/cannot 
answer/not 
relevant 
5 

6 

10 

71 

 
 
 
 
 
 
 
Figure 2 Q6 To what extent do you agree or disagree with the following statements concerning 
your registration process: 

Stron
gly 
agre
e 

Rath
er 
agre
e 

Neither 
agree nor 
disagree 

Rathe
r 
disag
ree 

Strong
ly 
disagr
ee 

Do not know/cannot 
answer/not relevant 
for me 

773 

1261 

1 368 

1 018 

510 

458 

1 689 

1 330 

1 218 

1 76
3 

1 83
1 

1 78
3 

442 

588 

655 

742 

97 

670 

575 

394 

715 

844 

527 

301 

It is transparent how 
independent experts are 
selected 

Overall, the registration 
process was smooth and clear 

Eligibility requirements to 
become an expert are 
balanced 

Information on how to 
become an independent 
expert is easy to find 

Table 8 Q7 Before you could be contracted as an expert, your identity and bank account details 
had to be verified (‘Legal Entity/Bank Account Validation’). This procedure is carried out 
separately from the registration process. Please assess the LE/BA validation procedure. 

Strong
ly 
agree 

Rather 
agree 

Rather 
disagr
ee 

Strong
ly 
disagr
ee 

Neithe
r 
agree 
nor 
disagr
ee 

2 892 

1 436 

397 

76 

91 

Do not 
know/
cannot 
answe
r/not 
releva
nt 
496 

2 190 

1 881 

573 

269 

130 

345 

3 325 

1 397 

282 

209 

124 

51 

2 774 

1 639 

623 

119 

84 

149 

2 601 

1 741 

423 

313 

182 

128 

3 181 

1 488 

286 

135 

94 

204 

2 428 

1 135 

599 

438 

333 

455 

I am satisfied with the support 
provided by the people who dealt with 
my file 

The IT tool in the Participants Portal 
related to LE/BA validation is easy to 
use 

It was clear to me why my identity 
details and bank account information 
needed to be checked 

I am confident that the information 
that I provided will be protected from 
unauthorised access and use 

I had no difficulties in providing the 
requested supporting documentation in 
relation to LE/BA validation 

The email notification that I received 
asking me to provide my identity and 
bank account details was clear 

I am aware that several bank accounts 
can be linked to my expert profile and 
that I can choose which one I want to 
be paid into when I submit my 
reimbursement claim 

72 

 
 
 
 
 
 
Table 9 Q9 Were you contacted with regard to your availability to work as an independent expert 
for a particular evaluation or project review before you received an expert contract to sign? 

Yes 

No 

4 581 

500 

Do not know/cannot answer/not relevant 

307 

Table 10 Q10. Please give your assessment of the following statements. 

Strong
ly 
agree 

Rathe
r 
agree 

Neither 
agree nor 
disagree 

Rathe
r 
disagr
ee 

Strongl
y 
disagre
e 

Do not 
know/cannot 
answer/not 
relevant 

2 540 

1 901 

498 

293 

97 

59 

2 593 

1 882 

327 

416 

149 

21 

Overall, the selection and 
contracting process was clear 

The contract was issued in sufficient 
time for you to organise your 
schedule/work 

Table 11 Q12Experts are required to declare conflicts of interest (CoI), either at the time of 
signature of the contract or as soon as a CoI is identified. Please assess the process in relation to 
CoI. 

Strongl
y agree 

Rathe
r 
agree 

Neither 
agree nor 
disagree 

Rather 
disagr
ee 

Strongly 
disagree 

Do not know/cannot 
answer/not relevant 

The consequences of a CoI are 
clear 

4 003 

1 059 

157 

101 

55 

I understand when and how to 
declare a CoI 

3 958 

1 198 

123 

It is clear to me under what 
circumstances I should declare 
a CoI 

3 906 

1 242 

99 

57 

96 

40 

38 

13 

12 

7 

Table 12 Q14 When an expert needs to travel for an assignment, travel costs are reimbursed to 
and from the expert's address as indicated in the contract (the ‘Point of Departure’ – POD). An 
expert who wishes to travel to and/or from a different address must request prior approval 
(‘request for change of point of departure’). Please assess the process in relation to requests to 
change the point of departure for assignments involving travel. 

Strongl
y agree 

Rather 
agree 

Neither agree 
nor disagree 

Decisions on a change of 
POD are made quickly 

899 

547 

530 

I understand when and how 
to request a change of POD 

2 427 

1 194 

336 

Rather 
disagre
e 

97 

134 

Strongly 
disagree 

Do not know/cannot 
answer/not relevant 

54 

52 

3 261 

1 245 

Table 13 Q15 Have you sought information or assistance from the REA during the contracting 
process? 

No 

3 081 

73 

 
 
 
 
 
 
 
 
Yes 

1 572 

Cannot remember 

735 

Table 14 Q16 Regarding place of work for the assignment, did you have to carry out your tasks: 

At the Commission/Agency premises and at your own premises and a (remote AND central) 

3 549 

At your own premises (remotely) 

At the Commission/Agency premises (centrally) 

At the premises of the project beneficiaries (for project monitors) 

Do not know/cannot answer 

2 782 

1 471 

501 

98 

Table 15 Q16a Please assess the extent to which you are satisfied with the quality of REA 
evaluation facilities in Brussels: 

Very 
satisfied 

Fairly 
satisfied 

Neither 

Not at 
all 
satisfie
d 

Not 
very 
satisfie
d 

Catering Facilities (cafeteria) 

1 386 

1 484 

Hotel List/accommodation 
availability 

1 283 

1 301 

580 

611 

Overall responsiveness of REA 
support services staff 

2 838 

1 068 

123 

IT tools, internet availability and 
assistance provided by IT staff  

2 131 

1 489 

262 

2 059 

1 625 

223 

114 

169 

12 

36 

32 

378 

393 

34 

180 

194 

The overall logistics and 
infrastructure (meeting rooms, 
working spaces, refreshments 
during meetings) 

The overall service provided at 
the reception desk (access to 
registration desk, badge, 
indication of meeting room) 

2 877 

1 125 

75 

13 

42 

41 

Do not 
know/ca
nnot 
answer/n
ot 
relevant 
227 

412 

94 

71 

38 

Table 16 Q17 To what extent do you agree or disagree with the following statements concerning 
your work as an expert: 

I knew either who to contact or 
where to get help regarding any 
questions I had when working on 
my tasks 
Overall, the whole workflow I was 
involved in as an expert went 
smoothly and did not cause 
unnecessary burden 
The templates for the reports I 
had to complete are fit for 
purpose 

Strongly 
agree 

Rather 
agree 

Neither 
agree nor 
disagree 

Rather 
disagree 

Strongly 
disagree 

3 462 

1 522 

195 

101 

40 

Do not 
know/ca
nnot 
answer/
not 
relevant 
68 

2 847 

1 880 

332 

201 

65 

63 

2 125 

2 112 

659 

286 

91 

115 

74 

 
 
 
 
 
 
I was continuously and 
appropriately briefed on the 
requirements for my work 
I have received individual 
attention on logistical issues from 
the REA staff during my work 
Tasks I had to carry out were 
clearly stated in the contract, 
ToRs, and other relevant 
documents 
The call coordinators/project 
officers/project assistants who I 
have worked with were qualified 
Information provided by the call 
coordinator/project officer/project 
assistant during my work as an 
expert was clear and sufficient 
The call coordinators/project 
officers/project assistants who I 
have worked with were 
responsive 

3 044 

1 744 

360 

125 

2 114 

1 443 

596 

148 

3 199 

1 701 

284 

96 

3 611 

1 236 

200 

42 

3 296 

1 728 

179 

109 

38 

64 

30 

16 

25 

77 

1023 

78 

283 

51 

3 862 

1 175 

148 

38 

19 

146 

Table 17 Q17 To what extent do you agree or disagree with the following statements concerning 
your work as an expert: 

Strongl
y agree 

Rather 
agree 

Neither agree 
nor disagree 

The online evaluation 
system is easy to use 

The EC online evaluation 
system is easy to access 

2 646 

1 633 

447 

2 799 

1 638 

359 

Rather 
disagre
e 

225 

171 

Strongly 
disagree 

Do not know/cannot 
answer/not relevant 

74 

58 

58 

58 

75 

 
 
 
 
 
Table 18 Q19 Did you visit the Reimbursement Help Desk during your evaluation in Brussels? 

Yes 

No 

3 296 

1 280 

Do not know/cannot answer/not relevant 

812 

Table 19 Q19a Please assess the quality of information/assistance provided by the REA: 

Strongl
y 
agree 

Rathe
r 
agree 

Neither 
agree nor 
disagree 

Rather 
disagr
ee 

Strongly 
disagree 

Do not know/cannot 
answer/not relevant 

The presentation was useful 

710 

234 

47 

10 

4 

67 

The presentation on 
reimbursement was clear 

726 

253 

30 

14 

7 

59 

The person who gave the 
presentation was responsive to 
my questions 

738 

182 

28 

12 

2 

102 

Table 20 Q18a Please assess the quality of information/assistance provided by the REA during the 
briefing: 

Strongly 
agree 

Rather 
agree 

Neither 
agree nor 
disagree 

Rather 
disagree 

Strongly 
disagree 

The presentation was useful 

2 040 

987 

214 

57 

8 

Do not 
know/can
not 
answer/n
ot 
relevant 
11 

 The presentation on 
reimbursement was clear 

2 158 

982 

115 

45 

11 

9 

 The person who gave the 
presentation was responsive 
to my questions 

1 545 

543 

199 

23 

5 

997 

76 

 
 
 
 
 
 
 
Figure 3 Q19 Did you visit the Reimbursement Help Desk during your evaluation in Brussels? 

Table 21 Q19 Did you visit the Reimbursement Help Desk during your evaluation in Brussels? 

No 

Yes 

3 607 

1 053 

Do not know/cannot answer/not relevant 

728 

Figure 4 Q20 Did you use the REA-Evaluation@ec.europa.eu functional mailbox? 

Table 22 Q20 Did you use the REA-Evaluation@ec.europa.eu functional mailbox? 

No 

2 804 

Do not know/cannot answer/not relevant 

1 636 

Yes 

948 

77 

 
 
 
 
 
 
 
Table 23 Q20a At what stage in the process did you use this service: 

Before signing my contract 

Before organising my travel arrangements 

290 

319 

After submitting my reimbursement request 

448 

Before submitting my reimbursement request 

428 

Table 24 Q20b Please assess the quality of information/assistance provided by the REA through 
the REA-Evaluation@ec.europa.eu mailbox. 

Strongl
y 
agree 

Rathe
r 
agree 

Neither 
agree nor 
disagree 

Rather 
disagr
ee 

Strongly 
disagree 

Do not know/cannot 
answer/not relevant 

559 

265 

55 

591 

237 

47 

36 

35 

28 

27 

6 

11 

590 

235 

58 

31 

29 

5 

The response time was 
appropriate 

The response I received 
provided me with the 
information I needed 

Overall, I am satisfied with the 
service I received via this 
mailbox 

78 

 
 
 
 
 
 
 
 
Table 25 Q21 To what extent do you agree with the following statements concerning payments 
for your services? 

Strongly 
agree 

Rather 
agree 

Neither 
agree 
nor 
disagree 

Rather 
disagr
ee 

Strongly 
disagree 

2 704 

1 026 

301 

59 

39 

Do not 
know/ca
nnot 
answer/
not 
relevant 
1 259 

2 609 

1 149 

279 

186 

110 

1 055 

2 922 

1 646 

330 

124 

65 

301 

3 820 

866 

171 

64 

44 

423 

2 931 

1 264 

298 

142 

111 

642 

2 749 

1 434 

296 

200 

69 

640 

1 095 

517 

386 

80 

71 

32 39 

2 790 

1 496 

449 

238 

114 

301 

3 855 

1 115 

127 

83 

44 

164 

1136 

1207 

574 

1262 

1035 

174 

2 830 

1 521 

436 

345 

140 

116 

2 607 

1 004 

234 

84 

34 

1 425 

The REA staff who processed my 
payment were responsive 

The reimbursement of my travel 
expenses was appropriate 

Overall I am satisfied with the 
way that my payment was 
handled 

The email notification informing 
me that I had been paid was 
clear 

I was able to understand the 
breakdown of the payment that I 
received 

It is clear which travel related 
expenses are not eligible for 
reimbursement 

The way in which complaints 
about payments are handled by 
the REA is satisfactory 

The time it took the REA to 
process my payment requests 
and make payments was 
satisfactory 

The email notification informing 
me that I could prepare and 
submit my payment request was 
clear 

The number of days I was paid 
for my remote work matched the 
effort I actually spent on my 
tasks 

I understood how to calculate 
the number of days for which I 
would receive payment for my 
remote work 

The email notification requesting 
me to provide further 
information/documentation 
supporting my payment request 
was clear 

Table 26 Q23 Would you like to work with the REA as an independent expert again in the future? 

Yes, certainly 

Yes, maybe 

No, probably not 

4 913 

404 

43 

79 

 
 
 
 
Do not know/cannot answer 

No, certainly not 

20 

8 

Table 27 Q23a What are the main reasons for indicating that you are unlikely to work with the 
REA as an independent expert again in the future? 

The professional fee rate (currently set at EUR 450) is too low 

Registration procedures to become an expert are too complex 

The reimbursement of my travel expenses was unreasonably low 

Administrative procedures related to working as an expert are too complex 

Personal/work reasons (e g retirement, changed area of work, too high workload) 

The amount of the daily and/or accommodation allowance is too low (currently EUR 92/day and 
EUR 100/night 

Other reasons (please specify):] 

The number of daily fees that I receive for my remote work does not reflect the amount of time I 
actually spend on the tasks  

27 

7 

10 

12 

6 

19 

27 

37 

80 

 
 
 
 
How to obtain EU publications 

Free publications: 

•  one copy: 

        via EU Bookshop (http://bookshop.europa.eu); 

•  more than one copy or posters/maps: 

        from the European Union’s representations (http://ec.europa.eu/represent_en.htm);  
        from the delegations in non-EU countries (http://eeas.europa.eu/delegations/index_en.htm);  
        by contacting the Europe Direct service (http://europa.eu/europedirect/index_en.htm) or 
        calling 00 800 6 7 8 9 10 11 (freephone number from anywhere in the EU) (*). 

        (*) The information given is free, as are most calls (though some operators, phone boxes or hotels may charge you). 

Priced publications: 

•  via EU Bookshop (http://bookshop.europa.eu);  

Priced subscriptions: 

•  via one of the sales agents of the Publications Office of the European Union    
   (http://publications.europa.eu/others/agents/index_en.htm). 

81 

 
 
         
 
 
 
 
[

C

a

t

a

l

o

g

u

e

n

u

m

b

e

r

]

K
I
-
0
2
-
1
6
-
5
1
1
-
E
N
-
N

Council  Regulation  (EC)  No  58/2003  related  to  the  establishment  of  Executive  Agencies  as  new 
management modes requires that the European Commission carries out an external evaluation of 
the operation of each of the Executive Agencies every three years. 

The  current (second)  evaluation of the  operation of  REA  covers the  period from July 2012 to July 
2015,  encompassing  two  different  legal  frameworks  and  two  different  programming  periods 
(Seventh  Framework  programme  (FP7)  and  Horizon  2020).  The  study  does  not  focus  on  the 
scientific achievements of these programmes. 

The  evaluation  was  carried  out  by  external  evaluators  (PPMI  -  Public  Policy  and  Management 
Institute) from July 2015 to February 2016.    

ISBN 978-92-79-58797-9 

82