COMMISSION STAFF WORKING DOCUMENT Evaluation of the Education, Audiovisual and Culture Executive Agency (EACEA) Accompanying the document Report from the Commission to the European Parliament, the Council and the Court of Auditors Evaluation of the Consumers, Health, Agriculture and Food Executive Agency, the Executive Agency for Small and Medium-Size Enterprises, the Innovation and Networks Executive Agency, the Education, Audiovisual and Culture Executive Agency, the Research Executive Agency and the European Research Council Executive Agency
Date: 2020-04-29
Institutions: European Commission, Secretariat-General
Subjects: EU aid, EU financing, EU programme, audiovisual industry, culture, education, executive agency, financial control, performance audit

EN 

  EN 

   EUROPEAN COMMISSION  Brussels, 29.4.2020  SWD(2020) 78 final  COMMISSION STAFF WORKING DOCUMENT Evaluation of the Education, Audiovisual and Culture Executive Agency (EACEA) Accompanying the document Report from the Commission to the European Parliament, the Council and the Court of Auditors Evaluation of the Consumers, Health, Agriculture and Food Executive Agency, the Executive Agency for Small and Medium-Size Enterprises, the Innovation and Networks Executive Agency, the Education, Audiovisual and Culture Executive Agency, the Research Executive Agency and the European Research Council Executive Agency  
 
 
TABLE OF CONTENTS 

1. 

INTRODUCTION .................................................................................................................... 2 

2.  BACKGROUND ...................................................................................................................... 3 

2.1.  Description of the intervention and its objectives ....................................................... 3 

2.2.  Baseline and points of comparison .............................................................................. 4 

3. 

IMPLEMENTATION / STATE OF PLAY ..................................................................................... 5 

4.  METHOD .............................................................................................................................. 7 

4.1.  Short description of methodology ............................................................................... 7 

4.2.  Limitations and robustness of findings ........................................................................ 8 

5.  ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS ................................................. 8 

5.1.  Effectiveness ................................................................................................................ 9 

5.2.  Efficiency ................................................................................................................... 19 

5.3.  Coherence .................................................................................................................. 23 

5.4.  Retrospective Cost-benefit analysis ........................................................................... 26 

6.  CONCLUSIONS .................................................................................................................... 27 

ANNEX - EVALUATION QUESTIONS ......................................................................................... 29 

1 

 
 
 
 
 
 
1. 

INTRODUCTION 

The purpose of this evaluation is the periodical (triennial) evaluation of the operation of the 
Education, Audiovisual and Culture Executive Agency (hereinafter EACEA). The EACEA is 
governed by: 

  The Framework Regulation for executive agencies1; 
  EACEA Act of Establishment2, which sets out its mandate; 
  EACEA Act of Delegation3 and its successive amendments, specifying the tasks to be 
carried  out  by,  and  the  powers  delegated  to,  the  Agency  in  order  to  perform  its 
mandate; 

  The decision appointing the members of the EACEA Steering Committee4. 

In  line  with  the  Commission’s  better  regulation  principles5,  the  evaluation  applies  several 
standard evaluation criteria. The evaluation assesses whether the Agency has fulfilled its tasks 
in  an  effective  and  efficient  way,  whether  there  are  overlaps  /  gaps  /  inconsistencies  in  the 
management  of  the  programme  portfolio  by  the  Agency,  and  whether  there  is  a  clear 
delineation of tasks between EACEA and the parent DGs (coherence). 

The  evaluation  assesses  whether  the  functioning  of  the  Agency  has  yielded  the  expected 
positive results as estimated in the cost-benefit analysis for delegating task to the Agency6 and 
identifies  potential  areas  of  improvement.  To  this  end,  the  estimations  of  the  cost-benefit 
analysis of 2013 have been tested to provide evidence on the validity of the assumptions made 
in  the  ex  ante  scenario  by  considering  the  actual  costs  and  benefits  of  programme 
implementation  by  the  Agency  in  a  structured  way.  The  aspects  to  be  covered  by  the  cost-
benefit analysis are specified in Article 3(1) of the Framework Regulation7 and the Guidelines 
on establishing and operating executive agencies8. 

The  evaluation  does  not  cover  the  achievements  of  the  programmes  managed  by  EACEA, 
which are subject to mid-term and ex post evaluations themselves. The evaluation of EACEA 
nevertheless  provides  useful  input  for  these  programme  evaluations,  considering  that  the 

1  Council  Regulation  58/2003  of  19  December  2003  laying  down  the  statute  for  executive  agencies  to  be 
entrusted with certain tasks in the management of Community programmes, OJ L 11 of 16 January 2003, p. 1. 
2  Commission  Implementing  Decision  2013/776/EU  of  18  December  2013  establishing  the  ‘Education, 
Audiovisual and Culture Executive Agency’ and repealing Decision 2009/336/EC. 
3 Commission decision C (2013) 9189 of 18 December 2013 delegating powers to the  Education, Audiovisual 
and  Culture  Executive  Agency  with  a  view  to  performance  of  tasks  linked  to  the  implementation  of  Union 
programmes  in  the  field  of  education,  audiovisual  and  culture  comprising,  in  particular,  implementation  of 
appropriations entered in the general budget of the Union and of the EDF allocations. 
4  Commission  Decision  C(2018)  4421  appointing  members  and  observers  to  the  Steering  Committee  of  the 
Education, Audiovisual and Culture Executive Agency and repealing Decision C(2015) 758. 
5 Commission’s Better Regulation Communication COM (2015) 215 final and its accompanying Staff Working 
Documents SWD (2015) 111 final and SWD (2015) 110 final. 
6  Cost-benefit  analysis  for  the  delegation  of  certain  tasks  regarding  the  implementation  of  Union  Programmes 
2014-2020 to the executive agencies - Final report for the Commission of 19 August 2013. 
7  Identification  of  the  tasks  justifying  outsourcing,  a  cost-benefit  analysis,  which  includes  the  costs  of 
coordination  and  checks,  the  impact  on  human  resources,  possible  savings  within  the  general  budgetary 
framework  of  the  European  Union,  efficiency  and  flexibility  in  the  implementation  of  outsourced  tasks, 
simplification of the procedures used, proximity of outsourced activities to  final beneficiaries, visibility of the 
EU as promoter of the EU programme concerned and the need to maintain an adequate level of know-how inside 
the Commission.  
8  Appendix  II  of  the  Guidelines  for  the  establishment  and  operation  of  executive  agencies  financed  from  the 
Union budget (C (2014) 9109 from 2 December 2014; pp. 64-72). 

2 

 
 
 
 
 
                                                           
performance  of  the  Agency  has  an  impact  on  the  efficiency  and  effectiveness  of  the 
programmes it manages. 

The evaluation  examines the efficient  use of resources and the  effective achievement of the 
tasks entrusted to it. In particular, it looks at whether: 

  The alignment of programme portfolios with  the Agency’s core competences and its 

brand identity delivered the estimated qualitative benefits;  

  The assembly of the management of different EU programmes delivered the estimated 

synergies, simplification and economies of scale;  

  The pooling of instruments guaranteed consistent service delivery;  
  There is scope for simplification and further efficiency gains.  

This evaluation – the fourth one since the establishment of EACEA – covers the period 2015-
2017. The evaluation is supported by a study carried out by an external contractor9. 
The results of this study are summarised hereafter and this evaluation will be presented to the 
European Parliament, to the Council and to the Court of Auditors in accordance with Article 
25(1) of the Framework Regulation.  

The results will feed into the reflection to assess the opportunity of expanding/ modifying the 
Agency's  mandate  in  view  of  the  delegation  of  the  implementation  of  2021-2027  EU 
programmes. 

2.  BACKGROUND  

2.1.  Description of the intervention and its objectives 

The  outsourcing  of  certain  management  tasks  to  the  executive  agency  according  to  the 
Framework Regulation and EACEA’s Establishing Act aims at: 

  Allowing the Commission to focus on its institutional tasks, i.e. tasks assigned to the 
institutions by the Treaty, which require discretionary powers in translating political 
choices into actions and should not be outsourced. 

  Enabling to achieve the goals of EU programmes more effectively. According to the 
cost-benefit  analysis  carried  out  in  2013,  delegating  tasks  to  the  Agency  was 
estimated  to  be  more  cost-efficient  than  an  in-house  scenario.  The  Act  of 
Establishment  estimated  that  the  alignment  of  more  coherent  programme  portfolios 
with  the  Agency’s  core  competences  and  its  brand  identity  would  bring  qualitative 
benefits.  In  addition,  it  projected  that  assembling  the  management  of  different  EU 
programmes would bring synergies, simplification and economies of scale.  

The original objective of the intervention is entrusting the Agency with the implementation of 
several EU programmes in parts or fully: 

  Erasmus+, including activities under external policies,(DG EAC)10 
  Creative Europe, (DG EAC, DG CNECT)11 
  Europe for Citizens, (DG HOME)12 

9  PPMI  Group,  Study  supporting  the  evaluation  of  the  Education,  Audiovisual  and  Culture  Executive  Agency 

(EACEA) (2015 – 2017). 

10 https://ec.europa.eu/programmes/erasmus-plus/node_en  
11 https://ec.europa.eu/programmes/creative-europe/  

3 

 
 
 
 
 
 
 
                                                           
  EU Aid Volunteers, (DG ECHO)13 
  European  Solidarity  Corps  (DG  EAC)  –  Programme  created  (and  delegated  to 

EACEA) in 201814 

  Actions under direct management via grants and other forms of funding. 

EACEA also implements the legacy of the predecessor programmes and actions. 
Over the evaluation period, and in relation to these programmes, EACEA was entrusted with 
the following tasks: 

  Monitoring  projects,  carrying  out  the  necessary  checks  and  recovery  procedures, 

performing budget implementation tasks covering revenue and expenditure: 

–  Awarding grants and managing the ensuing agreement or decision, including 
the operations required to launch and conclude grant award procedures;  

  Providing support in programme implementation, in particular:  

–  Collecting, processing and distributing data, in particular compiling, analysing 
and  transmitting  to  the  Commission  all  information  required  to  guide  the 
implementation  of  the  programme,  promote  coordination  with  other  Union 
programmes, the Member States or international organisations; 

–  Contributing to assessing the impact of the programme and to monitoring the 

actual effect of measures taken; 

–  Keeping a direct link with the target public of beneficiaries and recipients; 
–  Cooperating in the Commission’s own information and promotion campaigns; 
–  Contributing  to  studies  and  evaluations,  in  particular  the  annual  and/or  mid-
the 

term  evaluations  of  programmes 
preparation and the implementation of follow-up actions to evaluations;  
–  Preparing recommendations for the Commission on the implementation of the 

implementation;  contributing 

to 

programme; 

–  Ensuring control and producing supervision data; 
–  Participating  in  preparatory  work  on  work  programmes  and  financing 

decisions; 

–  Managing technical support services.  

2.2.  Baseline and points of comparison  

The current evaluation of EACEA operations during 2015-2017 assesses the actual costs and 
benefits  of  programme  implementation  by  EACEA  (executive  agency  scenario)  when 
compared with the alternative scenario of management by the Commission services (in-house 
scenario). 

Accordingly, the reference point for the present EACEA evaluation is the 2013 ex ante Cost-
benefit analysis, and the specific financial statement15 (SFS) of EACEA. 

12  http://ec.europa.eu/citizenship/europe-for-citizens-programme/index_en.htm.  As  from  1  January  2020,  the 

Europe for Citizens programme is managed by DG JUST. 

13 https://ec.europa.eu/echo/what/humanitarian-aid/eu-aid-volunteers_en  
14 https://europa.eu/youth/solidarity_en  
15 Original specific financial statement annexed to Commission Decision C(2013)9189 of 18 December 2013 and 
successively amended (including outside the evaluation period) by C(2014)4084 of 25 June 2014, C(2015)658 of 
12 February 2015, C(2016)401 of 1 February 2016, C(2016)1851 of 31 March 2016, C(2017)3049 of 12 May 
2017, C(2018)5011 of 1 August 2018, C(2018)7435 of 13 November 2018, C(2019)1299 of 19 February 2019.  

4 

 
 
 
 
 
                                                                                                                                                                                     
Wherever possible, the analysis of the performance of EACEA during the period of reference 
assesses the progress compared to the previous evaluation, covering the Agency’s operations 
during 2012-2014.  

According to the SFS of EACEA prepared in 2013: 

  The estimated efficiency gains of EUR 57,1 million over the period analysed;  
  The  total  number  of  full  time  equivalents  (FTEs)  required  to  manage  the  relevant 
programmes  in  year  2020,  the  peak  programming  year  in  terms  of  workload,  was 
estimated at 442 for EACEA16; 

  The  total  initial  operational  budget17  entrusted  to  EACEA  in  2020  was  estimated  at 
around  EUR  870  million  in  commitment  appropriations  and  EUR  778  million  in 
payment appropriations; 

  Considerable  efficiency  gains  were  expected  as  compared  to  the  in-house  scenario 
over  the  period,  along  with  non-quantifiable  benefits  such  as  improved  quality  of 
programme  management  and  service  delivery,  improved  visibility  of  the  EU 
programmes and proximity to beneficiaries. 

3. 

IMPLEMENTATION / STATE OF PLAY  

The  Agency’s  mandate  was  extended  by  the  Commission  Implementing  Decision  No 
2013/776/EU, which came into effect on 1 January 2014. Activities delegated to the Agency 
were expanded to include the following: 

  Creative  Europe,  which  replaced  the  MEDIA,  MEDIA  Mundus  and  Culture 

programmes; 

  Erasmus+,  which  replaced  seven  programmes  of  the  previous  period  (the  Lifelong 
Learning Programme, Youth in Action, Erasmus Mundus, Tempus, Alfa, Edulink and 
a programme for cooperation with industrialised countries); 

  Europe for Citizens, which is a continuation of the previous programme;  
  EU Aid Volunteers (an initiative for the cooperation  of volunteers and organisations 
from different countries that was delegated to the EACEA for the first time in 2014); 
  Projects  in  the  field  of  higher  education  under  the  multiannual  financial  framework 
regarding the  financing  of EU  cooperation for African, Caribbean and Pacific States 
and  Overseas  Countries  and  Territories  for  the  2014-2020  period  (10th  European 
Development Fund). 

As  regards  the  activities  delegated  to  EACEA,  over  the  2015-2017  period  the  Agency  had 
four  extensions  of  its  mandate18.  Comparison  with  other  executive  agencies  reveals  that  the 

16 The responsibility over certain competences (i.e. centralisation of validation tasks) originally under the remit 
of  individual  Commission  services  including  EACEA  was  reattributed  to  REA).  The  transfer  led  to  a 
compensation that resulted, for EACEA, in a decrease of 8 FTEs. 
17 This initial budget was estimated prior to the delegation of other spending programmes and activities launched 
in the course of the current MFF 2014-20 (e.g. European Solidarity Corps). 
18 Commission Decision C(2013)9189  delegating powers to the Education, Audiovisual and Culture Executive 
Agency with a view to performance of tasks linked to the implementation of Union programmes in the field of 
education,  audiovisual  and  culture  comprising,  in  particular,  implementation  of  appropriations  entered  in  the 
general  budget  of  the  Union  and  of  the  EDF  allocations  was  amended  by  the  following  four  Commission 
Decisions: 

  C(2015)658 
  C(2016)401 

5 

 
 
 
 
 
                                                           
 
EACEA  had  the  largest  number  of  changes  in  its  mandate  and  had  to  demonstrate  its 
flexibility in accommodating them.  

The new mandate also brought changes in the governance of the Agency as new parent DGs 
joined the Steering Committee (currently DG EAC, DG CNECT, DG HOME19, DG ECHO), 
and subsequently signed a Memorandum of Understanding (MoU) with EACEA20. The new 
MoU  defined  the  modalities  and  procedures  of  interaction  between  EACEA  and  its  parent 
DGs. In addition, it brought a clearer delimitation of the administrative activities between the 
Agency and its parent DGs. The distribution of tasks and responsibilities, as presented in the 
MoU, is included in section 5.3 of this document. Compared to this reference, this evaluation 
assesses whether this distribution  of roles has been respected for the 2015-2017 period, and 
the effects of the intervention on the coherence and the efficiency of EACEA’s action. 

The operational budget of the EACEA allocated to grants under 2014–2017 calls21 amounted 
to  EUR  2.35  billion  (of  which  EUR  71.3  million  related  to  EFTA,  third  countries’  and 
European Development Fund contributions) for the programmes listed under chapter 2 above. 
The Agency’s workload is closely linked with the allocated operational budget, the number of 
proposals received and the number of grants concluded/projects managed. During 2014-2017, 
the  actual  number  of  proposals  received  by  the  EACEA  was  22%  higher  compared  to  the 
CBA estimates.  

According to the study supporting the evaluation and based on a thorough review of activities 
carried  out  during  the  reference  period,  the  overall  number  of  grants  concluded  under  the 
2014-2017 calls22 was 12% higher compared to the CBA estimates. In detail: 

  19% higher for Erasmus+; 
  17% higher for Creative Europe; 
  20% lower for Europe for Citizens23. 

On  the  other  hand,  the  budget  per  operational  head  (in  commitments)  increased  from  EUR  1.35 
million  in  2015  to  EUR  1.61  million  in  2017.  On  average,  EACEA’s  staff  managed  an 
increasing volume of the operational budget during this period. 

EACEA remained substantially stable in size in accordance with the multi-annual planning stated in 
the SFS with 436 actual staff members in 201724. The Agency is based in Brussels. 
A  series  of  organisational  and  procedural  changes  started  with  the  implementation  of  the 
programmes under the multiannual financial framework (MFF) of 2014-2020: 

  Delegation of the management of additional programmes, new funding instruments and 

as a consequence new tasks and activities for the Agency25. 

  C(2016)1851 
  C(2017)3049. 

19 As of 1 January 2020 Directorate-General for Justice and Consumers (JUST) replaced DG HOME. 
20 Memorandum of Understanding between the EACEA and its parent DGs on 26 March 2015. 
21 Includes 2014 commitments and selected project grants as these elements have a direct impact on the 2015-
2017 EACEA’s workload. 
22  Excluding  grants  under  EU  Aid  Volunteers  and  Pan  Africa  programmes,  as  no  estimates  of  the  expected 
number of grants was included in the respective cost benefit analysis or Specific Financial Statement. 
23 The number of Europe for Citizens grants was 20% lower than initially estimated in the CBA and Commission 
annual Work Programme, which related to a higher than anticipated average grant size. 
24  Total number of staff, financed from EU Budget and contributions from EFTA/EEA and/or third countries. 

6 

 
 
 
 
 
 
 
 
                                                                                                                                                                                     
  Further  simplification  and 

improvement  of  grant  management  processes  and 

procedures, including through new business processes and IT tools.  

4.  METHOD 

4.1. 

Short description of methodology 

The  evaluation  was  supported  by  a  study  carried  out  by  external  contractors26.  The 
methodology  of  the  EACEA  evaluation  was  consistent  with  the  approaches  employed  for 
similar parallel evaluations of ERCEA, EASME, REA, INEA and CHAFEA, and built on the 
previous  evaluation  of  EACEA  operations  during  2012-2014.  Specific  methodological 
approaches were used to ensure data triangulation (confirmation from different sources). The 
study was structured around a series of evaluation questions available in Annex. 

Methods used for the analysis includes: 

Interview programme; 

  Desk research (in the form of a review of relevant documents and administrative data); 
  Statistical analysis of the EACEA’s administrative and monitoring data; 
  Survey programme; 
 
  Retrospective cost-benefit analysis; 
  Case studies/focus areas; 
  SWOT analysis; 
  Comparative analysis and benchmarking. 

Desk research carried out for the study consisted of two interrelated components – a review of 
relevant  documents  and  an  analysis  of  statistical  and  monitoring  data  collected  by  the 
EACEA. The evaluation relied also on the results of the 2016 staff opinion survey. 

As regards the survey and interview programmes, an extensive consultation was carried out to 
collect the stakeholders’ views on the functioning and outputs delivered by the Agency. This 
consisted of an extensive interview programme and two surveys: 

  30  semi-structured  in-depth  interviews  (44  persons  interviewed  in  total)  with 
Commission  and  EACEA  staff,  including:  members  of  the  EACEA  Steering 
Committee;  selected  Commission  staff  in  the  parent  DGs  and  in  other  DGs; 
Management of the EACEA; Representatives of the EACEA Staff Committee, other 
employees; 

  37 interviews with unsuccessful applicants and beneficiaries of different programmes 
managed by the EACEA who, while answering the survey, provided their agreement 
to be further contacted; 

  3 surveys targeting EACEA’s beneficiaries and unsuccessful applicants (overall 4879 
answers or 26% response rate), and experts (700 responses or 63% response rate).  

A retrospective cost-benefit analysis was performed. The methodology used was based on the 
assumptions, methods and results of the previous CBA, and the requirements of the legal acts 

25   The  Agency’s  mandate  was  extended  several  times  to  have  additional  activities  include  in  the  remit  of  the 
Agency (see above).  
26  Study supporting the evaluation of the EACEA, final report prepared by PPMI, April 2019. 

7 

 
 
 
 
 
 
 
                                                                                                                                                                                     
and  methodological  documents  setting  guidelines  for  the  establishment  and  operation  of 
executive agencies financed from the general budget of the Union. 

The  analysis  was  complemented  by  an  analysis  of  EACEA’s  performance  in  selected  areas 
through five  case  studies / focus areas. These focused  on specific business processes within 
the Agency. The rationale behind these case studies / focus areas is to shed greater light on 
success stories and  lessons  learned as a result  of key developments during the period 2015-
2017. 
Comparative analysis and benchmarking of the Commission’s various executive agencies was 
also  carried  out  against  a  set  of  (qualitative  and  quantitative)  indicators.  This  enabled  the 
evaluation  team  to  compare  the  performance  of  the  EACEA  effectively  with  that  of  other 
executive agencies. 

4.2.  Limitations and robustness of findings 

The range of stakeholders consulted and the size of the sample within each group make the 
consultation statistically significant. However, compared with the overall population reached, 
unsuccessful applicants participated less actively in the survey than programme beneficiaries. 
Furthermore, unsuccessful applicants and beneficiaries of the Erasmus+ programme were the 
most active group in answering the survey as compared to other applicants and beneficiaries 
contacted by the Agency. 

No  sampling  bias  was  observed  as  the  profile  of  the  respondents  to  the  surveys  was  very 
similar to the overall population, guaranteeing statistical representativeness. The non-response 
bias  (not  all  characteristics  of  the  group  that  did  not  reply  had  been  captured  in  full)  was 
mitigated  through  triangulation  with  the  results  of  follow  up  interviews.  The  triangulation 
approach,  using  multi-level  and  multi-stakeholder  dimension  in  the  data  collection,  ensured 
the robustness and reliability of the data and information used to draw up conclusions in the 
supporting study. 

Survey results were used extensively to answer the evaluation questions on effectiveness and 
efficiency  of  the  Agency.  Results  of  the  interviews  with  the  Commission  and  the  EACEA 
representatives  were  especially  relevant  for  answering  coherence  questions.  Whereas 
interviews with applicants and beneficiaries mostly informed efficiency questions, interviews 
provided both  first-hand factual  information  and opinions, and allowed to contextualise and 
explain  the  data  collected  through  surveys  and  desk  research.  Interviews  also  informed 
preparation  of  some  of  the  case  studies/focus  areas.  Some  views  from  Commission  and 
Agency  staff  gathered  from  the  interviews  and  mentioned  in  the  study  illustrate  personal 
experience,  although  they  may  not  be  representative  of  overall  relations  between  the 
Commission and the Agency. 

5.  ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS 

This  section  is  structured  around  the  evaluation  criteria,  effectiveness,  efficiency  and 
coherence.  Findings  are  presented  by  evaluation  criteria  and,  within  each  criterion,  by  the 
individual questions that form the basis for this evaluation. 

8 

 
 
 
 
 
 
 
 
5.1.  Effectiveness  

For  the  purpose  of  this  evaluation,  effectiveness  relates  to  how  successful  the  Agency  has 
been in achieving or progressing towards its objectives. 

Legal framework 

Mandate of the Agency 
The key documents defining EACEA’s current legal framework are its acts of Establishment27 
and Delegation28, as well as guidelines on executive agencies29. During the evaluation period, 
the Agency implemented its delegated tasks as defined by the legal framework.  

The core responsibility of the EACEA is to manage most (or all, depending on the programme 
delegated) of the phases of programme management and stages in the lifetime of projects and 
operating grants, including: 
  Calls for proposals; 
  Project evaluation and selection; 
  Conclusion of grant agreements; 
  Financial management; 
 
  Monitoring of projects; 
  Feedback to parent DGs and dissemination; 
  Calls for tender (e.g. training, insurance); 
  Management of framework and service contracts. 

Information and support to potential applicants, applicants and beneficiaries; 

Interviews carried out during this evaluation indicate that stakeholders are generally satisfied 
with  the  overall  effectiveness  of  the  Agency  and  how  the  tasks  delegated  to  it  are 
implemented.  

During the evaluation period, there were no changes in the legal framework itself, except for 
the updated mandate of the Agency. Updates occurred to accommodate the new policy needs 
of the Commission, with EACEA taking on board new actions delegated through extensions 
of  its  mandate,  with  particular  regards  to  new  activities  delegated  to  the  Agency  under  the 
Erasmus+  programme.  During  the  interviews  with  the  Commission  and  the  EACEA  some 
concerns were raised that the current legal framework is quite burdensome and not flexible. 

For example, the introduction of even minor changes in the mandate of the Agency creates the 
need  for  a  lengthy  legal  process.  Given  that  in  case  of  the  EACEA  these  changes  in  its 
mandate are quite frequent (due to large number of actions managed / number of parent DGs), 
dealing  with  this  administrative  procedure  requires  a  considerable  amount  of  time  and 
resources. 

27  Commission  implementing  decision  2013/776/EU  of  19  December  2013  establishing  the  ‘Education, 
Audiovisual and Culture Executive Agency’ and repealing Decision 2009/336/EC. 
28  Commission  Decision  of  18  December  2013  delegating  powers  to  the  Education,  Audiovisual  and  Culture 
Executive Agency with a view to performance of tasks linked to the implementation of the Union programmes in 
the  field  of  education,  audiovisual  and  culture  comprising,  in  particular,  implementation  of  appropriations 
entered in the general budget of the Union and of the EDF allocations, C(2013) 9189 final. 
29 Commission, Commission Decision of 2.12.2014 establishing guidelines for the establishment and operation 
of executive agencies financed by the general budget of the Union. Brussels, 2.12.2014, C(2014) 9109 final. // 
Annexes to the Commission Decision. 

9 

 
 
 
 
 
 
 
 
                                                           
Financial Regulation 
An important  issue has  been however identified  concerning the  compliance to  the Financial 
Regulation in relation to the management of programmes. This non-compliance was identified 
by the Internal Audit Service (IAS) in its 2017 audit on Grant Management - Phase I (from 
call  publication  to  contract  signature)  for  Erasmus+  and  the  Creative  Europe  programmes. 
The  IAS  recommendations  identified  the  need  for  actions  on  the  control  environment  for 
grants,  the  evaluation  process,  evaluation  committee  and  the  role  of  experts,  as  well  as  the 
contracting  phase.  As  a  consequence,  the  Agency  introduced  a  reservation  in  its  Annual 
Activity  Reports  for  2017  and  2018.  All  the  necessary  corrective  actions  were  taken  in  the 
course of 2018, so that at the beginning of 2019 all recommendations were closed30 and the 
Agency has lifted up its reservation in the 2019 Annual Activity Report.  

Implementation of delegated programmes and support services 
EACEA’s  tasks  and  objectives  were  set  out  in  its  Commission  Decisions  and  Acts  of 
Delegation. Priorities of the year, objectives, description of activities and the expected results 
are outlined in the work programmes and plans of the delegated programmes.  Each year the 
annual work programme of the EACEA defines several operational priorities with the specific 
KPIs. Operational priorities set in the annual work programmes of the EACEA between 2014 
and 2017 are linked to their parent DGs’ objectives and were quite stable. 

At the beginning of the MFF 2014-2020, an emphasis was placed on the successful launch of 
new  programmes  and  timely  selection  and  contracting  processes,  including  comprehensive 
guidance  for  beneficiaries  of  new  programmes,  while  continuing  to  ensure  the  financial 
management and monitoring of projects of the previous generation of programmes under the 
respective rules.  

Since 2016, when the implementation of new programmes was streamlined, the emphasis has 
been  placed  on  the  added  value  of  the  Agency.  The  need  for  enhancing  knowledge  sharing 
and  good  practice  has  started  to  be  underlined  since  2015.  Providing  policy  support  to  the 
Commission remained a stable priority from 2014 to 2017. 

Overall, four operational priorities of the EACEA guided its operations in 2015-2017: 

Improve performance in programme management; 

  Adapt to the new structure of the Commission (relevant in 2015); 
 
  Reinforce policy support to the Commission; 
  Enhance knowledge sharing and good practice. 

EACEA’s  Annual  Activity  Reports  confirm  that  the  Agency  has  been  effective  in 
implementation of its operational priorities. The Agency was establishing communication and 
setting certain practices in place to adapt to the new Commission structure and accommodate 
the needs of its new parent DGs. It was constantly introducing various measures to improve 
performance in programme management. It emphasised the importance of reinforcing policy 
support to the Commission and enhancing knowledge and good practice sharing and looked 
for the ways to respond to these needs. 

30 The IAS also identified other serious shortcomings affecting the effectiveness of the overall control system in 
its audit on Grant Management  - Phase II (from project monitoring to payment),  following  which the  Agency 
took a set of necessary corrective actions. The kick-off meeting of the third and final phase of the IAS audit on 
Grant Management, focussing on ex-post controls, took place on 9 December 2019. 

10 

 
 
 
 
 
 
 
 
                                                           
The  evaluation  of  the  EACEA’s  performance  is  primarily  based  on  the  analysis  and 
interpretation of the key performance indicators related to:  
  Timely execution of the delegated functions;  
  Cost-effectiveness of the management and control arrangements; 
  Effectiveness  of  the  established  supervisory  and  control  systems  in  ensuring  the 

legality and regularity of the programmes;  

  Budget execution of commitment and payment appropriations. 

Evaluation and selection of proposals, conclusion of grant agreements 
The main indicators describing the timeliness of the evaluation and the contracting process are 
the following: 

Indicator 

Definition 

Target 

Time-to-Inform (TTI) 

Time-to-Contract (TTC) 

Time duration between the 
application deadline and the 
grant award decision 

Time duration between the 
grant award decision and the 
signature of contracts/sending 
of decisions 

< 6 months 

< 3 months 

Overall Time-to-Grant (TTG) 

Time duration between the 
application deadline and the 
signature of contracts/sending 
of decisions 

< 8 months 

(EACEA-specific target, 
tighter than the 9 months 
foreseen in the Financial 
Regulation) 

During the evaluation period, the EACEA successfully met all targets. The increase for TTG 
in  2017 compared to  the previous period was related to  the revision of the TTG calculation 
methodology, which was aligned to the methodology specified in the Vademecum for Grant 
Management. 

EACEA’S PERFORMANCE IN TERMS OF TTI AND TTC  

Time-To-Inform 

Time-To-Contract 

3,1 

3,6 

3,6 

6 

4,1 

8

6

4

2

0

2014

2015

2016

2017

TTI all programmes

Target

Source: PPMI based on EACEA’s Annual Activity Reports. 

4,0

3,0

2,0

1,0

0,0

11 

3 

2,1 

1,7 

2,1 

3,0 

2014

2015

2016

2017

TTC all programmes

Target

 
 
 
 
 
 
 
 
 
 
The increased TTI and TTC in 2017 results from a revised calculation methodology of these 
indicators. 

EACEA’S PERFORMANCE IN TERMS OF TTG BY PROGRAMME  

s
h
t
n
o
M

10
9
8
7
6
5
4
3
2
1
0

9 
8 

7,1 

5,6 

5,2 

5,3 

Time-To-Grant 

7,3 

7,2 

6,4 

6,1 

6,4 

5,4 

5,3 

4 

5,4 

4,5 

3,7 

3,3 

5,8 

4,8 

4,9 

7,1 

4,8 

Time to grant all
programmes

Erasmus+

Creative Europe

Europe for
Citizens

EU Aid
Volunteers

Pan-Africa

2014

2015

2016

2017

EACEA target

FR target

5,2

5,3

5,6

7,1

8

9

4

5,4

5,3

7,3

8

9

6,4

6,1

6,4

7,2

8

9

3,3

3,7

4,5

5,4

8

9

4,8

4,9

5,8

8

9

4,8

7,1

8

9

Note: TTG for 2017 is based on the revised calculation methodology 
Source: PPMI based on EACEA’s Annual Activity Reports.  

Even though EACEA met the KPIs related to timely evaluation and selection of proposals and 
conclusion  of  grant  agreements,  in  surveys  and  interviews  applicants  and  beneficiaries 
expressed their desire for an even quicker evaluation and selection process. 

EACEA’s  performance  can  also  be  measured  in  terms  of  the  quality  and  timeliness  of 
information  and  guidance.  The  survey  revealed  that  applicants  and  beneficiaries  generally 
understood well the application and selection process (85% of respondents), the principles of 
the  selection  procedure  (79%)  and  the  priorities  and  objectives  of  the  Programme  Guide 
(81%).  The  respondents  were  generally  positive  on  the  clarity  and  ease  of  finding  of 
information material and agreed that the provided information enabled them to understand the 
application procedures. A somewhat lower satisfaction level (73%) was recorded in relation to 
the user-friendliness of the Programme Guide for applicants31. 

On  the  IT  side,  the  survey  revealed  that  the  perception  of  beneficiaries  and  applicants  as 
regards the proportionality of administrative requirements and user-friendliness of the IT tools 
remained  similar  to  the  previous  evaluation.  A  total  of  70%  of  respondents  agreed  that  the 
requirements  for  the  application  process  are  reasonable  and  proportionate,  though  a  lower 
share of respondents (66%) was satisfied with the user-friendliness of the Participant Portal32. 
Interviews  with  beneficiaries  and  unsuccessful  applicants  also  revealed  that  certain  parts  of 
the  application  procedure  are  seen  as  overly  complex:  while  the  process  itself  was  called 
‘straightforward’  and  ‘logically  coherent’,  some  of  the  applicants  complained  about  the 
application forms, which are said to be labour- and time-consuming. 

31 The Programme Guide is published by DG EAC and co-shared by EACEA and DG EAC. 
32 The Participant Portal is not managed by EACEA and DG EAC. 

12 

 
 
 
 
 
 
 
                                                           
 
Lastly,  interviews  revealed  that  applicants  are  generally  grateful  for  feedback  on  their 
application and find the comments and suggestions helpful for their future applications, even 
if  they  do  not  necessarily  agree  with  the  evaluators’  opinion  and  would,  in  some  cases, 
appreciate even more in-depth comments. 

The evaluation and selection of proposals was impacted by the conclusions of the 2017 IAS 
audit on Grant Management referred to above. Following this report, changes were introduced 
in the evaluation procedures for the 2018 calls for all the programmes managed by EACEA, 
which led to  improved controls  over potential cases of conflicts  of interests among external 
experts.  The  revised  evaluation  procedures  also  brought  about  a  redefinition  of  the  role  of 
evaluation committees as evaluators of all proposals, which implied a more active role of staff 
from  the  EACEA  and  the  Commission  in  the  evaluation  process.  All  these  changes  were 
introduced  rapidly  as  a  reaction  to  the  audit  conclusions,  in  parallel  with  the  evaluation 
process  of  2018  calls.  This  experience  is  being  taken  into  account  in  the  definition  of  the 
implementing provisions for the 2021-2027 MFF programmes33. 

Follow-up, monitoring and control of grant implementation, payments 
During  2015  –  2017,  the  EACEA  maintained  a  good  performance  in  terms  of  timely 
execution of payments to beneficiaries (measured as Time-to-Pay – TTP) with over 94% of 
payments  made  within  contractual  time  limits.  The  Agency’s  performance  hence  remained 
stable compared to the previous evaluation period.  

Good  performance  of  the  EACEA  in  terms  of  timely  processing  of  payments  was  also 
reflected in the results of the beneficiaries’ survey – 93% of beneficiaries were satisfied with 
the time period it took the Agency to make a pre-financing payment, 91% and 88% of them  
with the time period it took the EACEA to process interim and final payments, respectively. 
During  2015-2017,  the  EACEA  further  improved  project  monitoring  activities.  Like  in  the 
previous  evaluation  period,  EACEA’s  project  monitoring  framework  included  desk-based 
monitoring  (verification  of  various  reports  submitted  by  project  beneficiaries  and  other 
administrative data) and monitoring visits.  

The survey of EACEA’s beneficiaries revealed that beneficiaries were generally satisfied with 
most aspects related to the project implementation stage, and the satisfaction level improved 
compared  to  the  results  of  the  previous  evaluation.  Compared  with  the  results  of  the  staff 
opinion  survey  carried  out  in  2016,  more  respondents  agreed  that  the  project  reporting 
requirements  were  reasonable  taking  into  account  the  size  of  the  grant  and  the  scale  of  the 
project. 

As regards budget execution, in the period 2015- 2017, the EACEA managed to fully execute 
its  operational  budget,  both  in  terms  of  commitment  and  payment  appropriations,  hence 
meeting  both  targets:  98%  for  2015  –  2016  (raised  to  99%  in  2017)  for  commitment 
appropriations and 95% for 2015 – 2016 (raised to 98% in 2017) for payment appropriations. 

33 In 2018, in its audit on Grant Management - Phase II (from project monitoring to payment), the IAS identified 
shortcomings affecting also the effectiveness of the overall control system, following which the Agency took a 
set of necessary corrective actions. The full impact of the changes in the procedures implemented by EACEA in 
response to the findings of Phases I, II and III (started December 2019) of the audit will be assessed in full detail 
in the next triennial evaluation report. 

13 

 
 
 
 
 
 
 
 
                                                           
EXECUTION OF THE EACEA’S OPERATIONAL BUDGET 

100%

98%

96%

94%

92%

100% 
98% 

95% 

2015

100% 

100,0% 

100,0% 

98% 

95% 

2016

100% 

100% 
99% 

98% 

2017

Executed commitment appropriations

Executed payment appropriations

Target for commitments

Target for payments

Note: Based on C1 and C5 credits. Executed credits compared to the available budget at the end of the year (%). 
Source: PPMI, based on the EACEA’s Annual Activity Reports and Annual Work Programmes.  

Concerning the legality and regularity of the programme expenditure, executive agencies have 
to set up internal control processes ensuring adequate management of the risks related to the 
legality  and  regularity  of  the  underlying  transactions,  taking  into  account  the  multiannual 
character of the programmes and the nature of the payments concerned.  The related control 
objective is to ensure that the residual error rate does not exceed 2% on a cumulative basis by 
the  end  of  each  programme  implementation.  The  residual  risk  of  error  is  estimated  by  the 
residual  error  rate  obtained  from  an  examination  of  a  sample  of  randomly  selected 
transactions, less any corrections made as a result of the control systems that are in place. 

During the evaluation period, the estimated residual error rate was above 2% for some of the 
“legacy”  programmes,  i.e.  programmes  from  the  2007  –  2013  MFF.  Therefore,  EACEA 
introduced  reservations  in  its  Annual  Activity  Reports.  However,  due  to  the  increasingly 
smaller  proportion  of  the  financial  impact  on  the  overall  financial  portfolio  of  the  Agency, 
these reservations were lifted by the Agency as of its 2019 Annual Activity Reports34.  
The  residual  error  rates  were  estimated  for  the  2014-2020  MFF  programmes  starting  from 
2016. The first results for 2016 and 2017 were very good with an estimated residual error for 
all programmes being substantially below the 2% materiality threshold, as visible in the last 
three columns of the chart below. 

34  In  2017  and  2018,  the  Agency  also  introduced  a  reservation  in  its  Annual  Activity  Reports  concerning  its 
internal  control  system  assessed  as  partially  functioning  following  the  findings  of  the  IAS  audits  on  grant 
management  –  Phase  I  and  Phase  II.  Due  to  the  corrective  actions  put  in  place  and  the  improvement  of  the 
situation, this reservation was lifted by the Agency in its 2019 Annual Activity Report. 

14 

 
 
 
 
 
                                                           
RESIDUAL ERROR RATE 

10,00%
9,00%
8,00%
7,00%
6,00%
5,00%
4,00%
3,00%
2,00%
1,00%
0,00%

LLP 2007-
2013

Erasmus
Mundus and
Intra ACP
2007-2013

Tempus and
Bil. Coop.
2007-2013

Youth  2007-
2013

Culture 2007-
2013

2015

2016

2017

Target

4,52%

4,52%

4,42%

2,00%

0,19%

0,37%

0,34%

2,00%

0,94%

0,99%

1,52%

2,00%

3,00%

2,96%

3,20%

2,00%

9,13%

9,13%

9,18%

2,00%

Source: PPMI, based on the EACEA’s Annual Activity Reports.  

MEDIA and
MEDIA
Mundus
2007-2013

0,48%

0,53%

0,53%

2,00%

Europe for
Citizens
2007-2013

Erasmus+
2014-2020

Creative
Europe 2014-
2020

Europe for
Citizens 2014-
2020

0,78%

0,81%

0,81%

2,00%

0,08%

0,66%

2,00%

0,03%

0,44%

2,00%

0,00%

0,00%

2,00%

Overall satisfaction of EACEA’s beneficiaries 
Results  from  surveys  indicate  that  the  satisfaction  with  the  overall  performance  of  the 
EACEA  remained  stable  since  2012,  with  a  comparable  degree  of  satisfaction  across  the 
programmes managed by EACEA. 

In particular, the level of satisfaction expressed in terms of the share of beneficiaries agreeing 
with the statement ‘We are satisfied with the overall quality of the programme management 
services  provided  by  the  EACEA  during  the  whole  project  period’  increased  slightly  as 
compared to the previous evaluation and reached 88.8%. 

EACEA  beneficiaries  rated  most  positively  the  courtesy  (1.84),  commitment  (1.9)  and 
competence (2.01) of the Agency’s personnel (where 1 is the highest possible score, and 9 the 
lowest). 

SATISFACTION OF APPLICANTS AND BENEFICIARIES WITH THE OVERALL QUALITY AND 
TIMELINESS OF EACEA’S SERVICES 

QUESTION 

2008 

2010 

2012 

2015 

We  are  satisfied  with  the  overall  quality  of  the 
programme  management  services  provided  by  the 
EACEA during the whole project period  

2.8 

2.2 

2.8 

We received a reply on the selection results within the 
deadlines  announced  in  the  Call  for  Proposals  or 
Programme Guide 

3.9 

3.0 

2.9 

2.8 
(2.5 beneficiaries 
and 3.4 
unsuccessful 
applicants) 
2.7 
(2.5 beneficiaries 
and 3.0 
unsuccessful 
applicants) 

2018 (CURRENT 
EVALUATION) 

2.8 (2.3 
beneficiaries and 
3.5 unsuccessful 
applicants) 

2.9 (2.7 
beneficiaries and 
3.1 unsuccessful 
applicants) 

received 

After  being  informed  that  our  project  had  been 
selected,  we 
in  good 
time/without significant delays 
The  time  it  took  for  the  EACEA  to  process  grant 
amendment requests was appropriate 

the  contract 

2.4 

2.9 

2.4 

2.4 

Source: PPMI, Survey of EACEA’s unsuccessful applicants and beneficiaries, 2018 (average response on scale of 1-9 – 1=strongly agree and 
9=strongly disagree). 

15 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
OVERALL SATISFACTION WITH THE EACEA AMONG BENEFICIARIES AND UNSUCCESSFUL 
APPLICANTS BY PROGRAMME 

3,46 

2,84 

3,7 

2,93 

2,28 

2,27 

3,38 

2,76 

2,28 

4,33 

3,84 

3,77 

3,39 

2,92 

2,18 

2,62 

2,75 

2,27 

5

4

3

2

1

0

All programmes

Creative Europe
(N=1288)

Erasmus+ (N=2473)

EU Aid Volunteers
(N=25)

Europe for Citizens
(N=806)

Intra-Africa Academic
Mobility Scheme
(N=39)

All respondents

Beneficiaries

Unsuccessfull applicants

Average satisfaction of all programmes and all respondents

Note: on the scale from 1 to 9, with one being the most positive rating possible 
Source: PPMI, Survey of unsuccessful applicants and beneficiaries of the EACEA, 2018. 

Proximity to addressees and visibility of the EU  
The  nature  of  the  programmes  and  actions  managed  makes  EACEA  one  of  the  closest 
agencies to citizens, and results of the surveys carried out during the evaluation confirm that 
EACEA managed to remain close to its applicants and beneficiaries. 

PROXIMITY  OF  THE  EACEA  AND  ITS  STAFF  TO  BENEFICIAIRES  AND  UNSUCCESSFUL 
APPLICANTS 

We knew who to contact for any question(s) we had or where to get
help when preparing our application (N=4730)

We knew who to contact for any question(s) we had or where to get
help when submitting our application (N=4627)

The EACEA staff assigned to my project in the grant finalisation and
negotiation phase were easily available and responsive (N=1965)

The EACEA staff assigned to my project during the implementation
were easily available and responsive (N=1958)

The EACEA staff assigned to my grant amendment procedure were
easily available and responsive (N=486)

73% 

72% 

87% 

85% 

91% 

11% 

16% 

12% 

15% 

7% 

5% 

8% 

7% 

3% 

6% 

0%

10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Agree

Neutral

Disagree

Note: the first two statements represent a combined opinion of both beneficiaries and unsuccessful applicants; the remaining statements 
were addressed to and answered only by beneficiaries. 
PPMI, Survey of beneficiaries and unsuccessful applicants 2018.Questions 12, 15, 19 and 23. 

Thematic  cluster  meetings  help  discover  synergies  among  projects  and  create  additional 
opportunities. For instance, beneficiaries can interact and present their projects in an informal 
setting. By taking part in these events, project representatives get the necessary clarifications 
and learn from more experienced participants. 

External  experts  also  confirm  their  very  strong  satisfaction  with  EACEA’s  staff  ability  to 
provide  useful  answers  to  their  questions  and  their  responsiveness.  It  was  also  clear  for  the 
external experts whom to contact or where to get help regarding any questions they had when 
working  on  their  tasks.  90%  of  the  external  experts  interviewed  were  satisfied  with  their 
overall experience with the EACEA, with another 4% giving a neutral assessment. 

16 

 
 
 
 
 
 
PROXIMITY OF THE EACEA AND ITS STAFF TO EXTERNAL EXPERTS 

EACEA staff with whom I worked with provided useful answers
to my questions (N=688)

EACEA staff with whom I worked with were responsive (e,g, by
email or phone) (N=691)

I knew either who to contact or where to get help regarding
any questions I had when working on my tasks (N=691)

72% 

75% 

22% 

5% 

20% 

3% 

62% 

31% 

4% 

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Strongly agree

Rather agree

Neither agree nor disagree

Rather disagree

Strongly disagree

Source: PPMI, Survey of external experts of the EACEA, 2018. Question 20: To what extent do you agree or disagree with the following 
statements concerning your work as an expert for the EACEA? During the evaluation/monitoring: 

Applicants express, however, also several concerns. First, over the years it has become more 
difficult  to  get  a  prompt  response  from  EACEA.  Applicants  and  beneficiaries  would  also 
appreciate more overall stability in staff managing a project, as well as more on-site project 
visits. 

As regards the visibility of the EU as promoter of the programmes entrusted to the Agency: 

 

Interviews with beneficiaries and unsuccessful  applicants revealed that, when talking 
about the EACEA, they usually refer to the European Commission, which proves their 
awareness of EACEA managing programmes on behalf of the Commission; 

  When  asked  explicitly,  the  large  majority  of  respondents  states  being  aware  that 
EACEA  is  entrusted  with  the  management  of  programmes  by  the  European 
Commission; 

  Over two in three respondents to the surveys agreed that funding opportunities under 

programmes managed by the EACEA are well promoted. 

 

It should also be noted that the EACEA is using the same visual identity as the Commission.  

External communication 
Through  its  external  communication  activities,  the  EACEA  implements  one  of  its  tasks, 
namely  to  support  the  Commission  in  communication  and  promotion  of  the  programmes  it 
manages. 

Certain positive developments in external communication materialised during the evaluation 
period. In 2016, the culture of feedback was a priority for the EACEA, therefore collection, 
analysis  and  reporting  of  feedback  from  events  has  been  streamlined  and  a  systematic 
approach has been developed and tested for all infodays as well as several smaller events.  In 
2017 alone, a total of 40 such surveys were carried out with a view to identifying points for 
further improvement in the communication with stakeholders (both beneficiaries and experts).  
Advancements have also been achieved on the IT side, with continuous improvements to the 
website, which in 2017 ranked 7th on the list of the most visited sites on europa.eu portal. 

During  the  evaluation  period,  the  Agency  acted  as  a  focal  point  for  applicants  and 
beneficiaries of the programmes it manages. Its external communication helped to ensure that 
the  potential  applicants  are  informed  about  the  funding  possibilities  and  the  beneficiaries 
received  timely,  helpful  and  relevant  information  for  their  queries.  This  is  proved  by  the 
feedback  received  from  unsuccessful  applicants  and  beneficiaries  through  the  survey  and 
interview programmes, undertaken during the current evaluation. 

17 

 
 
 
 
 
 
 
 
 
EFFECTIVENESS OF COMMUNICATION CHANNELS IN COMMUNICATING WITH 
BENEFICIARIES 

E-mail contact (N=2008)

EACEA website (N=1896)

92% 

75% 

6% 

2% 

16% 

9% 

Face-to-face contacts (meetings, events, information sessions,…

Telephone contact (N=1533)

66% 

56% 

15% 

19% 

20% 

24% 

Online communities (N=1278)

41% 

22% 

37% 

News feeds (RSS) (N=1093)

Video conferences (N=1118)

30% 

30% 

24% 

20% 

46% 

50% 

0%

10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Provided with relevant and helpful information

Neutral

Did not povide with relevant and helpful information

Source: PPMI, Survey of beneficiaries and unsuccessful applicants 2018. Question 24: Please indicate on a scale from 1 to 9 (1 = to a large 
extent, 9 = not at all) to what extent the following communication channels used by the EACEA have provided you, if at all, with relevant 
and helpful information when you needed it. 

Interviews suggest that both unsuccessful applicants and beneficiaries evaluated the Agency’s 
communication activities as being effective, especially Agency’s efforts in promoting funding 
opportunities.  Although  some  of  the  interviewees  recommended  to  further  improve  the 
external  communication  by  employing  a  more  direct  approach  in  disseminating  the 
information to interested parties. In addition, some feedback received during interviews with 
applicants  and  beneficiaries  and  via  the  open-ended  survey  questions  indicated  that  the 
website should be further improved. 

SATISFACTION WITH VARIOUS EXTERNAL COMMUNICATION ASPECTS BY BENEFICIARIES 
AND UNSUCCESSFUL APPLICANTS 

The events (information days, project meetings, information visits,
cluster meetings, etc.) organised by the EACEA are useful (N=3938)

EACEA provides sufficient and user-friendly information about the
programmes it manages (N=4685)

EACEA’s communication provides relevant and useful information 
(N=4680) 

The information provided by the EACEA is easily accessible (N=4726)

EACEA’s website and information materials are visually appealing and 
user-friendly (N=4677) 

78% 

77% 

81% 

75% 

71% 

16% 

6% 

11% 

12% 

11% 

8% 

12% 

13% 

13% 

16% 

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Agree

Neutral

Disagree

Source: PPMI, survey of the EACEA’s beneficiaries and unsuccessful applicants 2018.  

 

 

 

Question 9: Please indicate on a scale from 1 to 9 (1 = strongly agree, 9 = strongly disagree) to what extent you agree or 
disagree with the following statements regarding the awareness of programmes managed by the EACEA; 
Question 12: Please indicate on a scale from 1 to 9 (1 = strongly agree, 9 = strongly disagree) to what extent you agree or 
disagree with the following statements concerning the application process; 
Question 29: Based on your experience, please indicate on a scale from 1 to 9 (1 = strongly agree, 9 = strongly disagree) to what 
extent you agree or disagree with the following statements concerning the EACEA’s actual performance and the services it 
provides 

18 

 
 
 
 
 
 
 
Opinions  on  the  effectiveness  in  disseminating  project  results,  identified  during  the 
interviews, were more divergent,  with  quite some respondents  being unaware where to  find 
such information or indicating that it should be made more user friendly. 

5.2.  Efficiency  

This  section  considers  the  relationship  between  the  resources  used  by  the  Agency  and  the 
output.  The  analysis,  among  other  factors,  also  includes  analysis  of  administrative  and 
regulatory burden and looks at aspects of simplification. 
The  ratio  between  the  administrative  budget  of  the  Agency  and  the  operational  budget 
managed  by  it,  was  during  2015-2017  in  the  range  of  7.0  -  7.7%  based  on  commitment 
appropriations of the operational budget and 7.4%  - 8.1% based on payment appropriations, 
i.e. similar to the values of the previous evaluation period. 

The relatively high administrative costs of the EACEA and the programmes managed by the 
Agency  were  related  primarily  to  the  specific  character  of  the  funding  schemes.  Their 
complexity and the large number of rather small grants (around half of projects managed by 
EACEA are below EUR 50.000) awarded to a large population of grant beneficiaries (which 
in many cases are small and possess limited administrative capacities) make implementation 
of  the  programmes  delegated  to  the  EACEA  on  average  more  expensive,  though  with 
significant variations across programmes:  

RATIO  BETWEEN  THE  EACEA’S  ADMINISTRATIVE  AND  OPERATIONAL  BUDGET  BY 
PROGRAMME 

Based on executed commitment appropriations of the operational budget 

11,5% 

12,9% 

9,0% 

7,7% 

7,5% 

7,7% 

5,2% 

9,0% 

7,6% 

7,5% 

7,7% 

8,4% 

7,6% 

6,8% 

9,3% 

7,1% 

7,4% 

6,7% 

7,0% 

6,2% 

7,2% 

7,5% 

8,9% 

8,0% 

7,3% 

14,0%
12,0%
10,0%
8,0%
6,0%
4,0%
2,0%
0,0%

2014

2015

2016

2017

2014-2017

Erasmus+

Creative Europe

Europe for Citizens

EU Aid Volunteers

All programmes

Note: the Agency’s administrative budget is based on executed commitment appropriations 
Source: PPMI, based on the EACEA’s Annual Activity Reports and data provided by the Agency.  

PERFORMANCE INDICATORS OF THE EACEA, 2015-2017 

2015 

2016 

2017 

Operational budget, commitments 

Actual  number  of  staff  (at  the  end  of  the 
year) 

597.31 

441 

Budget 
‘per 
commitments) 

head’ 

(according 

to 

1.35 

Proposals ‘per head’ 

28.25 

674.90 

442 

1.53 

28.28 

702.70 

436 

1.61 

28.90 

Running projects ‘per head’ 
Source: PPMI, based on data provided in the EACEA’s documents (annual activity reports for 2015, 2016 and 2017). 

13.64 

8.52 

13.74 

19 

 
 
 
 
 
 
 
  
 
A detailed analysis of the  EACEA cost-efficiency and results of the Cost-benefit analysis is 
presented under point 5.4. 

Human resource management 
The  number  of  staff  at  the  EACEA  remained  quite  stable  during  the  evaluation  period.  In 
2016,  each  EACEA  staff  member  managed  on  average  28.28  proposals  (compared  to  the 
average of 18.55 in all six executive agencies) and 13.64 ongoing projects (compared to the 
average of 8.67 in all six executive agencies). For the purpose of these calculations, the whole 
staff  of  executive  agencies  is  considered  (i.e.  not  limited  to  staff  members  with  project 
management functions).  

Interviewees indicated that overall the Agency had sufficient resources to implement the tasks 
delegated to it and, despite the relatively high amount of files handled by the Agency’s staff, 
in  the  2016  Commission  staff  opinion  survey  67%  of  EACEA’s  staff  agreed  that  their 
workload  was  acceptable  compared  to  59%  of  the  Commission’s  and  executive  agencies’ 
average. The actual workload of the Agency was generally in line with the estimates from the 
initial CBA estimates.  

In  terms  of  the  ratio  between  administrative  vs.  operational  staff  in  the  EACEA,  the 
Commission’s  screening  exercise  in  2017  indicated  that  the  share  of  administrative  support 
and coordination staff in the executive agencies was 9.5% on average. The EACEA employed 
slightly  more  staff  dedicated  to  general  coordination,  human  resources,  information 
technologies and other administrative job categories, with a share35 of 10.7%.  

Regarding the staff turnover and vacancy rate, during the period 2015-2017 the average job 
occupancy rate within the EACEA was around 95%. The Agency was therefore nearly fully 
staffed throughout the evaluation period. The rates of staff turnover in the years 2015, 2016 
and 2017 were 5.4%, 6% and 4.1%, respectively.  

The extent of internal mobility declined during the evaluation period, from 48 moves in 2015 
to 20 in 2017, although the Agency undertook some efforts to improve this towards the end of 
the evaluation  period. According to  the evaluation  evidence, the processes of staff selection 
and recruitment at the Agency ran smoothly. 

The results of the 2016 European Commission staff opinion survey made it possible to assess 
and compare the HR performance of the different Commission’s executive agencies. EACEA 
s results for staff engagement index36, staff wellbeing and overall job satisfaction were close 
to the average for the European Commission and its executive agencies: 

I have the appropriate and timely information to do my work well;  

35 The share of staff working in the Agency’s horizontal functions has gradually decreased since, to 9,6% in 2017 
and 8,5% in 2018. 
36 This index is composed of the following seven questions:  
• 
•  My colleagues are committed to doing quality work; 
• 
• 
• 
•  My manager seems to care about me as a person; 
•  My line manager helps me to identify my training and development needs. 

I have a clear understanding of what is expected from me at work; 
I have recently received recognition or praise for good work; 
I feel that my opinion is valued;  

20 

 
 
 
 
 
 
 
 
 
                                                           
STAFF  ENGAGEMENT  INDEX,  OVERALL  JOB  SATISFACTION  AND  WELLBEING  IN  THE 
EACEA, THE EXECUTIVE AGENCIES AND ALL COMMISSION SERVICES, 2016 

70 

65 

67 
66 
64 

55 

68 

70 

68 

70 

75 

76 

71 

72 

72 
70 

58 

47 

33 

49 
49 

35 

55 

49 

80

60

40

20

0

Staff engagement

Overall job satisfaction

Staff wellbeing

CHAFEA

REA

EACEA

INEA

ERCEA

EASME

Average of the Executive Agencies

Average of the Commission

Source: PPMI, based on the 2016 European Commission Staff Opinion Survey: Analysis of the findings, September 2016. 

The EACEA’s staff engagement score increased from 60 in 2014 to 65 in 2016, but this index 
has not yet returned to the level achieved at the Agency in 2013, when it scored 72. However, 
the EACEA saw an increase in its scores for overall job satisfaction between 2014 and 2016 
from 68% to 70%, and from 33% to 47% for staff wellbeing in the same period. 

However, staff’s perception of the Agency as a modern and attractive workplace was below 
the  average  for  the  Commission’s  executive  agencies  (56%  in  the  EACEA,  compared  with 
65% in all executive agencies). Another weak point in EACEA’s staff opinion survey results 
concerns career opportunities. 

These  results  provided  useful  guidance  when  the  Agency  defined  its  strategic  HR  priorities 
and  developed  an  action  plan  to  further  increase  staff  engagement  and  to  strengthen  the 
Agency as an attractive workplace, primarily by looking into how the career opportunities for 
its contract agents can be improved under the existing rules. Other points from the action plan 
that  were  implemented  during  the  evaluation  period  included  a  staff  day  focusing  on  staff 
engagement mobility and knowledge sharing, as well as a job-shadowing pilot.  

It needs however to be mentioned that the new staff opinion survey results of 2018, released 
after  the  evaluation  period,  have  shown  significant  drops  of  EACEA  staff  satisfaction  in  a 
variety of surveyed areas. The Agency with the support of its parent DGs is in the process of 
analysing  more  in  depth  those  results  and  their  origins  in  order  to  actively  address  them 
through appropriate action in the near future. 

The  findings  of  the  staff  opinion  survey  of  2018  were  addressed  mostly  by  the  EACEA 
Transformation  Project  (ETP).  The  purpose  of  the  ETP  was  in  particular  to  transform  the 
Agency  into  a  modern  organisation  ready  to  respond  to  the  future  challenges  of  the  current 
and  new  programmes,  and  thus  to  continue  delivering  high-quality  results  and  keep  staff 
engaged.  

The  Transformation  Project  was  designed  to  be  fully  participatory.  Staff  were  able  to 
contribute  directly  to  shaping  the  future  of  the  Agency  by  having  their  say  on  matters 
impacting  their  working  life.  The  project  was  carried  out  in  full  transparency  and  was 
designed to include all staff.  

In  addition,  on  10  July  2019,  the  Agency  presented  to  the  Steering  Committee  the  Draft 
Development Plan.  

21 

 
 
 
 
 
 
 
 
 
 
Specific  staff  engagement  actions  started  in  the  second  half  of  2019  are  “Connecting 
EACEA” (info sessions and conferences targeting topics related to the Agency’s portfolio and 
connecting to  the parent DGs) and  “EACEA Empowered” (bottom-up initiatives  on various 
topics of common interest).  

Organisational structure 
Regarding  the  organisational  structure  of  the  Agency,  the  internal  structure  of  the  EACEA 
had been adjusted to the new programmes under its administration after the start of the 2014-
2020  programming  period  (creation  within  Erasmus+,  hence  under  the  responsibility  of  the 
Head of Department, of the Unit responsible for Youth and Sport) and remained substantially 
unchanged during the evaluation period. 

Up  to  March  2020,  the  Agency  consisted  of  13  units:  seven  under  the  Department  of 
Erasmus+, and the remaining six reporting directly to the Director – two units for the Creative 
Europe programme, one for the Europe for Citizens programme, and three support units. 

Within its organisational structure, the EACEA has centralised its financial circuits somewhat 
by  merging  the  financial  resource  management  of  Units  A7  (Education  and  Youth  Policy 
Analysis) and A1, as well as merging the financial sector of A5 with A1, which increased the 
economy of resources.  

Despite  appointing  a  new  Head  of  Department  for  Erasmus+  units  during  the  previous 
evaluation  period,  the  organisational  structure  of  the  EACEA  has  remained  flatter  than  in 
most other executive agencies of the Commission.  

This could be reviewed taking into account the organisational characteristics of the Agency, 
the ongoing changes in the organisation and the various options for its organisational structure 
(including  the  possibility  of  introducing  new  departments).  A  new  organisational  structure 
could improve coordination of the implementation of programmes. 
The organisational chart of the EACEA is integrated in the organigrams of the parent DGs by 
linking various units of the Agency to the relevant directorates and units of the parent DG’s 
organigrams. This integration increased transparency and facilitated the collaboration between 
EACEA’s units and their mirror units in the parent DGs.  

Aspects of simplification  
Programmes managed by the EACEA benefit from the streamlining of various processes and 
practices.  For  instance,  the  implementation  of  the  Erasmus+  programme  benefits  from  the 
widespread use of simplified cost options covering travel, subsistence, staff salaries and other 
costs. The use of standard cost options in the Erasmus+ programme was expanded during the 
evaluation  period.  For  example,  simplified  grants  were  introduced  for  the  Collaborative 
Partnerships actions in Erasmus+ Sport programme’s 2017 calls.  

The  2007-2013  Europe  for  Citizens  programme  is  another  example.  The  programme  was 
often  used  as  a  testing  ground  to  pilot  simplified  cost  options  and  other  simplifications. 
Considering the positive effects of the simplification measures put in place for the 2007-2013 
generation programme, a financing system based on simplified costs was applied to all action 
grants of the new generation of the Europe for Citizens Programme 2014-2020. 

During the evaluation period, the EACEA also implemented a number of other administrative 
measures aimed at simplification and optimisation of programmes and grants management. 

22 

 
 
 
 
 
 
 
 
 
 
 
As  regards  simplification  of  financial  management,  in  cooperation  with  the  EACEA,  the 
Commission put a lot of effort into widening the use of standard cost options and simplifying 
financial  management  of  2014-2020  MFF  programmes  delegated  to  the  Agency.  The 
development of a system of lump-sums and unit costs simplified the financial management of 
the programmes both for the beneficiaries and for the Agency. 

The  use  of  eForms  (electronic  tool  for  the  submission  of  proposals)  was  significantly 
expanded  during  the  reporting  period,  with  over  94%  of  applicants  using  them.  Automatic 
checks were integrated into the eForms to facilitate the application process and to reduce the 
share  of  ineligible  proposals.  Such  developments  allowed  the  Agency  to  achieve  an  almost 
entirely paperless application and evaluation process.  
During  the  evaluation  period,  the  Agency  worked  to  replicate  the  success  of  eForms  by 
developing a generalised use of electronic interim and final reports (eReports) for the 2014-
2020 generation of programmes. Its aim was to simplify the grant administration process for 
beneficiaries  (by  automating  the  preparation  and  submission  of  reports,  facilitating 
collaboration between project partners, etc.), to improve productivity within the Agency (by 
avoiding the re-encoding of information already submitted), and to facilitate communication 
between the beneficiaries and the EACEA (faster receipt of reports, etc.) and the transition to 
a future corporate grant management system. 

The  use  of  eForms,  eReports  and  paperless  processes  produced  significant  efficiency  gains. 
The resources freed up by the simplification process were re-invested in many other processes 
of the Agency, in order to improve the quality of the service provided by the EACEA to its 
beneficiaries and stakeholders, in order to support policy objectives.  

EACEA  also  pursued  the  development  of  its  IT  systems,  i.e.  through  the  integration  of 
Pegasus (grant management system) with APPFIN (financial system). It is expected that such 
integration  would  be  a  significant  simplification  for  the  Agency,  allowing  it  to  use  a  single 
tool for grant management.  In 2015 the EACEA also introduced a new tool (Speedwell) for 
the payments of its administrative budget, which enabled a paperless workflow. ICT tools that 
support  online  interaction  with  experts  were  used  in  the  majority  of  project  selection 
procedures. 

During  2017,  the  EACEA  prepared  for  the  2018  adoption  of  the  Commission’s  Single 
Electronic  Data  Interchange  Area  (SEDIA).  This  provides  applicants  and  tenderers  (third 
parties)  with  a  single  entry  point  to  communicate  and  exchange  information  relating  to  the 
legal  and  financial  verification  process  that  the  EACEA  needs  to  perform,  prior  to  issuing 
contracts and making pre-financing payments. 

The survey shows that simplification is  appreciated by  applicants  and beneficiaries, who, at 
the  same  time,  would  appreciate  that  further  efforts  are  put  to  simplify  and  streamline  the 
administrative arrangements of the programmes managed by the EACEA. 

5.3.  Coherence  

Coherence  looks  at  any  overlaps  and  complementarities  within  the  programme  portfolio 
managed by the EACEA or delimitation of responsibilities between the EACEA and its parent 
DGs. 

23 

 
 
 
 
 
 
 
 
 
 
 
Coherence of portfolio 
As  noted  in  the  previous  evaluation  of  the  Agency,  the  EACEA  was  very  successful  at 
exploiting  synergies  resulting  from  the  optimisation  of  different  programmes  and  was 
extending its experience gained across the programmes. 

In terms of complementarities, a finding emerging interviews concerned links between Intra-
Africa,  the  Erasmus+  programme,  and  other  EU  initiatives  involving  collaboration  with 
Africa.  The  programmes  were  said  to  complement  each  other  very  well  and  encourage 
systemic  reforms.  The  EACEA  in  particular  was  praised  for  its  ability  to  understand  these 
links  and  reflect  upon  them.  The  Agency,  DG  DEVCO  and  DG  EAC  engaged  in  dynamic, 
triangular  cooperation  involving  joint  events  on  Intra-Africa  mobility  and  other  types  of 
meeting  involving  multiple  stakeholders  including  the  EACEA.  This  helps  the  Agency 
understand the context of systemic changes resulting from the mobility scheme. In return, the 
EACEA can promote the complementary activities of DG EAC and DG DEVCO, when they 
visit African countries on a monitoring mission.  

An additional instance where synergies were exploited is eReports (see above). After carrying 
out  an  initial  eReports  pilot  project  on  the  Europe  for  Citizens  programme,  the  EACEA 
extended  the  use  of  eReports  to  several  actions  of  the  Erasmus+  and  Creative  Europe 
programmes in 2015. 

The broadness of activities delegated to EACEA means that the Agency was confronted with 
quite diverse actions in terms of scope and scale. In the case of the Eurydice network37, for 
instance,  EACEA  successfully  ensured  the  management  of  a  network  characterised  by  a 
relatively small budget compared to the average of the programmes managed by the Agency. 

Relations with parent DGs and supervision 
The  evaluation  confirmed  that  responsibilities  of  the  Agency  are  clearly  defined  by  the 
existing  legal  framework  governing  the  interaction  and  roles  of  the  EACEA  and  its  parent 
DGs. In particular, the tasks delegated to the Agency and tasks reserved for the Commission 
are clearly defined in the Act of Delegation and the General Memorandum of Understanding 
(MoU) between EACEA and its parent DGs, updated in March 2015. 

In addition to general principles, the MoU defines how information exchange, administrative 
matters,  cooperation  in  relation  to  other  actors  and  institutions  and  other  modalities  and 
procedures of interaction are supposed to work. The MoU reduces the risk of overlaps or gaps 
which might originate from an inadequate formal delimitation of responsibilities borne by the 
EACEA and its parent DGs to a minimum.  

Based  on  the  evidence  drawn  from  interviews,  there  is  a  clear  understanding  of  the  general 
principle that defining the policy is in the hands of the Commission, whereas the EACEA is 
supposed to implement the defined policy goals.  

In terms of the Agency’s contribution to the preparation of the Work Programmes of its parent 
DGs, EACEA is closely consulted both formally and informally, with both sides expressing 
their satisfaction with how the cooperation takes place. 

37  In  2009,  the  EACEA  gained  responsibility  for  the  implementation  of  Eurydice,  the  European  network  of 
information  on  education  and  youth.  Eurydice’s  work  includes  the  collection,  analysis  and  dissemination  of 
information, as well as the production of studies and other publications. 

24 

 
 
 
 
 
 
 
 
 
 
                                                           
Lines  drawn  between  roles  of  the  Agency  and  its  parent  DGs  are  not  completely  static,  but 
adapt  to  circumstances:  in  times  of  need,  the  operational  units  in  the  Commission  have 
increased expectations and need more help from their colleagues in the Agency. Having close 
collaboration ties between the mirror units in the DG(s) and the Agency and some flexibility 
in  the  definition  of  EACEA’s  responsibilities  are  necessary  for  ensuring  the  Agency’ 
versatility and capacities to respond to expectations of its multiple parent DGs. 

As regards supervision, according to the Regulation 58/2003, the Commission sets the limits 
to the actions of each executive agency and exercises real control over its operation.  
Overall,  the  existing  supervision  arrangements  proved  to  be  effective  and  were  exercised 
thoroughly during the evaluation period.  

Yet,  even  though  the  arrangements  concerning  supervision  are  working  as  defined  in  the 
existing  legal  framework,  it  has  been  found  that  clarifications  in  the  definition  of  a 
supervisory  function  are  needed.  For  instance,  the  boundaries  of  supervision  should  be 
defined better, as it was not always obvious for the parent DGs how far the supervision should 
go  in  order  not  to  breach  the  autonomy  of  the  Agency  and  avoid  micro-management.  To 
address this issue, corporate discussions on the supervision of executive agencies have begun 
in 2018 in the Commission. 

It  should be mentioned that  the relocation of the Agency close to  the premises of its parent 
DGs in 2016 helped ensure better coordination. 

Policy Feedback 
Concerning policy feedback, EACEA is faced with the challenge of being able to address the 
policy-making  needs  of  its  parent  DGs  and  participating  DGs.  Indeed,  while  EACEA  is 
praised  for  its  capacity  to  manage  the  programmes  delegated  to  it,  this  recognition  is 
accompanied  by  increasing  expectations  and  pressure  to  improve  the  execution  of  its  other 
key  task  –  gathering,  analysis  and  transmitting  information  required  by  the  Commission  to 
guide the implementation of the programmes. 

The need for the EACEA to improve in this aspect was already emphasised by the previous 
external evaluation. The current evaluation finds that support to policy-making was defined as 
a priority for the Agency in the EACEA’s Annual Work Programme for 2016. 

Policy support was provided, among other means, by organising kick-off and thematic cluster 
meetings, placing greater emphasis on policy aspects and engaging all relevant stakeholders. 
EACEA staff provided support to several policy-related projects in DG EAC. The Agency has 
also  provided support and data  relating to  adult education for DG  EMPL, and responded to 
questions  of  the  European  Parliament.  In  addition,  the  Agency  supported  DG  NEAR  in  the 
evaluation  of  the  Fellowship  for  Youth  Action,  deepening  the  relationship  between  the 
Agency and this DG.  

At the level of actions, the EACEA has also implemented an analysis of the results produced 
the  projects  via  the  carrying  out  of  various  studies,  e.g.  Europe  for  Citizens  Programme 
Results 2017, and by disseminating success stories not only via the Project Results Websites 
(VALOR), but also through dedicated publications such as Europe for Citizens. 

However, the evaluation observed that the Agency’s efforts in relation to reporting the results 
and impacts of the programmes it manages lack either wider recognition, or better awareness 

25 

 
 
 
 
 
 
 
 
 
 
among  EACEA’s  parent  DGs.  Parent  DGs  expected  a  stronger  emphasis  in  EACEA’s 
reporting on the topics and outputs of awarded projects. 

Slight  dissatisfaction  was  expressed  in  relation  to  the  declining  resources  allocated  to 
monitoring  missions  and/or  the  changing  approach  of  the  EACEA  to  organising  missions. 
Some DGs also expected information from the EACEA on interim results, rather than just the 
final results of projects – particularly for newly delegated actions. 
The  reasons  behind  these  comments  by  parent  DGs  may  lay  in  the  shift  of  attention  of  the 
Agency  and  its  parent  DGs  towards  responding  to  the  outcomes  of  the  IAS  audit,  and  the 
significantly  increased  workload  associated  with  implementing  selection  procedures  that 
complied with its formal requirements. 

5.4.  Retrospective Cost-benefit analysis 

When looking at  the  ex ante cost-benefit analysis, the overall costs of the executive agency 
scenario was estimated at EUR 146.0 million in 2015-2017, i.e. lower than the estimated costs 
of the in-house scenario. The actual savings during 2015 -2017 period were found to be 6.5% 
higher  compared  to  the  initial  SFS  estimates  (EUR  57.1  million  compared  to  EUR  53.6 
million)  and  37.0%  higher  compared  to  the  initial  CBA  estimates  (EUR  57.1  million 
compared to EUR 41.7 million). As forecasted in the SFS and the ex ante CBA, savings of the 
executive  agency  scenario  primarily  resulted  from  a  higher  share  of  lower  cost  external 
personnel (CAs) employed within the Agency and lower overall number of staff: 

ESTIMATED SAVINGS OF THE EXECUTIVE AGENCY SCENARIO IN 2015-2017 (TITLE I AND 
TITLE II EXPENDITURE), EUR  

178.092.478 

178.875.478 

187.294.748 

136.409.419 

125.283.100 

130.195.868 

41.683.059 

53.592.378 

57.098.880 

200.000.000

150.000.000

100.000.000

50.000.000

0

CBA estimations

SFS estimations

Actual

In-house scenario

Executive agency scenario

Estimated savings of EA scenario

Sources: PPMI, based on CBA, SFS, final annual accounts, the EACEA’s AWPs. 

The  workload  initially  estimated  during  the  period  2015-2017  was  correct  as  overall  the 
EACEA  managed  to  cope  with  the  foreseen  human  resources  while  implementing  the 
programmes delegated to it. To ensure the optimal allocation of resources, and to respond to 
new  activity  requests,  the  Agency  revised  its  methodology  for  the  workload  indicators  and 
screened  the  allocation  of  human  resources  within  the  organisation.  In  addition  to  its 
authorised staff, the Agency employed interim staff to temporarily replace staff on long-term 
leave (such as maternity/parental leave). 

It  should  be  noted  that  costs  in  the  CBA  were  calculated  at  constant  2013  prices  (i.e. 
neutralising  the  effect  of  inflation).  However,  in  the  SFS  these  estimations  were  used  as 
current prices without any further indexation. In real terms, this constituted a reduction of the 
administrative  budget,  with  an  impact  on  staff  costs  (structural  under-estimation  of  staff 
expenditure).  

26 

 
 
 
 
 
 
 
 
6.  CONCLUSIONS 

The results of the evaluation of EACEA operations during 2015-2017 period confirm that the 
delegation of programme implementation to EACEA proved considerably more cost effective 
than the in-house scenario. 

While  successfully  implementing  its  annual  work  programmes,  EACEA  managed  to  further 
improve  the  efficiency  in  the  use  of  its  administrative  and  financial  resources,  to  develop  a 
more resilient and adaptive organisation. The retrospective cost benefit analysis found that the 
actual  savings  of  the  executive  agency  scenario  were  EUR  57.1  million  during  the  period 
2015-2017 compared with the in-house scenario. 

During the evaluation period, the EACEA achieved very good results in terms of almost all 
KPIs.  In  detail,  the  Agency  performed  well  in  ensuring  timely  execution  of  the  delegated 
functions  and  met  the  main  indicators  describing  timeliness  of  evaluation  and  contracting 
process  (Time-to-Award,  Time-to-Contract,  overall  Time-to-Grant)  and  timeliness  of 
processing payments to beneficiaries (Time-to-Pay). 

The overall satisfaction with the quality of services among its applicants and beneficiaries was 
rather  high  (79.2%),  though  somewhat  uneven  between  unsuccessful  applicants  and 
beneficiaries.  Applicants  and  beneficiaries  gave  a  very  positive  opinion  about  the 
simplifications already introduced in the programmes managed by the EACEA.  

The  coordination  and  communication  mechanisms  in  place  enabled  information  flows 
between  EACEA  and  the  Commission  services,  in  particular  with  regard  to  the  content  of 
projects  supported  and  their  results.  No  evidence  of  overlaps,  duplication,  gaps  and 
inconsistencies  within  the  programme  portfolio  and  support  services  managed  by  EACEA 
were identified. 

The current evaluation identifies a few strategic areas where further improvements would be 
desirable:  

1.  Continue adapting EACEA’s strategy, policies and procedures, particularly in light 
of the 2017 audit, so  as  to  strengthen the Agency  and its functioning. Ensure that 
the experience  gained and lessons learnt contribute to  making EACEA  even more 
resilient. 

2.  Review the  experience of the 2018 evaluation  and selection processes and use the 
lessons  learnt  from  this  experience  to  optimise  future  evaluation  and  selection 
procedures.  Take  into  account  this  experience  while  defining  implementation  acts 
and procedures for the management of the 2021-2027 MFF programmes. 

3.  Take action to update the existing supervision arrangements between EACEA and 

its parent DGs. 

4.  Explore  further  opportunities  to  improve  the  EACEA’s  reporting  of  programme 

results and policy feedback to its parent DGs.  

5.  Review  the  organisational  structure  of  the  EACEA,  taking  into  account  the 
organisational characteristics of the Agency, the recommendations of the 2017 IAS 
audit and the possibility of introducing new departments. 

Extensive  actions  have  already  been  undertaken  and  are  currently  being  (or  have  already 
been) implemented to address the issues identified under the points 1 to 3. Actions to respond 
to point 4 will be explored in more depth. As regards the organigramme of EACEA (point 5), 

27 

 
 
 
 
 
 
 
 
a reorganisation of the structure has been implemented by the new Director with effect on 16 
March 2020. 

The  parent  DGs  and  the  Agency  prepared  an  action  plan  to  address  the  shortcomings 
identified  in  this  evaluation.  This  action  plan  aims  at  increasing  the  Agency’s  performance 
and contribution to an efficient implementation of the delegated programmes. 
The Agency has also taken extensive corrective action to respond to the findings of the IAS 
audits  on  Grant  Management  -  Phase  I  (from  call  publication  to  contract  signature  –  all 
recommendations now closed) for Erasmus+ and the Creative Europe programmes, and Phase 
II  (from  project  monitoring  to  payment)  where  the  IAS  is  in  the  process  to  perform  the 
follow-up engagement. 

Actions were also taken to address the outcome of the Commission-wide staff opinion survey 
carried  out  in  the  course  2018,  which  showed  the  need  for  corrective  measures  to  be 
implemented in order to solve the identified issues and to increase the satisfaction of EACEA 
staff. 

More  globally  and  to  address  the  various  issues  outlined  above,  the  agency  has  been 
undertaking  since  2019  a  participatory  ‘Executive  Agency  Transformation  Project’  to  equip 
the  agency  with  an  improved  organisational  structure,  processes  and  internal  control 
framework  for  the  challenges  of  the  coming  years.  This  resulted  in  the  reorganisation 
mentioned above. 

28 

 
 
 
 
 
 
 
 
ANNEX - EVALUATION QUESTIONS 

Effectiveness  

  To  what  extent  has  EACEA  been  operating  according  to  the  legal  framework 

establishing it?  

  To  what  extent  has  EACEA  achieved  its  objectives  with  special  focus  to  (a)  the 
implementation  of  the  delegated  programmes4  and  (b)  the  implementation  of  the 
support services? What, if anything, could be done to render EACEA more effective in 
achieving these objectives?  

  To  what  extent  and  how  has  EACEA  contributed  to  an  improved  management  and 
visibility  of  the  delegated  programmes  and  better  services  and  satisfaction  to  the 
stakeholders and addressees in terms of the elements assessed in the 2013 CBA and as 
compared to the alternative options mentioned in that CBA?  

  To what extent does EACEA’s external communication function support the mission 

of the Agency?  

  To what extent has EACEA contributed to improved management of the programmes 

in terms of:  

o  Proximity  to  addressees  (e.g.  role  played  by  the  Agency  as  focal  point  for 
applicants  and beneficiaries of the programmes;  instruments and mechanisms 
put in place for communicating with the applicants and beneficiaries). 

o  Effective implementation of the programmes, taking into account the interests 

of the addressees and those of the EU:  

  Rate of execution of commitment appropriations;  
  Rate of execution of payment appropriations;  
  Time to grant;  
  Net time to pay;  
  Residual multi-annual error rate identified at ex post control;  
  Time to assess submitted reports  

o  Visibility  of  the  EU  as  promoter  of  the  programmes  entrusted  to  the  Agency 
(e.g.  compliance  with  the  Commission’s  guidelines  on  information  and 
visibility of programmes5, instruments  put  in  place to  ensure the visibility  of 
the EU as promoter of the programmes).  

  To what extent is the management of the Eurydice network by EACEA effective?  

Efficiency  

  To what extent has the EACEA carried out its work efficiently.?  
  To what extent have the actual costs (including cost of coordination and monitoring) 
of  EACEA  corresponded  to  the  estimates  of  the  2013  CBA?  If  not,  what  are  the 
reasons behind?  

  To  what  extent  have  the  actual  benefits  corresponded  to  the  estimates  of  the  2013 

CBA? If not, what are the reasons behind?  

  To  what  extent  has  the  management  and  execution  of  the  programmes  by  EACEA 

been cost-effective as compared to the alternative options6?  

  Examples  of  alternatives  to  the  Executive  Agency:  (i)  management  of  the 
programme(s) by the Commission, (ii) mixed Agency - Commission management, (iii) 
partial management by the Commission while outsourcing some activities to the extent 
legally possible.  

29 

 
 
 
 
 
 
 
  To what extent has EACEA contributed to improved management of the programmes 

in terms of:  

o  Simplification  of  the  procedures  and  flexibility  in  the  implementation  of 

delegated tasks (e.g. capacity to adapt to periods of high workload)?  

o  Which further scope for simplification exists?  

  To what extent is the management of the Eurydice network by EACEA cost-effective?  
  Which aspects/means/actors or processes render EACEA more or less efficient? What 

could be improved?  

  To what extent has the establishment of EACEA resulted in savings to the EU budget 
as  compared  to  the  alternative  options  (e.g.  difference  in  costs  between  the 
Commission option and the Agency option)?  

  To what extent have EACEA’s internal organisation and procedures been conducive to 

its efficiency? Is the size of the Agency adequate?  

  To  what  extent  is  EACEA’s  internal  organisation  capable  and  flexible  to  rapidly 

respond to resource needs due to uncertainties related to volumes of work?  

  To  what  extent  does  EACEA’s  human  resources  management  contribute  to  the 

achievement of the Agency’s objectives?  

o  Is the size and structure of the organisation appropriate?  
o  Is  the  balance  between  operational  staff  and  administrative  support  staff 

appropriate?  

o  Are the staff turnover and vacancy rate well managed?  
o  How are provisions on overtime and teleworking implemented in the Agency?  
o  How does the Agency follow up on the findings of staff surveys?  

Coherence  

  To what extent have there been overlaps / gaps / inconsistencies / complementarities 
within  the  programme  portfolio  and  support  services  managed  by  EACEA  and  how 
are these addressed?  
Is  there  a  clear  and  appropriate  delimitation  of  responsibilities  and  tasks  between 
EACEA  and  the  Parent  DGs?  Are  there  overlaps  or  gaps?  Are  the  different 
responsibilities adequately communicated to the beneficiaries?  

 

  To  what  extent  has  EACEA  enabled  the  Commission  to  better  focus  on  its  policy 

related tasks?  

o  Are  there  any  governance  (financial  and  policy)  issues  in  relation  to  the 
implementation  of  the  tasks  that  are  delegated  to  the  executive  agencies  / 
issues as  regards the Commission’s services  being able to  steer EU policy or 
budget implementation?  

o  Are  appropriate  mechanisms  and  instruments  in  place,  and  at  which  level,  to 
ensure  an  adequate  coordination  and  information  flow  between  EACEA  and 
the Commission services, notably on the content of the projects supported and 
their results?  

o  Does  EACEA  provide  useful  information  on  the  implementation  of  the 
delegated  programmes  and  their  progress  (in  terms  of  management  and 
content)  in  support  of  the  policy  process  (e.g.  information  required  for  the 
annual Management Plan of the Parent DGs)?  

  To  what  extent  have  the  activities  of  EACEA  resulted  in  unintended  effects  (both 

desirable and undesirable)?  

30 

 
 
 
 
 
  To what extent has the Commission, in the presence of EACEA, been able to maintain 
an  adequate  level  of  know-how  in  relation  to  the  programmes  entrusted  to  the 
Agency?  How  has  this  been  achieved?  What  are  the  feedback  channels,  means  and 
methods used for this purpose? What are areas for improvement, if any? 

  Have  the  monitoring,  reporting  and  supervision  arrangements  in  place  enabled  the 
Commission  to  benefit,  in  the  short  and  medium  term,  from  the  know-how  created 
within EACEA?  

  How effective is the flow of information and communication between EACEA and the 

Commission services (in particular Parent DGs)?  

  To what extent would the closing down of EACEA result in losing significant know-

how in relation to the management of the programmes entrusted to EACEA?  

31