COMMISSION STAFF WORKING DOCUMENT Evaluation of the Educational, Audiovisual and Culture Executive Agency (EACEA) Accompanying the document REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL AND THE COURT OF AUDITORS Evaluation of the Consumers, Health, Agriculture and Food Executive Agency, the Educational, Audiovisual and Culture Executive Agency, the Executive Agency for Small and Medium-Sized Enterprises, the European Research Council Executive Agency, the Innovation and Networks Executive Agency and the Research Executive Agency
Date: 2025-04-16
Institutions: Directorate-General for Research and Innovation, European Commission
Subjects: EU aid, EU financing, EU programme, audiovisual industry, culture, education, executive agency, financial control, performance audit

EN 

  EN 

   EUROPEAN COMMISSION  Brussels, 16.4.2025  SWD(2025) 89 final  COMMISSION STAFF WORKING DOCUMENT Evaluation of the Educational, Audiovisual and Culture Executive Agency (EACEA) Accompanying the document REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL AND THE COURT OF AUDITORS Evaluation of the Consumers, Health, Agriculture and Food Executive Agency, the Educational, Audiovisual and Culture Executive Agency, the Executive Agency for Small and Medium-Sized Enterprises, the European Research Council Executive Agency, the Innovation and Networks Executive Agency and the Research Executive Agency  {COM(2025) 171 final} - {SWD(2025) 87 final} - {SWD(2025) 88 final} - {SWD(2025) 90 final} - {SWD(2025) 91 final} - {SWD(2025) 92 final}  
 
 
Table of Contents 

1. 

INTRODUCTION ......................................................................................................................................... 3 

2.  WHAT WAS THE EXPECTED OUTCOME OF THE INTERVENTION? .......................................... 5 

2.1.  DESCRIPTION OF THE INTERVENTION AND ITS OBJECTIVES ....................................................................... 5 
2.2.  BASELINE AND POINT(S) OF COMPARISON ................................................................................................ 8 

3.  HOW HAS THE SITUATION EVOLVED OVER THE EVALUATION PERIOD? ............................ 9 

4.  EVALUATION FINDINGS (ANALYTICAL PART) ............................................................................. 11 

4.1.  TO WHAT EXTENT WAS THE INTERVENTION SUCCESSFUL AND WHY? ..................................................... 11 
4.2.  COST-BENEFIT ANALYSIS ....................................................................................................................... 15 

5.  WHAT ARE THE CONCLUSIONS AND LESSONS LEARNED? ...................................................... 16 

5.1 
5.2 

CONCLUSIONS ........................................................................................................................................ 16 
LESSONS LEARNED ................................................................................................................................. 17 

ANNEX I: PROCEDURAL INFORMATION ................................................................................................. 19 

ANNEX II: METHODOLOGY AND ANALYTICAL MODELS USED ...................................................... 21 

ANNEX III: EVALUATION MATRIX AND ANSWERS TO THE EVALUATION QUESTIONS (BY 
CRITERION) ...................................................................................................................................................... 23 

ANNEX IV: OVERVIEW OF BENEFITS AND COSTS ............................................................................... 24 

ANNEX V: COST BENEFIT ANALYSIS ........................................................................................................ 25 

ANNEX VI: STAKEHOLDERS CONSULTATION – SYNOPSIS REPORT .............................................. 28 

1 

 
 
 
 
 
 
 
Term or acronym 
Agency/Agencies 
Commission 
CBA 
CHAFEA 

EACEA 

EASME 

EDF 

EFTA 
ERCEA 

EU 

MFF 
REA 
SFS 

Glossary 

Meaning or definition 
Executive agency/agencies 
European Commission 
Cost-benefit analysis 
Consumers, Health, Agriculture and Food 
Executive Agency 
European Education and Culture Executive 
Agency 

Executive Agency for Small and Medium-Sized 
Enterprises 
European Development Fund 

European Free Trade Association 
European Research Council Executive Agency 

European Union 

Multiannual financial framework 
European Research Executive Agency 
Specific financial statement 

2 

 
 
 
 
 
 
 
 
 
 
 
 
1. 

INTRODUCTION 

Purpose and scope of the evaluation/fitness check 

The  purpose  of  this  periodic  (three-yearly)  evaluation  is  to  assess  the  operations  of  the 
Education,  Audiovisual  and  Culture  Executive  Agency  (‘EACEA’  or  ‘the  Agency’)  from 
1 January 2018  to  31 March 2021.  In  accordance  with  Article 25  of  the  Framework 
Regulation for executive agencies1, the Commission submits this evaluation to the European 
Parliament, the Council, the Court of Auditors and the Steering Committee of the Agency. 

During the evaluation period, EACEA was governed by: 

• 
• 
• 

• 

the Framework Regulation for Executive Agencies; 
its Establishment Act2, which sets out its mandate; 
the  Delegation  Act3,  which  specifies  the  tasks  to  be  carried  out  and  the  powers 
delegated to the Agency; and 
the Decision setting up the EACEA Steering Committee. 

EACEA  is  responsible  for  programme  management  that  does  not  entail  political  decision-
making but requires a high degree of technical and financial expertise throughout the project 
cycle.  The  Commission  Decision  of  18 December  2013  delegating  powers  to  the  Agency 
stipulates  that  ‘it  shall  not  perform  any  tasks  involving  a  large  discretion  implying  political 
choices‘. 

The  legal  framework  is  supplemented  by  the  General  Memorandum  of  Understanding 
between EACEA and its parent Directorates-General (signed in March 2015), which specifies 
the  arrangements  and  procedures  for  their  cooperation.  The  delimitation  of  responsibilities 
and  tasks  between  the  Agency  and  its  parent  Directorates-General,  as  set  out  in  the 
Memorandum  of  Understanding,  is  analysed  in  greater  detail  in  subsequent  evaluation 
questions on coherence. 

In  accordance  with  Article 25  of  the  Framework  Regulation4  for  executive  agencies,  the 
Commission  assess  the  operations  of  each  agency  every  three  years  and  performs  a  cost-
benefit  analysis.  It  submits  the  respective  report  to  the  agency’s  Steering  Committee,  the 
European Parliament, the Council and the Court of Auditors. 

This  staff  working  document  reports  on  the  three-yearly  evaluation  of  EACEA,  which  was 
part of a broader evaluation of the six executive agencies5. It concerns the period of EACEA’s 

1  

2  

3  

4  

5  

Council Regulation (EC) No 58/2003 of 19 December 2002 laying down the statute for  agencies to be entrusted with certain tasks in 
the management of Community programmes (OJ L 011, 16.01.2003, p.1, http://data.europa.eu/eli/reg/2003/58/oj). 
Commission  Implementing  Decision  2013/776/EU  of  19 December  2013  establishing  the  ‘Education,  Audiovisual  and  Culture 
Executive Agency’ and repealing Decision 2009/336/EC. 
Commission  Decision  of  18 December  2013  delegating  powers to the  Education,  Audiovisual  and Culture  Executive  Agency  with  a 
view to performance of tasks linked to the implementation of the Union programmes in the field of education, audiovisual and  culture 
comprising,  in  particular,  implementation  of  appropriations  entered  in  the  general  budget  of  the  Union  and  of  the  EDF  allocations, 
C(2013) 9189 final. 
Council Regulation 58/2003 of 19 December 2003 laying down the statute for executive agencies to be entrusted with certain tasks in 
the management of Community programmes (OJ L 011, 16.01.2003, p. 1). 
Consumers, Health, Agriculture and Food Executive Agency (CHAFEA), Executive Agency for Small and Medium-Sized Enterprises 
(EASME),  Innovation  and  Networks  Executive  Agency  (INEA),  Education,  Audiovisual  and  Culture  Executive  Agency  (EACEA), 
Research Executive Agency (REA) and European Research Council Executive Agency (ERCEA). 

3 

 
 
 
 
mandate  under  the  2014-2020  multiannual  financial  framework  (MFF)  that  ran  from 
1 January 2018 to 31 March 2021. 

In line with Article 12 of the Decision establishing the new agencies6, the evaluation covered 
all six agencies in a coordinated manner and followed a common evaluation methodology. It 
focused  on  their  operations  and  performance  during  the  evaluation  period  by  providing 
answers  to  specific  evaluation  questions  on  effectiveness,  efficiency  and  coherence,  and 
included a retrospective cost-benefit analysis. 

EACEA’s operations were evaluated to assess how the Agency had implemented the parts of 
the EU funding programmes entrusted to  it. This  concerns the six  programmes  managed by 
EACEA  between  2018  and  2021:  Europe  for  Citizens  (2014-2020),  Erasmus+  (2014-2020), 
Creative  Europe  (2014-2020),  the  European  Solidarity  Corps  (2018-2020),  EU  Aid 
Volunteers (2014-2020)  and the Pan-African  programme (2014-2020).  While the evaluation 
did not focus on their operational achievements, because such results are evaluated separately, 
the evaluator took into account the results of such completed evaluations where necessary. 

In line with the Commission’s  better regulation  principles7, the evaluation assesses whether: 
(i) the Agency has fulfilled its tasks effectively and efficiently; (ii) there are overlaps, gaps or 
inconsistencies  in  its  management  of  the  programme  portfolio;  and  (iii)  whether  there  is  a 
clear  delimitation  of  tasks  between  EACEA  and  the  parent  Directorates-General  or  other 
executive agencies (coherence)8. 

The evaluation also assesses whether the functioning of the Agency has produced the positive 
results  that  were  predicted  in  the  2013  cost-benefit  analysis  and  in  the  specific  financial 
statement  to  occur  as  a  result  of  the  delegation  of  tasks  to  the  Agency.  To  that  end,  the 
assessments  in  the  2013  cost-benefit  analysis  have  been  tested  to  prove  the  validity  of  the 
assumptions  in  the  ex ante  scenario  by  considering  in  a  structured  way  the  actual  costs  and 
benefits of programme implementation by the Agency. The aspects to be covered by the cost-
benefit  analysis  are  specified  in  Article 3(1)  of  the  Framework  Regulation9  and  in  the 
Guidelines on establishing and operating executive agencies10. 

6  

7  

8  

9  

Commission  Implementing  Decision  (EU)  2021/173  of  12 February  2021  establishing  the  European  Climate,  Infrastructure  and 
Environment Executive  Agency,  the European  Health  and  Digital Executive  Agency,  the  European Research Executive  Agency, the 
European  Innovation  Council  and  SMEs  Executive  Agency,  the  European  Research  Council  Executive  Agency,  and  the  European 
Education  and  Culture  Executive  Agency  and  repealing  Implementing  Decisions  2013/801/EU,  2013/771/EU,  2013/778/EU, 
2013/779/EU, 2013/776/EU and 2013/770/EU (OJ L 50, 15.2.2021, p. 9). 
Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the 
Committee  of  the  Regions  on  ‘Better  Regulation:  joining  forces  to  make  better  laws’,  COM(2021)219  final  of  29.4.2021,  and 
Commission Staff Working Document on ‘Better Regulation Guidelines’, SWD(2021)305 final of 3.11.2021. 
The assessment of ‘EU added value’, or why the EU should act, is not perceived to be a relevant criterion, as the agencies execute tasks 
transferred to them by the European Commission. The EU added value of the programmes that the agencies manage is assessed as part 
of the programme evaluation. The programme evaluation also assesses previous needs that the Agency has to address and whether they 
are still pertinent today (relevance). 
The  Cost-benefit  analysis  should  include  the  following:  (i)  the  identification  of  tasks  that  justify  being  outsourced;  (ii)  the  costs  of 
coordination  and  checks;  (iii)  the  impact  on  human  resources;  (iv)  possible  savings  within  the  general  budgetary  framework  of  the 
European Union; (v) efficiency and flexibility when implementing outsourced tasks; (vi) the simplification of the procedures used; (vii) 
the proximity of outsourced activities to the final beneficiaries; (viii) the EU’s visibility as promoter of the EU programme concerned; 
and (ix) the need to maintain an adequate level of know-how within the Commission. 

10   Appendix II of the Guidelines for the establishment and operation of executive agencies financed from the Union budget (C(2014) 9109 

from 2 December 2014. 

4 

 
 
 
The  evaluation  of  EACEA  provides  useful  input  for  evaluating  the  programmes,  as  the 
Agency’s performance affects the efficiency and effectiveness of the programmes it manages. 
The evaluation is based on a study carried out by an external contractor11.  

The  evaluation  of  the  Agency’s  performance  is  evidence-based  and  uses  a  mixed-methods 
approach, which combines: 

- 

- 

a  qualitative  approach,  based  on  open-ended  survey  questions,  interviews,  a 
documentary review and desk research, a qualitative cost-benefit analysis; and 
a  quantitative  approach,  based  on  administrative  and  monitoring  data,  surveys  and  a 
quantitative cost-benefit analysis. 

Relevant stakeholders were consulted via interviews and surveys (Commission staff, EACEA 
staff, beneficiaries, unsuccessful applicants and experts). Unsuccessful applicants participated 
less actively than beneficiaries. The low response rate (i.e. 15% on average) to the survey by 
beneficiaries and unsuccessful applicants could be due to survey fatigue. 

The evaluation  examines the efficient  use of resources and the  effective achievement of the 
tasks entrusted to the Agency. It assesses whether: 

• 

• 

the alignment of programme portfolios that are more coherent with the Agency’s core 
mandate and its brand identity delivered the estimated qualitative benefits; 
the  consolidation  of  the  management  of  different  EU  programmes  delivered  the 
estimated synergies, simplification and economies of scale; 
• 
the pooling of instruments guaranteed consistent service delivery; and 
•  whether there is scope for simplification and further efficiency gains. 

2.  WHAT WAS THE EXPECTED OUTCOME OF THE INTERVENTION? 

2.1.  Description of the intervention and its objectives 

As  outlined  in  the  Framework  Regulation  for  executive  agencies  and  in  EACEA’s 
Establishment  Decision,  the  outsourcing  of  certain  management  tasks  to  the  Agency  in  the 
2014-2020 MFF has a twofold purpose. 

Firstly, it allows the Commission to focus on its institutional tasks, i.e. tasks assigned to the 
institutions  by  the  Treaty  on  the  Functioning  of  the  European  Union  which  require 
discretionary  powers  in  translating  political  choices  into  action.  Such  institutional  tasks 
should not be outsourced. 

Secondly  it  enables  the  Commission  to  achieve  the  objectives  of  the  delegated  EU 
programmes more effectively. The 2013 cost-benefit analysis indicated that it was more cost-
efficient to delegate certain programme tasks to the Agency than to perform them in house. 

The Establishment Decision stated that aligning programme portfolios that are more coherent 
with  the  Agency’s  core  mandate  and  its  brand  identity  would  bring  qualitative  benefits  and 

11   Study supporting the evaluation of CHAFEA, EACEA, EASME, ERCEA, INEA & REA (2017/2018-2021), Final Report: EACEA,. 

5 

 
 
 
 
that  consolidating  the  management  of  different  EU  programmes  would  bring  synergies, 
simplification and economies of scale. 

The Agency has a varied portfolio of programmes and compared with other agencies, awards 
its beneficiaries smaller than average grants. During the evaluation period (1 January 2018 to 
31 March 2021), EACEA managed the following programmes or parts of programmes: 

-  Creative Europe, 
-  Erasmus+, 
-  Europe for Citizens, 
-  EU Aid Volunteers, 
- 
- 

the European Solidarity Corps, and 
the Pan-African programme. 

Not  only  has  EACEA’s  mandate  changed  with  regard  to  various  EU  programmes,  but  the 
Agency’s  parent  Directorates-General  have  also  changed  slightly.  During  the  current 
evaluation period, four parent Directorates-General supervised EACEA: 

• 

• 
• 

• 

the  Directorate-General  for  Education,  Youth,  Sport  and  Culture  (the  lead  parent 
Directorate-General); 
the Directorate-General for Communications Networks, Content and Technology; 
the Directorate-General for Migration and Home Affairs (replaced by the Directorate-
General for Justice and Consumers on 1 January 2020 due to the transfer of the Europe 
for Citizens programme to that Directorate-General); and 
the  Directorate-General  for  European  Civil  Protection  and  Humanitarian  Aid 
Operations. 

The intervention logic is in line with the Agency’s objectives and activities, as provided for in 
its legal basis. It has been adapted to the specific context, objectives, inputs, and processes for 
measuring  the  Agency’s  key  results  and  performance.  The  below  figure  summarises  the 
intervention logic used: 

6 

 
 
 
 
 
EACEA’s intervention logic: 

Source: Study supporting the triennial evaluation of the Agency 

7 

 
 
 
 
 
When  implementing  the  programmes  delegated  to  it,  EACEA  was  responsible  for:  (i) 
publishing  calls  for  proposals  after  their  approval  by  the  Commission;  (ii)  evaluating  and 
selecting  projects;  (iii)  signing  grant  agreements  and  contracts;  (iv)  monitoring  projects;  (v) 
carrying  out  the  necessary  checks  and  recovery  procedures;  and  (vi)  performing  budget 
implementation tasks covering revenue and expenditure under the EU Financial Regulation12.  
The budget implementation tasks involve, in particular: 

-  managing the operations and procedures leading to the adoption of Commission award 
decisions  and  to  the  conclusion  of  grant  agreements,  and  managing  the  ensuing 
decisions and agreements; 

-  providing support in programme implementation; 
-  performing  all the operations  required to  launch contests  and award  prizes under the 

- 

EU Financial Regulation; and 
concluding  public  procurement  procedures  and  managing  the  ensuing  contracts, 
including the operations required to launch and conclude such procedures. 

The  key  inputs  are  human  resources,  financial  resources,  IT  tools  and  communication 
channels. The key processes are set out in EACEA’s Act of Delegation, and the main outputs 
of  EACEA’s  performance  relate  to  the  management  of  the  delegated  programmes.  The 
achievement  of  the  outputs  depends  on  the  achievement  of  key  performance  indicators 
(standard  indicators  of  financial  management),  which  reflect  the  key  metrics  related  to 
proposal and project management life cycle (e.g. number of calls, number of ongoing projects, 
etc.). The key policy feedback-related outputs were also taken into consideration. 

Outcomes  relate  to  the  key  results  and  impacts  achieved  during  the  evaluation  period 
(medium-  to  long-term  outcomes).  The  satisfaction  of  unsuccessful  applicants,  beneficiaries 
and  experts  with  the  Agency’s  performance  was  one  of  the  key  indicators  used  to  measure 
those results. Outcomes also include better-informed policymaking by the parent Directorates-
General, as well as contribution to the policy priorities of the parent Directorates-General and 
the programmes managed by the Agency. 

The results of the quantitative cost-benefit analysis are also presented in this evaluation. They 
include  the  workload  analysis  and  the  evaluation  of  key  workload  drivers,  assumptions  and 
productivity  indicators  (operational  budget  delegated  to  the  Agency,  number  of  proposals, 
number  of  projects,  etc.)  underpinning  the  workload  analysis  and  staffing  estimates  in  the 
2013  cost-benefit  analysis  and/or  specific  financial  statement,  compared  with  the  current 
situation. 

2.2.  Baseline and point(s) of comparison 

This evaluation of EACEA’s operations from 1 January 2018 to 31 March 2021 assesses the 
actual  costs  and  benefits  of  programme  implementation  by  EACEA  (executive  agency 
scenario) compared with the alternative scenario of management by Commission departments 
(in-house scenario). 

12   Regulation (EU, Euratom) 2018/1046 of the European Parliament and of the Council of 18 July 2018 on the financial rules applicable 
to the general  budget of  the  Union, amending Regulations  (EU)  No 1296/2013,  (EU)  No  1301/2013,  (EU)  No  1303/2013,  (EU)  No 
1304/2013,  (EU)  No  1309/2013,  (EU)  No  1316/2013,  (EU)  No  223/2014,  (EU)  No 283/2014,  and  Decision  No  541/2014/EU  and 
repealing Regulation (EU, Euratom) No 966/2012 (OJ L 193, 30.7.2018, p. 1). 

8 

 
 
 
Accordingly,  the  reference  points  for  the  current  EACEA  evaluation  are  the  2013  ex ante 
cost-benefit analysis and EACEA’s specific financial statement. 

The costs of the executive agency scenario were much lower than the estimated costs of the 
in-house  scenario.  In  2018-2020,  the  cost  difference  between  the  executive  agency  scenario 
and  the  in-house  scenario  resulted  in  cost  savings  of  EUR 87.7  million  (or  36.4%  of  the 
estimated costs under the in-house scenario). 

As shown in  the intervention logic table above,  the main outputs of EACEA’s performance 
relate  to  the  Agency’s  implementation  of  the  EU  programmes  and  the  provision  of  policy 
feedback  to  parent  Directorates-General.  Key  performance  indicators,  standard  indicators  of 
financial  management,  and  key  metrics  for  the  proposal  and  project  management  lifecycle 
(e.g.  number  of  calls,  number  of  running  projects,  etc.)  were  used  to  assess  the 
implementation of EU programmes. The Agency’s key policy feedback-related outputs were 
also taken into consideration. 

The  satisfaction  of  unsuccessful  applicants,  beneficiaries  and  experts  with  the  Agency’s 
performance was considered a key output and was measured via a survey. The EU’s visibility 
as  the  promoter  of  the  programmes  entrusted  to  the  Agency  was  assessed  through  the 
Agency’s external communication and media campaigns and via a survey. 

3.  HOW HAS THE SITUATION EVOLVED OVER THE EVALUATION PERIOD? 

Current state of play 

EACEA was set up by Commission Decision No 2005/56/EC to manage EU activities in the 
fields of education, audiovisual and culture, and is one of the oldest executive agencies of the 
Commission.  Since  2006,  EACEA  has  been  responsible  for  programme  management  that 
does not  entail  political  decision-making, but  which requires  a high  degree of technical  and 
financial  expertise  throughout  the  project  life  cycle.  Its  programme  management  and  policy 
feedback activities help to implement the broader Commission priorities and the objectives of 
the parent Directorates-General. 

As  stipulated  in  the  EACEA  Delegation  Decision,  a  Memorandum  of  Understanding  was 
signed between EACEA and the parent Directorates-General. It established the detailed rules 
and  procedures  for  interaction  and  set  out  a  clear  delimitation  of  responsibilities  between 
EACEA  and  its  parent  Directorates-General.  This  evaluation  assesses  whether  that 
distribution of roles was complied with between 1 January 2018 and 31 March 2021, as well 
as the effects of the intervention on the coherence and the efficiency of EACEA’s activities. 

EACEA  successfully  adapted  to  the  challenges  of  the  COVID-19  pandemic  by  adapting  its 
activities to the remote working environment and enabling staff to work remotely. 

The retrospective cost-benefit analysis revealed that in 2018-2020, EACEA managed to cope 
using  the  staff  allocated  to  it.  The  composition  of  staff  (the  ratio  of  temporary  agents  to 
contract agents) corresponded to estimates in the specific financial statement. Over the course 
of  the  evaluation  period,  the  Agency  underwent  significant  organisational  changes,  which 
were  implemented  via  EACEA’s  transformation  project  and  led  by  the  newly  appointed 
Director  and  the  Steering  Committee.  The  Agency’s  reorganisation  was  carried  out  in  a 
participatory manner, informed by extensive consultations with its staff. As a result, a second 

9 

 
 
 
department  (Department  B:  Creativity,  Citizens  and  Joint  Operations)  and  two  horizontal 
support units were set up to strengthen coherence, synergies and administrative support across 
the Agency’s operational units, and to contribute to efficiency and compliance. 

EACEA’s  staff  engagement  index  increased  from  59%  in  2018  to  67%  (i.e.  2%  above  the 
threshold)  outside  the  evaluation  period  in  2021.  Although  the  Agency  did  not  achieve  its 
target for the percentage of women in middle management (which fell from 23% in 2019 to 
20%  outside  the  evaluation  period  in  2021),  the  responsibility  for  this  lies  with  the 
Commission, which appoints the Agency’s leadership. 

The  occupation  rate  at  EACEA  remained  stable  between  2018  and  2020,  while  the  staff 
turnover  rate  increased  slightly.  In  2021,  with  the  arrival  of  the  new  MFF  and  a  continued 
increase in activities and staff, the occupation rate was initially slightly lower than in previous 
years (91%), while the turnover rate was 5.5%. 

During 2018-2020, the administrative budget actually implemented by EACEA13 amounted to 
EUR 149.7  million.  Based  on  the  EU  contributions,  the  actual  administrative  budget  of 
EACEA  amounted  to  EUR 140.94  million  and  was  4.7%  lower  than  that  estimated  in  the 
specific  financial  statement,  representing  savings  of  EUR 6.95  million.  The  estimates  in  the 
specific  financial  statement  were  based  on  the  EU  contribution  only,  whereas  EACEA’s 
administrative  budget  also  included  contributions  from  EFTA,  non-Member  States  and  the 
European  Development  Fund  (EDF)  (EUR 8.746  million  in  2018-2020)  which  gave  the 
Agency an additional operational budget. 

EACEA’s actual operational budget14 in 2018-2020 amounted to EUR 2.65 billion, of which 
EUR 101.49 million came from contributions from EFTA, non-Member States and the EDF, 
which were not taken into consideration in EACEA’s specific financial statement. Excluding 
those  contributions,  EACEA’s  operational  budget  in  2018-2020  amounted  to  EUR 2.55 
billion,  which  is  4%  higher  than  the  initial  estimates  in  the  specific  financial  statement 
(EUR 2.45 billion). 

The  total  number  of  calls  for  proposals  launched  by  EACEA  remained  stable  overall 
throughout  the  previous  evaluation  periods  and  during  2018.  During  the  current  evaluation 
period,  the  number  of  calls  launched  each  year  varied  widely.  In  2019,  the  number  of  calls 
increased significantly to 51 (from 28 in 2018) but decreased sharply again in 2020. That fall 
could be attributed to the end of the 2014-2020 programming period, which led the Agency to 
focus on evaluating and finalising ongoing projects. The year in which the greatest number of 
calls for proposals was launched  (106) was 2021, which marked the beginning of the 2021-
2027 MFF. 

The number of grants concluded under 2018-2020 calls was very close to the estimates in the 
cost-benefit analysis. The number of grants was 19% higher for Erasmus+, which was mostly 
due  to  the  programme’s  higher  operational  budget.  The  lower  number  of  Creative  Europe 
grants  in  2019  and  2020  was  due  to  the  revision  of  three  schemes  under  the  MEDIA  sub-
programme,  which  resulted  in  a  reduction  in  the  number  of  low-value  grants  awarded.  The 

13   Based on executed commitment appropriations. 
14  

In commitment appropriations. Including EDF  – Intra-ACP (African, Caribbean and Pacific) allocations, which were not included in 
the draft annual accounts and financial reports presented in Annex 3 of EACEA’s Annual Activity Reports. 

10 

 
 
 
number  of  Europe  for  Citizens  grants  was  25%  lower  than  initially  estimated  in  the  cost-
benefit analysis owing to higher-than-anticipated average grant sizes. 

The number of signed grants increased slightly throughout the period (i.e. from 3 606 in 2018 
to 3 932 in 2020). 

During the evaluation period, EACEA continued to manage a broad but thematically coherent 
portfolio  of  programmes,  requiring  high  levels  of  expertise  in  several  policy  fields 
(citizenship,  education  and  training,  media,  youth,  sport,  culture,  volunteering,  and 
humanitarian  aid).  The  programmes  (or  parts  of  programmes)  managed  included  Creative 
Europe, Erasmus+, Europe for Citizens, EU Aid Volunteers, the Pan-African programme, the 
European  Solidarity  Corps  and  some  projects  from  the  2007-2013  legacy  programmes. 
EACEA’s mandate was expanded four times to include: 

•  youth, social inclusion, volunteering and policy support in 2018; 
• 

the European Universities initiative and the management of European Youth Together 
later in 2018; 

•  parts  of  the  management  of  the  European  Solidarity  Corps  programme,  which  were 
delegated to the Agency in November 2018 (the key change to its portfolio during the 
evaluation period); and 
 higher  education,  vocational  education  and  training,  social  inclusion,  volunteering, 
and humanitarian aid in 2019. 

• 

4.  EVALUATION FINDINGS (ANALYTICAL PART) 

4.1. To what extent was the intervention successful and why? 

Effectiveness 

For this evaluation, effectiveness relates to how successful the Agency has been in achieving 
or  progressing  towards  its  objectives  as  set  out  in  the  annual  work  programmes  during  the 
evaluation  period.  Other  questions  concerned  the  Agency’s  ability  to  react  and  adapt  to  the 
challenges posed by the COVID-19 pandemic. 

The  analysis  of  EACEA’s  operations  concluded  that  it  operated  within  its  legal  framework 
during the evaluation period, continuously improved its operations and achieved a high level 
of  overall  effectiveness,  as  demonstrated  by  key  performance  indicators  and  the  high 
satisfaction rate expressed in surveys by the stakeholders. 

Overall, EACEA performed well about programme implementation and achieved most of the 
key performance indicators, as mentioned below. 

The  Agency  smoothly  and  effectively  accommodated  changes  in  its  mandate.  It  responded 
well  to  the  Commission’s  changing  political  priorities  and  to  the  COVID-19  pandemic. 
EACEA was successful in producing its intended outputs and results and in achieving most of 
the key performance indicators. Overall, EACEA’s stakeholders (parent Directorates-General, 
beneficiaries,  unsuccessful  applicants,  and  external  experts)  were  satisfied  with 
its 
performance. 

The  Agency  took  steps  to  improve  its  key  external  communication  channels  and  measures 
during the evaluation period and met all its various external communication targets for 2018-
2020. 

11 

 
 
EACEA  provided  effective  support  and  remained  accessible  to  unsuccessful  applicants, 
beneficiaries, and external experts during the different stages of the proposal and project life 
cycle. However, the study found that the Agency had room to further improve the support it 
provided during the application phase, for example, by: 

• 

• 

• 

encouraging Erasmus+ and European Solidarity Corps national agencies to coordinate 
their interaction with the applicants; 
 identifying  and  addressing  gaps  in  support  for  and  communication  with  applicants; 
and 
sharing good practices and lessons learned. 

The Agency should improve or formalise cooperation with Erasmus+ and European Solidarity 
Corps national agencies to share and implement good practices. 

Other  improvement  measures  include  facilitating  the  application  process,  enhancing  the 
accessibility of funding opportunities and providing applicants across different programmes, 
actions  and  countries  with  access  to  relevant  guidance,  information  materials,  upcoming 
information sessions, workshops and training. 

The Agency’s mandate has been extended several times. Overall, based on the findings of the 
document  analysis  and  interviews  with  Commission  officials  and  Agency  staff,  EACEA’s 
activities  were  consistent  with  its  mandate  and  tasks  during  the  evaluation  period,  and  the 
evaluation  team  did  not  identify  any  discrepancies.  According  to  interviews  with  both 
Commission and EACEA staff, the Agency aligns its activities with its mandate so that it can 
perform its tasks coherently. 

Efficiency 

This  section  considers  the  relationship  between  the  resources  used  by  the  Agency  and  the 
output.  Among  other  factors,  it  covers  the  management  and  execution  of  programmes 
managed by the Agency, and how sound the financial and human resource management was 
during the evaluation period. It also includes an analysis of the administrative and regulatory 
burden  and  looks  at  aspects  of  simplification.  Those  aspects  are  covered  by  key  indicators 
taken from Agency performance and by additional survey data. 

Overall,  EACEA  managed  and  executed  the  programmes  efficiently  and  performed  well  in 
terms  of  operational  efficiency  during  the  evaluation  period.  The  Agency  exceeded  the 
estimated  ‘budget  per  head’  ratio  and  spent  less  than  the  estimated  administrative  budget, 
thereby decreasing the ratio of the administrative to the operational budget compared with the 
previous  evaluation  period.  In  addition,  the  parent  Directorates-General  highlighted  the 
Agency’s  long  experience  in  programme  implementation,  specialised  processes  and 
knowledge, the number of available staff members and their proximity to beneficiaries. 

However,  in  line  with  the  previous  evaluation  period,  EACEA’s  costs  remain  higher 
compared with other agencies evaluated. This is because its projects are smaller than those of 
other  agencies  and  because  it  has  a  large  number  of  different  types  of  applicants  and 
beneficiaries, including small civil-society organisations. 

EACEA  introduced  changes  and  simplifications  in  preparation  for  the  2021-2027  MFF, 
mainly  by onboarding the corporate  IT tool eGrants  to  streamline business  processes  across 
programmes, which temporarily increased the applicants’ and beneficiaries’ workload. Future 
simplifications  and  changes  to  processes  should  be  linked  to  specific  outcomes,  and  a  clear 

12 

 
 
approach  must  be  taken  to  monitoring  and  evaluating  their  effectiveness  across  different 
programmes/measures, to identify potential gaps. 

EACEA  achieved 
the  evaluation  period.  EACEA 
implemented its budget in line with sound financial management principles. The Agency also 
had good results in relation to financial management and control system indicators. 

its  operational  objectives  during 

EACEA  launched  an  in-depth  revision  of  its  internal  control  system  and  made  consistent 
improvements  throughout  the  evaluation  period  to  address  the  audit  recommendations 
received. In this respect, EACEA also implemented the ‘EACEA Transformation Project‘ and 
achieved internal  synergies by centralising  certain business  processes  and procedures across 
programmes in horizontal support units. This gave the opportunity to carry out a substantial 
internal  mobility  exercise  to  support  career  development  and  the  best  use  of  staff 
competencies.  By  the  end  of  the  evaluation  period,  EACEA  demonstrated  an  effective  and 
reliable internal control system aligned with the principles of sound financial management. 

As  the  result  of  an  Internal  Audit  Service  Audit  on  Erasmus+  and  Creative  Europe  grant 
management, the Service recommended that EACEA’s anti-fraud strategy should be updated. 
The  Commission  revised  its  2019  anti-fraud  strategy  and  its  accompanying  action  plan  in 
2023.  It  concluded  that  the  2019  anti-fraud  strategy  remains  valid  and  that  the  action  plan 
merely needs to be updated. The EACEA anti-fraud strategy for 2021-2023 was approved in 
June 2021. 

During  the  evaluation  period,  EACEA  was  monitored  through  five  key  performance 
indicators. The Agency met most of those key performance indicators in 2019-2021, with the 
exception  of  the  ‘time-to-pay’  targets.  The  content  of  some  key  performance  indicators 
changed during the evaluation period, with significant changes occurring in 2021. 

The ‘time-to-grant’ indicator remained stable during the evaluation period with a target of 9 
months, which was achieved (7 months in 2018, falling to 5.52 months in 2021). The Agency 
met the ‘time-to-pay’ indicator in 2018 (achieving 98% vs a target of 90%), was close to the 
target in 2021 (97% vs a target of 100%) but fell below the target value of 100% in 2019 and 
2020 (achieving 95% in both years). The deviation from the target in 2020 was mainly due to 
the lockdown and enforced remote working, which slowed down the processes. 

Although  the  Agency’s  operational  and  administrative  budget  increased  during  2018-2021, 
EACEA  proved  to  be  an  efficient  and  cost-effective  structure  for  managing  the  delegated 
programmes.  This was demonstrated by the programme management  cost ratio between the 
Agency’s administrative budget and the operational budget it manages. 

EACEA  almost  achieved  full  budget  execution,  with  97%  of  commitments  and  94%  of 
payments  executed  in  2018,  rising  to  100%  and  99%  respectively  in  2021.  The  payments 
executed dropped to 91% in 2020 as a result of the COVID-19 pandemic and the lockdown 
policies that halted the implementation of some projects. 

In general, the Agency’s administrative budget rose steadily from EUR 49.6 million in 2018 
to  EUR 53.9  million in  2021.  The  largest  rise  was  seen  in  the  budget  for  infrastructure  and 
operating expenditures (from EUR 8 million in 2018 to EUR 9.1 million in 2021) and in staff 
expenditures (from EUR 35.9 million in 2018 to EUR 39.5 million in 2021). 

13 

 
 
In  general,  EACEA’s  operational  budget  has  been  fluctuating  since  2018.  It  rose  from 
EUR 752  to  EUR 970  million  in  2018-2020  but  fell  to  EUR 834  million  (EUR 136  million 
less than the year before) in 2021. 

There  were 1 026  experts  in  2020 and 806 in 2021, but  the evaluation  report does not  state 
how many experts were contracted during 2018-2019. 

The  costs  for  experts  varied  in  line  with  the  operational  budget.  The  average  expert 
it  peaked  at 
expenditure  was  at 
EUR 473/proposal in 2020 and then declined to EUR 401/proposal in 2021. 

in  2018  at  EUR 354  per  proposal, 

lowest 

its 

The Agency’s application, evaluation and selection processes were transparent, user-friendly 
and  clear  overall.  However,  as  in  the  previous  evaluation  period,  EACEA’s  programme 
management costs remain higher than those of the other executive agencies evaluated. In this 
context, it should be noted that the average size of the projects managed by the Agency are 
smaller  than  those  of  other  agencies,  and  it  works  with  a  large  number  of  applicants  and 
beneficiaries of various types, including small civil-society organisations. 

During the evaluation period, EACEA introduced changes and simplifications in preparation 
for the 2021-2027 MFF. The Agency increased digitalisation (by implement remote working 
and  streamlining  and  digitalising  workflows),  switched  to  corporate  IT  tools  (e.g.  the 
onboarding of the corporate grant management tool eGrants) and created horizontal units. The 
expansion  of  simplified  cost  options  has  eased  the  work  of  beneficiaries  and  reduced 
EACEA’s administrative burden. The Agency expects those simplifications to help maintain 
low error rates and contribute to faster payment and application processes. 

However,  based  on  interviews  with  the  Agency  and  its  parent  Directorates-General,  the 
transition  increased  the  workload for  applicants  and beneficiaries during  some stages of the 
application and project cycle. 

Coherence 

Coherence  looks  at  any  overlaps  and  complementarities  within  the  programme  portfolio 
managed by the Agency and at the delimitation of responsibilities between the Agency and its 
parent  Directorates-General.  Other  questions  related  to  the  Agency’s  ability  to  provide 
feedback on policy to its parent Directorates-General. 

The study confirmed EACEA’s ability to fulfil its responsibilities efficiently and effectively. 
The  Agency’s  significant  expertise  in  programme  management  in  the  specific  fields  of  the 
delegated programmes enables the Commission to focus on its primary policymaking role. 

EACEA  took  a  coherent  approach  to  implementing  programmes  and  coordinating  with  the 
Commission.  The  revisions  and  updates  of  the  Agency’s  organisational  structure,  various 
communication  and  working  arrangements  and  streamlining  of  processes  facilitated  that 
progress. 

To  ensure  overall  coherence,  the  legal  framework  and  the  supplementary  Memorandum  of 
Understanding  set  out  clear  provisions  on  the  division  of  responsibilities  between  EACEA 
and its parent Directorates-Generals. EACEA’s transformation project successfully exploited 
opportunities  to  enhance  coherence.  The  project  addressed  the  organisational  setup  to 
accommodate the increasing complexity of the Agency’s mandate and its interactions with a 
growing  number of  Directorates-General. The organisational  restructuring brought  clarity to 

14 

 
 
the  overall  management  of  the  Agency.  In  addition,  overall  efficiency  and  coherence  were 
increased  through  further  streamlining  of  processes,  setting  up  working  arrangements  and 
harmonising the points of contact between the Agency and its parent Directorates-General. 

Overall, 
communication arrangements and practices between EACEA and the Commission. 

the  evaluation  revealed  significant 

improvements 

to 

the  coordination  and 

The  cooperation  between  the  Agency  and  the  Commission  in  preparing  work  programmes 
was  positive  overall,  but  the  study  found  that  it  could  be  improved  by  ensuring  the  earlier 
involvement of the Agency and by further exploiting the potential for smoother planning and 
implementation. 

Feedback  to  policy  by  the  Agency  could  benefit  from  a  more  qualitative  approach  to  better 
reflect  the  real-life  impact  of  the  programmes.  The  parent  Directorates-General  and  the 
Agency should work closely together to set specific feedback to policy arrangements. 

4.2. Cost-benefit analysis 

The retrospective  cost-benefit analysis for 2018 to 31 March 2021 was carried out based on 
the results of the 2013 ex ante cost-benefit analysis, the assumptions in the specific financial 
statement and EACEA’s actual costs. 

For EACEA, the costs of the executive agency scenario were much lower than the estimated 
costs  of  the  in-house  scenario.  In  2018-2020,  the  difference  in  costs  between  the  executive 
agency  scenario  and  the  in-house  scenario  is  EUR 87.7  million.  When  the  savings  initially 
estimated in the specific financial statement and the cost-benefit analysis were compared with 
the  actual  savings  from  delegating  tasks  to  EACEA,  the  actual  savings  during  2018-2020 
were  found  to  be  24.7%  higher  than  the  estimates  in  the  initial  specific  financial  statement 
(EUR 87.7  million  compared  with  EUR 70.3  million).  The  savings  in  the  executive  agency 
scenario  resulted  primarily  from  a  higher  proportion  of  lower-cost  contract  agents  being 
employed in the Agency, and a lower overall number of staff members. 

During 2018-2020, the administrative budget actually implemented by EACEA15 amounted to 
EUR 149.7  million.  The  estimates  in  the  specific  financial  statement  were  based  on  the  EU 
contribution only, whereas EACEA’s administrative budget also included contributions from 
EFTA, non-Member States and the EDF (EUR 8.746 million during 2018-2020) to manage an 
additional operational budget. Based on the EU contributions, the actual administrative budget 
of  EACEA  amounted  to  EUR 140.94  million  and  was  4.7%  lower  (representing  savings  of 
EUR 6.95 million) than that estimated in the specific financial statement. 

The  specific  financial  statement  estimates  were  based  on  the  EU  contribution  alone,  but 
EACEA’s administrative budget also included contributions from EFTA, non-Member States 
and  from  the  EDF  (EUR 8.746  million  during  2018-2020)  which  gave  the  Agency  an 
additional operational budget. Based on the EU contributions only, the  actual administrative 
budget of EACEA amounted to EUR 140.94 million, 4.7% which was lower than estimated in 
the specific financial statement, with savings of EUR 6.95 million. 

15   Based on executed commitment appropriations. 

15 

 
 
 
Between 2018 and 2020, the number of staff at the Agency decreased  slightly (from 442 to 
432  actual  staff).  The  planned  number  of  staff  decreased  from  462  in  2018  to  455 in  2020. 
There were fewer posts at the Agency in 2018 compared with 2017 due to the move to SEDIA 
(Single  Electronic  Data  Interchange  Ares)  and  a  decrease  in  posts  financed  by  R0  (‘Third 
Countries’) and EDF credits. In 2021, which is largely outside the period of this evaluation, 
the  total  number  of  posts  increased  to  50016,  of  which  454  were  occupied  at  the  end  of  the 
year17. The Agency received additional posts under the 2021-2027 MFF and launched efforts 
to bolster its staff in 2021. 

Estimated costs and savings under the executive agency scenario in 2018-2020, in EUR 
million 

191,47

143,80

47,67

195,71

125,34

70,37

221,69

133,95

87,74

250,00

200,00

150,00

100,00

50,00

0,00

CBA estimations

SFS estimations

Actual

Costs of the in-house scenario Costs of the Executive Agency (EA) scenario

Savings of the EA scenario

Source: Study supporting the triennial evaluation of the Agency 

Staff-related expenditure (Title I) was higher than in the initial estimates because of average 
staff costs being higher than initially estimated. Therefore, average staff cost estimates in the 
specific financial statement were  constant for 2014-2020, but actual average staff costs rose 
during that period due to salary indexation, promotions, seniority increments18. During 2018-
2020,  actual  average  staff  costs  at  EACEA  were  higher  than  the  average  costs  for  the 
Commission, in terms of both temporary agents and contract agents. 

The  actual  infrastructure  and  operating  expenditure  (overheads,  Title  II)  during  2018-2020 
was  11%  below  the  specific  financial  statement  estimates.  EACEA’s  actual  average 
overheads  were  also  lower  than  the  average  costs  for  the  Commission.  Actual  programme 
support expenditure (Title III) in 2018-2020 was 25% below the specific financial statement 
estimates. The increase in Title I expenditure could be offset by the lower actual expenditure 
under Title II and Title III. 

5.  WHAT ARE THE CONCLUSIONS AND LESSONS LEARNED? 

5.1  Conclusions 

The  results  of  the  evaluation  of  EACEA  confirm  that  the  delegation  of  programme 
implementation generated significant cost savings in EU budget and high value for money and 
was therefore considerably more cost-effective than the in-house scenario, as demonstrated by 
the retrospective cost-benefit analysis in the evaluation report. 

16   EACEA’s Annual Work Programme for 2021. 
17   EACEA’s Annual Activity Report for 2021 
18   The  issue  of  higher  staff  costs  was  already  highlighted  during  the  previous  evaluations  of  EACEA  and  corresponding  retrospective 

CBAs. 

16 

 
 
 
 
 
The  Agency  was  effective  and  efficient  in  achieving  its  objectives,  which  are  to  implement 
the parts of EU programmes that have been delegated to it and to provide administrative and 
logistical support to Commission services and other agencies. This is demonstrated by the key 
performance indicators and the high levels of stakeholder satisfaction. 

The  Agency  and  the  relevant  Commission  departments  communicated  through  well-
functioning mechanisms, which were formalised and improved over the evaluation period. No 
evidence of overlaps, gaps or inconsistencies in the Agency’s management of the programme 
portfolio was identified. There was a clear delineation of tasks between the EACEA and the 
parent Directorates-General. 

With  pressure  on  the  EU  budget  rising,  EACEA  is  encouraged  to  strengthen  the  internal 
synergies between the different programmes it manages and to improve its cooperation with 
other  agencies.  It  should  do  this,  in  particular,  by  continuing  the  positive  trend  of  sharing 
knowledge and good practices so that it can maintain and improve its performance as a lean 
and flexible administration. 

5.2  Lessons learned 

In the light of the above, EACEA may wish to consider developing measures to improve the 
support provided to potential beneficiaries during the application stage. Such measures could 
serve to: (i) encourage national bodies to streamline existing support; (ii) identify and address 
gaps in support for - and communication with - applicants; and (iii) share good practices and 
lessons learned. Several  steps in this direction have already been taken (e.g. formalising the 
cooperation  between  EACEA  and  national  agencies,  expanding  the  use  of  multilingual 
resources, and improving information sharing and communication procedures). 

Another recommendation that is already being addressed by EACEA and the Commission is 
to ensure that current and future harmonisation  and simplification efforts serve the needs of 
EACEA’s programmes and stakeholders. Over the years, the central Commission departments 
have  taken  robust  steps  towards  harmonising  processes  and  procedures  across  the 
Commission  and  its  agencies,  while  still  addressing  the  needs  for  flexibility  specific  to 
different  policy  areas.  With  the  2021-2027  MFF,  programmes  and  the  IT  systems  that 
underpin  them  have  been  designed  to  address  gaps  and  lessons  learn  from  the  previous 
programmes. 

EACEA  underwent  a  significant  organisational  change  during  the  evaluation  period.  This 
resulted  in  the  introduction  of  a  second  department  and  two  horizontal  units  that  aim  to 
improve  coherence,  synergies  and  administrative  support  across  operational  units.  These 
changes are starting to show results in 2023. The Commission and EACEA will continue to 
monitor  and  track  the  Agency’s  progress  to  identify  not  only  opportunities  for  further  fine-
tuning  and  improvement,  but  also  potential  best  practices  that  could  help  other  agencies 
undergoing similar transformations. 

The  evaluation  study  has  confirmed  that  the  monitoring,  reporting  and  supervision 
arrangements put in place by the Agency and the Commission were effective. They ensured 
accountability and sufficient knowledge transfer between EACEA and its parent Directorates-
General.  The  Agency  and  its  parent  Directorates-General  should  address  a  point  for 
improvement by strengthening their cooperation during the preparation of work programmes. 

17 

 
 
Through its feedback to policy activities, the Agency provides its parent Directorates-General 
with  extremely  useful  insights  on  the  functioning  of  the  programmes  it  implements.  The 
Agency  could  further  strengthen  this  work,  by  focusing  more  on  qualitative  data  to  better 
reflect the programmes’ real-world impact. 

18 

 
 
ANNEX I: PROCEDURAL INFORMATION 

The  evaluation  was  supported  by  an  external  study  and  followed  the  Commission’s  Better 
Regulation  principles,  by  applying  the  standard  evaluation  criteria.  This  evaluation  of  EACEA’s 
operations had a clearly defined scope: 

-  The reference period of the evaluation was 1 January 2018 – 31 March 2021. 

-  The  scope  of  the  evaluation  covered  EACEA’s  mandate,  which  consisted  of  programme 

implementation and administrative and providing logistical support services. 
With regard to programme implementation, EACEA continued to manage between January 
2018- March 2021 the following programmes: Europe for Citizens (2014-2020), Erasmus+ 
(2014-2020), Creative Europe (2014-2020), the European Solidarity Corps (2018-2020), EU 
Aid Volunteers (2014-2020) and the Pan African Programme (2014-2020). 

-  The  current  generation  of  agencies  established  under  the  2021-2027  multiannual  financial 

framework was outside the scope of this evaluation. 

the  present  evaluation  will 
Knowledge  from 
implementation by the agencies of the programmes of the Commission. 

inform  on  possible 

improvements 

in 

the 

The evaluation of EACEA was part of the evaluation of all the six agencies, which was made in a 
coordinated  manner  based  on  a  supporting  study  carried  out  by  an  external  contractor.  The 
preparations for the evaluation  started in  2022, when an  interservice group was set  up by all  lead 
parent Directorates-General and the concerned central services (Directorate-General for Budget and 
the  Secretariat-General).  The  six  group  meetings  were  chaired  by  the  Common  Implementation 
Centre,  set  up  within  the  Directorate-General  for  Research  and  Innovation,  who  also  ensured  the 
secretariat. The evaluation of each agency was led by the respective lead parent Directorate-General 
(PLAN/2022/1912). The Commission also launched a ‘call for evidence’ on 10 March 2023 on its 
‘Have your say’ portal. The call was open for feedback on agencies activities until 7 April 2023. 

The methodology used for the evaluation was consistent across the agencies in accordance with the 
criteria  provided  in  the  Commission’s  Better  Regulation  guidelines.  The  assessment  of  the 
evaluation  criteria  ‘EU  added  value’,  i.e.  why  the  EU  should  act,  was  not  considered  relevant  as 
each agency carries out tasks which the Commission transferred to it. The EU added value of the 
programmes that each agency manages is assessed in the context of the programme evaluations. 

Evidence  was  taken  from  sources  such  as  the  Commission  databases,  annual  reporting  exercises, 
adopted  decisions.  The  supporting  study  was  prepared  using  a  mixed-methods  approach  at  the 
levels of methodologies and methods and worked on a qualitative methodological approach (based 
on documentary review and desk research, interviews, answers to open-ended survey questions and 
qualitative cost-benefit analysis) combined with a quantitative methodological approach (based on 
administrative  and  monitoring  data,  surveys  and  quantitative  cost-benefit  analysis),  as  detailed  in 
Annex II. 

Furthermore,  the  results  of  this  evaluation  will  be  useful  for  accountability  purposes.  The  final 
evaluation reports on the performance of the six agencies will allow the Commission to report the 
results of the retrospective cost-benefit analysis to the budgetary authority while informing various 
European  Union  (EU)  institutions  on  the  value  of  the  cost  savings  achieved  as  a  result  of  the 
executive agency scenario compared with the costs of in-house management by the Commission, as 
well  as  identifying  potential  areas  for  improvement.  The  results  of  this  evaluation  will  be 
communicated  to  the  Steering  Committee  of  the  Agency,  to  the  European  Parliament,  to  the 
Council, and the European Court of Auditors. 

19 

 
 
Along  with  changes  to  EACEA’s  mandate  for  various  EU  programmes,  the  Agency‘s  parent 
Directorate  General  have  also  changed  slightly.  During  the  present  evaluation  period,  four  parent 
Directorates-General supervised EACEA:  

-  Directorate-General  for  Education,  Youth,  Sport  and  Culture  (the  lead  parent  Directorate-

General) 

-  Directorate-General for Communications Networks, Content and Technology 
-  Directorate-General  for  Migration  and  Home  Affairs  (replaced  by  Directorate-General  for 
Justice  and  Consumers  on  1  January  2020,  due  to  the  transfer  of  the  Europe  for  Citizens 
programme to that Directorate-General), and  

-  Directorate-General for European Civil Protection and Humanitarian Aid Operations. 

20 

 
 
 
 
ANNEX II: METHODOLOGY AND ANALYTICAL MODELS USED 

Short description of methodology 

As  mentioned  in  Section  1,  the  evaluation  was  supported  by  a  study  carried  out  by  an  external 
contractor.  The  study  covered  the  evaluation  of  the  six  executive  agencies  (INEA,  EACEA, 
EASME,  ERCEA,  CHAFEA  and  REA)  in  the  period  between  2017/2018  and  March  2021  in  a 
coordinated manner, based on the same methodology. The study was structured around a series of 
evaluation questions outlined in Annex to the study. 

A  combination  of  evaluation  methods  providing  for  the  collection  of  qualitative  and  quantitative 
information and evidence including: 

•  Extensive documentary review and desk research concerning all relevant documentation 
relating  to  EACEA’s  legal  base,  its  relation  with  the  parent  Directorates-General,  the 
delegation  of  programme  and  its  financial  and  non-financial  performance,  and  program 
management,  including  the  memorandum  of  understanding,  the  annual  work  programmes, 
the annual activity reports, the European Court of Auditors and Internal Audit Service audit 
reports,  the  previous  evaluations  and  cost-benefit  analysis;  the  results  of  the  2018  staff 
satisfaction  survey.  As  part  of  the  desk  research,  an  in-depth  analysis  was  conducted  on 
communication and policy feedback. 

•  Interview program: this targeted two types of key stakeholders: 

(1) The Commission and EACEA staff; and 

(2)  EACEA’s  beneficiaries,  unsuccessful  applicants  and  external  experts.  It  included  39 
interviews  involving  47  interviewees  were  conducted  between  18  August  2022  and  24 
February 2023. 

This is in line with the initial plan to conduct around 40 interviews for the evaluation of EACEA. 

•  Survey program: Two surveys were conducted as part of the evaluation: 

-  Survey A: EACEA beneficiaries and unsuccessful applicants.  

This  survey  addressed  all  of  the  Agency’s  beneficiaries,  and  a  random  sample  of 
5,000 unsuccessful applicants to Erasmus+, Creative Europe, Europe for Citizens, EU 
Aid Volunteers, the European Solidarity Corps and the Pan-African Programme, who 
had applied for or received funding during the evaluation period. In total, we received 
3,296 responses to this survey from EACEA beneficiaries, and 929 from unsuccessful 
applicants.  This  constitutes  a  response  rate  of  29%  for  beneficiaries  and  19%  for 
unsuccessful applicants, with an overall response rate for Survey A of 26%. 

-  Survey B: external experts contracted by EACEA.  

This survey targeted all external experts contracted by EACEA during the evaluation 
period. There were 1,341 responses, representing a response rate of 50%. 

Due to  the slightly lower response  rate achieved (compared with  the previous evaluation), 
partial responses were also used to inform the evaluation. Excluding these partial responses, 
the response rate for Survey A (beneficiaries) was 24%, while the response rate for Survey 
A (unsuccessful applicants) was 15%. 

•  Cost-benefit  analysis  included  a  workload  analysis  and  the  cost-effectiveness  and  actual 
savings from  the delegation of programme implementation,  when by comparing  the actual 
indicators  with  the  estimates  foreseen  in  the  2013  ex  ante  cost-benefit  analysis  and 
respectively the specific financial statement. 
21 

 
 
The conclusions and the recommendations proposed by the study were discussed with each EACEA 
and their lead parent Directorates-General. 

Limitations and robustness of the findings 

All  relevant  stakeholders  were  consulted  via  interviews  and  surveys  (Commission  staff,  EACEA 
staff,  beneficiaries,  unsuccessful  applicants,  experts).  Unsuccessful  applicants  participated  less 
actively compared with beneficiaries. The survey response rates were slightly lower than those for 
the previous evaluation, amounting to 29% for beneficiaries (compared with 40% for the previous 
round of evaluation) and 50% for external experts (compared with 63% for the previous round of 
evaluation). The specific measures applied by the study team had only a minor influence on these 
response rates. The comparatively lower interest on the part of the survey respondents could mostly 
be explained by survey fatigue among  respondents.  However,  while the response rate achieved is 
lower than for the previous survey, it is still sufficient in absolute numbers to perform analyses per 
target group (unsuccessful applicants, beneficiaries, external experts). 

No sampling bias was observed as the profile of the respondents to the surveys was very similar to 
the  overall  population,  guaranteeing  statistical  representativeness.  The  non-response  bias  (not  all 
characteristics  of  the  group  that  did  not  reply  had  been  captured  in  full)  was  mitigated  through 
triangulation with the results of follow-up interviews. 

The  triangulation  approach,  using  multi-level  and  multi-stakeholder  dimension  in  the  data 
collection,  ensured  the  robustness  and  reliability  of  the  data  and  information  used  to  draw  up 
conclusions  in  the  supporting  study. It  utilized  diverse  data  sources,  including  documentary 
reviews, desk research, surveys, and interviews, ensuring a comprehensive assessment of EACEA's 
operations. 

22 

 
 
 
 
 
 
 
 
ANNEX  III:  EVALUATION  MATRIX  AND  ANSWERS  TO  THE  EVALUATION 
QUESTIONS (BY CRITERION) 

The  purpose  of  the  evaluation  was  to  respond  to  evaluation  questions  relating  to  the  criteria  of 
effectiveness, efficiency, coherence, as defined by the Better Regulation. Evidence-based answers, 
including the points of comparison and sources, can be found in Section 4 of the external evaluation 
report19. 

19   EACEA evaluation report 

23 

 
 
 
 
 
 
ANNEX IV: OVERVIEW OF BENEFITS AND COSTS 

The benefits and costs are detailed in the Cost Benefit Analysis in Section 4.2 and Annex V below. 

24 

 
 
 
 
ANNEX V: COST BENEFIT ANALYSIS 

A cost-benefit analysis was performed to assess whether the conclusions of the ex ante assessments 
(the estimates of savings provided in the specific financial statement,) are still valid when compared 
with the actual situation, and what the overall possible savings are. 

The following approach was adopted: 

-  To  use  data  on  the  actual  performance  of  EACEA  (actual  execution  of  the  administrative 

budget, actual staffing, etc.) for the executive agency scenario. 

-  To  follow  the  assumptions  laid  down  in  the  specific  financial  statement  to  ensure  the 
comparability and validity of results, and to provide estimates of the comparable ‘actual’ in-
house scenario (the comparator), which would best reflect the actual situation. 

-  To  assess  based  on  these  estimates  whether  the  conclusions  of  the  ex  ante  assessments 
provided  in  the  specific  financial  statement  are  still  valid  when  compared  with  the  actual 
situation, and what the overall possible savings are. 

To  deconstruct  the  ‘actual’  in-house  scenario  (the  ‘comparator’),  estimates  were  based  on  the 
following assumptions: 

-  Number  and  composition  of  staff  at  the  Commission  and  EACEA  under  an  in-house 
scenario  corresponds  with  specific  financial  statement  estimates20.  Further  contract  agents 
were added to the estimated number of staff at the Commission during the period 2018-2020 
(20 contract agents in 2018, 18 in 2019, and 17 in 2020), to reflect the additional authorised 
staff at EACEA financed from the contributions of the EFTA, EDF and the participation of 
candidate  countries  and/or  third  countries,  to  manage  additional  operational  budget  not 
covered in the cost-benefit analysis/specific financial statement resource calculations. 

-  Commission  staff  costs  and  overheads  correspond  to  the  average  costs  set  by  Directorate-
General  for  Budget  for  the  estimates  of  human  resources  and  overheads  in  legislative 
financial statements for the respective year. 

-  Programme  support  expenditure  (Title  III)  remains  the  same  under  both  the  in-house 

scenario and the executive agency scenario. 

The  table  below  presents  the  results  of  the  analysis  of  the  estimated  actual  costs  of  the  in-house 
scenario and the actual costs of the executive agency scenario. 

20   Additional resources allocated under Specific Financial Statement amendments are also considered. 

25 

 
 
 
 
 
Estimated  actual  costs  of  the  in-house  scenario  and  the  actual  costs  of  the  executive  agency 
scenario, EUR 

2018 

2019 

2020 

No 

Cost 

No 

Cost 

No 

Cost 

Total 2018-
2020 

In-house scenario  

Commission 

Title I. Staff-related expenditure 

574.7 

55,520,800 

584.8 

58,667,600  618.8 

63,616,000  177,804,400 

Establishment plan posts 

388.2 

46,195,800 

398 

48,954,000  422.6 

52,825,000  147,974,800 

Contract agents 

186.5 

9,325,000 

186.8 

9,713,600 

196.2 

10,791,000  29,829,600 

Title II. Infrastructure and 
operating expenditure 
Title III. Programme support 
expenditure 
TOTAL COST 

Executive Agency scenario  

EACEA 

Title I. Staff-related expenditure 

Establishment plan posts 

Contract agents 

Interim supportive agents and 
trainees 
Professional development  
and recruitment costs 
Title II. Infrastructure and  
operating expenditure 
Title III. Programme support  
expenditure 
Total EACEA cost: 

Commission 

Title I. Staff-related expenditure 

Establishment plan posts 

Contract agents 

Title II. Infrastructure and  
operating expenditure 
Total Commission cost: 

13,792,800 

14,620,000 

15,470,000  43,882,800 

5,382,974 

7,611,700 

6,139,886 

19,134,560 

574.7 

74,696,574 

584,8 

80,899,300  618,8 

85,225,886  240,821,760 

442 

100 

342 

35,645,355 

429 

35,707,175  432 

35,971,057  107,323,587 

12,633,939 

97 

12,682,051  98 

12,856,435  25,490,374 

18,318,288 

332 

18,837,909  334 

19,343,418  71,559,528 

2,896,561 

2,381,302 

2,094,676 

4,994,676 

1,796,567 

1,805,914 

1,676,529 

5,279,010 

7,601,641 

7,780,365 

7,846,653 

23,228,660 

5,382,974 

7,611,700 

6,139,886 

19,134,560 

48,629,970 

51,099,240 

49,957,597  149,686,806 

7.7 

7.7 

916,300 

916,300 

- 

184,800 

7.7 

7.7 

947,100 

947,100 

- 

192,500 

7.7 

7.7 

962,500 

962,500 

- 

2,825,900 

2,825,900 

- 

192,500 

569,800 

1,101,100 

1,139,600 

1,155,000 

3,395,700 

TOTAL COST 

449.7 

49,731,070 

436.7 

52,238,840  439.7 

51,112,597  153,082,506 

ESTIMATED SAVINGS 

125.0 

24,965,504 

148.1 

28,660,460  179.1 

34,113,290  87,739,253 

Source: Study supporting the triennial evaluation of the Agency 

The analysis concluded that: 

•  The overall actual costs of the executive agency scenario21 amounted to EUR 153.1 million for 
the  period  2018-2020.  To  evaluate  the  extent  to  which  the  actual  costs  corresponded  to  the 
initial specific financial statement estimates, it is important to follow the same assumptions that 

21  

Including the cost of coordination and monitoring by the Commission and the costs covered from EDF and third-country contributions. 

26 

 
 
 
 
 
 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
 
  
 
 
  
  
  
  
  
  
 
 
led  to  such  specific  financial  statement  estimates.  The  specific  financial  statement  estimates 
(EUR 151.0 million for the period 2018-2020) were based on the EU contribution  alone, but 
EACEA’s  administrative  budget  also  included  contributions  from  the  EFTA,  third  countries 
and  the  EDF  (EUR  8.7  million  during  the  period  2018-2020)  to  manage  its  additional 
operational budget. Consequently, based on the EU contribution alone, the actual costs of the 
executive  agency  scenario  amounted  to  EUR  144.3  million,  which  means  that  the  actual 
savings  amounted  to  EUR  6.6  million,  corresponding  to  4.4%  of  the  specific  financial 
statement estimates. 

•  The costs of the executive agency scenario were much lower than the estimated costs of the in-
house  scenario.  In  2018-2020,  the  actual  cost  savings  deriving  from  the  difference  in  cost 
between  the  executive  agency  scenario  and  the  in-house  scenario  amounted  to  EUR  87.7 
million (or 36.4% of the estimated costs under the in-house scenario). 

•  When  comparing  the  savings  initially  estimated  in  the  specific  financial  statement  and  cost-
benefit  analysis  with  the  actual  savings  from  the  delegation  of  tasks  to  EACEA,  the  actual 
savings  during  the  2018-2020  period  were  24.7%  higher  than  the  initial  specific  financial 
statement estimates (EUR 87.7 million, compared with EUR 70.3 million), and 84.1% higher 
than  the  initial  cost-benefit  analysis  estimates  (EUR  87.7  million,  compared  with  EUR  47.7 
million). As forecast  in the  specific  financial  statement and the  ex  ante  cost-benefit  analysis, 
the savings from the executive agency scenario resulted primarily from a higher share of lower-
cost external personnel (contract agents) employed within the executive agency, and an overall 
lower number of staff22. 

22   The number of staff at the  executive  agency scenario was reduced; however, the in-house scenario was not modified in the  specific financial 

statement. 

27 

 
 
 
 
 
ANNEX VI: STAKEHOLDERS CONSULTATION – SYNOPSIS REPORT  

This annex summarises the consultation of stakeholders, which was undertaken for the evaluation 
and  was  performed  on  the  basis  of  the  Better  Regulation  Guidelines.  A  similar  approach  was 
adopted for the evaluation of all six agencies. 

1.  Outline of the consultation strategy 

The consultation strategy was designed as follows: 

Source: Study supporting the triennial evaluation of the Agency 

EACEA’s stakeholders are grouped as follows: 

-  Agency staff; 
- 
- 

staff from the five parent Directorates-General; 
applicants (successful and unsuccessful) to the funding programmes managed by EACEA; 
and 
experts hired by EACEA to assess funding proposals. 

- 

A  survey  programme  was  carried  out  for  applicants  and  experts,  and  an  interview  programme 
targeted all four groups of stakeholders. 

Consultation activities and target groups covered 

Consultation 
activity 

Surveys 

Target groups 

Date 

Survey A 

Beneficiaries (successful candidates) of 
and unsuccessful applicants to the 
programmes managed by EACEA 

Survey B 

External experts contracted by EACEA 

29 November to 
13 December 2022 

25 November to 
8 December 2022 

Interview programme 

•  EU officials from parent 
Directorates-General; 
•  Directorate of EACEA; 
•  Heads of Sector and Heads of 

• 

Unit; 
representatives of EACEA’s 
Staff Committee; 

28 

18 August 2022 to 
24 February 2023 

 
 
 
 
 
•  members of the EACEA 
Steering Committee; 
•  beneficiaries (successful 

candidates) of and unsuccessful 
applicants to the programmes 
managed by EACEA; and 

•  external  experts  contracted  by 

EACEA. 
Source: Study supporting the triennial evaluation of the Agency 

Those  stakeholder  consultations  complemented  one  another  and  informed  the  answers  to  the 
evaluation  questions  on  effectiveness  and  efficiency.  Interviews  with  Commission  officials  and 
EACEA staff not only provided information on those questions but also allowed coherence-related 
evaluation questions to be covered. 

Consultation activity and evaluation criteria covered 

Effectiveness 

Efficiency 

Coherence 

Interviews with the European Commission 

Interviews with representatives of EACEA 

Interviews with unsuccessful applicants, 
beneficiaries and experts 

Survey of EACEA’s beneficiaries and 
unsuccessful applicants 

Survey of external experts contracted by 
EACEA 

✓ 

✓ 

✓ 

✓ 

✓ 

Source: Study supporting the triennial evaluation of the Agency 

2.  Key results of the consultation activities 

2.1  Summarised results of the survey 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

Two surveys were performed using the Alchemer online survey tool. 

The  survey  programme  targeted  applicants  to  all  programmes  managed  by  EACEA  during  the 
evaluation period23. Survey A included unsuccessful applicants who  had applied to EACEA calls 
launched between 1 January 2018 and 31 March 2021, and beneficiaries, whose projects had started 
within the same period. Applicants from all six  programmes participated in the survey.  Survey B 
addressed experts who were contracted also within the same timeframe. 

-  Survey  A:  EACEA  beneficiaries  and  unsuccessful  applicants.  This  survey 
addressed all the Agency’s beneficiaries, and a random sample of 5 000 unsuccessful 
applicants  to  Erasmus+,  Creative  Europe,  Europe  for  Citizens,  EU  Aid  Volunteers, 
the European Solidarity Corps and the Pan-African programme, who had applied for 
or received funding during the evaluation period.  Survey A had an overall response 
rate  of  26%,  with  3 296  responses  (29%  response  rate)  from  EACEA  beneficiaries 
and 929 responses (19% response rate) from unsuccessful applicants. 

23   Erasmus+, Creative Europe, Europe for Citizens, the European Solidarity Corps, EU Aid Volunteers and the Pan-African programme. 

29 

 
 
 
 
 
 
 
 
 
-  Survey  B:  external  experts  contracted  by  EACEA.  This  survey  targeted  all  the 
external  experts  contracted  by  EACEA  during  the  evaluation  period.  There  were 
1 341 responses, representing a response rate of 50%. 

Full launch of 
the survey 
(majority of 
invitations 
sent on this 
date) 
29 November 
2022 

Main statistics related to the implementation of the surveys: 
Target group 

Survey 
closure date 

Number of 
responses 
received 

Response 
rate* 

Population 
targeted / 
number of 
invitations 
sent out 

Beneficiaries 

Unsuccessful 
applicants 

Experts 

13 December 
2022 

11 276 

Partial: 548 
Complete: 2 748 
Total: 3 296 

Total: 29% 
Only including 
complete: 24% 

4 933 

25 November 
2022 

8 December 
2022 

1 341 

Partial: 188 
Complete: 741 
Total: 929 
Partial: 61 
Complete: 604 
Total: 665 

Total: 19% 
Only including 
complete: 15% 
Total: 50% 
Only including 
complete: 45% 

Number of 
responses 
included in 
the analysis 
after cleaning 

Partial: 241 
Complete: 
2 744 
Total: 2 985 
Partial: 90 
Complete: 741 
Total: 831 
Partial: 38 
Complete: 602 
Total: 640 

Note: the response rate was calculated based on the formula: 

Response rate = (No. of responses received/ No. of invitations sent out) * 100%  
Source: Study supporting the triennial evaluation of the Agency 

Owing to the slightly lower response rate achieved (compared with the previous evaluation), partial 
responses were also used to inform the evaluation. Excluding those partial responses, the response 
rate for Survey A was 24% for beneficiaries and 15% for unsuccessful applicants. 

Effectiveness 

Overall,  beneficiaries  were  satisfied  with  the  quality  of  services  provided  by  the  Agency  at  the 
different stages of project management. The respondents were the most satisfied with the payment 
process (94%, 1 708 out of 1 819 respondents agreed/strongly agreed that this process was smooth), 
followed  by  the  grant  amendment  (89%,  735  out  of  824),  granting  (87%,  1 753  out  of  2 023), 
project  implementation  (86%,  1 626  out  of  1 886),  and  monitoring/audit  (84%,  249  out  of  298) 
processes.  Experts  generally  had  the  same  positive  attitudes  towards  the  services  provided  by 
EACEA,  with  95%  (566  out  of  593)  of  them  agreeing  or  strongly  agreeing  that  the  quality  of 
EACEA’s services during the execution of expert tasks was satisfactory. 

Most  of  the  surveyed  beneficiaries,  unsuccessful  applicants,  and  experts  were  satisfied  with  the 
Agency’s response to the COVID-19 pandemic. Their responses can be broken down as follows: 

•  87% (2 094 out of 2 395) replied that EACEA had acted promptly and flexibly to maintain 

the quality of its services during the pandemic; 

•  91% (2 197 out of 2 430) agreed that the Agency and/or the Commission had been flexible 

in adjusting deadlines or arranging other changes to the project; 

•  85% (1 934 out of 2 276) found the transition to virtual activities smooth; and 

•  86%  (2 058  out  of  2 381)  found  the  response  to  be  effective  in  terms  of  outreach  and 

dissemination of information. 

30 

 
 
 
 
As in previous evaluations, beneficiaries and unsuccessful applicants had different opinions on the 
Agency’s  effectiveness  during  the  evaluation  and  selection  stage.  Beneficiaries  were  significantly 
more satisfied with the clarity and transparency of the process (86%, 2 302 out of 2 663 respondents 
agreed/strongly agreed) than unsuccessful applicants (55%, 397 out of 722). The latter highlighted 
the need for more information on the decision-making process and better feedback to enable them 
to  improve  their  applications.  Most  of  the  respondents  from  both  groups  felt  that  information  for 
applicants related to the call and the application process was easy to find (80%, 2 899 out of 3 620 
and also clear enough for them to understand the application procedures (81%, 2 806 out of 3 487). 

Overall,  beneficiaries  and  unsuccessful  applicants  considered  the  application  process  to  be  clear. 
Compared  with  other  aspects,  they  were  somewhat  less  satisfied  with  the  proportionality  of  the 
application process to the administrative burden experienced when filling in the application (75%, 
2 615  out  of  3 472  of  respondents  agreed/strongly  agreed  that  they  were  proportionate).  Overall, 
both  groups  considered  the  procedures  related  to  the  application  process  to  be  user-friendly. 
However,  they  considered  that  there  is  still  some  room  for  improvement,  especially  to  make  IT 
tools  more  user-friendly  (69%,  2 380  out  of  3 430  respondents  agreed/strongly  agreed  that  they 
were user-friendly). 

Beneficiaries  considered  the  various  communication  channels  to  be  very  effective,  with  most 
respondents indicating that they provided them with relevant and helpful information.  The highest 
ratings  were  for  direct  email  contact  (93%,  1 846  out  of  1 986)  and  the  Agency’s  website  (71%, 
1 306 out of 1 837), whereas video conferences, face-to-face contacts and telephone contacts scored 
somewhat lower24. Similar trends were observed among external experts, who awarded the highest 
score for support by EACEA’s staff (94%, 209 out of 223). 

In response to open questions on areas for improvement, the applicants  expressed discontent with 
the  bureaucratic  language  of  applications,  the  length  and  complexity  of  the  application  form,  and 
the  time  needed  to  fill  in  and  submit  the  application.  A  particular  source  of  dissatisfaction  was  a 
perceived  need  to  repeat  similar  information  throughout  the  form.  The  following  potential 
weaknesses and suggestions for improvement were mentioned: 

- 

a lack of multilingual resources and services, including guidelines, help desks, web pages, 
and especially IT tools in languages other than English; 

-  unclear communication by the Agency on channels for direct contact; 
- 
- 
- 

the need to improve the usability of e-forms for Mac OS users; 
the need for more good examples throughout the application platform; and 
the need to make the existing guidelines less complex, long and unclear, and more directly 
relevant (although some applicants found them helpful). 

Efficiency 

Overall, surveyed beneficiaries and unsuccessful  applicants  provided largely  positive feedback on 
all  the  simplifications  introduced.  In  particular,  they  praised  the  move  towards  paperless 
procedures,  including  the  increased  use  of  electronic  signatures  (87%,  2 865  out  of  3 283 
respondents  agreed/strongly  agreed  that  this  simplified  the  project  application/implementation 
process). The respondents also provided very positive feedback on the use of simplified cost options 
instead of actual cost funding (83%, 2 588 out of 3 102), and the simplification and clarification of 
call guidelines (83%, 2 675 out of 3 237). Compared with all the other simplifications, they reported 
somewhat  lower  satisfaction  with  the  introduction  of  eGrants  and  the  Funding  and  Tender 

24   However,  a  significant  share  of  respondents  selected  ‘Do  not  know  /  not  applicable’  when  assessing  these  channels,  indicating  that  many 

respondents may not have used them at all during the evaluation period. 

31 

 
 
 
Opportunities  Portal  (68%,  383  out  of  565  for  the  application;  66%,  361  out  of  547  for  the 
implementation)25. 

Beneficiaries  were  generally positive about  the grant  agreement finalisation stages, except  for the 
user-friendliness of the electronic tools used to validate and assess beneficiaries and the smoothness 
of the validation process. Only 18% of beneficiaries (351 out of 1 957) did not agree that the tools 
were user-friendly, and 13% (240 out of 1 911) considered the validation process too cumbersome 
or demanding. Most of the surveyed beneficiaries whose grant agreement was amended agreed that 
the related processes were smooth (89%, 735 out of 824) and clear (90%, 741 out of 824), and that 
assigned EACEA staff were easily contactable and responsive (91%, 754 out of 827). 

The  beneficiaries  strongly  agreed  that  the  project  implementation  process  was  smooth,  with  86% 
(1 626 out of 1 886) responding positively. On average, 78% to 88% of beneficiaries agreed that the 
reporting instructions were clear and user-friendly (83%, 1 539 out of 1 855) and reasonable (84%, 
1 548  out  of  1838).  Surveyed  experts  also  reported  a  generally  positive  opinion  on  the  reporting 
process, agreeing that the reporting requirements were reasonable (84%, 495 out of 591). 

The  beneficiaries  who  were  monitored  and/or  audited,  generally  had  positive  experiences.  More 
than  80%  of  beneficiaries  (strongly)  agreed  that  the  time  it  took  the  experts  or  EACEA  to 
monitor/audit them was appropriate (82%, 242 out of 296), and that the amount of information to be 
provided was reasonable (85%, 259 out of 305) and clearly indicated (86%, 262 out of 304). The 
surveyed  experts  and  beneficiaries  held  similar  opinions  on  the  smoothness  of  the  monitoring  or 
audit/evaluation  process.  84%  (249  out  of  298)  of  beneficiaries  thought  that  the  monitoring/audit 
process was smooth, while 10% (30) had a neutral opinion on that statement. Similarly, 91% (543 
out of 593) of experts assessed the evaluation/project monitoring process positively, while 3% (19) 
had a neutral opinion. 

2.2 Summarised results of the interview programme 

The interview programme encompassed scoping, the main phase and follow-up interviews with the 
representatives  of  the  Commission  and  EACEA,  as  well  as  with  beneficiaries,  unsuccessful 
applicants and external experts contracted by the Agency. The interview programme was designed 
to  cover  a  wide  variety  of  views,  including  those  of  representatives  of  the  Commission  and  the 
Agency, as well as those of other stakeholders. 

The interviews were conducted using a standardised questionnaire, which included questions on the 
evaluation  criteria  and  which  was  adjusted  to  the  experience  of  individual  stakeholders.  Each 
interview  was  recorded,  and  interview  notes  and/or  transcripts  were  prepared  on  the  basis  of  the 
recording. In line with the planned number of interviews, a total of 39 interviews were conducted 
with 47 interviewees, which can be broken down as follows: 

-  10 interviews (11 interviewees) with the representatives of parent Directorates-General; 
-  17 interviews (23 interviewees) with the representatives of EACEA; 
-  8  interviews  (8  interviewees)  with  the  unsuccessful  applicants  or  beneficiaries  of 

programmes managed by EACEA; 

-  2 interviews (2 interviewees) with the external experts contracted by EACEA; 
-  2 interviews (3 interviewees)  conducted for follow-up and validation purposes  with  senior 
staff from the Directorate-General for Education, Youth, Sport and Culture and EACEA. 

The  interview  data  was  used  to  answer  the  relevant  evaluation  questions,  in  particular  those 
questions  where  the  opinions  of  respondents  were  of  prime  importance.  All  information  from  the 
interviews was incorporated into the final evaluation report. 

25   Only  respondents  who  took  part  in  calls  piloting  eGrants  during  the  evaluation  period  were  asked  to  assess  the  tool,  resulting  in  a  smaller 
proportion of responses compared  with all other statements. The proportion of respondents who selected  ‘Do not know / not applicable’ was 
higher for both eGrants statements than for any of the other statements, which contributed to the more negative assessment. 

32 

 
 
 
Effectiveness 

In  general,  the  interviewed  members  of  the  parent  Directorates-General  evaluated  the  programme 
management performance of the Agency positively. Although the respondents also mentioned a few 
areas  for  improvement,  they  praised  EACEA  for  successfully  managing  the  flagship  initiatives 
(such  as  the  European  Universities  initiative)  and  commended  its  performance  in  satisfying  the 
needs of beneficiaries in the evaluation period. According to the interview programme, the Agency 
also demonstrated efficient results in carrying out tasks related to providing policy feedback. This 
included providing the parent Directorates-General with useful information for their policy work. 

A lack of awareness of funding opportunities within the sectors targeted by the calls was identified 
during  several  interviews  with  beneficiaries  and  unsuccessful  applicants  who  had  experience  in 
applying for EU funding. They suggested that the Agency should promote the funding opportunities 
more actively  through social media channels  such as Facebook or Twitter, or through newsletters 
and mailing lists. 

Interviewees’  satisfaction  with  the  communication  channels  used  during  the  application  and 
implementation  phase  varied.  Several  interviewees  either  did  not  consider  contacting  the  Agency 
for  support  during  the  application  phase  and  solved  any  issues  themselves  or  indicated  that  they 
were not able to identify any available communication channels at that stage. A few interviewees 
commented very positively on the usefulness of support and guidance provided by national bodies 
during  that  process.  Both  applicants  and  beneficiaries  considered  meetings  organised  by  EACEA 
(e.g. call info days, sessions for newcomers, meetings for beneficiaries) to be particularly helpful in 
helping  them  understand  funding  procedures  (although  one  interviewee  highlighted  the  need  to 
make  the  corresponding  recordings  more  accessible).  Several  of  them  also  emphasised  the 
importance of direct contact with staff and the possibility to ask for help when needed. 

Several  beneficiaries  noted  that  external  constraints  as  a  result  of  COVID-19  negatively  affected 
their  projects  and  resulted  in  delays  in  the  implementation.  However,  interviewees  assessed  the 
support  of  EACEA’s  staff  during  the  pandemic  very  positively  overall  and  highlighted  the 
Agency’s flexibility in allowing them to adjust activities and project timelines. 

Efficiency 

Some interviewees from EACEA highlighted that local IT tools for grant management (which will 
eventually be replaced by eGrants but remain in use for legacy projects) are creating an additional 
workload  for  staff,  who  need  to  switch  between  multiple  tools.  Several  interviewees  from  the 
Directorates-General  and  the  Agency  highlighted  challenges  resulting  from  the  complexity  of  the 
eGrants  tool.  According  to  the  interviewees  from  the  Agency,  it  does  not  sufficiently  address  the 
needs of their stakeholders (i.e. the high number of smaller organisations and lower value grants). 

Several interviewees from EACEA highlighted imbalances in workload that were noticeable during 
the  substantial  changes  and  processes  that  the  Agency  underwent  during  the  evaluation  period 
(implementation  of  audit  recommendations  made  by  the  Internal  Audit  Service,  reorganisation, 
onboarding of eGrants, COVID-19 and preparation for the new MFF). The workload imbalance was 
noticeable  between  different  units  or  sectors  of  the  Agency  and  depended  on  the  number  of  calls 
and the stage of the project cycle, resulting in increased pressure on some staff. 

Interviews with beneficiaries and unsuccessful applicants revealed that they consider the application 
procedure to be manageable, although the process was described as complex and requiring a lot of 
time  and  effort.  Interviewees  emphasised  that  this  stage  is  less  challenging  for  experienced 
applicants  but  can be very complicated, burdensome and bureaucratic for newcomers and smaller 

33 

 
 
organisations  with  fewer  resources.  A  few  interviewees  found  it  challenging  to  understand  the 
application requirements and what specific information they were being asked for. They also noted 
that  guidelines  and  application  forms  should  be  less  complex  and  that  good  examples  throughout 
the process would be useful. Two interviewees described the narrative-based application templates 
as more challenging to complete compared with the previous matrix-based forms. One interviewee 
highlighted that, given the complexity of the application, the info days for potential applicants were 
held too close to the deadline. 

Most  interviewees  (beneficiaries  and  unsuccessful  applicants)  considered  the  evaluation  report 
summaries  to  be  useful,  also  for  preparing  future  applications.  A  few  interviewees  noted  that 
evaluations should be more transparent and that their approach and quality varied. One interviewee 
indicated that they received very generic and nearly identical comments for several applications. 

A  few  interviewees  among  the  beneficiaries  and  unsuccessful  applicants  described  increasingly 
digital  processes,  such  as  the  use  of  electronic  signatures,  as  positive  developments  that  facilitate 
funding  processes.  Several  interviewees  also  highlighted  that  the  increased  use  of  simplified  cost 
options (lump sums) reduces administrative burdens and makes budgeting and reporting procedures 
easier. 

Interviewed  beneficiaries  and  unsuccessful  applicants  provided  various  ideas  on  how  to  improve 
funding processes. They suggested: 

- 

improving the promotion of calls outside the Funding and Tenders Portal, i.e. through social 
media channels such as Facebook or Twitter, or through newsletters and mailing lists; 

-  holding  outreach  activities,  information  sessions,  seminars  and  targeted  courses  for 

- 

- 

applicants more frequently and making all recordings of online events available; 
improving  the  availability  of  helpdesks  and  support  during  the  application  phase  and  the 
frequency of direct exchanges with project officers during implementation; 
simplifying  application  forms  and  guidelines  and  providing  detailed  Q&A  sections  for 
actions; 

- 

-  providing  more  support  for  smaller  organisations/companies  to  succeed  when  competing 
against larger ones with substantially more resources, and reducing administrative burdens; 
enhancing  knowledge  exchanges  among  applicants/beneficiaries,  and  facilitating  regional 
events; and 
allowing  the  budget  to  be  allocated  flexibly  throughout  the  project  to  reflect  changes  in 
activities during longer projects. 

- 

Coherence 

Overall, the Commission officials who were interviewed strongly acknowledged the importance of 
the Agency model in helping them focus on policymaking tasks. Delegating tasks to EACEA added 
significant  value  owing  to  the  Agency’s  long-term  experience  and  expertise  in  both  programme 
management and the specific fields of the programmes. In addition, the benefits of the more flexible 
Agency model were noted, such as the ability to process payments more promptly, and to contract 
in 
and  manage  human  resources  more  efficiently,  which  allows  for  greater  flexibility 
accommodating additional tasks. 

34 

 
 
 
The  interview  programme  revealed  a  clear  understanding  among  the  interviewees  in  the  Agency 
and the Commission of the overall delimitation of responsibilities between EACEA and its parent 
Directorates-General.  No  examples  of  overstepped  boundaries  on  EACEA’s  part  or  of  micro-
management by the parent Directorates-General were reported. 

Overall, during the interviews, EACEA staff praised the establishment of two horizontal units  for 
helping  to  improve  and  standardise  tasks  and  processes  within  the  Agency.  Many  staff  members 
considered this to have significantly clarified and harmonised the Agency’s overall work. However, 
some noted that it was still too early to form an opinion. Interviewees at the Agency also pointed 
out  that  the  horizontal  units  have  taken  on  an  oversight  function  to  a  certain  degree.  This  was 
mostly  described in  a positive light, as it helped bring about  more quality  control  in  the common 
processes. However, a smaller group thought that the horizontal units have created an extra layer of 
control without any tangible benefits so far. 

When considering EACEA’s involvement in preparing work programmes, interviewees from both 
the Agency and the Commission maintained that the Agency’s involvement was sufficient.  It was 
informed  of  the  policy  goals  once  they  had  been  drawn  up  by  the  Commission  and  was  able  to 
provide information on the implementation aspects. However, interviewees at both the Commission 
and  the  Agency  mentioned  that  involving  the  Agency  earlier  (while  respecting  the  delineation  of 
roles) would help improve the strategic quality of the document. 

The interview programme revealed a small gap between what was expected and what was delivered 
in  terms  of policy feedback. The interviewees from  the parent  Directorates-General outlined their 
need for more proactive and improved qualitative, impact-based policy feedback from the Agency, 
while  noting  that  reporting  outputs  were  too  process-oriented  during  the  evaluation  period. 
However, the interview programme also identified room for enhanced participation and dialogue on 
the part of parent Directorates-General to calibrate the specific policy feedback arrangements with 
the  Agency.  It  was  not  always  clear  to  the  Agency  what  policy  feedback  outputs  would  be  most 
valuable to its parent Directorates-General. 

The increased supervision measures taken over by the Agency were seen as appropriate to mitigate 
risks identified by the Internal Audit Service audit during the evaluation period. Some interviewees, 
nevertheless, noted that carrying out increased supervision arrangements was highly work-intensive 
on both sides. 

2.3 Comparison of the results of consultation activities 

The table below presents the key results per consultation activity, organised by evaluation criteria, 
as  well  as  by  the  level  of  consistency,  complementarity  and  convergence  of  results  across 
consultation activities. 

Overall,  as  summarised  in  the  table  below,  there  was  high  convergence  in  the  results  of  different 
consultation activities. 

35 

 
 
 
Summary of key findings 

Evaluation 
criterion 

Survey 

Interview programme 

Effectiveness 

Efficiency 

High satisfaction overall among 
beneficiaries with the quality of services. 

Beneficiaries were satisfied with the 
clarity and transparency of the process, 
whereas unsuccessful applicants expected 
better feedback. 

Overall, beneficiaries and unsuccessful 
applicants considered the application 
process to be clear. 

Lower satisfaction among surveyed 
applicants with the proportionality of the 
application process and the administrative 
burden experienced. 

High satisfaction with the Agency’s 
response to the COVID-19 pandemic and 
its communication channels. 

Largely positive feedback on all 
simplifications introduced. 

Beneficiaries generally had positive 
opinions about the grant agreement 
finalisation stages, except for the user-
friendliness of the electronic tools. 

The beneficiaries strongly agreed that the 
project implementation process was 
smooth and the reporting requirements 
reasonable. 

The beneficiaries who were monitored 
and/or audited, generally had a positive 

Interviewed members of the parent Directorates-General 
evaluated the programme management performance of the 
Agency positively. 

A lack of awareness of funding opportunities within the sectors 
targeted by the calls was identified during several interviews with 
beneficiaries and unsuccessful applicants who had experience in 
applying for EU funding. 

Interviewees’ satisfaction with the communication channels used 
during the application and implementation phase varied. 

Overall, beneficiaries assessed the support of EACEA’s staff 
during the pandemic very positively and highlighted the Agency’s 
flexibility in allowing them to adjust activities and project 
timelines. 

Interviewees from EACEA noted that the simplification of 
funding schemes had greatly simplified the work for 
beneficiaries. 

The Agency’s staff noted imbalances and increases in workload 
during the substantial changes and processes that the Agency had 
undergone. 

Interviews with beneficiaries and unsuccessful applicants 
revealed that they consider the application procedure manageable, 
although the process was described as complex and requiring a lot 
of time and effort. 

36 

Consistency of 
results across 
consultation 
activities 

Complementarity 
of results across 
consultation 
activities 

Contradiction 
of results 
across 
consultation 
activities 

High 

High 

Low 

High 

High 

Low 

 
 
Evaluation 
criterion 

Survey 

Interview programme 

Consistency of 
results across 
consultation 
activities 

Complementarity 
of results across 
consultation 
activities 

Contradiction 
of results 
across 
consultation 
activities 

opinion of those processes. 

Respondents indicated high levels of 
understanding of the different roles 
performed by the Agency and its parent 
Directorates-General. 

Coherence 

Most interviewees (beneficiaries and unsuccessful applicants) 
considered the evaluation report summaries useful. 

A few interviewees among the beneficiaries and unsuccessful 
applicants described increasingly digital processes as positive 
developments. 

Overall, the Commission officials who were interviewed strongly 
acknowledged the importance of the Agency model in helping 
them focus on policymaking tasks. 

The clear delineation of tasks between the Agency and the parent 
Directorates-General was highlighted. 

The establishment of two horizontal units was praised for 
standardising tasks and processes within the Agency. 

Interviewees from both the Agency and the Commission 
maintained that the Agency’s involvement in the preparation of 
work programmes was sufficient. However, earlier involvement 
was also noted as an underexploited opportunity to improve the 
strategic quality of the document. 

A gap between the feedback to policy expected by the 
Commission and the feedback to policy delivered by the Agency 
was noted. However, the parent Directorates-General need to be 
more involved in calibrating specific arrangements. 

The supervision arrangements were considered appropriate to 
mitigate risks during the evaluation period. 

Source: Study supporting the triennial evaluation of the Agency 

37 

High 

High 

Low