COMMISSION STAFF WORKING DOCUMENT Evaluation of the Executive Agency for Small and Medium-Sized Enterprises (EASME) Accompanying the document REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL AND THE COURT OF AUDITORS Evaluation of the Consumers, Health, Agriculture and Food Executive Agency, the Educational, Audiovisual and Culture Executive Agency, the Executive Agency for Small and Medium-Sized Enterprises, the European Research Council Executive Agency, the Innovation and Networks Executive Agency and the Research Executive Agency
Date: 2025-04-16
Institutions: Directorate-General for Research and Innovation, European Commission
Subjects: EU aid, EU financing, EU programme, executive agency, financial control, performance audit, small and medium-sized enterprises

EN 

  EN 

   EUROPEAN COMMISSION  Brussels, 16.4.2025  SWD(2025) 90 final  COMMISSION STAFF WORKING DOCUMENT Evaluation of the Executive Agency for Small and Medium-Sized Enterprises (EASME)         Accompanying the document REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL AND THE COURT OF AUDITORS Evaluation of the Consumers, Health, Agriculture and Food Executive Agency, the Educational, Audiovisual and Culture Executive Agency, the Executive Agency for Small and Medium-Sized Enterprises, the European Research Council Executive Agency, the Innovation and Networks Executive Agency and the Research Executive Agency                                          {COM(2025) 171 final} - {SWD(2025) 87 final} - {SWD(2025) 88 final} - {SWD(2025) 89 final} - {SWD(2025) 91 final} - {SWD(2025) 92 final}  
 
 
Table of Contents 

1. 

INTRODUCTION .................................................................................................................................. 3 

2.  WHAT WAS THE EXPECTED OUTCOME OF THE INTERVENTION? ................................... 4 

2.1 
2.2 

DESCRIPTION OF THE INTERVENTION AND ITS OBJECTIVES ............................................................... 4 
POINT(S) OF COMPARISON ................................................................................................................ 7 

3.  HOW HAS THE SITUATION EVOLVED OVER THE EVALUATION PERIOD? ..................... 8 

4.  EVALUATION FINDINGS (ANALYTICAL PART) ...................................................................... 11 

4.1 
4.2 

TO WHAT EXTENT WAS THE INTERVENTION SUCCESSFUL AND WHY? ............................................. 11 
COST-BENEFIT ANALYSIS ............................................................................................................... 33 

5.  CONCLUSIONS AND LESSONS LEARNED ................................................................................. 35 

5.1 
5.2 

CONCLUSIONS ................................................................................................................................ 35 
LESSONS LEARNED ......................................................................................................................... 38 

ANNEX I: PROCEDURAL INFORMATION ........................................................................................... 40 

ANNEX II: METHODOLOGY AND ANALYTICAL MODELS USED ................................................ 41 

ANNEX III: EVALUATION MATRIX AND, WHERE RELEVANT, DETAILS ON ANSWERS TO 
THE EVALUATION QUESTIONS (BY CRITERION)........................................................................... 47 

ANNEX IV: OVERVIEW OF BENEFITS AND COSTS ......................................................................... 48 

ANNEX V: COST BENEFIT ANALYSIS ................................................................................................. 49 

ANNEX VI: STAKEHOLDER CONSULTATION - SYNOPSIS REPORT .......................................... 64 

 
 
 
 
 
 
Glossary 

Term or 
abbreviation 

CIP 
Commission 

COSME 
DG 

DG GROW 
EASME 
EISMEA 
EIC 
EMFF 
ERCEA 
ESI FUNDS 
EU 
HaDEA 
Horizon 2020 
HR 
IAS 
IEE II 
INEA 
INNOSUP 
LIFE 
REA 
SILC II 

Meaning or definition 

Competitiveness and Innovation Programme 
European Commission 
Programme for Competitiveness of Enterprises and Small and Medium-sized 
Enterprises 
Directorate-General 
European Commission Directorate-General for Internal Market, Industry, 
Entrepreneurship and SMEs 
Executive Agency for Small and Medium-sized Enterprises 
European Innovation Council and SMEs Executive Agency (former EASME) 
European Innovation Council 
European Maritime and Fisheries Fund 
European Research Council Executive Agency 
European Structural and Investment Funds  
European Union 
European Health and Digital Executive Agency 
Framework Programme for Research and Innovation 
Human resources 
Internal Audit Service 
Intelligent Energy Europe Programme 
Innovation and Networks Executive Agency 
Horizon 2020 “Innovation in SMEs” programme 
Programme for the Environment and Climate 
European Research Executive Agency 
Sustainable Industry Low Carbon Scheme 

 
 
 
1. 

INTRODUCTION 

The purpose of this evaluation is the periodical (triennial) evaluation of the operation of the Executive 
Agency for Small and Medium-Sized Enterprises (EASME).  

EASME is governed by: 

• 

• 

• 

• 

its Act of Establishment1, which sets out the mandate of the agency  

the  Act  of  Delegation2,  which  specifies  the  tasks  to  be  carried  out  by  the  agency  and  the 
powers  delegated  to  to  it  for  this  purpose.  These  tasks  relate  to  the  implementation  of 
dedicated parts of certain EU financial programmes.  

the Framework Regulation for executive agencies3  

the decision establishing the EASME steering committee.  

In line with the Commission’s Better Regulation principles4, the evaluation applies different criteria 
to guide the analysis of the agency’s performance. The evaluation assesses whether the agency has 
fulfilled  its  tasks  in  an  effective  and  efficient  way,  whether  there  are  overlaps,  gaps  and/or 
inconsistencies in the way the agency manages its programme portfolio, and whether there is a clear 
delineation of tasks between EASME and the parent Directorates-General (DGs) or other executive 
agencies (coherence). 

The evaluation also assesses whether the functioning of the agency has yielded the expected positive 
results, as estimated in the cost-benefit analysis for delegating tasks to the agency5. And it identifies 
potential areas of improvement. However, because this evaluation concerns the executive agencies 
set up by European Commission, there is no assessment of ‘EU added value’ or relevance6. 

The  estimations  of  the  2013  cost-benefit  analysis  have  been  tested  to  provide  evidence  about  the 
validity  of  the  assumptions,  by  considering  the  actual  costs  and  benefits  of  programme 
implementation by the agency in a structured way. The aspects covered by the cost-benefit analysis 
are specified in Article 3(1) of the Framework Regulation7 and the guidelines on establishing and 
operating executive agencies8.  

The evaluation does not cover the achievements of the programmes managed by EASME, which are 
subject  to  separate  mid-term  and  ex  post  evaluations.  The  evaluation  of  EASME  is,  nevertheless, 
providing useful input for these programme evaluations, as the performance of an agency affects the 
efficiency and effectiveness of the programmes it manages.  

1  

2  
3  

4  

5  

6  

7  

8  

2013/771/EU:  Commission  Implementing  Decision  of  17  December  2013  establishing  the  ‘Executive  Agency  for  Small  and  Medium-sized 
Enterprises’ and repealing Decisions 2004/20/EC and 2007/372/EC. 
Commission Decision C(2013)9414 of 23 December 2013. 
Council  Regulation  58/2003  of  19  December  2003  laying  down  the  statute  for  executive  agencies  to  be  entrusted  with  certain  tasks  in  the 
management of Community programmes, OJ L 11 of 16 January 2003, p. 1. 
Commission’s Better Regulation guidelines SWD (2021) 305 final and toolbox. 
Cost-benefit analysis for the delegation of certain tasks regarding the implementation of Union Programmes 2014-2020 to the executive agencies 
- Final report for the Commission of 19 August 2013. 
The assessment of the evaluation criterion ‘EU added value’, i.e. why the EU should act, is not perceived to be a relevant criterion for the evaluation 
of EASME, as the agency carries out tasks which the Commission has transferred to it. The EU added value of the programmes that EASME 
manages is assessed when evaluating the programmes themselves. The previous needs which EASME was meant to address and whether they are 
still pertinent (relevance) will also be assessed as part of the programme evaluation. 
The cost-benefit analysis takes into account a number of factors such as: identification of the tasks justifying outsourcing; the costs of coordination 
and  checks;  the  impact  on  human  resources;  possible  savings  within  the  EU’s  general  budget  framework;  efficiency  and  flexibility  in  the 
implementation of outsourced tasks; simplification of the procedures used; proximity of outsourced activities to final beneficiaries; visibility of 
the EU as promoter of the EU programme concerned; and the need to maintain an adequate level of know-how inside the Commission.  
Appendix II of the Guidelines for the establishment and operation of executive agencies financed from the  EU budget (C (2014) 9109 from 2 
December 2014; pp. 64-72). 

3 

 
 
 
The evaluation  examines the efficient  use of resources and the effective  achievement of the tasks 
entrusted to the agency. It also looks at whether the delegation to the agency of the management of 
different EU programmes delivered the anticipated simplification and economies of scale and whether 
there is scope for simplification and further efficiency gains.  

Also the continued relevance of all objectives and tasks entrusted to EASME were assessed, as well 
as the coherence of its programme portfolio and the delineation of tasks between it and other executive 
agencies.  

The evaluation covers all the tasks carried out by the agency during 2017-2021. The evaluation is 
supported  by  a  study  carried  out  by  an  external  contractor9,  and  builds  on  the  work  of  another 
evaluation  that  covered  the  first  3  years  of  EASME  under  the  2014-2020  multiannual  financial 
framework.  

2.  WHAT WAS THE EXPECTED OUTCOME OF THE INTERVENTION? 

2.1  Description of the intervention and its objectives  

The reference period of the evaluation was from January 2017 to March 2021, so it does not cover 
the  executive  agencies  set  up  to  manage  programmes  under  the  2021-2027  multiannual  financial 
framework, including EASME’s successor, the European Innovation Council and SMEs Executive 
Agency (EISMEA).  

The evaluation covered the programmes delegated in part to and managed by EASME in 2017-2021, 
namely:  

•  part of the Programme for the Competitiveness of Enterprises and small and medium-sized 

enterprises (COSME) 2014-202010 

•  part II and part III of the EU Framework Programme for Research and Innovation11 

•  part of the Programme for the Environment and Climate Action (LIFE) 2014-202012, 

•  part of the European Maritime and Fisheries Fund (EMFF)13  

•  part of the legacy of the Competitiveness and Innovation Programme (CIP) 2007-201314.  

New evaluation knowledge will inform internal decision-making in the successor agency of EASME 
– the European Innovation Council and SMEs Executive Agency (EISMEA) – as well as the parent 
DGs,  about  possible  improvements  to  the  implementation  of  the  legacy  programmes  and  the  next 
generation of programmes in the Commission.  

Furthermore,  the  results  of  this  evaluation  will  be  useful  for  accountability  purposes.  The  final 
evaluation reports on the performance of the six agencies will allow the Commission to report the 
results of the retrospective cost-benefit analysis to the budgetary authority (the European Parliament 
and the EU Council). And it will inform various EU institutions about the cost savings achieved as a 

9  

PPMI Consortium: Public (Policy Management Institute and IDEA Consult). Study supporting the evaluation of CHAFEA, EACEA, EASME, 
ERCEA, INEA & REA (2017/2018-2021), Final Report: EASME. 

10   Regulation - 1287/2013 - EN - EUR-Lex (europa.eu)  
11   Regulation - 1291/2013 - EN - EUR-Lex (europa.eu) 
12   Regulation - 2021/783 - EN - EUR-Lex (europa.eu) 
13   Regulation - 508/2014 - EN - EUR-Lex (europa.eu) 
14   Competitiveness and Innovation Framework Programme (CIP) (2007-2013) | EUR-Lex (europa.eu) 

4 

 
 
 
result of using the executive agencies (instead of in-house management by the Commission), as well 
as identifying potential areas for improvement.  

The results of this evaluation will be communicated to the steering committee for each agency, the 
European Parliament, the EU Council and the European Court of Auditors. 

EASME’s mandate was to provide high quality support to its beneficiaries, turning EU policy into 
action. Furthermore, it sought to ensure that projects funded by the programmes it managed would 
deliver results and provide the Commission with valuable input for its policymaking tasks15.  

The intervention logic (below) closely follows the provisions and logic provided in the key documents 
defining the objectives, tasks and activities of EASME. The overall needs that EASME was set up to 
address  were  derived  from  Council  Regulation  (EC)  No  58/200316  laying  down  the  statute  for 
executive agencies, which defines the factors against which the need to establish an executive agency 
should be assessed.  

Commission Decision C(2014)910917, along with its annex, provides guidelines for the establishment 
and operation of executive agencies. As regards the key contextual factors and events that frame the 
relevant evaluation period, these are derived from the context of EASME. 

The objectives of the agency reflect those set out in the decision on EASME’s establishment18, and 
they relate to the overall objective of implementing certain parts of the delegated programmes and 
legacy  programmes.  It  should  be  noted,  however,  that  throughout  the  evaluation,  the  general  and 
specific objectives of the programmes delegated to the agency are considered, as well as the agency’s 
operational objectives and their related indicators, as defined in its annual work programmes. 

The key inputs are human resources, financial resources, IT tools and communication channels.  

The key processes  are summarised from  EASME’s  delegation acts,  which provide a very detailed 
description of EASME’s business processes in relation to each programme managed. 

The  main  outputs  of  EASME’s  performance  relate  to  the  management  of  the  programmes.  The 
achievement is determined by meeting key performance indicators (standard indicators of financial 
management),  and  these  are  reflected  through  the  key  metrics  related  to  proposal  and  project 
management lifecycle (e.g. number of calls, number of running projects, etc.). The key feedback to 
policy-related outputs were also considered. 

Outcomes relate to the key results (immediate outcomes) achieved during the evaluation period and 
impacts (medium-to-long term outcomes). One of the key measures that informed the measurement 
of these results was the satisfaction of unsuccessful applicants, beneficiaries and experts with regard 
to the performance of the agency.  

Furthermore, facilitating a positive public perception of the EU was among EASME’s objectives as 
regards impacts. This was expected to be achieved by strengthening engagement and communicating 
effectively.  In addition,  better-informed policymaking  by the parent  DGs and  contribution  to  their 
policy priorities (and those of the programmes managed by the agency) are also among the outcomes 
assessed.

15   Annual activity report of EASME, 2018, p. 3. 
16   Council Regulation (EC) No 58/2003 of 19 December 2002 
17   Commission Decision C(2014) 9109 of 2 December 2014 
18  

2013/771/EU:  Commission  Implementing  Decision  of  17  December  2013  establishing  the  ‘Executive  Agency  for  Small  and  Medium-sized 
Enterprises’ and repealing Decisions 2004/20/EC and 2007/372/EC. 

5 

 
 
 
Figure 1. EASME’s intervention logic: 

Source: Study supporting the triennial evaluation of the Agency 

6 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
This  evaluation  looks  at  the  tasks  assigned  to  the  agency  as  regards  the  programme. 
EASME was responsible for monitoring  the projects,  making the necessary checks  and 
recovery procedures, and performing budget implementation tasks covering revenue and 
expenditure within the meaning of the EU Financial Regulation19, and in particular: 

−  managing the operations and procedures leading to the adoption of Commission award 
decisions  and  to  the  conclusion  of  grant  agreements,  and  managing  the  ensuing 
decisions and agreements; 

−  providing support to implement programmes; 

−  concluding  public  procurement  procedures  and  managing  the  ensuing  contracts, 

including the operations required to launch and conclude such procedures. 

The  results  of  the  quantitative  cost-benefit  analysis  are  also  presented.  These  include  a 
workload analysis and evaluation of key workload drivers, assumptions and productivity 
indicators (operational budget delegated to the agency, number of proposals, number of 
projects and average grant size, etc.). These factors underpinned the workload analysis and 
staffing estimates in the 2013 cost-benefit analysis and/or the specific financial statement, 
compared with the actual situation. 

Issues which impacted this evaluation were linked to the staffing changes which occurred 
in 2021, with the revised format for the executive agencies. While information and data 
were available, it took some time to identify the  relevant staff. Interviews varied in the 
level of detail or context provided, as not all interviewees were able to provide exhaustive 
insights on the performance of EASME in all the years under consideration.  

Other  interviewees  were  only  able  to  focus  on  a  specific  programme  –  which  provides 
useful detail on agency processes but may be limited as regards the aspects linked to the 
agency  and  its  parent  Directorates-General.  See  Annex  II  for  more  detail  on  the 
methodology. 

2.2  Point(s) of comparison  

The current  evaluation of EASME assesses the actual costs and benefits  of  programme 
implementation (executive agency scenario) when compared with the alternative scenario 
of management by the Commission (in-house scenario). 

Accordingly,  the  reference  points  for  the  present  evaluation  are  the  2013  ex  ante 
cost-benefit analysis and the specific financial statement drafted at the time the agency was 
set up. 

The analysis of the agency’s performance during the reference period assesses the progress 
made since the previous evaluation (2014-2016). 

The  2013  cost-benefit  analysis  estimation  was  that  the  savings  resulting  from  the 
delegation of tasks to EASME in 2017–2020 would be EUR 68.15 million, compared to 

19   Regulation  (EU,  Euratom)  2018/1046  of  the  European  Parliament  and  of  the  Council  of  18  July  2018  on  the  financial  rules 
applicable  to  the  general  budget  of  the  Union,  amending  Regulations  (EU)  No  1296/2013,  (EU)  No  1301/2013,  (EU)  No 
1303/2013, (EU) No 1304/2013, (EU) No 1309/2013, (EU) No 1316/2013, (EU) No 223/2014, (EU) No 283/2014, and Decision 
No 541/2014/EU and repealing Regulation (EU, Euratom) No 966/2012, OJ L 193, 30.7.2018, p.1. 

7 

 
 
 
the  in-house  scenario.  The  specific  financial  statement  estimated  efficiency  gains  of 
EUR 78.61 million over the same period. 

The  specific  financial  statement  estimated  the  total  number  of  full  time  equivalents 
required  to  manage  the  programmes  delegated  to  EASME  in  2020  to  be  502  (the  peak 
programming year in terms of workload).  

total 

initial  operational  budget  for  EASME 

The 
to 
EUR 6 719 billion  in  commitment  appropriations  and  EUR 5 699 billion  in  payment 
appropriations.  

in  2020  was  estimated 

Considerable efficiency  gains  were  expected  over the period,  compared to the in-house 
scenario,  along  with  non-quantifiable  benefits  such  better  programme  management  and 
service  delivery,  more  visibility  for  the  EU  programmes  and  closer  contacts  with 
beneficiaries. 

The  evaluation  study  used  the  following  data  collection  methods,  with  specific 
methodological  approaches  applied  to  ensure  data  triangulation  (confirmation  from 
different sources): 

•  document review and desk research;  

•  semi-structured interviews;  

•  online survey and statistical analysis of survey data;  

• 

retrospective cost-benefit analysis. 

The results of the previous evaluation also served as a basis for comparison, in order to 
continue with a similar survey basis. 

3.  HOW HAS THE SITUATION EVOLVED OVER THE EVALUATION PERIOD? 

Current state of play 

EASME was created in 201320 by expanding powers previously entrusted to EACI21 by 
delegating operational tasks to implement parts of Horizon 2020, COSME, LIFE and the 
EMFF. It also dealt with certain legacy actions from the CIP which ran from 2007-2013, 
with projects finalised by 2019.  
There  were  7  parent  DGs22,  and  there  is  a  strong  focus  on  small  and  medium-sized 
enterprises running through the projects. 

This  evaluation  assesses  whether  the  aims  of  delegating  the  programmes  have  been 
achieved (see section 2), and whether the distribution of tasks, agreed in the memorandum 
of understanding, has been respected for the 2017-2021 period.  

20   Commission  implementing  decision  of  17  December  2013  establishing  the  ‘Executive  agency  for  Small  and  medium-sized 

Enterprises’ and repealing Decisions 2004/20/EC and 2007/372/EC (2013/771/EU). 

21   Commission Decision 2007/372/EC of 31 May 2007 amending Decision 2004/20/EC in order to transform the Intelligent Energy 

22  

Executive agency into the Executive agency for Competitiveness and Innovation. 
(1) DG for Internal Market, Industry, Entrepreneurship and SMEs (DG GROW), (2) DG for Research and Innovation (DG RTD), 
(3) DG for Communications Networks, Content and Technology (DG CNECT), (4) DG for Climate Action (DG CLIMA), (5) DG 
for Energy (DG ENER), (6) DG for Environment (DG ENV) and (7) DG for Maritime Affaires and Fisheries (DG MARE). 

8 

 
 
 
An important aspect of the evaluation is to assess whether the cost-benefit analysis carried 
out in 2013 has yielded the expected results, and what simplification and economies of 
scale  have  occurred.  Building  from  the  evaluation  carried  out  for  2014-2016,  this 
evaluation  looks at the period from  1 January  2017 to  end March 2021 (when the new 
agencies were established).  

From a headcount of 417 in 2016, EASME’s staff increased progressively to 434 in 2017 
and 483 in 2020. The operational budget for commitments delegated to EASME increased 
from  EUR  1 383.7  million  in  2017  to  EUR  1 421.8  million  in  2020.  The  agency’s 
administrative budget also increased, from EUR 43 million in 2017, to EUR 49.2 million 
in 2020.  

The series of organisational and procedural changes that started with the implementation 
of  the  programmes  under  the  2014-2020  multiannual  financial  framework  continued, 
including maximising the use of IT tools, further work on simplifying procedures and the 
delegation of decision-making to lower levels of management.  

Both the EMFF and COSME had higher numbers of projects implemented via procurement 
rather than grants, which means less streamlining of procedures within sectors due to the 
different  needs  and  workflows.  However,  the  use  of  IT  tools  and  simplification  in 
procedures positively impacted all projects managed by the agency. 

The  complexity  of  the  different  programmes  delegated  to  the  Agency  impacted  the 
implementation of common IT tools, but also the sharing of good practice. Feedback to the 
parent DGs on policy was hampered by the diversity of the programmes, meaning that one 
system could not be implemented across the agency but rather each programme used its 
own system. Developing synergies across programmes was also limited, despite agency 
efforts. 

EASME  bilateral  meetings  with  programme  staff  in  the  DGs  worked  to  close  the  gaps 
between  what  was  feasible  and  desired  at  technical  level,  both  for  projects  and  budget 
management.  In  addition,  the  steering  committee,  by  bringing  parent  DGs  at  senior 
management level together with the agency, allowed for transparent communication and 
sharing of ideas on policy and synergies, in addition to progress reports on the management 
of the agency and its activities. 

External  stakeholders  included  project  beneficiaries,  intermediary  organisations  and 
unsuccessful applicants, as well as external experts to evaluate large calls for proposals. 
The agency continued its efforts to communicate with all internal and external stakeholders 
in the reference period, and this included agency staff.  

Staffing23 of EASME was at 95% of the staffing plan by the end of the evaluation period. 
Attracting staff from the Commission continued to be challenging, although recruitment 
overall  remained  positive.  Internally,  EASME  continued  to  apply  flexibility  to  staffing 
allocations  where  possible,  looking  at  areas  where  programmes  or  departments  needed 
reinforcement in peak periods.  

23   See section 2.2.6 of the supporting study for more detail on staffing numbers and policy and section 5.2 for costs. 

9 

 
 
 
The use of a flexibility mechanism (endorsed by the steering committee), which allowed 
up to 10% of programme staffing to be reallocated24 according to real needs, was mainly 
triggered by the understaffing of the LIFE programme. 

The communication setting up the agency, along with the specific financial statement, had 
envisaged staffing decreasing from 38 in 2016 to 33 in 2017, based on the phasing out of 
legacy  projects  under  CIP  eco-innovation.  However,  the  workload  for  LIFE  was 
underestimated in the cost benefit analysis. While the analysis assumed that the number of 
proposals to be evaluated by the agency would be around 450, in reality the number is in 
the order of 1 50025. 

Towards the end of the evaluation period, preparations for the new multiannual financial 
framework started, and this, together with the impact of the COVID-19 pandemic, meant 
considerable uncertainty for staff in 2020.  

The agency reacted with several measures both for current staff and to aid recruitment. HR 
activities took place remotely to ensure business continuity and to deliver on the workplan. 
New online courses, which were designed specifically for the pandemic, aimed to help and 
encourage staff (courses such as mindfulness and personal efficiency). Other HR activities 
ensured that recruitment could continue to attract new staff into the agency. 

IT developments and increased use of IT tools brought changes, as they impacted on the 
staffing  required  for  call  processes  but  also  on  the  time  required  by  applicants  and 
beneficiaries. The programmes delegated to EASME were not all equally suitable for the 
use of these IT tools and a period of adaptation was required for some programmes.  

In  addition,  the  programmes  with  a  higher  use  of  procurement  had  different  needs, 
including  the  use  of  less  standardised  grants.  The  inclusion  of  standardised  grant 
agreements  and  co-financing  rates  simplified  the  process  for  programmes  other  than 
Horizon 2020. IT improvements are also positively reflected in the feedback from both 
successful beneficiaries and unsuccessful applicants. 

External challenges such as Brexit and in particular the COVID-19 pandemic (see section 
4.1) meant that the agency had to react and adjust to continue delivering on the programme 
implementation and project supervision, and this was a particular factor in the evaluation 
period. 

The preparation of the supporting study encountered some problems, which led to delays 
during the process of collecting and harmonising data. Moving EASME’s programmes and 
related databases to four different agencies (EISMEA, HaDEA, REA, CINEA), along with 
considerable staff turnover in 2021, added complexity to information gathering.  

This included data on programmes and calls, desk material and documents, as well as lists 
of potential interviewees from projects that had been managed by EASME between 2017 
and 2021. 

24   Redeployment could not exceed 10% of staffing allocated to a given programme in a given year 
25   Steering committee reporting 2017 

10 

 
 
 
4.  EVALUATION FINDINGS (ANALYTICAL PART) 

4.1  To what extent was the intervention successful and why?  

4.1.1 Effectiveness 

The data evaluated  under the ‘effectiveness’ heading looked at whether the agency had 
implemented the delegated programmes within the objectives of the framework laid down 
by the Commission. A further question is to what extent this was done in the most effective 
manner and whether stakeholders are satisfied. A key question relates to the ability of the 
agency to provide feedback to policy to its parent DGs, and an assessment was also carried 
out on the ability of the agency to react and adapt to the particular challenge of the COVID 
19 pandemic.  

In 
the 
interviews,  unsuccessful  applicants,  beneficiaries  and  experts  all  found 
administrative  burden  reasonable,  often  indicating  it  is  easier  than  national  funding 
programmes and that this has improved over time. On the other hand, some respondents 
mentioned that fewer reporting requirements would be welcome, especially for SMEs.  

From the  evaluation of  the EIC Pilot,  it can be  concluded that beneficiary SMEs under 
Innosup  consider  the  administrative  burden  of  the  pilot  projects  to  be  acceptable.  They 
particularly appreciate the structure of lump sum payments in certain Innosup projects that 
reduce  the  administrative  burden  and  allow  for  resources  to  be  focused  on  project 
execution. However, start-ups and smaller companies did indicate that the process was too 
complex and cumbersome for them26.  

Within the agency, the diversity of calls combined with the high number of calls puts a 
strain on EASME staff. From applicants’ perspective, EU programmes are becoming more 
complex and companies face difficulties in navigating and using programmes. 

The  input  by  EASME  staff  during  the  launch  procedures  for  projects  was  highly 
appreciated by beneficiaries. Beyond the intended structuring and launching of the project, 
these  initiatives  provided  a  platform  to  these  companies  for  networking  and  business 
development.  

The application procedures for project calls have the potential to be further simplified as 
for many of the calls applicants find that preparing a proposal is cumbersome and resource 
intensive.  

Both  unsuccessful  applicants  and  beneficiaries  have  suggested  that  digital  portals  be 
simplified and standardised instruction materials offered on navigating such platforms. The 
digital platforms used for grant preparation are perceived by unsuccessful applicants and 
beneficiaries to be cumbersome and difficult to navigate27. 

26   European Commission, Evaluation study on the European Innovation Council (EIC) Pilot, April 2022, ISBN 978-92-76-53253-8, 

p. 47, p. 134 

27   Supporting study - Interviews with unsuccessful applicants and beneficiaries  

11 

 
 
 
Unsuccessful  applicants  described  the level  of  explanation offered by  EASME  for why 
their  call  was  rejected  as  very  specific,  detailed  and  helpful  for  designing  future 
applications28. 

Companies who have applied for grants and have had prior experience with other funding 
opportunities  have  indicated  the  need  to  hire  external  consultants  to  help  them  submit 
proposals.  In  fact,  half  of  the  applicants  spent  between  1%  and  10%  of  their  proposal 
budget on external consultancy work29. This constitutes a barrier for companies to apply 
for EU funding.  

Concerning  the  application  process  and  the  information  provided  to  applicants,  most 
beneficiaries  and  unsuccessful  applicants  who  responded  to  the  survey  (strongly)  agree 
that requirements for the application process are reasonable and proportionate, and that the 
application  process  was  clear  and  transparent.  However,  concerning  the  evaluation 
process, responses differ greatly between beneficiaries and unsuccessful applicants.  

Most of the beneficiaries found that they received useful feedback on their application and 
rated  the  evaluation  process  as  clear  and  transparent.  Nonetheless,  several  interviewed 
beneficiaries indicate that the reviewers did not always understand the true value of their 
proposal,  and  that  the  evaluation  process  generally  depends  too  much  on  an  individual 
reviewer. This view was given credence by the fact that some beneficiaries are very happy 
about the feedback they received, while others are not30.  

On  the  other  hand,  unsuccessful  applicants,  on  balance,  have  negative  views  of  the 
evaluation process: 48% of respondents found that they did not receive useful feedback; 
43% found they had not received a clear explanation for rejection; and 44% found that the 
outcome of their application was not fair. However, 44% of respondents considered that 
the evaluation process was clear and transparent. 

A good indicator for the success of the evaluation process, is the number of re-evaluations 
sought out by applicants. The number of re-evaluation requests varied between 32 and 157 
per year between 2017 and 2020, which is between 0.0% and 0.7% of the total number of 
proposals submitted – well below the target of 3%.  

The number of re-evaluation requests that lead to re-evaluation ranged between 2 and 3 
per year in 2017-2020, indicating the robustness of the evaluation process.  

Experts contracted to evaluate proposals consider that the selection and registration process 
for them was clear, and that the contract procedures were completed in a timely manner, 
with necessary documents and information available.  

The parent DGs were satisfied with the level of information provided by the experts and 
their capacity to contract experts with the required expertise31.  

28   European Commission, Supporting study on the evaluation of EASME, PPMi, IDEA Consult (2023) Interviews with unsuccessful 

applicants and beneficiaries 

29   Supporting study - results of the surveys of beneficiaries and unsuccessful applicants 
30  
31   European Commission, Supporting study on the evaluation of EASME, PPMi, IDEA Consult (2023) Interview with parent DGs 

Interviews with beneficiaries  

12 

 
 
 
Feedback  to  policy  is  an  essential  part  of  the  agency  tasks.  The  previous  evaluation 
(2014-2016) identified a need for increased capacity both in terms of staffing and mandate, 
to deliver the expected feedback to the parent DG and improve its reporting32.  

For  the  Directorate-General  for  Energy,  the  feedback  to  policy  was  instrumental  in 
enriching the policy design relating to energy efficiency. Innosup projects (the Helpdesk 
project  in  particular)  have  proven  very  effective  in  providing  relevant  feedback  that 
EASME could deliver to the relevant parent DGs33.  

For the LIFE programme, feedback to policy is especially important, as its main rationale 
is  to  develop  and  improve  environmental  and  climate  policy.  The  memorandum  of 
understanding  with  Directorate-General  for  Environment  and  Directorate-General  for 
Climate Action specifies who oversees which policy integration activities (for example the 
DGs are responsible for analysing policy needs in the call for proposals, while EASME is 
responsible  for  information  and  data  on  key  strategic  indicators  relevant  to  the  DGs’ 
priority objectives).  

EASME performs its feedback to policy to parent DGs both in an informal and a structured 
manner.  In  June  201934,  EASME  launched  a  working  group  on  feedback  to  policy  in 
response to the discussions between the Commission and the executive  agencies. It was 
concluded that, given the large divergence between programmes, there is no one-size-fits-
all in providing feedback.  

Individual feedback to policy was perceived to be the most common practice applied by 
EASME. More structured feedback could be considered for more complex programmes 
and  is  preferred  by  certain  DGs  (Directorate-General  for  Communications  Networks, 
Content and Technology, Directorate-General for Research and Innovation).  

For  instance,  under  societal  challenge  ‘Raw  materials‘,  EASME  staff  started  writing 
biannual  reports  for  their  parent  DG  in  2016,  with  factsheets  for  each  project.  These 
biannual reports account for 50% of the feedback to policy, while the other 50% consists 
of regular interaction at meetings, one-off requests and ongoing collaboration35.  

Some areas for improvement can also be raised concerning feedback to policy in EASME. 
According  to  interviewees  from  the  EASME  staff,  the  Commission  could  more  clearly 
define its feedback expectations towards the agency. DGs should also regularly exchange 
information on future policy development, to be able to provide fit-for-purpose feedback 
to policy36.  

The agency proposed to improve feedback to policy by inviting the DGs to nominate one 
representative  each,  for  the  purpose  of  establishing  a  continuous  dialogue  between 
themselves  and  the  agency  on  improvements  in  feedback  mechanisms37  –  although 

32   European Commission, Evaluation of the Executive Agency for Small and Medium-sized Enterprises (EASME), June 2019, ISBN 

978-92-76-09259-9, pp. 11-12. 

33   European Commission, Evaluation study on the European Innovation Council (EIC) Pilot, April 2022, ISBN 978-92-76-53253-8, 

p. 71 

Interviews with EASME staff 

34   EASME, 26th meeting of the EASME steering committee, 26 June 2020, Agenda item 3.3: feedback to policy 
35  
36   EASME, 26th meeting of the EASME steering committee, 26 June 2020, Agenda item 3.3: feedback to policy 
37   EASME steering committee Secretariat, Final minutes of the EASME steering committee 21st July 2019 

13 

 
 
 
concerns  about  resource  allocation  for  this  feedback  process  were  also  considered, 
especially in the context of the draft cost-benefit analysis38.  

While  executive  agencies  do  have  a  mandate  to  provide  feedback  to  policy,  the  key 
performance  indicators  used  do  not  reflect  performance  in  the  feedback  given  to 
policymaking DGs.  

For communication, the agency indicated three key objectives, addressing both  internal 
and external communication:  

1)  promoting the programmes, resulting project successes and EU added value, 

2)  increasing staff motivation and engagement, to bring out full potential at work,  

3)  maintaining a positive image of EASME among key stakeholders within the EU 

institutions. 

According  to  the  survey  supporting  this  evaluation,  e-mails  were  the  communication 
channel  that  have  provided  beneficiaries  with  the  most  relevant  information.  Video 
conferences,  telephone  contact,  face-to-face  exchanges  as  well  as  EASMES’s  website 
were also considered useful by beneficiaries in their search for information.  

By contrast, EASME’s social media was considered useful by only 32% of beneficiaries. 

Interviewed  stakeholders  (beneficiaries,  unsuccessful  applicants,  experts  and  national 
coordination offices, NCOs) are generally very positive about their relations with project 
and financial officers at EASME. Multiple interviewees specifically mentioned the value 
of project officers in answering questions in a timely manner.  

The  2020  qualitative  analysis  of  the  costs  and  benefits  of  delegating  certain  tasks  to 
executive agencies39 found that project advisers/officers lacked the time to maintain close 
relationships with clients, due to their high workload. However, the survey conducted for 
this  study  paints  a  more  positive  picture,  indicating  that  most  beneficiaries  and 
unsuccessful applicants are satisfied with EASME’s ability to respond to questions within 
a reasonable time during the application and selection process (62%), while only 15% of 
respondents were not satisfied.  

Moreover,  the  great  majority  of  respondents  indicated  that  the  selection  results  were 
announced within the communicated deadlines in the call for proposals.  

On the other hand, 25% and 26%, respectively, of respondents did not know who to contact 
when submitting or preparing their application.  

Concerning  the  implementation  phase  of  their  project  or  grant,  satisfaction  rates  of 
beneficiaries with EASME’s communication efforts are even higher. Most beneficiaries 
indicate  that  they  are  (very)  satisfied  with  EASME’s  ability  to  respond  to  questions  in 

38   EASME steering committee Secretariat, Final minutes of the EASME steering committee 21st February 2020 
39   Lapatinas, A., Katay, G. Vollbracht, I., Funck, A., Grigoleto, M. (2020), CBA for the delegation of certain tasks regarding the 

implementation of EU programmes 2021-2027 to the executive agencies: Qualitative Analysis 

14 

 
 
 
reasonable  time  (88%),  as  well  as  the  availability  and  responsiveness  of  EASME  staff 
(89%).  

Moreover,  the  answers  and  guidance  provided  by  EASME  was  deemed  satisfactory  by 
most beneficiaries. Finally, only 11% of beneficiaries did not know how or who to contact 
during the implementation phase, which is much lower than during the application phase. 

Regarding the guidance provided by EASME to experts, the survey indicates that different 
levels  of  guidance  are  given  to  different  experts:  a  large  portion  of  experts  (41%)  only 
receive initial guidance before their first assessment. However, a similarly large portion of 
experts (36%) indicate they receive continuous support, or they receive guidance for every 
assessment  and  task.  Another  18%  indicates  that  they  only  receive  sporadic  guidance. 
Nonetheless, most experts are satisfied with the provision of information by EASME.  

To conclude, EASME applies guiding documents for staff on how to interact with clients, 
ensuring  proximity  to  applicants,  beneficiaries  and  experts.  Most  beneficiaries  and 
unsuccessful applicants are satisfied with the relationships they have with EASME staff 
and  say  the  response  time  is  satisfactory.  The  same  applies  to  the  level  and  quality  of 
information received. Nonetheless, there is still room for improvement on clarifying who 
applicants can contact during their application process.  

During the evaluation period, EASME organised many events in close collaboration with 
its  parent  DGs,  including  information  days,  kick-off  meetings,  networking  workshops, 
R&I Days, the Raw Materials Week and the EU Sustainable Energy Week (EUSEW).  

EASME  also  took  part  in  organising  international  events  promoting  the  results  of 
EU-funded projects. Moreover, the agency also contributed directly to other stakeholder 
events  promoting  funding  opportunities  and  guiding  potential  applicants  in  relevant 
procedures, using social media as one of their main channels for announcing the launch of 
calls, news about EU programmes and the timing of events such as information days. The 
EASME website was also used actively to promote its projects40. 

Raw Materials week was organised annually with panels, presentations, booths, etc., with 
the content being prepared in close collaboration with DG GROW and results of the event 
as  well  as  the  achievements  and  results  of  projects  themselves  being  communicated 
through a biannual report to DG GROW41. 

Communication and – via proxy – knowledge valorisation from the Commission could be 
further improved through the work of the executive agencies. One example put forward by 
Directorate-General  for  Climate  Action  is  that  of  green  hydrogen  as  an  opportunity  to 
further  enforce  communication  channels  between  parent  DGs,  EASME  and  external 
stakeholders in the EASME network42. 

The channels used by EASME to communicate on EU funding opportunities and the role 
of the EU were reasonably successful, as reflected by how beneficiaries and unsuccessful 
applicants receive the communications message and materials prepared by the agency.  

40   EASME, Annual Activity Reports 2017, 2018, 2019, 2020 
41   EASME, Annual Activity Report 2019 
42  

Interview with parent DG 

15 

 
 
 
31%  of  potential  applicants  learn  about  funding  opportunities  through  the  websites  of 
EASME or the Commission, while 23% learn about it from professional colleagues in their 
own or other organisations, and 15% from other national sources.  

The COVID-19 pandemic had a significant impact on the agency’s work, as was the case 
for  all  departments  and  services  of  the  European  Commission  dealing  with  the 
implementation of projects and programmes. There was a need to launch calls and actions 
which would respond to the specific needs of the pandemic, and this also increased the 
workload in the financial units.  

In terms of programme implementation, minor delays occurred in implementing delegated 
projects, due to the outbreak of the pandemic and the resulting economic situation in the 
Member States43. 

The  Enterprise  Europe  Network  was  quick  to  react  to  the  pandemic  challenges  to  help 
SMEs in the most flexible way. Special services were put in place to help companies source 
urgently-needed medical or protective equipment, to adapt production lines or to find new 
buyers or suppliers.  

Furthermore, a lot of face-to-face brokerage events and capacity-building activities were 
moved online to make sure companies could still benefit from Enterprise Europe Network 
services despite the lockdown.  

EASME  made  IT  tools  available,  helping  the  network  collaborate  more  efficiently. 
Training  events  were  moved  online  and  contacts  between  staff  and  beneficiaries  were 
intensified, to give them as much help as possible to implement their projects. Within the 
network, SME clients appreciated how the network stayed by their side during this crisis, 
with 85% being either satisfied or very satisfied44. 

Together  with  the  Task  Force  of  the  European  Innovation  Council,  the  agency’s  unit 
dealing  with  the  EIC-accelerator  has  organised  a  call  for  proposals  to  find  innovative 
solutions helping contain this crisis, as well as (co-) hosting a pan-European hackathon to 
connect  civil  society,  innovators,  partners  and  investors  across  Europe  in  developing 
innovative solutions for coronavirus-related challenges (20 900 participants)45.  

EASME  collaborated  with  the  Directorate  General  for  Research  and  Innovation  for  an 
extensive COVID-19 portfolio analysis. The agency contacted beneficiaries with potential 
contributions through the projects either for combating the crisis or for dealing with the 
aftermath46. 

The EIC Pilot is deemed by users to be unfit to respond to short term needs as was the case 
during the early stages of the COVID-19 pandemic. In 2020 the EIC Pilot Accelerator put 
in  place  several  calls  that  have  received  unprecedented  amounts  of  applications.  The 
quality of submitted projects, however, was lower than previous rounds47.  

43   EASME 2021 HIGH RISKS REGISTER (November 2020) 
44   EASME steering committee Secretariat, Final minutes of the EASME steering committee 26th June 2020 
45   EASME steering committee Secretariat, Final minutes of the EASME steering committee 26th June 2020 
46   EASME steering committee Secretariat, Final minutes of the EASME steering committee 26th June 2020 
47   European  Commission,  Study  on  the  effectiveness  of  public  innovation  support  for  SMEs  in  Europe,  Annex  E  INNOSUP 

evaluations, March 2021, ISBN 978-92-9460-580-1 

16 

 
 
 
The increased number of applications in 2020 following additional COVID-related calls 
resulted in delays, high workload for staff and project complications. Overall, beneficiaries 
were  satisfied  but  they  do  comment  on  frustration  with  staff  turnover  at  EASME. 
Nonetheless, beneficiaries in the EIC Pilot appreciate the flexibility and agility of EASME. 

The flexibility  and continued professionalism  proven by the  agency is  reflected in  how 
beneficiaries,  unsuccessful  applicants  and  experts  perceived  EASMEs  response  to  the 
COVID pandemic, as shown by the following figures:  

-  92%48 of (574) beneficiaries appreciate the flexibility shown by EASME and the 
Commission  in  adapting  deadlines  and  agreeing  to  other  changes  necessary  to 
preserve projects despite COVID-19 restrictions.  

-  82% of (825) beneficiaries and unsuccessful applicants and 91% of (646) experts 

consider that the agency smoothly transitioned to virtual meetings. 

-  80% of (834) beneficiaries and applicants and 90% of (733) experts found that the 

agency maintained the quality of its services throughout the pandemic. 

-  80% of (817) beneficiaries and applicants and 90% of (724) experts found that the 

agency effectively disseminated relevant information on calls and projects.  

The agency followed the instructions of the Directorate-General for human resources and 
security and the Secretariat General regarding staff management in the pandemic period, 
and  an  interagency  task  force  was  established  in  the  second  half  of  2020  to  tackle  the 
consequences  of  the  crisis.  The  task  force  was  led  by  the  Head  of  Department  for 
Administration  and  Finance  and  included  colleagues  from  the  Business  Continuity 
Management team and logistics. It was intended to coordinate the presence of staff in the 
building,  the reporting of COVID-19 cases  as  well as  teleworking  conditions  and other 
measures.  

The agency conducted a risk assessment exercise (involving the whole agency) to monitor 
the impact of COVID-19. New staff and especially single people were struggling to adapt 
to the new work arrangements early on during the pandemic. Burnout rates among EASME 
staff have been increasing since COVID-1949. 

According to the large majority of interviews from both beneficiaries and staff in parent 
DGs, EASME managed to adjust its operations with a very high level of flexibility and 
competence, while maintaining business continuity50. 

4.1.2  Efficiency 

Efficiency covers the management and execution of programmes managed by the agency, 
and  how  sound  the  financial  and  human  resource  management  has  been  during  the 
evaluation  period.  These  aspects  are  covered  by  key  indicators  taken  from  agency 
performance but also with additional survey data.  

48   Strongly agree or agree 
49   Supporting study - interviews with EASME staff 
50   Supporting study - interviews with EASME staff and beneficiaries  

17 

 
 
 
The timely execution of programme management functions is evaluated by looking at the 
performance on key indicators such as time-to-inform, time-to-grant, time-to-pay and the 
share  of  payments  that  is  executed  within  the  legal  deadline.  Survey  responses  from 
beneficiaries and experts included in this section reflect their satisfaction level with regards 
to the timely execution of payments.  

The average time-to-inform was not later than the target for most years and programmes. 
The  two  exceptions  were  in  2020  -  for  the  EIC  Pilot  Phase  II  and  for  Fast  Track  to 
Innovation.  

The average time-to-grant – which combines the time-to-inform with the time-to-sign – 
was  also  not  later  than  the  target  for  most  of  the  programmes  and  years.  However,  the 
target was systematically missed for Horizon EIC PILOT Phase I, due to the high number 
of proposals each year.  

The share of payments  executed within the legal  deadline was above 98% for all years 
covered  by  this  evaluation.  The  share  of  timely  payments  increased  between  2017  and 
2019, but then fell slightly in 2020 and in the first quarter of 2021.  

In the table below the share of payments made within the legal deadlines is represented per 
programme. Most payments are executed in time, ranging between 97.4% and 100% per 
programme.  

Share of payments within the legal deadlines, 2017-2020 

COSME 
Horizon INNOSUP 
Horizon EIC PILOT & Fast Track to 
Innovation 
EMFF 
Horizon Energy 
Legacy IEE 
Horizon ENV & RESOURCES 
LIFE 
Legacy CIP ECO-innovation 

2017 
99.9% 
100.0% 

2018 
100.0% 
100.0% 

2019 
100.0% 
100.0% 

2020 
100.0% 
100.0% 

97.4% 

99.5% 

99.8% 

99.7% 

100.0% 
99.0% 

99.4% 

97.4% 

100.0% 
98.5% 
100.0% 
100.0% 
97.8% 
100.0% 

100.0% 
100.0% 

100.0% 
100.0% 

100.0% 
98.6% 
100.0% 

100.0% 
98.3% 

Source: Study supporting the triennial evaluation of the Agency, taken from annual activity reports and 
annexes, 2017, 2018, 2019, 2020 

The average time-to-pay has remained relatively stable for each legal deadline over 2017-
2019 and always remained within the legal deadline51.  

94% of interviewed beneficiaries are satisfied with the payments and their timeliness52. 
The  prefinancing  and  mid-term  payments  are  highly  appreciated  by  beneficiaries. 
Nonetheless, several beneficiaries mentioned that the retention of funding until the end of 
the project is an issue for cascade funding directed towards SMEs, and they suggest that 
the retention of funding should not apply for third party funding.  

51   For 2020, no average time-to-pay was presented in the annual activity report. 
52   N=549 (516 out of 549) 

18 

 
 
 
 
 
 
 
 
Furthermore,  one  interviewee  (beneficiary)  mentioned  there  is  too  little  flexibility  in 
payments where one partner within the project consortium is not delivering, as this leads 
to no one getting paid.  

Experts  find  that  the  payment  terms  are  satisfactory.  On  the  other  hand,  only  63%53  of 
surveyed experts agree that their payment matched the efforts they had spent on the task. 
From  the  interview  programme  it  appears  that  most  of  the  experts  found  their  fee 
reasonable,  though  one  expert  did  indicate  that  the  fees  received  by  experts  are  low 
compared to industry rates, but that it remains worthwhile as it helps them gain market 
intelligence.  

The IAS audit on the management of experts in Horizon 2020 grants also recommended 
that “EASME should ensure that the paid amount corresponds to the days in the attendance 
session” and that a “clear justification for the rejections should be recorded in the payment 
summary sheet”. In the corresponding action plan54, EASME ensured that this would be 
implemented by March 2021.  

To  conclude,  EASME  was  operating  efficiently  regarding  timely  execution  of 
programme’s  management  functions.  Key  indicators  for  timely  execution  are  the 
time-to-inform  and  time-to-grant,  which  generally  remained  below  the  set  targets. 
Furthermore, between 98% and 99.8% of payments  were executed on time on a yearly 
basis between 2017 and 2020, close to the target of 100%.  

Beneficiaries and experts are also satisfied with the timeliness of payments (94% (of 549) 
beneficiaries  agreed  that  the  payment  process  was  smooth,  while  92%  (of  995)  experts 
agreed the process had gone smoothly.  

The management and control arrangements cost is the ratio between administrative and 
implemented operational budget, and the evaluation looks at whether these arrangements 
were cost effective. The ratio was at an average of 3.0% for commitments (ranging between 
2.6%  and  3.4%),  and  3.3%  for  executed  payments  between  2017-2020.  The  ratio  for 
commitments and payments remained well within the target of 5.5%55. 

In  comparison  to  other  Commission  executive  agencies,  EASME’s  ratio  between 
administrative  and  operational  budget  for  commitments  and  payments  was  close  to  the 
average of 3.4% in 202056. 

Retrospective checks are aimed at detecting and correcting instances of error or fraud that 
were  not  identified  in  the  advance  (ex  ante)  checks.  The  main  result  finding  the 
retrospective  checks  was  the  residual  error  rate,  i.e.  the  level  of  errors  in  financial 
transactions  which  remain  undetected  and  uncorrected  until  the  end  of  the  programme 
lifecycle.  

The residual error rate remained within target for most programmes. The exception was 
the legacy programmes CIP-IEE II and CIP-Eco-Innovation, where the residual error rate 
was systematically exceeded, and COSME, where the residual error rate was exceeded in 

53   N=1011 Strongly agree or agree (637 out of 1011) 
54   EASME (2021 February 26), Action plan EASME – IAS audit on the management of experts in Horizon grants  
55   EASME, Annual Activity Report 2017 
56   See Annex 6, Study on the evaluation of EASME PPMI, IDEA Consult (2023) 

19 

 
 
 
2018 and 2020. In every case where the targets were exceeded, mitigating measures were 
taken,  such  as  better  informing  beneficiaries  and  improving  advance  checking 
mechanisms. 

In  terms  of  budget,  the  execution  of  the  operational  budget  was  at  100%  for  the  years 
2017-2020, both for commitment and payment appropriations57. It is noteworthy, however, 
that  EASME’s  2020  risk  register  points  out  a  risk  of  double  funding  of  the  projects 
managed by EASME because of a lack of tools for project officers to perform effective 
double funding checks on project applications.  

For the administrative budget58, between 98% and 100% of commitments and payments 
were executed, which is close to the target of 100%. With regards to payments, between 
85% and 92% were executed. The forecast administrative budget, which covers the running 
costs of the agency (i.e. staff expenditure, office costs, IT and other services) should not 
be  fully  spent  when  it  is  not  deemed  necessary,  and  the  agency  carried  out  budget 
amendments several times a year to adjust the forecast. 

Process efficiency is the extent to which grants and tenders were managed efficiently. An 
indicator in this regard is the number of proposals and ongoing projects ‘per operational 
head’ (in full-time equivalents, or FTEs). Compared to other agencies, EASME had the 
largest number of proposals per FTE59, even given the significant decrease in 2020.  

On  the  other  hand,  the  number  of  running  projects  ‘per  operational  head’  is  average 
compared to other agencies. The actual operational budget ‘per operational head’ (in FTEs, 
and  in  terms  of  commitments)  within  EASME  was  also  average  compared  to  other 
agencies, amounting to EUR 3.91 million in 2020.  

This  indicates  that  EASME  staff  were  relatively  efficient  compared  to  other  agencies, 
given that they manage many more proposals, while being able to manage a similar number 
of projects and operational budget.  

The  high  level  of  proposals  but  similar  number  of  running  projects  compared  to  other 
agencies indicates relatively low success rates, i.e. the number of selected (funded) projects 
vs eligible proposals, for the programmes managed by EASME.  

The total success rate for EASME has decreased over the evaluation period, from 8% in 
2017 to 5% in 2020. The low success rate is mainly due to the competitive nature of the 
EIC pilot’s SME instrument: the success rate of the EIC Pilot is comparable to those of 
private investment funds and acceleration programmes60.  

For  other  programmes,  such  as  COSME  or  Horizon  INNOSUPP,  the  success  rates  are 
considerably higher, though they vary largely between years. This variation is mainly due 
to  changes  in  the  number  of  funded  proposals,  rather  than  the  number  of  submitted 
proposals.  

57   Budget and financial monthly indicators, January to December 2017, 2018, 2019, 2020 
58   Administrative budget refers to the budget allocated for the day-to-day running of the agency 
59   See Annex 6, Supporting study on the evaluation of EASME, PPMI, IDEA Consult (2024) 
60   EASME, Annual Activity Report 2018 

20 

 
 
 
To  conclude,  EASME  has  achieved  its  targets  of  operational  budget  execution,  as  it 
executed  100%  of  its  operational  budget.  Regarding  the  administrative  budget,  budget 
execution in 2020 was at 98% for commitments and payments, and at 92% for payments.  

Furthermore: 

•  EASME has a very high number of proposals ‘per operational head’ compared to 

other agencies (up to 61 proposals per FTE in 2019);  

•  The number of running projects ‘per operational head’ (10 in 2020) was more in 

line with other agencies.  

•  Success rates, which can be regarded as one of the indicators for cost-effectiveness, 
were rather low and decreased further from 8.0% in 2017 to 5.0% in 2020 – though 
this can be attributed to the low success rate of EIC Pilot proposals. 

Simplification is a core area for efficiency gains. Two main aspects were evaluated: 

• 

IT tools;  

•  procedures.  

IT tools have enabled significant time savings and staff rationalisation. This is particularly 
the case for call processes. For instance, for the management of SME Instrument phase 1 
grants, IT tools (standardised communication, automatic reminders, etc.) helped reduce the 
size of the staff required to manage the grants from 15 to 5 people61.  

For Horizon, the use of common IT tools and a support centre has brought significant gains 
in terms of efficiency and a consistent approach toward beneficiaries62. The results of the 
survey conducted among beneficiaries and unsuccessful applicants support this statement. 
Respondents agree that a wider use of common IT-tools across different programmes and 
programmes  strands  has  made  the  process  of  project  application/implementation  much 
more efficient63. 

EASME  has  also  invested  proactively  in  developing  its  digital  tools  during  the  period 
covered by the evaluation: 

• 

in March 2017, an internally developed IT system called ePAD was rolled out for 
the planning and monitoring of EMFF actions64;  

•  EASME is the first agency to use e-submission for procurement procedures65. This 

resulted in a more efficient submission and evaluation process66.  

Interviews with unsuccessful applicants, beneficiaries and EASME staff 

61  
62   EASME (2019), SB simplification measures 2017 – 2019 
63   Results of the surveys of beneficiaries and unsuccessful applicants 
64   EASME steering committee Secretariat, 14th meeting of the EASME steering committee 30th June 2017 
65   EASME steering committee Secretariat, Final minutes of the EASME steering committee 14th December 2017 
66   EASME steering committee Secretariat, Final minutes of the EASME steering committee 21st February 2019 

21 

 
 
 
•  subsequently the agency extended the eGrants suite to EMFF grant projects67. 

•  EASME also made IT tools available to the Enterprise Europe Network to support 
its response to COVID-related calls that required swift implementation (see 4.1.1). 

To achieve efficiency gains, EASME introduced e-tendering and e-submissions in 2017 
for  several  key  procurement  procedures.  In  2018,  the  agency  then  phased  in  electronic 
invoicing, including its suppliers68 as well as the eGrants system. This helped it manage a 
high volume of proposals and grant agreements and simplified administrative procedures 
for applicants.  

In  2017,  IT  tools  for  evaluating  grant  applications  were  not  working  as  expected  (SEP 
Submission  and  SEP  Evaluation),  especially  for  remote  work  using  the  PEAS  tool 
(keyword matching between expert and proposal). This was due to teething problems with 
the tools and the situation has improved since then with regular training for EASME staff 
and refinements to the tools themselves69.  

Another  issue  identified  in  2017  was  that  EASME  had  no  mandate  for  the  internal 
governance structure for LIFE IT systems, as they are owned by the Directorate-General 
for  the  Environment.  In  general,  the  agency  has  neither  a  sufficiently  formal  role  nor 
resources to influence the governance of corporate IT systems. 

Nevertheless, reliable and efficient IT tools necessitate continued investment. While the 
introduction of eGrants has resulted in more efficient procedures, there are still problems 
with specific programmes such as COSME, due to the unique nature of small and recurrent 
contracts with small beneficiaries.  

A  qualitative  analysis  of  the  costs  and  benefits  of  delegating  certain  tasks  to  executive 
agencies70 found that constant improvements to IT tools are needed to simplify procedures 
and ensure both grant agreement preparation and project management are effective.  

Interviews  with  unsuccessful  applicants,  beneficiaries  and  EASME  staff  reveal  that  the 
way users are involved in developing IT tools can be further improved. The intended users 
were not sufficiently involved in the design, development and updating of IT tools, leading 
to  applicants  being  dissatisfied71.  They  specifically  mention  that  user  friendliness  is 
suboptimal, and in some instances, users have to rely on help from external consultants to 
navigate the IT tools designed for submitting proposals and reporting within their projects.  

In EASME’s 2020 risk register72, the transition from LIFE IT tools to Horizon 2020 tools 
was recorded as challenging. It was not clear how the dedicated support for LIFE IT would 
transition to the Horizon 2020 tools, and continuity and comparability between the tools 

67   EASME steering committee Secretariat, Final minutes of the EASME steering committee 14th December 2017 
68   Annual 

financial  year  2017 

report  on  EU 

agencies 

(OJ  C,  C/434,  30.11.2018,  p.  213, 

the 

for 

https://www.europol.europa.eu/sites/default/files/documents/eca-
annual_report_on_the_eu_agencies_for_the_financial_year_2017.pdf) 

69   Risk register 2017 EASME, Ref. Ares(2018)453546 - 25/01/2018 
70   Lapatinas, A., Katay, G. Vollbracht, I., Funck, A., Grigoleto, M. (2020), CBA for the delegation of certain tasks regarding the 

implementation of 2021-2027 EU programmes to the executive agencies: Qualitative Analysis 

71   Results of the surveys of beneficiaries and unsuccessful applicants 
72   Risk register 2020, EASME 

22 

 
 
 
was also flagged as an issue. These key aspects led to inefficient project management in 
the short term. 

According  to  the  internal  Scoreboard  for  Simplification  measures,  such  measures  were 
introduced  at  different  levels  and  for  different  implementation  mechanisms 
in 
2017-202073.  A  taskforce  with  representatives  from  different  units  issued  concrete 
proposals,  adopted  by  management  as  from  1  January  2020.  The  main  changes  made 
resulted in shorter approval flows and faster booking of  business travel, the removal of 
redundant documents and tasks, and more cost-effective controls.  

The internal scoreboard also refers to other simplification measures such as: 

• 

removing the approval requirement for the monthly forecast of business travel,  

•  eliminating the need for a separate travel request form,  

• 

• 

the number of project monitoring visits using a risk-based approach, 

travel orders to be authorised by the Head of Unit instead of Head of Department 
(with a few exceptions to approved by the Head of Department and the Director), 

•  measures to reduce the environmental impact of business travel. 

Delegating tasks internally is a practice that has  greatly contributed to making EASME 
procedures  simpler  and  more  flexible  and  has  enabled  EASME  to  operate  in  a  more 
seamless way. Delegating by giving more power to lower levels led to time savings and 
streamlined processes.  

For instance, delegating authorisation to lower levels (where possible) reduced the time 
needed for the approval of commitments and payments. This is the case  with the EDES 
approval  flow  for  bankruptcy  cases  and  non-payment  of  taxes.  The  final  authorisation 
could be given by the Head of Department C instead of the Director. The need for approval 
from the legal department was also reduced in 201974.  

Another example of this practice is the possibility to delegate authorisation power to Head 
of Sector when the Heads of Unit are unavailable75. Successful delegation practices can 
also be found in HR processes, as evidenced by the simplification of HR approval flows 
for  selection  procedures.  Decision-making  power  to  launch  selection  procedures  to  fill 
existing  posts  by  means  of  internal  mobility  within  the  agency,  and  publication  of  the 
related internal vacancy notice76, was delegated from Director to Head of Unit C2.  

Nevertheless,  delegating  also  has  its  limits.  Delegating  small  budgets  and  short-term 
measures was not considered efficient in all instances, as it duplicates operational, financial 
and coordination costs – as demonstrated by a qualitative analysis of costs and benefits of 
delegating certain tasks to executive agencies77.  

73   EASME (2019), SB simplification measures 2017–2019. 
74   EASME (2019), SB simplification measures 2017 – 2019. 
75   EASME (2019), SB simplification measures 2017 – 2019. 
76   EASME (2019), SB simplification measures 2017 – 2019. 
77   Lapatinas, A., Katay, G. Vollbracht, I., Funck, A., Grigoleto, M. (2020), CBA for the delegation of certain tasks regarding the 

implementation of 2021-2027 EU programmes to the executive agencies: Qualitative Analysis 

23 

 
 
 
For example, managing EMFF-Direct was a big challenge due to its small scale and diverse 
set of projects. The unit responsible also lacked stability in terms of priorities and rules, 
such as eligibility criteria, and standardised procedures and IT tools.  

In  the  previous  evaluation  (2014-2016),  grant  evaluation  and  planning  of  calls  were 
identified as the main areas for further potential simplification78. According to interviews 
with  EASME  officials,  as  part  of  the  qualitative  analysis  of  the  costs  and  benefits  of 
delegating  certain  tasks  to  executive  agencies,  EASME  has  simplified  organisational 
processes and procedures to the extent that very few additional efficiency gains could still 
be realised.  

One good practice identified was for example the introduction of eGrants for EMFF, which 
resulted in a faster and more efficient submission and evaluation process79. Nevertheless, 
according  to  interviewees  from  the  agency,  there  are  some  areas  where  improvements 
could still be made as regards receiving information on calls80 in due time. 

Reducing  paperwork  and  administrative  burden  is  also  important  for  simplifying 
procedures.  This  applies  both  to  EASME  and  to  applicants.  Beyond  the  delegation 
measures  mentioned  above,  EASME  has  put  in  place  a  series  of  measures  that  have 
reduced  paperwork.  For  instance,  a  paperless  policy  that  helped  further  reduce  paper 
signatories and documents in 201881 and several simplification measures related to HR and 
benefitting staff82.  

On the applicants’ side, EASME introduced two-stage calls for the Environment strand, 
which reduced paperwork for unsuccessful applicants and led to an increase in proposals 
for the LIFE-programme.  

For COSME, the application of lump sums for some parts of the programme (e.g. European 
Destination of Excellence Network) helped reduce the administrative burden. The use of 
simplified cost options (lump sums, unit costs and flat rates) instead of real-cost funding 
is also considered by beneficiaries and unsuccessful applicants to be an effective solution, 
making project application/implementation easier83. 

Under the SME Instrument, the management of phase 1 grants (of which there were over 
1 000  per  year)  required  15  people.  This  number  was  reduced  to  5  through  simpler 
reporting,  IT  tools,  standardised  communication,  automatic  reminders,  etc.  The 
beneficiaries benefited from internal simplification at the agency in terms of reduced time 
to pay and time to inform84. 

Simplified reporting also made the operation of the agency more efficient. In cooperation 
with  the  Directorate-General  for  Maritime  Affairs  and  Fisheries,  EASME  prepared  a 
Commission Decision that allows the use of simplified unit costs for salary costs of SME 

78   European Commission, Evaluation of the Executive Agency for Small and Medium-sized Enterprises (EASME), June 2019, ISBN 

978-92-76-09259-9, pp. 38-39. 

79   Lapatinas, A., Katay, G. Vollbracht, I., Funck, A., Grigoleto, M. (2020), CBA for the delegation of certain tasks regarding the 

implementation of 2021-2027 EU programmes to the executive agencies: Qualitative Analysis 
Interviews with EASME staff 

80  
81   EASME (2019), SB simplification measures 2017 – 2019. 
82   EASME Annual Activity Report 2017 - simplification of the catering order process; simplification of the approval flow for interim 

and external training requests; paperless policy; and the reduction of paper signatories and documents 

83   Results of the surveys of beneficiaries and unsuccessful applicants 
84  

Interviews with EASME staff 

24 

 
 
 
owners. This simplifies reporting by beneficiaries and verifications by EASME staff85. The 
simplification  of  reporting  templates  for  a  structured  collection  of  results  was  also 
considered to have made project implementation and reporting easier86. 

Regarding  monitoring  and  reporting  processes,  beneficiaries  report  a  high  degree  of 
satisfaction in project implementation. Specifically:  

-  88% (506 out of 575) considered that project reporting requirements were clear and 
consistent  throughout  project  implementation,  and  84%  (484  out  of  576) 
considered these requirements to be in line with the scope of the project.  

-  Monitoring activities carried out by EASME staff were perceived to be useful for 
projects  execution  (86%  or  495  out  of  576)  and  the  processes  were  clear  to  the 
affected stakeholders (88% or 508 out of 577).  

Retrospective  (ex  post)  audits  are  considered  to  be  arduous  and  time  consuming  but 
relevant.  The COSME  programme in  particular saw extensive contradictory procedures 
resulting from such audits (sometimes due to misunderstanding of the rules). Introducing 
rules that are better adapted to the specific nature of these projects may potentially have 
avoided this situation and improved efficiency.87 

The qualitative analysis of the costs and benefits of delegating certain tasks carried out in 
2020 to executive agencies88 indicates that insufficient resources were allocated to EASME 
because of a lack of stability in delegated actions (EMFF and COSME), and the additional 
tasks assigned to EASME in 2014-2020, which created additional work for EASME staff, 
without any increase in FTEs.  

It was concluded that any further increase in delegation should come with an increase in 
staffing  levels.  It  was  also  noted  that  specific  attention  should  be  given  to  the  high 
workload associated with feedback to policy when calculating resources – this was flagged 
as a critical task of an executive agency.  

Along  the  same  lines,  in  2017  the  IAS  audit  on  management  of  human  resources  in 
EASME89  recommended  that  EASME  should  further  improve  its  methodology  and 
guidance  for  workload  assessment,  re-run  the  workload  assessment  and  identify  gaps 
between  resources  needed  and  resources  available.  EASME  implemented  these 
recommendations in 201890 in line with the approved action plan91.  

Estimations of staffing requirements in the agency were carried out by EASME based on: 

1)  cost-benefit analysis,  

2)  multi-annual analysis of the workload for Horizon, COSME, LIFE, EMFF and the 

legacy programmes managed by EASME92,  

85   EASME steering committee Secretariat, Final minutes of the EASME steering committee 14th December 2017 
86   Results of the surveys of beneficiaries and unsuccessful applicants 
87  
88   Lapatinas, A., Katay, G. Vollbracht, I., Funck, A., Grigoleto, M. (2020), CBA for the delegation of certain tasks regarding the 

Interviews with EASME staff 

implementation of 2021-2027 EU programmes to the executive agencies: Qualitative Analysis 

89   European Commission, Final Audit Report on Management of Human Resources in EASME, June 2017 
90   EASME Annual Activity Report 2018 
91   European Commission, Action Plan: IAS audit on Management of Human Resources in EASME 
92   European Commission, Final Audit Report on Management of Human Resources in EASME, June 2017, p. 5 

25 

 
 
 
3)  a data collection method assessing the workload for the agency93,  

4)  assessment  of  the  evolution  of  business  needs,  based  on  workload  indicators, 

leading to a competency framework analysis. 

The  final  workload  assessment  also  took  into  consideration  the  establishment  plan  and 
flexibility mechanism agreed by the parent DGs, which was approved in 2017 to reallocate 
resources towards the LIFE programme, which had been structurally understaffed94.  

This issue follows from a significant increase in the number of projects to be managed, 
combined with a very limited increase in resources95. The actual workload related to the 
management of LIFE therefore deviated from the initial  cost-benefit  analysis estimates. 
This understaffing was already raised as an issue in the 2014-2016 evaluation of EASME 
and remained an issue during the current evaluation period. 

Figure 2. Filling rate of staffing plan (actual staff divided by planned number of staff in 
staffing plan) 

Source: Study supporting the triennial evaluation of the Agency, based on EASME annual activity reports 
and annexes 2017, 2018, 2019, 2020 

95% of posts were filled by the end of 2020, which is lower than the set target of 98% and 
represents a slight decrease compared to 2017 (97%) (Figure 2).  

Filling rates for contract agents, which represent the bulk of staff, are high. On the other 
hand, only 79% of the planned number of seconded officials in the staffing plan were in 
place in 2020. 

The  average  programme  expenditure  (in  commitments)  per  staff  member  was  between 
3.20 and 2.94 million EUR (Figure 3). A decrease was observed in 2020, resulting from a 
decrease in the operational budget and an increase in the number of staff. 

93   European Commission, Action Plan: IAS audit on Management of Human Resources in EASME, July 2017, p. 1 
94   EASME steering committee Secretariat, Final minutes of the EASME steering committee 17th February 2017 
95   European Commission, Evaluation of the Executive Agency for Small and Medium-sized Enterprises (EASME), June 2019, ISBN 

978-92-76-09259-9, p. 48 

26 

 
 
 
 
Figure 3. Average programme expenditure (in commitments) per staff member, in million 
EUR  

3,20

3,70

3,68

2,94

4,00

3,00

2,00

1,00

0,00

2017

2018

2019

2020

Source: Study supporting the triennial evaluation of the Agency based on EASME annual activity reports 
and annexes 2017, 2018, 2019, 2020 

The total number of vacancies at the end of the year increased steadily, from 13 in 2017 to 
28 in 2020 (Figure 4). Given the relatively low number of total positions for temporary 
agents and seconded officials, the open vacancies for these positions are more problematic 
(as also indicated by the filling rate in Figure 2).  

The delays in recruiting seconded officials were largely due to a lack of interest in joining 
EASME, as well as lengthy procedures96. The 2020 cost-benefit analysis97 also indicated 
that EASME is generally successful in attracting sufficient and qualified employees.  

However, officials (especially at Head of Sector level) are less attracted to positions in the 
executive  agencies.  EASME  is  aware  of  this  difficulty  and  marked  the  difficulties  in 
recruiting and replacing seconded officials as a risk in its 2017 risk register98. 

Figure 4. Number of vacancies by the end of each year, 2017-2020 

Source: EASME annual activity reports 2017, 2018, 2019, 2020 

The average time to recruit (measured from the moment a post becomes vacant until the 
new recruit is in place) increased yearly from 2018 (2.6 months) to 2020 (6.6 months).  

96   EASME annual activity report 2017 
97   Lapatinas, A., Katay, G. Vollbracht, I., Funck, A., Grigoleto, M. (2020), CBA for the delegation of certain tasks regarding the 

implementation of 2021-2027 EU programmes to the executive agencies: Qualitative Analysis 

98   Risk register 2017 EASME - 25/01/2018 

27 

 
 
 
 
 
In 2019, the increase was attributed to the high number applications received (93% increase 
in CV’s screened). In 2020, the increase was attributed to the impact of COVID-19, which 
affected the timing of the tests, as well as the freeze in certain recruitments due to the new 
Commission mandate99.  
The  IAS  audit  on  management  of  human  resources  in  EASME100  recommended  that 
EASME should further improve its management of recruitment and selection processes. In 
response, the agency created a recruitment taskforce and implemented the recommended 
changes101 in line with the approved action plan.  

Finally, the turnover rate fell from 6.58% in 2017 to 5.23% in 2020 (see Figure 5 below). 
EASME  indicates  that  most  of  the  staff  that  left  took  up  positions  in  other  EU 
institutions102.  

Figure 5. Turnover rate, 2017-2020 

6,58%

5,41%

5,41%

5,24%

8,00%

6,00%

4,00%

2,00%

0,00%

2017

2018

2019

2020

Source: Study supporting the triennial evaluation of the Agency, based on EASME annual activity reports 
and annexes 2017, 2018, 2019, 2020 

The number of vacancies increased sharply from 13 in 2017 to 28 in 2020  (as shown in 
Figure 4) and is especially problematic in the case of seconded officials. EASME indicates 
this is a risk and points to a lack of attractiveness of the job and lengthy procedures. The 
average time to fill a position also increased over the years, reaching 6.6 months in 2020. 
Finally, the turnover rate decreased to 5.24% in 2020.  

In terms of wellbeing, 72% of staff indicated that they felt that the agency cared about this 
aspect, an increase compared to 2016 (70%) although this decreased to 51% in 2020103, 
perhaps understandable given the pandemic situation and the uncertainties linked to the 
change in the agency’s structure in 2021. 

A  Commission  staff  survey  was  held  in  2018  and  EASME  gauged  staff  satisfaction  in 
2020.  The  staff  engagement  index,  which  measures  staff’s  emotional,  cognitive  and 
physical connection to the job, organisation and people within it, remained stable in 2020 
at around 70%, but did not reach its target of 72%.  

99   EASME annual activity report 2020 
100   European Commission, action plan: IAS audit on Management of Human Resources in EASME, July 2017 
101   EASME annual activity report 2018 
102   EASME annual activity reports and annexes 2017, 2018, 2019, 2020 
103   EASME annual activity report 2020 

28 

 
 
 
 
The  previous  staff  survey  results  from  2016  showed  two  main  areas  for  improvement: 
managerial  excellence  and  career  development  of  staff.  The  previous  evaluation  also 
highlighted a perception of limited career opportunities for EASME staff104.  

In its 2017 risk register, EASME indicates staff retention and decreasing staff engagement 
levels  as  a  risk,  and  attributes  this  to  limited  job  profiles,  limited  scope  of  technical 
knowledge required from staff, strict regulations reducing the internal mobility of contract 
agent  staff  and  the  agency’s  dependence  on  EPSO  for  organising  CAST  selections  for 
contract  agents.  This  dependency  slows  down  their  selection,  especially  for  specialist 
profiles, but also makes career progression for staff very limited105. 

To improve the career development of staff and managerial excellence, EASME adopted 
an HR strategy and action plan106. To boost career opportunities, several initiatives were 
taken, such as job shadowing, a competency framework (IAS also recommended in the 
2017  IAS  audit  to  address  identified  gaps  between  workforce  needs  and  current 
workforce), and better training of HR officers, to allow career guidance advice to be given 
to staff. A “360° feedback” exercise took place for managers. 

In addition to its initiatives on increasing career opportunities and improving managerial 
excellence, EASME also made efforts to increase staff engagement in 2017 by creating 
“The  Sounding  Board”  (an  initiative  that  was  maintained  in  the  following  years  and  to 
which  80  colleagues  actively  contributed  in  2018107),  organising  TED  talk  sessions  to 
encourage  staff  to  network  and  get  inspired,  and  by  continuing  its  onboarding  plan  for 
newcomers.  

Nonetheless, the targets  set in 2017108 for the 2018 staff survey in terms of satisfaction 
around  career  opportunities  and  managerial  excellence  were  not  reached,  though  they 
generally improved. The satisfaction of employees with their professional future improved 
compared to 2016, and came closer to the target of 42%, but remained relatively low, with 
an overall general satisfaction level of 41%.  

In terms of managerial excellence, staff satisfaction with middle management was at 54% 
in 2018 and remained below its target of 60%, and satisfaction with senior management 
was 66% - below its target of 70%109. Finally, only 55% of EASME staff thought effective 
action had been taken on the results of the 2016 survey, and only 59% thought that effective 
action would be taken on the results of the 2018 survey.  

The EASME development plan, following the Commission staff survey in 2018, indicated 
several areas for improvement, with quick wins identified for 2019, as well as action in the 
medium term (2020) and long term (2021). The areas of improvement that were identified 
include the professional future of its staff and middle management.  

In July 2018, a multi-annual HR strategy was adopted, closely aligned with the identified 
improvement areas and was the result of a participatory process with contributions from 

104   European Commission, Evaluation of the Executive Agency for Small and Medium-Sized Enterprises (EASME), SWD(2020) 

final, 29.04.2020, p. 16.  

105   Risk register 2017 EASME - 25/01/2018 
106   EASME steering committee Secretariat, Final minutes of the EASME steering committee 30th June 2017 
107   EASME annual activity report 2018 
108   EASME annual activity report 2017 
109   European Commission, SPEAKUP Staff survey 2018, EASME results 

29 

 
 
 
management, the staff committee, sounding board groups and staff110. The strategy is built 
around three strands - creating an attractive workplace, building a modern leadership and 
developing professional growth111.  

One  of  the  initiatives  was  the  creation  of  the  Staff  Engagement  and  Business 
Transformation  team  (SEBT)  or  “EASME  together”,  to  help  EASME  become  a  highly 
modern and attractive workplace in 2018112.  

The results of this were already visible in the 2018 staff survey, where satisfaction in terms 
of collaboration increased (to 86%), and a large majority of staff felt that listening to staff 
is important within EASME (79%) and that their opinion was of value (79%).  

The  staff  engagement  index,  which  is  a  key  indicator  in  the  Commission  staff  survey, 
remained relatively constant, at 71% in 2018 and 70% in 2020. Key areas of improvement 
remained  the  same  across  the  evaluation  period  (managerial  excellence  and  career 
development  for  staff).  After  each  staff  survey,  an  action  plan  was  developed  with 
initiatives clustered around these two areas.  

4.1.3  Coherence 

This section looks at tasks and responsibilities and whether there are any overlaps, gaps or 
inconsistencies  in  programme  portfolio  or  delimitation  of  responsibilities  between  the 
agency and its parent DGs.  

As per the previous evaluation (2014-2016) the division of tasks was clearly outlined in 
the legal framework (Act of Delegation, Memorandum of Understanding between EASME 
and all the parent DGs). However, it appeared from the interviews that this legal framework 
might not have been clear or known to all Commission and EASME staff. For example, in 
the  area  of  feedback  to  policy,  DGs  had  different  views  of  the  level  of  the  agency’s 
involvement and the level of information to be shared with the DGs.  

EASME  staff  have  confirmed  that  the  role  of  the  various  parent  DGs  has  developed 
organically as part of the bilateral partnerships113. The interviews with EASME staff noted 
a  lack  of  standard  practice  across  the  programmes.  Each  programme  and  parent 
Directorate-General  partnership  is  managed  differently,  with  a  specific  underlying 
memorandum of understanding and delegations of power114. 

In March 2018, EASME made the financial unit a separate entity the operational unit, to 
differentiate the two and give more operational autonomy to the financial unit. The same 
structure  was  emulated  in  CINEA  and  continues  under  EISMEA.  The  result  of  this 
separation is perceived to be successful, as the financial unit has gained more operational 
autonomy, and has made decisions that are now perceived to be more objective. However, 
there are indications that it makes collaboration more cumbersome and less efficient, as 

110   EASME steering committee Secretariat, Final minutes of the EASME steering committee 10th October 2018 
111   EASME annual work programme 2019 
112   EASME annual activity report 2018 
113   Based on the interview programme carried out for the supporting study 
114   European Commission MoUs with EASME 

30 

 
 
 
the contact is not as fluent as when the financial unit was integrated into the operational 
unit115. 

Looking at how the role and responsibilities of the agency were communicated to outside 
stakeholders, there is an awareness of EU funding but not as much knowledge of the role 
of the agency.  

While  98%  of  potential  applicants  and  beneficiaries  are  well  aware  of  the  fact  that  the 
projects  are  funded  by  the  EU  (911  out  of  930),  awareness  of  the  role  of  EASME 
specifically  is  less  apparent,  as  only  47%  (438  out  of  933)  know  what  programme 
management  tasks  the  agency  is  responsible  for  and  what  tasks  the  Commission  is  in 
charge of. 

In the context of the SME Instrument, beneficiaries believe that the agency staff do not 
always have the relevant profile to oversee highly complex projects that require technical 
expertise.  This  feedback  is  also  echoed  by  beneficiaries,  who  commented  on 
communication difficulties when presenting the content of their proposals116.  

Concerning  experts  hired  to  evaluate  proposals,  the  mandate  of  EASME  is  somewhat 
clearer,  with  69%  (689  out  of  999)  indicating  that  the  delineation  of  tasks  between  the 
agency and the Commission is clear to them and 85% (853 out of 1004) indicating that 
they knew the programme management itself was outsourced to the agency. 

With regard to whether or not the delegation of the programmes and other tasks to EASME 
enabled  the  Commission  to  better  focus  on  its  policy-related  tasks,  overall  this  is 
considered  to  be  effective.  This  ensures  that  parent  DGs  were  able  to  focus  on  the 
policy-making  aspects  of  the  programme,  but  also  that  the  agency  is  effectively 
implementing those initiatives for which they have operational responsibility117. (See also 
section 4.1 Effectiveness). 

Regarding how the Commission monitored the developments in – and operations of – the 
agency,  the  focus  is  on  the  extent  to  which  the  supervision  structures,  monitoring,  and 
reporting requirements, as well as other forms of information flow and feedback, provided 
an adequate overview of programme implementation. 

According to interviewees from both EASME and parent DGs, weekly meetings involving 
deputy heads of unit and heads of unit were a key element in streamlining the information 
flow  between  EASME  and  parent  DGs.  The  director  of  EASME  put  an  emphasis  on 
communicating information between relevant stakeholders via the agency website. This 
contributed  to  an  improved  transfer  of  knowledge  between  the  agency,  the  DGs  and 
relevant stakeholders.  

Organising regular meetings with different units was also described as good practice for 
financial units. These meetings are considered a constructive way for financial units to gain 
insights into the challenges faced by the different programmes.  

Interviews with EASME and steering committee members 
Interviews with beneficiaries carried out for the supporting study  

115  
116  
117   Lapatinas, A., Katay, G. Vollbracht, I., Funck, A., Grigoleto, M. (2020), CBA for the delegation of certain tasks regarding the 

implementation of 2021-2027 EU programmes to the executive agencies: Qualitative Analysis 

31 

 
 
 
Nevertheless, some interviewees from parent DGs and members of the EASME steering 
committee  point  out  that  improvements  could  be  made  regarding  knowledge  sharing 
between the DGs and the agency. The Directorate-General for Communications Networks, 
Content  and  Technology  highlighted  a  lack  of  time  and  effort  put  into  understanding 
mutual expectations. Interactions between the parent DGs and EASME were thought to 
take  place  mainly  in  times  of  difficulty,  rather  than  as  a  way  of  regularly  sharing 
knowledge. Other interviewees indicate that the physical distance between the agency and 
the parent Directorate-General is a barrier to an effective information flow.  

Regarding  the  exchange  of  reports,  interviews  with  the  Directorate-General  for  Energy 
indicate  that  it  was  very  satisfied  with  the  information  flow  to  and  from  EASME.  The 
EASME  structure  has  proven  effective  in  employing  experts  that  extract  relevant 
information and transfer this information in a timely manner to the parent DG.  

The success is attributed to the dynamic and flexible structure of the agency. On the other 
hand, quarterly reports were initially requested from EASME, but this has proven to be 
cumbersome  and  inefficient.  Interviewees  from  the  DGs  indicate  that  biannual  reports, 
complemented with regular liaison meetings were sufficient to provide  DG ENER with 
relevant input118. 

Among innovative HR tools that contributed the sharing of knowledge  both within and 
between the agencies, it is worth mentioning the two job-shadowing exercises that took 
place in 2017 and 2019-20.  

The first took place from the beginning of November 2017 to mid-January 2018. It was an 
intra-agency exercise that involved 45 participants. Participants expressed a high degree 
of  satisfaction,  and  the  exercise  was  perceived  as  very  useful  to  mutual  learning.  The 
job-shadowing allowed staff to share best practice and facilitated knowledge-sharing and 
collaboration across the agency.  

The main difficulties were related to the timing and duration of the exercise. The exercise 
was carried out in a demanding period of high workload for most participants, and they 
indicated that a  different  period (either some additional months  or a year-long  activity) 
would have been beneficial. It would allow them to follow the entire project life cycle over 
a year and not just a small part of it. Endorsement from management was also considered 
essential to ensure the success of the exercise119. 

The second job-shadowing exercise was an inter-agency exercise carried out from October 
2019  to  April  2020120,  and  extended  until  end  of  June  2020.  It  concluded  with  a  high 
satisfaction  rate  among  participants.  The  report  found  positive  impacts  on  knowledge 
sharing among the agencies. The project promoted the exchange of best practice, favoured 
cooperation and networking across departments.  

It  also  has  far-reaching  impacts,  as  the  bonds  and  networks  created  during  the 
job-shadowing endured and continue to enrich the job of participants beyond the exercise 
itself. It also allowed EASME to improve the support it provides in terms of programme 

118  
119  
120  

Interviews with parent DGs 
JOB SHADOWING PILOT PROJECT 2017 FINAL REPORT, February 2018 
Inter-Agency Job shadowing 2019-2020, Final report, Feedback and evaluation, p.7 

32 

 
 
 
implementation  and  to  facilitate  mutual  understanding.  “Job  shadowing  proved  to  be  a 
more  effective  learning  opportunity  than  classical  training  courses”121.  The  report  also 
contains recommendations to transform job shadowing into a consolidated work practice. 

According  to  interviewees  from  both  EASME  and  parent  DGs,  while  staff  mobility  is 
described as a form of continuous capacity building within the parent DGs, staff mobility 
from DGs to agencies is still somewhat lacking, as Commission staff appear reluctant to 
join agencies122. 

4.2  Cost-benefit analysis 

The retrospective cost–benefit analysis for 2017 to March 2021 was based on the results 
of  the  2013  advance  cost-benefit  analysis,  the  assumptions  laid  down  in  the  specific 
financial statement123 and the actual costs of EASME.  

The  2013  advance  (ex  ante)  cost-benefit  analysis  estimated  that  delegating  programme 
implementation  to  EASME  would  result  in  savings  of  around  EUR  97.9  million  over 
2014-2020, compared to having them managed by the Commission (the in-house scenario).  

The  retrospective  cost-benefit  analysis  revealed  that  the  actual  costs  of  the  executive 
agency scenario amounted to EUR 189.3 million during 2017-2020. To evaluate the extent 
to  which  the  actual  costs  corresponded  to  the  initial  specific  financial  statement  (SFS) 
estimates, the same assumptions that led to these estimates were followed124.  

The  statement  estimates  (EUR 241.1  million  in  2017-2020)  were  based  on  the  EU 
contribution,  but  EASME’s  administrative  budget  also  included  contributions  from  the 
EEA/EFTA  and  non-EU  countries  (EUR 3.9  million  during  2017-2020)  to  manage  its 
additional operational budget.  

Consequently, based on the EU contribution alone, the actual costs of the executive agency 
scenario  were  EUR 185.4  million,  which  means  that  savings  amounted  to  EUR 55.7 
million and accounted for 23.1% of the specific financial statement estimates.  

Staff-related  expenditure  (Title  1)  was  higher  than  estimated  in  the  specific  financial 
statement  due  to  salary  indexation,  promotions  and/or  increasing  staff  seniority.  For 
example  estimated  costs  in  2018  were  EUR 29.6  million,  rising  to  EUR 31.7  in  2020, 
whereas actual costs were EUR 34.4 in 2018, rising to EUR 39.4 in 2020. 

The costs of the executive agency scenario were much lower than the estimated costs of 
the  in-house  scenario.  During  2017-2020,  the  actual  cost  savings  deriving  from  a  cost 
difference between the executive agency scenario and the in-house scenario amounted to 
EUR 98 million (or 34.1% of the estimated costs under the in-house scenario). 

Comparing  the  savings  initially  estimated  in  the  specific  financial  statement  and 
cost-benefit analysis with the actual savings from delegating tasks to EASME, it was found 

Inter-Agency Job shadowing 2019-2020, Final report, Feedback and evaluation, p.7 
Interviews with EASME staff and parent DGs 

121  
122  
123   The specific financial statement for EASME C(2013)9414 was updated via C(201)6944, to take account of the addition of the 

“Fast Track to Innovation” pilot scheme. 

124   Costs  in  CBA  were  calculated  in  constant  2013  prices;  the  specific  financial  statement  used  these  estimates  without  further 

indexation. 

33 

 
 
 
 
that  the  actual  savings  during  2017-2020  were  24.7%  higher  than  the  initial  statement 
estimates (EUR 98 million, compared with EUR 78.6 million), and 43.8% higher than the 
initial cost-benefit analysis estimates (EUR 98 million, compared with EUR 68.2 million) 
(see figures 6 and 7). 

Figure 6: Estimated costs and savings of the executive agency scenario 2017-2020, EUR million 

400,00

300,00

200,00

100,00

0,00

319,68

241,07

78,61

287,31

189,31

98,00

SFS estimations

Actual

Costs of the in-house scenario

Costs of the executive agency (EA) scenario

Savings of the EA scenario

Source: Study supporting the triennial evaluation of the Agency 

Figure 7: Estimated costs and savings of the executive agency scenario 2017-2020 (Title I and 
Title II expenditure), EUR million 

244,42

176,27

249,43

275,81

170,82

177,81

68,15

78,61

98,00

300,00

250,00

200,00

150,00

100,00

50,00

0,00

CBA estimations

SFS estimations

Actual

Costs of the in-house scenario

Costs of the executive agency (EA) scenario

Savings of the EA scenario

Source: Study supporting the triennial evaluation of the Agency 

The workload analysis in the retrospective  cost-benefit analysis revealed that the actual 
operational budget, in terms of executed commitment appropriations for 2017-2020 was 
close  to  the  specific  financial  statement  estimates  (EUR  6.707  billion  actual  instead  of 
EUR 6.719 billion in terms of commitment appropriations).  

An additional budget coming from EEA/EFTA and other non-EU countries amounted to 
EUR 90 million125 in the same period, and this is not accounted for in the specific financial 
statement.  

125   The bulk of this budget EUR 89.3 million relates to Horizon 2020, with EUR 0.58 million coming from COSME. 

34 

 
 
 
 
 
 
The  actual  operational  budget  executed  by  EASME  in  2017-2020  was  higher  for  LIFE 
(109% of the specific financial statement estimates) and Horizon (101.5% of the specific 
financial  statement  estimates,  with  the  higher  operational  budget  for  Horizon  mainly 
related to EEA/EFTA and non-EU contributions).  

The actual commitments for COSME and EMFF were lower than initially estimated and 
constituted respectively 95% and 47% of the specific financial statement estimates. Since 
2018,  the  agency  managed  a  redesigned  SME  instrument  and  Fast  Track  to  Innovation 
schemes under the umbrella of the EIC pilot. Due to the changes in the operational budget 
and  functioning  of  the  agency,  the  number  of  authorised  posts  in  EASME  was 
correspondingly adjusted through the budget procedure126. 

Implementation instruments such as grants for various types of projects, tenders and the 
distribution  of  the  programme  budget  between  tenders  and  grants  are  all  factors  which 
affect  workload.  Other  influencing  factors  include  the  number  of  applications  and  the 
corresponding  success  rate  and  the  number  of  grants/contracts  and  the  average 
grant/contract size.  

The  workload  level  related  to  managing  the  SME  instrument  in  the  2013  cost-benefit 
analysis was estimated based on the number of grants. However, the workload related to 
evaluating a very high number of applications was not properly addressed.  Some of the 
programmes managed by EASME (such as COSME, LIFE and EMFF) were implemented 
through complex and fragmented implementation instruments, which increased the level 
of workload.  

To meet these challenges, the agency monitored the workload level within and across the 
programmes  and  management  tasks,  initiated  requests  for  additional  resources  and 
redeployed its resources, subject to the permitted flexibility level. 

The productivity level initially estimated in the 2013 cost-benefit analysis was achieved 
by the agency (in terms  of budget  per head). This increased  by 34%  – from  EUR 2.85 
million in 2013 to EUR 3.83 million in 2020. 

5.  CONCLUSIONS AND LESSONS LEARNED 

5.1   Conclusions 

EASME has ensured the effective implementation of the programmes. The execution of 
the operational budget was at 100% in all years of the evaluation period (2017-2020). In 
terms of administrative budget, between 98% and 100% of commitments were executed, 
and between 85% and 92% of payments. 

126   Due to the impact of the European Fund for Strategic Investment (EFSI) and the correspondingly reduced EASME’s operational 
budget, the number of authorised staff was reduced (-6 posts), additional posts were allocated for LIFE programme to reflect a 
higher  operational  budget  (+6  posts),  a  post  was  deducted  for  SEDIA  (-1  post  from  2019)  and  additional  posts  allocated  for 
managing the European Innovation Council (+5 posts from 2019). 

35 

 
 
 
 
The  ratio  between  the  administrative  and  operational  budget  –  an  indicator  for  the 
management and control cost of the agency – was at an average of 3% for commitments 
and remained well within the target of 5.5%127.  

Moreover, the supervisory and control systems in place proved to be generally effective, 
though they could be improved for some programmes.  

The  residual  error  rate,  i.e.  the  level  of  errors  in  financial  transactions  which  remain 
undetected and uncorrected during the advance checks, is an important indicator in this 
regard. For the legacy programmes CIP-IEE II and CIP-Eco-Innovation, the residual error 
rate was systematically exceeded, and for COSME the residual error rate was exceeded in 
2018 and 2020.  

However, in all of these cases, mitigating actions were taken by EASME – such as better 
informing beneficiaries and improving advance check mechanisms. 

Programme implementation was executed in a timely manner, with key indicators such as 
time-to-inform and time-to-grant generally remaining below the set targets. Payments were 
also largely executed on time, with 98.0% to 99.8% of payments made on time every year 
between 2017 and 2020, close to the target of 100%128.  

IT tools have enabled significant time savings and allowed for staff rationalisation. This is 
particularly  the  case  for  calls  processes,  and  the  agency  has  invested  proactively  in 
developing  its  digital  tools.  These  tools  are  one  way  that  simplification  measures  are 
implemented.  They  facilitate  grant  agreement  preparation  procedures  as  well  as  project 
management operations – though they do require constant improvement.  

In  a  process  initiated  in  2016,  EASME  extended  IT  tools  to  cover  e-tendering  and 
e-submissions for procurement procedures, eventually including electronic invoicing and 
leading to significant efficiency gains for the agency. These tools were further refined with 
the help of EASME staff and feedback from applicants and experts. Improvements to and 
simplifications  of  the  tools  still  needed  to  be  addressed  when  transitioning  to  the  new 
agency structure in 2021. 

Simplification  measures  were 
levels  and  for  different 
implementation mechanisms in 2017-2020, with a high degree of success. Delegating by 
giving more decision-making authority to lower levels of management led to efficiency 
gains and streamlined processes. EASME adopted simplification measures that made the 
process of project application/implementation easier. 

introduced  at  different 

Beneficiaries  were  satisfied  with  key  procedures,  such  as  grant  implementation, 
amendment, checks and audits, as well the promptness with which EASME staff provided 
information on grant preparation and announcements.  

In  terms  of  staffing,  EASME  may  not  be  sufficiently  staffed  to  ensure  effective  and 
efficient operation. For the LIFE programme, EASME was structurally understaffed, due 

127   EASME, annual activity report 2017 
128   The above-mentioned indicators are referred in the efficiency section. 

36 

 
 
 
to the expansion in the workload, compared with the initial assumptions in the cost-benefit 
analysis.  

The number of vacancies increased sharply, from 13 in 2017 to 28 in 2020. It is especially 
problematic in the case of seconded officials (only a 79% staff filling rate, with 7 open 
vacancies for seconded officials in 2020). The turnover rate decreased slightly from 6.6% 
in 2017 to 5.2% in 2020.  

A healthy work-life balance and work environment were one of EASME’s priorities. In 
2018, 72% of EASME staff felt the agency cared about their well-being, but this decreased 
sharply in 2020 to 51%, most likely due to COVID-19 restrictions. 

The staff engagement index remained stable for EASME at between 70% and 71%. Key 
areas  of  improvement  remain  the  same  -  career  development  of  staff  and  managerial 
excellence. 

The  results  of  the  2017-2020  retrospective  cost-benefit  analysis  showed  that  the  actual 
costs  of  the  executive  agency  scenario  were  below  the  specific  financial  statement 
estimates. The overall actual costs of the executive agency scenario129 constituted EUR 
189.3 million during 2017-2020.  

As  noted  in  section  4.2,  the  assumptions  which  led  to  the  specific  financial  statement 
estimates  have  been  followed,  so  only  the  EU  contribution  is  taken  into  account.  The 
specific  financial  statement  estimates  (EUR  241.1  million  during  2017-2020)  were 
compared to the actual costs of the executive agency scenario (EUR 185.4 million), which 
means that the actual savings amounted to EUR 55.7 million, accounting for 23.1% of the 
specific financial statement estimates.  

The costs of the executive agency scenario were much lower than the estimated costs of 
the in-house scenario, and the actual cost savings (difference between the executive agency 
scenario  and  the  in-house  scenario)  constituted  EUR  98.0  million  (or  34.1%  of  the 
estimated costs under the in-house scenario).  

Comparing the savings initially estimated in the specific financial statement/cost-benefit 
analysis with the actual savings from delegating tasks to EASME, the actual savings during 
the 2017-2020 period were found to be: 

•  24.7%  higher  than  the  initial  specific  financial  statement  estimates  (EUR  98.0 

million compared to EUR 78.6 million) 

•  43.8%  higher  than  the  estimates  in  the  initial  cost-benefit  analysis  (EUR  98.0 

million compared to EUR 68.2 million).  

As forecast in the specific financial statement and the advanced cost-benefit analysis, the 
savings  in  the  executive  agency  scenario  resulted  primarily  from  a  higher  share  of 

129  

Including  the  cost  of  coordination  and  monitoring  by  the  Commission  and  the  costs  covered  from  EEA/EFTA  and  non-EU 
contributions. 

37 

 
 
 
 
lower-cost external personnel (CAs) employed within EASME and a lower overall number 
of staff. 

The programme portfolio managed by EASME during the evaluation period corresponded 
to the portfolio envisaged for delegation in the advance (ex ante) cost-benefit analysis and 
specific financial statement.  

The actual operational budget in executed commitment appropriations for the 2017-2020 
period was close to the specific financial statement estimates.  

At the same time, part of the parameters for the delegated programmes deviated from the 
initial cost-benefit analysis assumptions. EASME monitored the actual workload and the 
main  factors  contributing  to  it  and  consequently  initiated  the  redeployment  of 
administrative resources (subject to the allowed flexibility level). 

Analysis  revealed  that  EASME  managed  to  achieve  the  productivity  level130  initially 
estimated  in  the  2013  cost-benefit  analysis  and  in  the  Commission  Communication  on 
delegation. The budget ‘per head’ ratio in EASME increased from EUR 2.85 million in 
2013 to EUR 3.83 million in 2020131 (an increase of 34%). 

5.2   Lessons learned 

The main lessons identified in this evaluation are linked to the set up and structure of the 
agency. More coordinated arrangements between the lead DGs and the agency would lead 
to a smoother running process for the programmes.  

Delegating similar programmes to the same agency should lead to a more straightforward 
and  efficient  delegation  process,  while  having  fewer  parent  DGs  can  also  simplify  the 
relationship and procedures between the DGs and the agency.  

Although  the  delegation  decisions  have  been  taken  for  2021-2027,  the  review  of 
programme  allocation  has  seen  this  reduction  take  place  in  practice  in  EISMEA,  the 
successor  agency.  A  more  coordinated  approach  to  preparing  the  memorandum  of 
understanding  in  the  current  implementation  period  would  appear  to  have  met  some  of 
these concerns. 

Within programmes there is an advantage to running projects for longer periods or with a 
repetitive structure and content. This allows the agency to build up expertise, which then 
translates into efficiency gains. 

The  IT  tools  have  created  a  more  efficient  and  user-friendly  interface  for  both  the 
agencies/DGs and stakeholders. However, there is room to improve, and this should remain 
a  continuous  process,  for  example  by  building  in  further  capacity  to  assist  feedback  to 
policymaking  and  key  performance  indicators  for  projects.  The  tools  have  helped  with 
efficiency gains, but they should not create an extra administrative burden for stakeholders. 
They should take stakeholder feedback into account. 

130  
131  

In terms of the ‘budget per head’ ratio. 
In commitment appropriations. 

38 

 
 
 
 
Human resources are key to ensuring the success of the agency, and flexibility in resource 
allocation is important to ensure that changes during the programming period can be met. 
Continued  attention  should  also  be  given  to  the career  development,  to  ensure  staff  are 
motivated  and  engaged.  The  new  agency  structure  provides  a  more  reactive  human 
resource policy, including working across agencies to provide more flexibility. 

39 

 
 
 
 
ANNEX I: PROCEDURAL INFORMATION 

The  evaluation  of  all  the  six  agencies  was  made  in  a  coordinated  manner  based  on  a 
supporting  study  carried  out  by  an  external  contractor  (PPMI  Consortium).  The 
preparation for the evaluation started in 2022, when an Interservice group was set up by 
all lead parent DGs and the concerned central services (Directorate-General for budget 
and the Secretariat-General). The group meetings (i.e. 6) were chaired by the Common 
Implementation Centre, set up within the Directorate-General for research and innovation, 
who also ensured the secretariat. The evaluation of each agency was led by the respective 
lead  parent  Directorate-General  (PLAN/2022/1912).  The  Commission  also  launched  a 
‘call for evidence’ on 10 March 2023 on its ‘Have your say’ portal. The call was open for 
feedback on agencies activities until 7 April 2023.  

The methodology used for the evaluation was consistent across the agencies in accordance 
with  the  criteria  provided  in  the  Commission’s  Better  Regulation  guidelines.  The 
assessment of the evaluation criteria ‘EU added value’, i.e. why the EU should act, was 
not  considered  relevant  as  each  agency  carries  out  tasks  which  the  Commission 
transferred to it. The EU added value of the programmes that each agency manages is 
assessed in the context of the programme evaluations.  

Evidence was taken from sources such as the Commission databases, annual reporting 
exercises, adopted decisions. The supporting study was prepared using a mixed-methods 
approach  at  the  levels  of  methodologies  and  methods  and  worked  on  a  qualitative 
methodological approach (based on documentary review and desk research, interviews, 
answers to open-ended survey questions and qualitative cost-benefit analysis) combined 
with  a  quantitative  methodological  approach  (based  on  administrative  and  monitoring 
data, surveys and quantitative cost-benefit analysis), as detailed in Annex II. 

40 

 
 
 
 
ANNEX II: METHODOLOGY AND ANALYTICAL MODELS USED 

1.  Evidence-based and mixed-methods approach 

Taking into consideration its nature and purpose, this evaluation relied on an evidence-based 
approach. Data were gathered and analysed according to the logic of triangulation, meaning 
that the answer to each of the evaluation question is based on several sources of information. 
However,  the  scope  for  triangulation  depends  on  the  specificities  of  each  question.  For 
example,  some  evaluation  questions  –  or,  rather,  evaluation  criteria  –  only  require 
information from one specific source (e.g. surveys). Throughout the Report, the sources of 
evidence used to inform the answers to specific evaluation questions are clearly indicated. 
Proposed  conclusions  and  policy  recommendations  were  discussed  with  the  Interservice 
Steering Group to ensure broad agreement and understanding of the evaluation process and 
its results. 

Since this project required the evaluation of the operations of multiple agencies and their 
performance  results,  a  mix  of  different  methods  was  used  during  the  evaluation  process. 
This  mixed-methods  approach  is  defined  as  the  coordinated  use  of  more  than  one  social 
science  paradigm,  methodology  and/or  method  to  improve  the  understanding  of  the 
phenomena  under  investigation.  For  this  evaluation  of  the  operation  of  the  agencies,  the 
evaluation team working on the supporting study employed a mixed-methods approach at 
the levels of methodologies and methods by connecting: 

▪  a qualitative methodological approach, which is based on documentary review and 
desk research, interviews, answers to open-ended survey questions and qualitative 
cost-benefit analysis; 

▪  a  quantitative  methodological  approach,  which  is  based  on  administrative  and 

monitoring data, surveys and quantitative cost-benefit analysis. 

These  methodological  approaches  to  mixing  methods  were  used  in  a  complementary 
manner,  i.e.  the  results  of  quantitative  methods  were  used  to  enhance  the  team’s 
understanding  of  the  qualitative  results,  and  vice  versa.  These  methods  were  applied  in 
differentiated  manner,  according  to  the  nature  of  the  evaluation  questions  and  the  issues 
outlined  in  the  technical  specifications.  While  all  of  the  study  questions and  issues  were 
analysed  both  qualitatively  and  quantitatively,  in  answering  some  of  the  evaluation 
questions,  greater  focus  placed  on  the  qualitative  methodological  approach,  while  other 
questions were addressed primarily by relying on quantitative evidence. A more in-depth 
description of how different methods of data collection and analysis informed specific tasks 
and evaluation questions is provided in the subsection below. 

2.  Organisational model for the evaluation of the executive agencies  

This evaluation followed a holistic approach to organisational analysis, which identifies all 
important  elements  of  an  organisation’  activities,  and  the  general  relationships  between 
them. To properly organise a descriptive, explanatory and prescriptive study of the agencies, 
it is necessary to identify the main sets of factors operating in each organisation’s context of 
organisations, as  well as the enablers  (causes) and results of organisational  performance, 
and to link them together into a single framework. To achieve this, an organisational model 

41 

 
 
 
developed  based  on  the  model  used  in  the  Common  Assessment  Framework  Common 
Assessment Framework, 2013 was used.132  

The  structure  of  the  organisational  model  used  in  this  study  comprises  three  sets  of 
organisational factors: (1) regulatory and operational framework, (2) enablers and (3) results 
(see the figure below). All of these organisational factors are integrated into to the evaluation 
questions and their operationalisation. 

Organisational model for the evaluation of the Commission’s executive agencies 

Source:  Study  supporting  the  triennial  evaluation  of  the  Agency,,  based  on  the  Common  Assessment 
Framework 2013 model. 

3.  Extensive consultation strategy 

The evaluation was informed by a strategy of extensive consultation with key 
stakeholders. This aimed to gather factual information, data and knowledge about the way 
each of the agencies performed its tasks, as well as the subjective opinions and views of 
stakeholders regarding the performance of the agencies during the evaluation period. The 
design for the stakeholder consultation comprised surveys of unsuccessful applicants, as 
well as beneficiaries, and external experts who assisted the agencies during the evaluation 
period, plus an extensive interview programme involving relevant DGs of the Commission 
and the staff of the agencies. More information about the scope of the stakeholder 
consultation is provided in the subsection below. 

a)  Agency-specific methods and data that informed the evaluation findings 

The methods applied to inform the supporting study and this evaluation, are desk research, 
scoping and main phase interviews, cost-benefit analysis (cost-benefit analysis) and survey 
programme.  

The main problems encountered were related to substantial setbacks during the collection 
and  harmonisation  process  of  data  and  information  received.  The  start  of  the  evaluation 
study in the middle of the summer break delayed some initial steps such as identification 
and execution of scoping interviews, or transmission of documents and data. On top of that, 
the  shift  of  the  full  EASME’s  portfolio  and  related  databases  to  four  different  agencies 
(EISMEA, HaDEA, REA, CINEA, including associated staff turnover) in 2021 added an 

132   The CAF Resource Centre of the European Institute of Public Administration, CAF 2013: Improving Public Organisations through 

Self-Assessment, p. 9.  

42 

 
 
 
 
additional level of complexity to the transmission of information (programmes’ and calls’ 
data,  desk-material  and  documents,  lists  of  potential  interviewees  etc.)  that  had  been 
managed by EASME between 2017 and 2021. Data and information  had  to  be compiled 
from four different agencies, and in some cases, staff who were able to compile and send 
the data had to be identified first. As a result, all components of the analysis (desk-research 
and processing of documentary evidence, identification of interviewees  and execution of 
interviews,  identification  of  applicants,  beneficiaries  and  experts  to  be  included  in  the 
survey, survey launch etc.) were substantially delayed. Programme data, desk-material and 
documents  had  been  required  on  25/08  by  e-mail  but  were  not  available  until  mid-
November. The survey launch was delayed for reasons of quality control and late delivery 
of data from  other agencies,  although  the results are fully integrated in  the analysis.  The 
information received from the four agencies involved (EISMEA, HaDEA, REA, CINEA) 
provided a broad and good basis for analysis of different elements in the evaluation. The 
results of the cost-benefit analysis are complemented with a workload analysis. Finally, the 
analysis integrates the findings from the surveys with beneficiaries, unsuccessful applicants, 
and experts. 

b)  Documentary review and desk research 

The  implementation  of  the  documentary  review  and  desk  research  consisted  of  two 
interrelated sets of evidence: 

•  A review of relevant literature/documents, informing the analysis of the political, legal 
and  administrative  context  the  agency  operates  in,  as  well  as  shedding  light  on 
performance  dimensions  of  the  agency  such  as  IT  organisation,  HR  management, 
administrative  simplification,  experts’  selection,  calls  management  etc.,  and 
supplementing  interviews’  findings.  These  included  previous  evaluations  of  EASME 
(internal  or  external),  legal  acts,  Memoranda  of  Understanding  between  EASME  and 
several  parent  DGs,  annual  activity  reports,  annual  work  programmes,  IAS  and  CAS 
reports,  risk  registers,  steering  committee  meeting  minutes,  Coordination  meetings 
minutes,  action  plans  (e.g.  report  on  staff  mobility,  report  on  COVID-19  impact,  HR 
Strategies  and  action  plans  etc.),  internal  memos  and  notes,  previous  or  internal 
evaluations, among others. In total more than 320 documents were processed.  

•  An analysis of statistical and monitoring data collected by the agency on the financial 
and non-financial performance. The main data sources were the 2017, 2018, 2019, 2020 
and 2021 Activity Annual Reports and the corresponding annexes, Dashboards 2017-
2021, Final Accounts 2017-2020, and data on beneficiaries, non-successful applicants 
and experts received directly from the agencies. Also, other more specific sources on IT 
strategy, HR strategy and data, etc. complemented this analysis.  

In addition to being one of the key sources of evidence to answer the evaluation questions, 
the desk research has also been used by the supporting study team to prepare the interview 
and survey questionnaires and the survey contact lists. 

c)  Interview programme 

During August and early September 2022, 3 scoping interviews133 were conducted with 
EASME officials and Directorate-General GROW.  

133  The contractor also conducted 1 main-phase interview during this period, as one of the interviewees from DG CLIMA was retiring 

on 31/08/2022.  

43 

 
 
 
 
 
The  supporting  study  team  have  completed  38  interviews  (scoping,  main-and  follow-up 
phase) with 47 interviewees134.  

d)  Survey programme 

The survey programme for EASME included all programmes and targeted running projects 
in the evaluation period 2017-2021. It included the following target groups:  

•  Survey  A:  Survey  of  beneficiaries  and  applicants,  i.e.  both  granted  organisations 
(SME, company, public organisation (for/not for profit), government body etc.) with 
a proposal submitted and evaluated in the evaluation period and project running in 
the  evaluation  period,  and  unsuccessful  applicants  with  a  proposal  submitted  and 
negatively evaluated in the evaluation period; 

•  Survey B: Survey of external experts / evaluators involved in evaluations between 

2017 and April 2021. 

For  beneficiaries  and  external  experts/evaluators,  the  full  population  was  targeted.  For 
unsuccessful  applicants,  a  randomly  chosen  sample  of  maximum  5,000  contacts  were 
selected from the population of unsuccessful applications for invitation to the survey.  

As regards the scope of the beneficiaries’ survey, the following three categories of projects 
were selected: 

1.  Projects submitted, evaluated and completely executed between January 2017 and 

01/04/2021; 

2.  Projects  submitted,  evaluated  and  largely  executed  between  January  2017  and 

01/04/2021, but still on-going after 01/04/2021; 

3.  Projects  submitted,  evaluated  and  with  Grant  Agreement  signed  between  January 

2017 and 01/04/2021, but not necessarily started on 01/04/2021. 

As regards the scope of the unsuccessful applicants’ survey, projects that were submitted 
and negatively evaluated between 01/01/2017 and 01/04/2021 were targeted.  

Data  allowing  the  identification  of  panels  of  applicants,  beneficiaries  and  experts  (i.e. 
participation  data  to  calls  under  the  different  programmes  managed  by  EASME)  were 
delivered in  batches between  end September and Mid-November 2022.  Some data could 
have been further complemented but since they were not critical for the subsequent stages 
of operation, they were left aside135. 

After  cleaning  and  harmonisation  procedure,  the  final  panels  ready  for  survey  launch 
consisted of 5.830 unique respondents for the beneficiaries and 4.797 unique respondents 
for the unsuccessful applicants (survey A) and 4.300 unique respondents  for the experts’ 
survey (survey B).  

For Survey A (beneficiaries and unsuccessful applicants), the response rates are as follows 
(see  Table  6  below):  10%  response  rate  overall  (9%  after  cleaning),  of  which  8%  with 
complete  answers  (8%  after  cleaning)  and  2%  with  partial  answers  (1%  after  cleaning). 
Beneficiaries  recorded,  as  expected,  a  higher  response  rate  than  unsuccessful  applicants 
(13% versus 8%; 11% versus 6% after cleaning). In total 159 responses were deleted; the 

134   For specific programmes that were managed by EASME, a number of group interviews were conducted with several interviewees 

were carried out. Therefore, the number of interviews differs from the number of interviewees.  

135   E.g.  in  LIFE  data  it  was  not  possible  to  make  the  distinction  between  coordinators  and  partners.  Since  only  coordinators  were 

surveyed, the issue was solved by adding a question and a routing in the survey. 

44 

 
 
 
 
largest  majority  (149/159)  because  they  took  less  than  one  minute  to  answer  the  entire 
survey or because they only answered the background questions. In the analysis answers 
from  beneficiaries  and  from  unsuccessful  applicants  were  treated  together.  In  Survey  B 
(experts),  response  rates  are  higher,  as  expected:  29%  response  rate  overall  (24%  after 
cleaning), of which 25% with complete answers (24% after cleaning). In total 210 responses 
were  deleted;  the  largest  majority  (194/210)  because  they  took  less  than  one  minute  to 
answer  the  entire  survey  (n=88),  because  they  only  answered  the  background  questions 
(n=39), because they answered partially (n=39), or because they were experts only in 2021 
(n=28)136. 

Main statistics related to the implementation of the surveys  

INVITATIONS 
SENT 

RESPONSES 
SUBMITTED 

REMOVED 
AFTER 
CLEANING 

RESPONSES 
INCLUDED IN 
THE ANALYSIS 

RESPONSE 
RATE 

Survey A: 
unsuccessful 
applicants 
Survey A: 
beneficiaries 
Total survey A 
Survey B: 
external experts 

4797 

5830 

10627 
4300 

367 

731 

1098 
1236 

59 

100 

159 
210 

308 

631 

939 
1026 

8% 

13% 

10% 
29% 

4.  Cost-benefit analysis 

The  quantitative  cost-benefit  analysis  is  presented,  covering  in  particular  the  workload 
analysis and evaluation of key workload drivers, assumptions and productivity indicators 
(operational budget delegated to the Agency, number of proposals, number of projects and 
average grant size, etc.) – underpinning the workload analysis and staffing estimates in the 
2013  cost-benefit  analysis  and/or  Specific  financial  statement,  compared  with  the  actual 
situation.  It  also  provides  conclusions  regarding  the  cost-effectiveness  of  the  executive 
agency scenario, and the actual savings due to externalisation. 

The qualitative aspects of the cost-benefit analysis which reflect the cost-benefit analysis 
questions provided in Article 3(1) of Regulation (EC) No 58/2003were integrated into the 
overall evaluation framework. 

5.  Limitations of the methodological approach and findings 

The high quality, variety and abundance of the administrative and monitoring data collected 
and provided by the Agency, as well as the information in their annual activity reports and 
other  sources,  allowed  the  study  team  to  perform  sound  analyses  of  the  performance  of 
EASME. In addition, the longitudinal dimension of the data collected enabled the study team 
to examine and assess changes in the Agency’s performance over time and to identify both 
positive and negative trends. The foreseen number of interviews was largely carried out in 
due time, allowing to incorporate also qualitative information and specific insights into the 
analysis and to  validate  findings  from  the quantitative analyses.  Further  detailed insights 
were collected through surveys and allowed to integrate the beneficiaries, applicants, and 
expert perspectives in the analysis. 

136   The data did not allow identifying the specific call, i.e. before or after April 2021.  

45 

 
 
 
 
 
Nonetheless,  this  evaluation  is  not  without  limitations.  First  of  all,  response  rates  to  the 
survey, which aimed to corroborate and further expand the analysis of administrative and 
monitoring  data,  could  have  been  higher  (8%  for  unsuccessful  applicants,  13%  for 
beneficiaries and 29% for experts). This is due to a combination of specific factors such as:  

•  The  nature  of  the  target  group  and  type  of  organisations  surveyed  (large  part  of 

respondents being from SMEs and/or small organisations). 

•  The fact that the surveys were kept open for a relatively short period of time (ten 

days including two weekends).  

•  Finally, some of the target groups were already surveyed recently in the context of 
other  (evaluation)  studies.  Although  the  study  team  anticipated  on  this  in  the 
invitation emails, survey fatigue can therefore not be excluded. 

While  the  response  rate  could  have  been  higher,  it  is  still  largely  sufficient  to  perform 
analyses and the total number of observations is higher than in previous evaluations137.  

Some dimensions of the analysis could be biased by a lower number of observations. For 
instance, the very large majority of experts were recruited for the evaluation of proposals, 
while  a  small  minority  of  them  were  involved  in  monitoring  and  evaluation  of  projects. 
Elements related to monitoring and evaluation will therefore be based on less observations 
than those related to the application and selection phases. 

The analysis usually makes the distinction between applicants and beneficiaries. In the cases 
where it doesn’t, the results might be skewed by a sample bias given the fact that the sample 
contains slightly more beneficiaries than unsuccessful applicants. The latter, for instance, 
tend to report an overall lower level of satisfaction towards the Agency than beneficiaries. 
This sample bias remains, however, very limited (8% of unsuccessful applicants’ population 
versus 13% of beneficiaries’ population). 

This  evaluation  also  comprised  exploratory  and  in-depth  interviews  to  improve  the 
understanding  of  the  governance,  interactions,  internal  processes  and  performance  of  the 
agency and the value it brings to its stakeholders. While all the interviews conducted proved 
to  be  especially  useful  for  the  contextualisation,  complementing  and  deepening  of  the 
quantitative  findings  and  statistical  trends  observed,  not  all  interviewees  were  able  to 
provide  exhaustive  insights  on  the  performance  of  EASME  in  all  the  years  under 
consideration. For instance, some interviewees from parent DGs were interacting with the 
agency  only  recently  and  were  only  able  to  shed  light  on  recent  interactions.  Other 
interviews focused on very specific processes or programmes (instruments) only, which was 
very  useful  to  allow  the  supporting  study  team  to  deepen  their  understanding  of  these 
processes or programmes (instruments) but contributed only to a limited extent to the more 
overarching  analysis  of,  for  instance,  the  relation  between  the  agency  and  the  parent 
Directorate-General.  Nevertheless,  the  collection  of  interviews  jointly  provided  largely 
sufficient inputs.  

137   European Commission, Evaluation of the Executive Agency for Small and Medium-sized Enterprises (EASME), June 2019, ISBN 
978-92-76-09259-9, pp. 3-4. In this evaluation (covering the 2014-2016 period), only 16 complete responses for beneficiaries were 
recorded (against more than 600 in the current case); the total number of responses (unsuccessful applicants, beneficiaries and experts 
together) amounted 1864 units in the 2014-2016 evaluation against more than 2300 units in this supporting study.  

46 

 
 
 
 
 
ANNEX III: EVALUATION MATRIX AND, WHERE RELEVANT, DETAILS ON ANSWERS TO THE 
EVALUATION QUESTIONS (BY CRITERION) 

Please  refer  to  section  4  of  the  external  supporting  study  for  detailed  answers  to  the 
evaluation questions. 

47 

 
 
 
 
 
ANNEX IV: OVERVIEW OF BENEFITS AND COSTS 

The benefits and costs are detailed in the Cost Benefit Analysis in section 4.2 and annex V. 

48 

 
 
  
 
 
ANNEX V: COST BENEFIT ANALYSIS 

1.  Background to the quantitative cost-benefit analysis 

A  detailed  cost-benefit  analysis  (cost-benefit  analysis)  of  all  the  executive  agencies 
(including  EASME)  was  conducted  in  2013  for  the  2014-2020  Multiannual  Financial 
Framework.  This  cost-benefit  analysis  compared  the  following  four  scenarios,  based  on 
varying levels of delegation and the distribution of programmes between different executive 
agencies: 

•  An  in-house  scenario  that  assumed  new  programmes  would  be  managed  by  the 
Commission,  while  the  executive  agencies  would  remain  responsible  for  the  delivery  of 
legacy work (2007–2013 Multiannual Financial Framework programmes).  
•  An initial delegation scenario defined by the Commission. 
•  Two  alternative  delegation  scenarios  with  different  options  for  delegation  among 
executive agencies. 

A description of the programme portfolio of EASME under different scenarios is presented 
in the table below. 

Table 1. Programme portfolio of the EASME under different cost-benefit analysis 2013 
scenarios 

Initial scenario 

scenario 

Alternative 
1 
(changes relative to the initial 
scenario) 

scenario 

Alternative 
2 
(changes relative to the initial 
scenario) 

• 
• 

• 

New programmes (2014-2020 
Multiannual Financial Framework) 
COSME  
Parts of Horizon 2020: Leadership in 
enabling industrial technologies:  
Space research  
The Sustainable Industry Low Carbon 
Scheme initiative  
Light and fast (ODI) scheme (ODI 
budget will be fully implemented 
through the SME instrument)  
Parts of Horizon 2020: Innovation in 
SMEs  
Horizon 2020: Climate action, resource 
efficiency and raw materials  
Eco-innovation  

• 
•  Materials demonstration projects  
• 

New programmes (2014-2020 
Multiannual Financial Framework) 
Management  of  EMFF  scientific 
advice  measures  by  EASME  (in 
addition to IMP and control measures) 
Legacy programmes (2007-2013 
Multiannual Financial Framework) 
Transferring the legacy of FP7-Space 
to EASME 

New programmes (2014-2020 
Multiannual Financial Framework) 
Delegation of the new space 
programme to REA (instead of 
EASME) 
EASME to manage CIP-IEE II and 
IEE III (instead of INEA) 
Centralised management of the entire 
Horizon SME instrument in EASME 
Management of EMFF scientific 
advice measures by EASME (in 
addition to IMP and control 
measures) 

Environment and climate action 
research projects  
Parts of Horizon 2020 SME instrument  
The new LIFE programme  
European Maritime and Fisheries Fund 
(EMFF) – Integrated Maritime Policy 
(IMP) and control measures  
Legacy programmes (2007-2013 
Multiannual Financial Framework) 
EIP eco-innovation 
EIP – Enterprise Europe Network, Your 
Europe Business portal and the IPR 
helpdesk 
Source: Study supporting the triennial evaluation of the Agency, based on cost-benefit analysis. 

It was concluded that alternative scenario 2 (hereafter, the ‘executive agency scenario’) was 
the most efficient in terms of cost savings and qualitative benefits. It was estimated that to 

49 

 
 
manage  EUR  13,267  million  (an  increase  of  127%  compared  to  2013),  the  six  agencies 
would need 2,887 full-time employees (FTEs) by 2020 (an increase of 71%, i.e. an additional 
1,200 FTEs compared to 2013). This compared favourably to the ‘in-house scenario’, which 
would require 3,088 FTEs to manage the same programmes. A further conclusion was that 
the executive agencies will benefit from economies of scale as they become larger. 

In order to  achieve further efficiency gains,  the  Commission proposed a few adjustments 
stemming from  an improved level  of productivity and aimed at  containing  administrative 
costs through a 5% staff reduction. This significantly increased the ratio of budget “per head” 
in the case of the EASME – from EUR 2.85 million in 2013 to EUR 3.80 million in 2020 
(Table  2).  Overall,  the  Agency  was  expected  to  achieve  efficiency  gains  stemming  from 
various sources: 

•  Simplification measures proposed for the 2014-2020 programmes;  
•  Continuous innovation and learning striving for organisational excellence;  
•  Optimising the delivery of some functions 

Table 2. Budget managed and human resources in EASME compared to all executive 
agencies between 2013 and 2020 

Executive 
agency 

Budget 
managed 
in 2013 
(EUR, 
million) 

FTEs 
in 
2013 

Budget 
per head 
2013 
(EUR, 
million) 

Budget to 
be 
managed 
by Agency 
in 2020 
(EUR, 
million) 

Envisioned 
FTEs in 
2020 

Envisioned 
FTEs in 
2020 
(adjusted) 

Budget 
per head 
in 2020 
(EUR, 
million) 

Budget per 
head in 
2020 
(EUR, 
million) 
(adjusted) 

EASME 

453 

159 

All 
agencies 

5,846  1,687 

2.85 

3.47 

1,946 

13,267 

537 

498 

2,887 

2,630 

3.62 

4.60 

3.80 

5.46 

Source: Study supporting the triennial evaluation of the Agency, based on Communication to the Commission 
on the delegation of the management of the 2014-2020 programmes to executive agencies (SEC (2013) 493) 

These  adjusted  results  of  the  cost-benefit  analysis  were  used  in  the  specific  financial 
statement138. With regard to the forecasts of the administrative budget, the specific financial 
statement differs from the cost-benefit analysis in the following aspects: 

•  As  explained  above,  the  staff  number  in  EASME  was  reduced  under  the  executive 
agency scenario. The in-house scenario was not adjusted. 
•  The  costs  in  the  cost-benefit  analysis  were  calculated  in  constant  2013  prices  (i.e. 
neutralising  the  effect  of  inflation).  However,  in  the  specific  financial  statement  these 
estimates  were  used  as  current  prices  without  any  further  indexation.  In  real  terms,  this 
constituted another reduction of the administrative budget with the impact of such reduction 
gradually increasing over time. 

The table below summarises the assumptions used in the cost-benefit analysis and specific 
financial statement for both scenarios (in-house and executive agency). 

138  

specific financial statement for Establishing the Executive Agency for Small- and Medium-sized Enterprises and repealing Decisions 
2004/20/EC and 2007/372/EC. 

50 

 
 
 
 
Table 3. Assumptions used in ex ante cost-benefit analysis and specific financial statement 

Assumptions cost-benefit analysis 

Assumptions specific 
financial statement 

In-house scenario 

Description of the scenario 

▪  Legacy 2007-2013 programme managed by EASME until 2017. Any left-over legacy 

would then be handed over to the Commission. 
▪  New programmes managed by the Commission. 

Staffing mix 
European Commission: 

▪  Establishment plan posts/ Temporary Agents (Tas) – 70%; 
▪  External personnel/ Contract Agents (Cas) – 30%. 

EASME: 

▪  Establishment plan posts (Tas) – 25%; 
▪  External personnel (Cas) – 75%. 

No. of staff 
See Figure 2. 
Average cost assumptions (EUR per employee) 
European Commission: 

▪  Establishment plan posts (Tas) – EUR 108,000; 
▪  External personnel (Cas) – EUR 47,000; 
▪  External personnel/ Seconded national experts (SNEs) –EUR 55,000;  
▪  Overheads – EUR 23,000. 

EASME 

▪  Establishment plan posts (Tas) – EUR 103,250; 
▪  External personnel (Cas) – EUR 49,672; 
▪  Overheads – EUR 21,775. 

Average  cost  assumptions  are  based  on  Directorate-General  for  budget  estimations. 
Costs relating to programme support[5] (Title III expenditure) have not been included in 
the calculations, as these are likely to be the same across all scenarios. As such, these 
do not affect the cost difference between the different scenarios. 

Executive agency scenario 

Description of the scenario 
New and legacy programmes managed by EASME.  
Staffing mix 
Same as the in-house scenario 
No. of staff 
See Figure 2. 
Number  of  staff  includes  12.9  FTEs  in  the  Commission  for  the  supervision  and 
coordination with EASME. 
Average cost assumptions  
Same as the in-house scenario 

The  cost-benefit  analysis 
were 
assumptions 
not 
modified 
the  specific 
in 
financial  statement  (except 
for 
current 
prices). 

constant 

vs 

Description of the scenario 
Same  as  in  the  cost-benefit 
analysis.  
Staffing mix 
Same  as  in  the  cost-benefit 
analysis. 
No. of staff 
See Figure 2. 
includes 
Number  of  staff 
the 
11.9 
Commission 
the 
supervision and coordination 
with EASME. 
Average cost assumptions  
Same  as  in  the  cost-benefit 
analysis. 
Also  cost  estimates  for  Title 
III  expenditure  (Programme 
support  expenditure)  were 
added. 

in 
for 

FTEs 

Source: Study supporting the triennial evaluation of the Agency, based on cost-benefit analysis and specific 
financial statement. 

51 

 
 
  
EASME’s specific financial statement was updated in 2014139, which related to the allocation 
of an additional programme Fast Track to Innovation Fast Track to Innovation pilot scheme 
to  the  mandate  of  the  Agency.  This  amendment  had  an  impact  on  the  administrative 
resources allocated to the Agency and the Commission and therefore was considered in the 
retrospective cost-benefit analysis. 

2.  Actual staffing and costs of EASME 

During the 2017-2020 period of operations, the administrative budget actually implemented 
by  EASME140  amounted  to  EUR  182.4  million  (see  table  below).  The  specific  financial 
statement  estimations  (EUR  233.4  million)  were  based  on  the  EU  contribution  only,  and 
EASME’s  administrative  budget  also  included  contributions  from  EEA/EFTA  and  third 
countries  (EUR  3.9  million  during  the  period  2017-2020)  to  manage  an  additional 
operational budget. Based on the EU contributions, the actual administrative budget of the 
EASME  amounted  to  EUR  178.5  and  was  23.5%  lower  than  estimated  in  the  specific 
financial statement, with savings of EUR 54.9 million. However, as explained below, all of 
these savings were related to Title III expenditure, which was overestimated in the specific 
financial statement. 

The  costs  in  the  2013  cost-benefit  analysis  were  calculated  in  constant  2013  prices  (i.e. 
neutralising the effect of inflation), but in the specific financial statement these estimations 
were used as current prices without any further adjustment for inflation. In order to analyse 
this, the table below also presents a comparison of the specific financial statement estimated 
and EASME’s actual budget, where specific financial statement estimations were adjusted 
to reflect current prices. Current prices were established using a fixed 2% annual deflator 
starting from 2014141. 

Table 4. EASME administrative budget 2017-2020, million EUR 
2018 

Administrative budget 

2017 

2019 

2020 

Total 
2017-
2020 

Title  I.  Staff  Related 
Expenditure 

specific 
statement 

Title  II.  Infrastructure 
Operating 
and 
Expenditure 

Actual 

specific 
statement 

Actual 

III.  Programme 

Title 
Support Expenditure 

specific 
statement 

Total 

Actual 

specific 
statement 

Actual 

financial 

28.571 

29.611 

30.716 

31.793  120.691 

31.243 

34.494 

36.776 

39.426  141.939 

financial 

9.970 

10.393 

10.835 

11.284 

42.482 

6.509 

6.523 

8.619 

7.332 

28.983 

financial 

12.682 

16.463 

19.786 

21.323 

70.254 

4.293 

2.639 

2.949 

1.619 

11.500 

financial 

51.223 

56.467 

61.337 

64.400  233.427 

EEA/EFTA 
contribution 

and 

42.045 

43.656 

48.344 

48.377  182.422 

0.677 

1.003 

1.100 

1.151 

3.932 

139   C(2014)6944  of  2.10.2014,  amending  Decision  (2013)9414  adding  the  ‘Fast  Track  to  Innovation  pilot  scheme  to  the  EASME 

mandate. 

140   Based on executed commitment appropriations 
141   2% annual deflator is provided for in Article 6(2) of the Multiannual Financial Framework Regulation. 

52 

 
 
 
 
participation 
candidate 
and/or third countries 

of 
countries 

Actual 
contribution) 

(EU 

41.368 

42.653 

47.244 

47.226  178.491 

Savings (specific financial statement-Actual) 

9.178 

12.811 

12.993 

16.023 

51.005 

Savings  (specific  financial  statement-Actual  EU 
contribution) 

specific  financial  statement  adjusted  for  current 
prices using 2% annual deflator  

Savings  (specific  financial  statement  adjusted  for 
current prices-Actual) 

9.855 

13.814 

14.093 

17.174 

54.936 

55.445 

62.344 

69.075 

73.975  260.840 

13.400 

18.688 

20.731 

25.599 

78.418 

Savings  (specific  financial  statement  adjusted  for 
current prices-Actual EU contributions) 
Source: Study supporting the triennial evaluation of the Agency, based on specific financial statement, final 
annual accounts, draft general budget of the EU. 

14.078 

19.691 

26.749 

21.831 

82.350 

Staff-related expenditure (Title I) was higher than the initial estimates. This resulted from 
average  staff  costs  being  higher  than  initially  estimated  –  the  average  actual  costs  for 
temporary  agents  during  2017-2020  amounted  to  EUR  115.5  thousand  (cost-benefit 
analysis/specific financial statement estimates – EUR 103.3 thousand), the actual costs for 
contract agents were EUR 62.8 thousand (cost-benefit analysis/specific financial statement 
estimates – EUR 49.7 thousand). Such deviation reflected unsustainable initial cost-benefit 
analysis/specific financial statement assumptions: 1) as noted before, the average costs were 
not adjusted for inflation while the actual average staff related costs were rising over 2014-
2020 due to salary indexation, promotions and increasing staff seniority and 2) the average 
costs for contract agents in EASME were based on the weighted average staff costs of all 
agencies  disregarding  the  fact  that  actual  contract  agents  costs  in  EASME  were  higher. 
During the 2017-2020 period, the actual average staff costs of temporary agents at EASME 
were  lower  compared  to  the  average  staff  costs  set  by  Directorate-General  for  budget, 
however, the actual costs of contract agents were higher (Table 16). 

The actual number of EASME staff142 financed from the EU contribution was within specific 
financial  statement  estimated  range  in  2017-2020  (as  indicated  in  Table  5  below).  The 
composition of staff (the ratio between temporary agents and contract agents) corresponded 
to  the  specific  financial  statement  estimates.  EASME  also  employed  additional  contract 
agents financed from EEA/EFTA and third country contributions to manage the additional 
budget. 

Table 5. Actual and estimated number of EASME staff 

Planned No of EASME staff according to the specific financial statement (financed 
from the EU general budget)143 

temporary agents  

contract agents  

Total 

Authorised posts through the budgetary procedure[11] 

2017 

2018 

2019 

2020 

110 

337 

447 

116 

352 

468 

120 

365 

485 

126 

376 

502 

142   Actual number of staff at the end of the year. 
143   According to the latest specific financial statement version, including posts for the management of Fast Track to Innovation. 

53 

 
 
  
  
  
  
  
  
  
  
  
 
temporary agents  

contract agents financed from the EU general budget contributions 

Total authorised posts financed from the EU general budget 

contract  agents  financed  from  EEA/EFTA  and  participation  of  candidate  countries 
and/or third countries 

Total 

Actual (occupied at the end of the year) 

temporary agents  

contract agents financed from the EU general budget contributions 

Seconded National Experts 

110 

337 

447 

115 

347 

462 

5 

120 

369 

489 

5 

126 

380 

506 

5 

447 

467 

494 

511 

101 

333 

106 

342 

107 

360 

108 

372 

Total authorised posts financed from the EU general budget 

434 

448 

467 

480 

contract  agents  financed  from  EEA/EFTA  and  participation  of  candidate  countries 
and/or third countries 

5 

5 

5 

Total 

434 

453 

472 

485 

Source: Study supporting the triennial evaluation of the Agency, based on specific financial statement, 
final annual accounts and draft general budget of the EU. 

Infrastructure and operating expenditure (overheads, Title II) was around 30% lower than 
estimated in the specific financial statement. The actual average overheads of EASME were 
also lower compared to the average costs set by Directorate-General for budget (Table 16).  

The actual programme support expenditure (Title III) was significantly (over 80%) below 
the  specific  financial  statement  estimates  with  increasing  savings  over  2017-2020,  which 
was related to the overestimated Title III budget in the specific financial statement.  

Lower  actual  expenditure  under  Title  II  and  Title  III  allowed  for  offsetting  higher 
expenditure under Title I. 

Table 6. Estimated and actual average staff costs and overheads, EUR  

Average  costs  set  by  Directorate-General  for  budget  for  the  estimates of  human  resources  and  overheads  in 
legislative financial statements  

Average staff costs: temporary agents  

115 000 

119 000 

123 000 

125 000 

2017 

2018 

2019 

2020 

Average staff costs: contract agents 

Overheads 

Actual average costs144 of EASME 

47 000 

23 000 

50 000 

52 000 

55 000 

24 000 

25 000 

25 000 

Average staff costs: temporary agents  

108 643 

114 857 

116 512 

121 807 

Average staff costs: contract agents 

Overheads 

58 551 

14 998 

61 311 

63 818 

67 373 

14 399 

18 261 

15 117 

Source: Study supporting the triennial evaluation of the Agency 

144   Average staff costs include professional development and social expenditure. 

54 

 
 
  
  
  
  
  
  
  
  
  
  
 
  
 
 
 
3.  Cost-effectiveness  of  the  executive  agency  scenario  and  actual  savings  due  to 

externalisation 

To  assess  whether  the  conclusions  of  the  ex  ante  assessment  (the  estimates  of  savings 
provided in the cost-benefit analysis and specific financial statement) are still valid when 
compared with the actual situation, and what the overall possible savings are, the following 
approach was adopted: 

1.  To use data on the actual  performance of EASME (actual execution of the 

administrative budget, actual staffing, etc.) for the executive agency scenario.  

2.  To  follow  the  assumptions  laid  down  in  the  specific  financial  statement  to 
ensure the comparability and validity of results, and to provide estimates of the 
comparable  ‘actual’  in-house  (Commission)  scenario  (the  comparator),  which 
would best reflect the actual situation.  

3.  To  assess  based  on  these  estimates  whether  the  conclusions  of  the  ex  ante 
assessments  provided  in  the  specific  financial  statement  are  still  valid  when 
compared with the actual situation, and what the overall possible savings are.  

To deconstruct the ‘actual’ in-house (Commission) scenario (the ‘comparator’), estimates 
were based on the following assumptions:  

•  Number  and  composition  of  staff  in  the  Commission  under  an  in-house  scenario 
corresponds to specific financial statement estimations145. Additional contract agents 
(5) were added to the estimated Commission staff number for 2018-2020, to reflect 
additional authorised staff in EASME financed from the contributions of EEA/EFTA 
and participation of candidate countries and/or third countries to manage additional 
operational  budget  not  covered  in  the  cost-benefit  analysis/specific  financial 
statement resource calculations. 

•  Commission  staff  costs  and  overheads  correspond  to  the  average  costs  set  by 
Directorate-General for budget for the estimates of human resources and overheads 
in legislative financial statements for the respective year. 

•  Programme support expenditure (Title III) remains the same under both the in-house 

scenario and the executive agency scenario.  

The table below presents the results of the analysis of the estimated actual costs of the in-
house (Commission) scenario and the actual costs of the executive agency scenario.  

145   Additional resources allocated under the SFS amendment/CBA for the delegation of FTI are also considered. 

55 

 
 
 
 
Table 7 - Estimated actual costs of the in-house (Commission) scenario and the actual costs of the executive agency scenario, EUR  

2017 

2018 

2019 

2020 

Total 

No. 

Cost 

No. 

Cost 

No. 

Cost 

No. 

Cost 

2017-2020 

In-house scenario  

Commission 

Title I. Staff related expenditure 

519.5 

49 141 300  549.4 

53 765 900 

569.9 

57 708 200  589.6 

61 072 000 

221 687 

400 

Establishment Plan Posts 

363.6 

41 814 000  381.1 

45 350 900 

395.4 

48 634 200  409.2 

51 150 000 

186 949 

100 

Contract Agents 

155.9 

7 327 300  168.3 

8 415 000 

174.5 

9 074 000  180.4 

9 922 000 

34 738 

300 

Title 

II. 

Infrastructure 

and 

operating 

11 948 500 

13 185 600 

14 247 500 

14 740 000 

54 121 

expenditure 

600 

Title III. Programme support expenditure 

4 292 636 

2 638 953 

2 949 327 

1 618 886 

11 499 

802 

TOTAL COST 

519.5 

65 382 436  549.4 

69 590 453 

569.9 

74 905 027  589.6 

77 430 886 

287 308 

802 

Executive agency scenario  

EASME 

Title I. Staff related expenditure 

434 

31 243 193 

453 

34 494 486 

472 

36 775 698 

485 

39 425 942 

141 939 

319 

Establishment Plan Posts 

101 

10 555 273 

106 

11 686 730 

107 

11 952 762 

108 

12 634 525 

46 829 

290 

Contract Agents 

333 

18 120 350 

347 

19 676 987 

365 

21 540 005 

377 

23 582 426 

82 919 

768 

Interim supportive agents and trainees 

Professional development and recruitment costs 

772 885 

1 794 685 

1 044 769 

2 086 000 

1 015 431 

2 267 500 

871 088 

3 704 173 

2 337 903 

8 486 088 

56 

 
 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
2017 

2018 

2019 

2020 

Total 

No. 

Cost 

No. 

Cost 

No. 

Cost 

No. 

Cost 

2017-2020 

Title 

II. 

Infrastructure 

and 

operating 

6 509 165 

6 522 876 

8 619 320 

7 331 885 

28 983 

expenditure 

246 

Title III. Programme support expenditure 

4 292 636 

2 638 953 

2 949 327 

1 618 886 

11 499 

802 

Total EASME cost: 

42 044 994 

43 656 315 

48 344 345 

48 376 713 

182 422 

367 

Commission 

Title I. Staff related expenditure 

Establishment Plan Posts 

Contract Agents 

Title 

II. 

Infrastructure 

and 

operating 

expenditure 

Total Commission cost: 

TOTAL COST 

11.9 

11.9 

1 368 500 

1 368 500 

11.9 

11.9 

1 416 100 

1 416 100 

11.9 

11.9 

1 463 700 

1 463 700 

11.9 

11.9 

1 487 500 

5 735 800 

1 487 500 

5 735 800 

0 

273 700 

0 

285 600 

0 

297 500 

0 

0 

297 500 

1 154 300 

1 642 200 

1 701 700 

1 761 200 

1 785 000 

6 890 100 

445.9 

43 687 194  464.9 

45 358 015 

483.9 

50 105 545  496.9 

50 161 713 

189 312 

467 

ESTIMATED SAVINGS 

73.6 

21 695 242 

84.5 

24 232 438 

86.0 

24 799 482 

92.7 

27 269 173 

97 996 

335 

Source: Study supporting the triennial evaluation of the Agency 

57 

 
 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
 
  
  
  
  
  
  
  
  
  
  
  
  
 
 
The analysis concluded that the overall actual costs of the executive agency scenario146 constituted 
EUR 189.3 million during 2017-2020. The specific financial statement estimates (EUR 241.1 million 
during  2017-2020)  were  based  on  the  EU  contribution,  but  EASME’s  administrative  budget  also 
included contributions from EEA/EFTA and third countries (EUR 3.9 million during 2017-2020) to 
manage its additional operational budget. Consequently, based on the EU contribution only, the actual 
costs of the executive agency scenario constituted EUR 185.4 million, which means that the actual 
savings amounted to EUR 55.7 million and accounted for 23.1% of the specific financial statement 
estimates.  However,  all  of  these  savings  were  related  to  Title  III  expenditure,  which  was 
overestimated in the specific financial statement. 

The costs of the executive agency scenario were much lower than the estimated costs of the in-house 
scenario. In 2017-2020, the actual cost savings deriving from a cost difference between the executive 
agency scenario and the in-house scenario constituted EUR 98 million (or 34.1% of the estimated 
costs under the in-house scenario). 

Comparing the savings initially estimated in the specific financial statement and cost-benefit analysis 
with the actual savings from the delegation of tasks to EASME, the actual savings during the 2017-
2020 period were 24.7% higher compared to the initial specific financial statement estimates (EUR 
98.0 million compared to EUR 78.6 million), and 43.8% higher compared to the initial cost-benefit 
analysis estimates (EUR 98.0 million compared to EUR 68.2 million). See section 4.2, figures 6 and 
7. As forecasted in the specific financial statement and the ex ante cost-benefit analysis, savings of 
the executive agency scenario resulted primarily from a higher share of lower-cost external personnel 
employed within the agency and a lower overall number of staff147. 

4.  Workload analysis  

The  programme  portfolio  managed  by  EASME  during  the  evaluation  period  corresponded  to  the 
portfolio envisaged for delegation in the ex ante cost-benefit analysis and specific financial statement. 
After  the  2013  cost-benefit  analysis,  the  EASME’s  mandate  was  expanded  to  include  additional 
actions  (such  as  the  Fast  Track  to  Innovation  pilot  scheme)  and  additional  functions  (such  as  the 
management  of  the  European  Innovation  Council).  These  changes  to  EASME’s  mandate  were 
reflected in the updated specific financial statement or the revision of the number of authorised posts 
through the budgetary procedure. 

According to the specific financial statement estimates, EASME’s operational budget in commitment 
appropriations  allocated  in  2017-2020  amounted  to  EUR  6.719  billion  (Table  8).  The  actual 
operational  budget  in  executed  commitment  appropriations  for  the  same  period  was  EUR  6.707 
billion, close to the specific financial statement estimates. In addition to the EU contributions, EASME 
also managed the operational budget allocated by EEA/EFTA and third countries which amounted to 
EUR 90 million over 2017-2020 (1.3 % of the overall operational budget of the Agency), which was 
not accounted for in the specific financial statement. Consequently, the actual operational budget of 
EASME in 2017-2020 based on the EU contribution only was around 1.5% lower compared to the 
specific financial statement estimates. Most of EEA/EFTA and third countries’ contributions related 
to  Horizon  (EUR  89.3  million  over  2017-2020  or  2%  of  the  Horizon  operational  budget  of  the 
Agency).  EEA/EFTA  and  third  countries’  contributions  to  COSME  constituted  EUR  0.58  million 
(0.12% of the COSME operational budget of the Agency). 

Including the cost of coordination and monitoring by the Commission and the costs covered from EEA/EFTA and third country contributions. 

146  
147   The number of staff in the executive agency scenario was reduced; however, the in-house scenario was not modified in the  specific financial 

statement. 

58 

 
 
 
 
Table 8. Specific financial statement estimated and actual EASME’s operational budget 2017-2020 
(commitment appropriations) by programme, million EUR 

Programme 

2017 

2018 

2019 

2020 

Total 2017-
2020 

Actual/specific 
financial statement 
estimates 

specific financial 
statement 

COSME 

Horizon 

EMFF 

LIFE 

Total 
Actual[2] 

COSME 

Horizon 

EMFF 

LIFE 

Total 

107.085 

118.272 

130.053 

142.098 

497.508 

976.693 

1 036.391 

1 106.732 

1 209.236 

4 329.052 

87.315 

88.913 

89.706 

94.440 

274.081 

388.039 

424.079 

445.426 

1 445.174 

1 631.615 

1 750.570 

1 891.200 

112.32 

946.59 

38.25 

308.20 

115.03 

117.89 

126.74 

1112.99 

1 124.81 

1 207.55 

41.36 

423.32 

42.81 

464.87 

45.96 

478.14 

1 405.36 

1 692.70 

1 750.38 

1 858.39 

360.374 

1 531.625 

6 718.559 

471.98 

4 391.94 

168.38 

1 674.53 

6 706.83 

94.9% 

101.5% 

46.7% 

109.3% 

99.8% 

97.2% 

Actual/specific 
financial statement 
estimates 
Source: Study  supporting  the  triennial  evaluation  of  the  Agency,  based  on  specific  financial  statement and  EASME’s 
AARs. 

100.0% 

103.7% 

98.3% 

99.8% 

The actual operational budget executed by EASME in 2017-2020 was higher for LIFE (109% of the 
specific  financial  statement  estimates)  and  Horizon  (101.5%  of  the  specific  financial  statement 
estimates,  a  higher  operational  budget  of  Horizon  mainly  related  to  EEA/EFTA  and  third  country 
contributions). The actual commitments for COSME and EMFF were lower than initially estimated 
and constituted respectively 95% and 47% of the specific financial statement estimates. In 2018 the 
European Commission launched a new EIC pilot initiative to strengthen breakthrough innovation and 
boost high-growth companies coming out of Horizon. Consequently, since 2018, the Agency managed 
under  the  umbrella  of  the  EIC-pilot  a  redesigned  SME-instrument  and  Fast  Track  to  Innovation 
schemes and supported the work of EIC. Due to the changes in the operational budget and functioning 
of the Agency, the number of authorised posts in EASME was correspondingly adjusted through the 
budgetary procedure148. 

EASME’s  operational  budget  in  terms  of  executed  payment  appropriations  during  2014-2020  was 
10% lower than estimated in the specific financial statement (Table 9). This was primarily related to 
lower payments under the legacy programmes, and EMFF and LIFE programmes. 

148   The authorised number of staff differs from the initially programmed in the specific financial statement, which was related to the impact of the 
European Fund for Strategic Investment (EFSI) and correspondingly reduced EASME’s operational budget and the number of authorized staff (-
6 posts), additional posts for LIFE programme (+6 posts), deducted post for SEDIA (-1 from 2019) and additional post for the management of the 
European Innovation Council (+5 from 2019).  

59 

 
 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
 
 
 
 
Table 9. Specific financial statement estimated and actual EASME’s operational budget 2017-2020 
(payment appropriations) by programme, million EUR 

Programme 

2017 

2018 

2019 

2020 

Total 
2020 

2017-

Actual/specific 
financial  statement 
estimates 

specific  financial 
statement 

COSME 

Horizon 

EMFF 

LIFE 

Legacy IEE II  

Legacy ECO-I  

Total 

Actual 

COSME 

Horizon 

EMFF 

LIFE 

Legacy IEE II  

Legacy ECO-I 

131.615 

877.091 

55.792 

196.791 

26.744 

12.300 

57.514 

159.032 

75.417 

423.578 

948.628 

59.798 

260.952 

16.700 

5.941 

954.813 

1 019.383 

3 799.915 

62.993 

75.091 

253.674 

334.706 

362.351 

1 154.800 

4.106 

0.900 

0.720 

48.270 

19.141 

1 300.333 

1 349.533 

1 516.550 

1 532.962 

5 699.378 

130.40 

840.02 

19.64 

116.36 

30.76 

10.06 

64.53 

915.33 

27.28 

181.92 

7.00 

4.69 

100.72 

1 047.55 

37.40 

244.73 

0.42 

132.91 

853.09 

33.75 

317.52 

428.56 

3 655.99 

118.07 

860.53 

37.76 

15.17 

Total 

1 147.24 

1 200.75 

1 430.82 

1 337.27 

5 116.08 

101.2% 

96.2% 

46.5% 

74.5% 

78.2% 

79.3% 

89.8% 

94.3% 
Source: Study supporting the triennial evaluation of the Agency,, based on specific financial statement and EASME’s 
annual activity reports. 

88.2% 

89.0% 

87.2% 

89.8% 

Besides the allocated operational budget, the Agency’s workload was closely linked to the parameters 
of  the  delegated  programmes.  These  parameters  included  the  programmes’  implementation 
instruments such as grants to various types of projects, tenders, distribution of the programmes’ budget 
between tenders and grants. Additionally, factors such as the stability in delegated actions, the number 
of applications and the corresponding success rate, the number of grants/contracts and the average 
grant/contract  size  also  influenced  the  Agency’s  workload.  The  workload  level  related  to  the 
management of the SME instrument in 2013 cost-benefit analysis was estimated based on the number 
of grants. However, the workload related to the evaluation of a very high number of applications was 
not properly addressed.  

A  part  of  the  programmes  managed  by  EASME  (such  as  COSME,  LIFE  and  EMFF)  were 
implemented through complex and fragmented implementation instruments, which increased the level 
of workload. Such a diversity  of instruments  meant  that the actual characteristics of the delegated 
programmes  were  complex  and  for  some  programmes  deviated  from  the  initial  2013  cost-benefit 
analysis assumptions. The Agency monitored the workload level within and across the programmes 
and  management  tasks,  initiated  requests  for  additional  resources149  and  redeployed  its  resources 
subject to the allowed flexibility level.  

149  

Including staff in the Commission for ensuring supervision and coordination with EASME. 

60 

 
 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
 
 
 
Overall conclusions of the workload analysis 

The  programme  portfolio  managed  by  EASME  during  the  evaluation  period  corresponded  to  the 
portfolio envisaged for delegation in the ex ante cost-benefit analysis and specific financial statement. 
The actual operational budget in executed commitment appropriations for the 2017-2020 period was 
close to the specific financial statement estimates. At the same time, a part of the parameters of the 
delegated programmes deviated from the initial cost-benefit analysis assumptions. EASME monitored 
the actual workload and the main factors contributing to it and consequently initiated the redeployment 
of administrative resources (subject to the allowed flexibility level). 

Analysis revealed that EASME managed to achieve the productivity level initially estimated in the 
2013  cost-benefit  analysis  and  the  Communication  to  the  Commission  on  the  delegation  of  the 
management  of  the  2014-2020  programmes  to  executive  agencies  (please  see  chapter  1.1  of  the 
supporting study for details). The budget ‘per head’ ratio in EASME increased from EUR 2.85 million 
in 2013 to EUR 3.83 million in 2020 (an increase of 34%).  

Estimated and actual budget ‘per head’ in EASME in 2013 and 2020, EUR million150 

3,62

3,80

3,83

2,85

4,50

4,00

3,50

3,00

2,50

2,00

1,50

1,00

0,50

0,00

2013

2020 CBA

2020 adjusted by the
Commission

2020 actual

Source: Study supporting the  triennial evaluation of the Agency, based on Communication to the  Commission on the 
delegation  of  the  management  of  the  2014-2020  programmes  to  executive  agencies  (SEC  (2013  )493)  and  EASME’s 
annual activity report. 

5.  Qualitative analysis 

Qualitative  aspects  of  the  cost-benefit  analysis  (which  reflect  the  cost-benefit  analysis  questions 
provided  in  Article  3(1)  of  the  Regulation  (EC)  No  58/2003)  were  integrated  into  the  overall 
evaluation framework. 

Identification of the tasks justifying outsourcing 

The  delegation  of  programme  implementation  tasks  is  deemed  effective  overall  and  enabled  the 
Commission to focus on policy-related aspects of the programmes. On the other hand, the delegated 
tasks specified in the Memorandum of Understanding between the Commission and EASME are not 
always clearly defined, for example on how to provide feedback to policy or if input is requested for 
drafting  work  programmes  and  calls.  Moreover,  there  is  no  uniform  structure  for  the  specific 
Memorandum of Understanding per programme, and a low level of alignment in delegated tasks. A 
more coherent programme portfolio where delegated tasks are similar for the programmes managed, 

150  

In commitment appropriations 

61 

 
 
 
 
would  be  beneficial.  This  has  improved  considerably  with  the  transformation  of  EASME  into 
EISMEA.  

Costs of coordination and checks 

The results of the 2017-2020 retrospective cost-benefit analysis showed that the actual costs of the 
executive agency scenario were below the specific financial statement estimates. The overall actual 
costs of the executive agency scenario151 constituted EUR 189.3 million during 2017-2020. In order 
to  evaluate  to  what  extent,  the  actual  costs  corresponded  to  the  initial  specific  financial  statement 
estimates,  it  is  important  to  follow  the  same  assumptions  that  have  led  such  specific  financial 
statement estimates. The specific financial statement estimates (EUR 241.1 million during 2017-2020) 
were based on the EU contribution, but EASME’s administrative budget also included contributions 
from EEA/EFTA and third countries (EUR 3.9 million during 2017-2020) to manage its additional 
operational budget. Consequently, based on the EU contribution only, the actual costs of the executive 
agency scenario constituted EUR 185.4 million,  which means that  the actual savings amounted to 
EUR 55.7 million and accounted for 23.1% of the specific financial statement estimates. However, all 
these savings were related to Title III expenditure, which was overestimated in the specific financial 
statement. 

Impact on human resources 

Retrospective cost-benefit analysis revealed that during 2017-2020 EASME managed to cope with 
the  foreseen  human  resources.  The  composition  of  staff  (the  ratio  between  temporary  agents  and 
contract agents) corresponded to the specific financial statement estimates.  

Nonetheless, there are indications that EASME may not be sufficiently staffed to ensure effective and 
efficient operations, as pointed out by the cost-benefit analysis done by the JRC in 2020152. This is 
especially  due  to  the  lack  of  stability  in  delegated  actions  (in  case  of  EMFF  and  COSME)  and 
additional tasks that were assigned in the period 2014-2020 without an increase in staff. Further, more 
attention  should  be  given  to  the  high  level  of  workload  attached  to  effective  feedback  to  policy. 
Moreover, EASME has struggled with offering career perspective for its employees, which is partly 
due to the limited job profiles, the limited scope of technical knowledge required from staff, and strict 
regulations reducing internal mobility of contract agent staff. 

Possible savings within the general budgetary framework of the European Union 

The costs of the executive agency scenario were much lower than the estimated costs of the in-house 
scenario. In 2017-2020, the actual cost savings deriving from a cost difference between the executive 
agency scenario and the in-house scenario constituted EUR 98.0 million (or 34.1% of the estimated 
costs under the in-house scenario). Comparing the savings initially estimated in the specific financial 
statement and cost-benefit analysis with the actual savings from the delegation of tasks to EASME, 
the actual savings during the 2017-2020 period were found to be 24.7% higher compared to the initial 
specific financial statement estimates (EUR 98.0 million compared to EUR 78.6 million), and 43.8% 
higher compared to the initial cost-benefit analysis estimates (EUR 98.0 million compared to EUR 
68.2 million). As forecasted in the specific financial statement and the ex ante cost-benefit analysis, 
savings of the executive agency scenario resulted primarily from a higher share of lower-cost external 
personnel (Cas) employed within the Executive Agency and a lower overall number of staff. 

Including the cost of coordination and monitoring by the Commission and the costs covered from EEA/EFTA and third country contributions. 

151  
152   Lapatinas, A., Katay, G. Vollbracht, I., Funck, A., Grigoleto, M. (2020), CBA for the delegation of certain tasks regarding the implementation of 

Union Programmes 2021-2027 to the executive agencies: Qualitative Analysis 

62 

 
 
 
 
 
Efficiency and flexibility in the implementation of outsourced tasks 

According to the large majority of interviews (from both beneficiaries and parent DGs staff), EASME 
has  adjusted  its  operations  with  a  very  high  level  of  flexibility,  competence,  while  maintaining 
business continuity. This was especially relevant  given the impact of COVID-19 and the resulting 
need for flexibility.  

Moreover, key indicators with regards to efficient operation of EASME were within target. In terms 
of timeliness, the Time-To-Inform and Time-To-Grant were within target, and the share of payments 
executed in time was between 98.0% and 99.8%. EASME’s management and control cost was also 
below target and averaged 3.0% during the evaluation period.  

Simplification of the procedures used 

EASME has implemented several simplifications in IT tools and management procedures to improve 
efficiency in program management and implementation. This includes e-submission for procurement 
procedures (ePAD), delegation of tasks to lower levels, reducing paperwork, and using lump sums for 
parts  of  the  COSME  programme.  EASME  has  also  supported  the  Commission  Decision  to  use 
simplified  unit  costs  for  SME  owners  and  actively  contributed  to  the  next  Multiannual  Financial 
Framework  by  proposing  measures  to  increase  efficiency.  Although  improvements  were  made, 
efficiency gains could still be achieved by enhancing the interoperability of IT tools.  

Proximity of outsourced activities to final beneficiaries 

The  survey  indicates  that  most  beneficiaries  and  unsuccessful  applicants  are  satisfied  with  the 
relationships  they  have  with  EASME  staff,  both  in  terms  of  response  time  and  the  quality  of 
information received. However, there is still room for improvement in clarifying who applicants can 
contact during their application process.  

Visibility of the Community as a promoter of the Community programme concerned 

EASME ensured a high degree of visibility of the EU institutions, amongst others by co-hosting a 
multitude of events. The survey further indicates that 98% of applicants and beneficiaries are aware 
that their programme is funded through EU budget.  

Need to maintain an adequate level of know-how inside the Commission 

An  important  aspect  in  maintaining  an  adequate  level  of  know-how  inside  the  Commission  is 
organising feedback to policy from the agencies to the Commission. EASME provides feedback in 
both ad hoc and structured manners, depending on the parent Directorate-General. A good practice is 
the  creation  of  Communities  of  Practice  (COPs),  which  can  contribute  to  breaking  silos  between 
different funding programmes and foster synergies in feedback to policy. Another good practice are 
job-shadowing exercises which is very beneficial for mutual learning and facilitating collaboration. 
Staff mobility between the Commission and the agencies are also very beneficial for capacity building.  

To ensure an adequate level of know-how, a direct link between project and policy officers should be 
cultivated  and  a  structure  should  be  set  up  for  feedback  to  policy  with  clear  expectations  towards 
EASME.  

63 

 
 
 
 
 
ANNEX VI: STAKEHOLDER CONSULTATION - SYNOPSIS REPORT  

This annex presents a concise overview and conclusions on the work carried out during the evaluation 
of EASME. It covers both types of stakeholder consultation activities (surveys and interviews) carried 
out by the study team.  

In accordance with the requirements set out in the terms of reference and the Commission’s Better 
Regulation guidelines and toolbox153, it provides an outline of the consultation strategy, explains the 
implementation of each consultation activity undertaken and how they are aligned with the original 
consultation strategy, as well as summarising and comparing the results of the consultation activities. 

The  purpose  of  this  annex  is  to  inform  policymaking  about  the  outcome  of  all  the  consultation 
activities and inform stakeholders about how their input has been taken into account. 

1.  Outline of the consultation strategy  

Stakeholder  consultation  is  a  formal  process  by  which  the  Commission  and  its  contractors  gather 
information and views from stakeholders about the Commission’s policies. The consultation strategy 
should clearly define the scope of the consultation and specify which stakeholders are to be reached, 
through which consultation tool and for what purpose. 

In  designing  and  executing  the  consultation  strategy,  the  supporting  study  followed  the  relevant 
principles  and  steps  for  stakeholder  consultation  outlined  in  the  Commission’s  Better  Regulation 
guidelines.154 The main steps in the stakeholder consultation included: 

▪  designing the consultation strategy; 

▪  conducting the consultation work; 

▪ 

informing  policymaking  by  drawing  up  reports  (see  the  figure  below,  summarising  the 
approach to the design and execution of the consultation strategy). 

Design and execution of the consultation strategy 

Design of the consultation strategy

Setting 
consultation 
objectives

Mapping 
stakeholders

Determining 
consultation 
methods

Conducting consultation work

Surveys

Informing policy making

Interviews

Preparation of the evaluation 
reports

Presentations to the ISG and the 
steering committees of agencies

Source:  Study  supporting  the  triennial  evaluation  of  the  Agency,  based  on  the  Commission’s  Better  Regulation 
Guidelines. 

153   European Commission, Commission Staff Working Document. Better Regulation Guidelines. Brussels, 3.11.2021 SWD(2021) 305 final. Available 
at:  https://commission.europa.eu/system/files/2021-11/swd2021_305_en.pdf;  European  Commission,  ‘Better  regulation’  toolbox  –  November 
2021 edition. Available at: https://commission.europa.eu/system/files/2022-06/br_toolbox-nov_2021_en_0.pdf 
154   European Commission, Commission Staff Working Document. Better Regulation Guidelines, tools 53 and 54.  

64 

 
 
 
 
The purpose of the consultation carried out as part of this evaluation was to gather the views of key 
stakeholders and the data needed to inform responses to the evaluation questions. Given the scope 
of the agency’s activities, the consultation sought to include various categories of stakeholders, in 
particular: 

the staff of the agency;  

•  Commission officials from the parent DGs;  
• 
•  beneficiaries and unsuccessful applicants to the programmes managed by EASME; 
•  external experts contracted by the agency.  

The implementation of the consultation strategy was also coordinated with the evaluations of other 
agencies, to the extent necessary. 

The methods used for stakeholder consultation included two surveys and an extensive interview 
programme. During the evaluation, the contractor triangulated various data sources. The table 
below provides details of the types of stakeholders engaged for each consultation method, the 
results of which are presented in the forthcoming sections of this annex. 

Main information on stakeholder consultation activities 

CONSULTATION 
ACTIVITY  

From  

Survey 
A  
Survey 
B  

Interview programme  

TARGET GROUPS  

DATE  

Beneficiaries and unsuccessful applicants of the programmes 
managed by the agency  
External experts contracted by the agency  

1–13 December 2023 

1–13 December 2023 

-  Commission officials from parent DGs  
-  EASME Directorate  
-  Heads of sector and unit  
-  Representatives of the EASME staff committee  
-  Members of the EASME steering committee  
-  Beneficiaries, NCOs and unsuccessful applicants for the 

programmes managed by EASME  
-  External experts contracted by EASME  

1–13 December 2023 

Source: Study supporting the triennial evaluation of the Agency 

These different types of stakeholder consultation activities complement each other. While the survey 
of and interviews with unsuccessful applicants, beneficiaries and external experts informed questions 
regarding the agency’s effectiveness and efficiency, interviews with the Commission and EASME 
officials  provided  information  not  only  on  these  questions,  but  also  allowed  coherence-related 
evaluation questions to be covered. 

Consultation activity and evaluation criteria covered 

Effectiveness  

Efficiency  

Coherence  

Interviews with the European 
Commission  
Interviews with EASME 

✓  

✓  

✓  

Interviews with unsuccessful applicants, 
beneficiaries and experts  
Survey of EASME beneficiaries and 
unsuccessful applicants  
Survey of external experts contracted by 
EASME  
Source: Study supporting the triennial evaluation of the Agency 

✓  

✓  

✓  

✓  

✓  

✓  

✓  

✓  

✓  

65 

 
 
 
  
  
  
  
2.  Key results of consultation activities  

2.1 Summarised results of the survey  

The evaluation team implemented two surveys: survey A and survey B.  

The  design  of  survey  A  was  presented  to  applicants  and  allowed  us  to  differentiate  between 
beneficiaries and unsuccessful applicants.  

Survey B was presented to external experts.  

The  questions  in  both  surveys  were  carefully  crafted  to  ensure  comparability  across  groups  of 
respondents,  as  well  as  complementarity  with  the  other  evaluation  methods  used,  including 
interviews.  The  surveys  were  launched  for  EASME  and  were  active  for  around  2  weeks  on  the 
Alchemer survey tool. 

The  data  gathered  through  the  two  surveys  were  analysed  using  both  quantitative  and  qualitative 
methods.  Because  most  of  the  survey  questions  were  ‘closed’  (many  of  them  being  numerical  in 
nature, due to Likert-style response scales) and therefore result in structured responses, researchers 
primarily used Excel and statistical software (STATA, SPSS) to run descriptive statistics, such as 
cross tabulations of key variables to identify patterns across/among different groups of respondents.  

More detailed information on the survey methodology and tools used to process the data is presented 
in Annex 2.  

Main statistics related to the implementation of the surveys 

Target 
group 

Full 
launch of 
the survey 
(majority 
of all 
invitations 
sent on 
this date) 

Survey 
closure 
date 

Population 
targeted/no. 
of 
invitations 
sent out 

No. of responses 
received 

Response rate* 

No. of responses 
included in the 
analysis after 
cleaning 

Beneficiaries 

1 

13 

December 

December 

Unsuccessful 

1 

13 

applicants 

December 

December 

Partial*: 130 

Total: 13% 

Partial*: 60 

5 830 

Complete: 601 

Only including 

Complete: 571 

Total: 731 

Partial*: 92 

complete: 10% 

Total: 631 

Total: 8% 

Partial*: 42 

4 797 

Complete: 275 

Only including 

Complete: 266 

Total: 367 

complete: 6% 

Total: 308 

Experts 

23 

8 

November 

December  

4 300 

Partial*: 167 

Total: 29% 

Partial*: 0 

Complete: 1 055 

Only including 

Complete: 10 26 

Disqualified: 11 

complete: 25% 

Total: 1 026 

Empty: 3 

Total: 1 236 

Note: response rate was calculated using the formula:  

𝑅𝑒𝑠𝑝𝑜𝑛𝑠𝑒 𝑟𝑎𝑡𝑒 =

𝑁𝑜. 𝑜𝑓 𝑟𝑒𝑠𝑝𝑜𝑛𝑠𝑒𝑠 𝑟𝑒𝑐𝑒𝑖𝑣𝑒𝑑
𝑁𝑜. 𝑜𝑓 𝑖𝑛𝑣𝑖𝑡𝑎𝑡𝑖𝑜𝑛𝑠 𝑠𝑒𝑛𝑡 𝑜𝑢𝑡

∗ 100% 

Source: Study supporting the triennial evaluation of the Agency 

The survey data fed into all of the evaluation questions, in particular those aspects on which the 
opinions of respondents are of prime importance. All evidence from the surveys has been 

66 

 
 
 
incorporated into the evaluation’s final report. A summary of the findings from the survey data is 
presented below. 

Effectiveness 

During the evaluation period (January 2017 to March 2021) the overall level of satisfaction was 55% 
for unsuccessful applicants (102 out of 217) and 90% for beneficiaries (488 out of 548).  

They were satisfied with the overall quality of services provided by EASME.  

The  specific  statement  for  beneficiaries  was  ‘We  are  satisfied  with  the  overall  quality  of  the 
programme  management  services  provided  by  the  EASME  during  the  whole  application  and  (if 
relevant) project implementation period’. 

The slightly adapted version for unsuccessful applicants was ‘We are satisfied with the overall quality 
of the services provided by the EASME during the whole application process’.  

The level of satisfaction of the overall communication by and interaction with EASME during the 
application process and project implementation period (for beneficiaries) are similar to the overall 
satisfaction level: 89% for beneficiaries and 47% for unsuccessful applicants.  

In total, 73% (596 out of 838) of EASME’s unsuccessful applicants and beneficiaries indicated that 
they  will  certainly  apply  for  the  funding  again,  and  another  18%  (143  out  of  838)  will  “maybe” 
reapply.  

The underlying reasons for not considering applying in future are related to the low success rate of 
applications for the programmes managed by EASME and the cumbersome application procedure.  

Regarding the external experts contracted by EASME, 86% (870 out of 1 016) indicated that they 
would certainly like to work in this capacity for EASME in future, with 12% (124 out of 1 1 016) 
indicating “maybe”.  

The main reasons indicated for not considering this is that the total remuneration does not correspond 
to the actual time spent on the task and that the professional fee rate (currently 450€ /day) is too low.  

The agency’s unsuccessful applicants and beneficiaries rated various aspects related to its external 
communication  very  positively.  They  agreed  that  the  agency  provides  relevant  and  useful 
information, and that this information is user-friendly.  

Both groups indicated that they most often find information about the funding possibilities through 
the  Commission’s  website.  Whereas  other  communication  channels,  such  as  EU  delegations, 
advertising, media and social media were mentioned much less frequently.  

Regarding the effectiveness of communication channels used by EASME while communicating with 
its  beneficiaries,  direct  contact  via  email  was  identified  as  the  most  relevant  and  helpful 
communication tool, while other channels such as EASME’s social media was ranked less relevant.  

Video conferences and face-to-face contact were relevant channels for more than 50% of respondents 
(respectively 66% and 61% – meaning 289 out of 439 and 247 out of 405).  

There was  a  significant  percentage of  “Do not  know/ cannot  answer” responses for  the  following 
channels: 

•  social media (28% of total responses: 159 out of 571); 

• 

• 

face-to-face contact (30% of total responses: 172 out of 577); 

telephone contact (25% of total responses: 146 out of 581); 

67 

 
 
•  video conferences (24% of total responses: 138 out of 577).  

This might indicate that these channels are used by beneficiaries much less frequently. 

The most important factors mentioned by the respondents when dealing with EASME are: 

• 

• 

the accessibility and clarity of information provided by EASME (87%: 674 out of 777); 

the knowledge and courtesy of EASME’s employees (87%: 648 out of 743); 

•  EASME’s ability to perform the service promptly (86%: 640 out of 747); 

•  EASME’s willingness to help and cooperate with them and provide individualised attention 

(86%: 639 out of 745) 

50% (255 out of 586) of the beneficiaries underwent grant amendment procedures in the project.  

Only 13% (58 out of 439) of projects were audited and 30% of projects underwent monitoring.  

The respondents that were audited or underwent monitoring were very satisfied with the follow-up, 
checks  and/or  audit procedures  related to  the grant  (88%: 132 out  of 150,  with  another 6% being 
neutral: 11 out of 150). 

Efficiency 

Our analysis of the applicants and beneficiaries survey revealed that they were satisfied overall (83% 
satisfied:  700 out  of 842,  with  another 8% being  neutral: 69 out  of 842) with  the  amount  of time 
available to prepare and submit their applications.  

80%  (568  out  of  706)  of  respondents  agreed  (another  13%  being  neutral:  89  out  of  706)  that  the 
selection results were announced within the timeframe announced in the call for proposals.  

The same proportion of respondents declared the time period from the call deadline to the time the 
outcome of the proposal was announced was appropriate.  

Beneficiaries were also satisfied with the timeliness of the contracting process (85% satisfied: 473 
out of 557, with another 8% being neutral: 45 out of 557).  

Respondents  agree  that  EASME  was  efficient  in  implementing  COVID-related  provisions  (83%: 
between 403 and 484 out of between 447 and 609, with another 13% being neutral: between 26 and 
101 out of between 403 et 484 respondents). 

The  survey  shows  that  a  total  of  75%  (638  out  of  849)  of  respondents  agreed  (another  9%  being 
neutral:  76  out  of  849)  that  the  requirements  for  the  application  process  (e.g.  the  volume  of  the 
proposal, requirements for supporting documents, etc.) were reasonable and proportionate.  

84%  (484  out  of  576)  of  respondents  agreed  (another  7%  being  neutral:  46  out  of  576)  that  the 
application process was clear and transparent. 

Concerning  the  monitoring  and  reporting  processes,  EASME’s  beneficiaries  indicated  they  were 
largely satisfied that the project implementation and reporting requirements remained stable during 
the project execution: 89% (457 out of 515).  

They also agreed (87%: 443 out of 511) that the monitoring activities carried out by EASME staff or 
external experts working for EASME were useful for the implementation of the grant/project, and 
that the process of monitoring grants was sufficiently clear.  

The survey of beneficiaries revealed that respondents were satisfied with the grant conclusion process 
(80% satisfied: between 408 and 501 out of between 558 and 587, with another 9% being neutral: 
between 38 and 68 out of between 558 and 587 with the overall granting process).  

68 

 
 
However, satisfaction was lower with: 

• 

• 

• 

the  process  of  validating  beneficiaries  (77%  satisfied:  429  out  of  558,  with  another  11% 
neutral: 61 out of 558); 

the understanding of the contract (76% satisfied: 445 out of 587, with another 12% neutral: 
68 out of 587); 

the  user-friendliness  of  the  electronic  tools  used  for  the  validation  and  assessment  of 
beneficiaries (72% satisfied: 408 out of 565, with another 10% neutral: 58 out of 565). 

EASME’s  good  performance  in  terms  of  timely  processing  of  payments  was  also  reflected  in  the 
results of the beneficiaries’ survey: 

•  95%  (479  out  of  506)  were  satisfied  with  the  time  it  took  the  agency  to  make  a  pre-

financing payment; 

•  93% (308 out of 331) and 91% (203 out of 223) of them, respectively, were satisfied with 

the time it took to process interim and final payments.  

The survey showed a positive opinion from respondents on the simplifications already introduced in 
the programmes managed by the EASME, such as: 

• 

the simplification of reporting templates (79%: 541 out of 685); 

•  simplification of call guidelines and application forms (77%: 557 out of 717); 

• 

the establishment of a single point of contact (81%: 546 out of 674).  

However, other simplification measures would allow for a better and more efficient implementation 
of the programmes managed by EASME in the future. Respondents agree that other simplifications 
could be made, such as: 

•  simplifying granting and reporting requirements (88%: 478 out of 543); 

• 

• 

increasing the user-friendliness of IT tools for project management and reporting (87%: 668 
out of 768); 

increasing the user-friendliness of the IT tool for application submissions (86%: 681 out of 
793) 

34%  (268  out  of  832)  of  respondents  used  the  services  of  external  consultants  to  prepare  their 
applications. Among them, 50% (109 out of 219155) paid the consultancy firm between 1 and 10% of 
the budget proposal.  

20% (109 out of 544) of beneficiaries paid a consultancy firm to run administrative/management tasks 
for the project. The share of the project budget dedicated to these consultancy firms was, in most 
cases, between 1 and 5%.  

In most cases, administrative tasks required between 1 and 10% of the project budget and between 6 
and 10 FTEs. 

Coherence 

The survey of unsuccessful applicants and beneficiaries confirms that there are no issues with the 
visibility of the EU in terms of programmes managed by the agency. In total, 70% (631 out of 899) 

155   Some respondents did not answer follow-up questions. 

69 

 
 
 
 
of respondents agreed that they were aware that the management of the programmes was delegated 
to EASME by the Commission.  

An absolute majority of respondents (98%: 882 out of 901) are also aware that the programmes were 
funded from  the EU budget  and they  believe  that their project  participants  are also  aware  of this. 
Overall, 65% of respondents (583 out of 893) agreed that funding opportunities under programmes 
managed by the EASME are well promoted. 

It was not clear for close to half of respondents who they should contact with any questions they had 
or where to find help, both when preparing the application and when submitting the application (56% 
agreed with these statements: 430 out of 772).  

However,  the  agency  managed  to  ensure  that  its  staff  was  available  and  responsive  during  the 
contracting phase (86% of beneficiaries agreed: 481 out of 558) and the project implementation phase 
(89% of beneficiaries agreed: 472 out of 5309) – and especially when dealing with grant amendments 
(93% of beneficiaries agreed: 233 out of 250).  

The slightly larger level of satisfaction with help and information received during the contracting, 
implementation and amendment phases could be explained by the fact that unsuccessful applicants, 
who in general were more negative, did not assess these two statements. 

72%  (600  out  of  835)  of  respondents  agreed  (with  another  11%  neutral:  95  out  of  835)  that  the 
evaluation process was clear and transparent.  

However, the most negative responses were related to the outcomes of evaluation and feedback to 
unsuccessful applicants: 

•  only 41% (106 out of 258) agreed (another 16% being neutral: 40 out of 258) that the outcome 

of their application process was fair and they had no objections to it; 

•  only  41%  (106  out  of  260)  agreed  (another  16%  being  neutral:  42  out  of  260)  that  the 

explanation on why the application was rejected was clear; 

•  only  38%  (96  out  of  257)  indicated  (another  15%  being  neutral:  38  out  of  257)  that  the 

feedback on the application was useful and practical. 

2.2 Summarised results of the interview programme  

The  interview  programme  encompassed  scoping,  main  phase  and  follow-up  interviews  with  the 
representatives of the Commission and the agency, as well as beneficiaries, unsuccessful applicants 
and external experts contracted by the agency. As such, it was designed to embrace a wide variety of 
views.  

The interviews were conducted following a standardised questionnaire,  including questions on the 
evaluation  criteria,  adjusted  to  the  experience  of  individual  stakeholders.  Each  interview  was 
recorded, with interview notes and/or transcripts based on the recording. As initially planned, a total 
of 38 interviews was conducted, summarised in the table below.  

70 

 
 
 
 
 
 
EASME                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
Commission officials  
EASME 
National contact organisation 
Experts 
Unsuccessful applicants 
Beneficiaries 
Second rank beneficiaries 

Phases and participants of the interview programme  

Interview phase 

Target group 

No. of completed interviews (No. 
of interviewees) 

Exploratory/scoping 

Lead parent DG (DG GROW) 

Main phase 

EASME 
Parent DGs: 

DG GROW 
DG RTD 
DG for Climate Action 
DG for the Environment 
DG for Energy 
DG 
Networks, 
Technology 
DG  for  Maritime  Affairs  and 
Fisheries 

Communications 
and 
Content 

for 

2 (2) 

1 (1) 

2(3) 
2 (2) 
1 (1) 
1 (1) 
1 (1) 
1 (1) 

1 (1) 

Follow-up 

8 (12) 
1(4) 
1 (2) 
4(4) 
2 (2) 
2 (2) 
6 (6) 
2 (2) 
38 (47) 

Total amount of interviews 

35-41 

Source: Study supporting the triennial evaluation of the Agency 

The interview data fed into responses to the relevant evaluation questions, in particular the aspects of 
these  questions  where  respondents  opinions  are  of  prime  importance.  All  evidence  from  the 
interviews  was  incorporated  into  the  final  evaluation  report.  A  summary  of  the  findings  from  the 
interview data is presented below.  

Mission and governance 

Planning  and  programme  implementation.  Despite  improvements  observed  from  2017-2020, 
EASME officials continue to point out the lag in DG GROW’s announcement of programmes, while 
DG GROW cites insufficient input from EASME as a reason for delays in policy preparation.  

Interviewees pointed out that one of the primary obstacles faced by agencies is the delayed receipt of 
the  documents  related  to  calls  for  proposals  from  DGs.  There  is  a  deadline  for  using  the  agency 
operational  budget,  but  EASME  relies  on  the  parent  DGs  to  implement  the  calls,  leading  to 
inefficiencies. The key problem is that the DGs do not adhere to their stated schedule.  

Definition of programmes. Executive agencies have the potential to  improve cross-collaboration 
and  provide  strategic  advice  to  the  Commission.  Specifically,  EASME  is  well-connected  to 
companies  and  has  insight  into  market  practices  and  other  ground-level  stakeholders,  providing  a 
comprehensive understanding of industry needs and evolution. The Commission’s expertise in areas 
such as AI, digital, and quantum could be delegated to these agencies. By working closely together, 
the executive agencies and Commission could jointly develop work programmes. 

Importance of the steering committee. The steering committee played a crucial role in ensuring 
proper official oversight and financial management. It was essential to bring in the necessary expertise 

71 

 
 
 
 
 
to fulfil this role effectively. In addition to these responsibilities, the steering committee also served 
as a key facilitator for coordination and collaboration between the parent DGs and the exploitation of 
synergies between them.  

This was crucial for ensuring successful outcomes. However, the role of the steering committee varied 
between  different  programmes  and  initiatives.  For  example,  the  steering  committee  for  Horizon 
functioned differently compared to that for COSME. Nevertheless, both of these roles were critical 
to the success of their respective programmes. 

A  diversity  of  agencies.  Some  parent  DGs  have  expressed  the  opinion  that  there  were  too  many 
parent  DGs  for  EASME.  Initially,  the  mandate  for  the  executive  agencies  was  believed  to  be  the 
complete  financial  management  of  programmes.  To  increase  efficiency  and  reduce  Commission 
oversight, the mandate for executive agencies could be expanded. 

Effectiveness  

External  communication  and  information  on  funding  opportunities.  Several  interviewees 
mention that it is difficult to find relevant calls for their projects and to have an overview of all funding 
possibilities, especially for young researchers with little experience of EU programmes, and SMEs.  

The National Contact Points seem to be very valuable in this regard but are not always responsive. 
One interviewee suggests that the communication of funding opportunities could be improved and 
made more appealing by having videos explaining the core programmes and topics and where to find 
more  information.  Finally,  another  interviewee  pointed  out  the  lack  of  coordination  regarding 
communication initiatives between executive agencies.  

Communication  with  EASME.  Interviewees  (beneficiaries,  unsuccessful  applicants,  experts  and 
national  coordination  offices,  NCOs)  are  generally  very  satisfied  with  their  cooperation  and 
communication  with  EASME.  Multiple  interviewees  specifically  mentioned  the  value  of  project 
officers in answering questions in a timely manner. One NCO interviewee mentioned issues around 
high staff turnover, which made it unclear who was responsible for their project and who they could 
contact. Several NCOs also mentioned that there is a shortage of staff in the current EISMEA period 
responsible for Enterprise Europe Network leading to low response rates and a lack of guidance from 
EASME staff.  

COVID-19 responses. The COVID pandemic caused difficulties for new staff. Burnout rates among 
colleagues increased, according to interviewees. The pandemic also led to changes in the setup of the 
agencies and political priorities.  

Intended outputs and results. According to most interviewees in parent DGs, and supported by the 
cost-benefit analysis and key performance indicators, EASME was successful in accomplishing its 
primary goals from 2017 to April 2021, given the extensive and varied portfolio of programmes they 
managed.  

One  interviewee  from  the  Commission  notes  that  EASME  lacked  environmental  criteria  in  its 
programmes, such as the absence of requirements for Horizon projects to meet specific Green Deal 
criteria, such as a 5% reduction in emissions per year. The interviewee thinks environmental criteria 
should be clearer and not open to interpretation. The mandate of policy officers regarding the Green 
Deal and climate targets is not sufficiently explicit. 

Feedback to policy. There is no standard method for EASME to provide feedback to its parent DGs. 
Feedback  is  mainly  based  on  individual  exchanges  at  meetings,  rather  than  structured  reports. 
Commission  staff  have  varying  experiences  with  regards  to  agency  recommendations  for  work 
programmes, and DGs do not always ask for EASME’s input.  

72 

 
 
In the energy efficiency field, policymakers at the Commission appreciated the regular interactions 
with  EASME  staff,  as  EASME  provided  valuable  market  feedback.  The  flexible  structure  of  the 
agency allowed for this dynamic interaction. For policy-related projects without investment goals, 
EASME was also effective in providing support.  

The agency needs sufficient time to provide feedback to policy. According to a Commission official 
there are not enough resources directed towards feedback to policy and staffing should be increased 
in  this  regard.  Finally,  Commission  officials  interviewed  stress  that  agencies  should  have  key 
performance  indicators  for  feedback  and  that  the  views  of  the  parent  DGs  should  be  collected 
regularly.  

Need  for  long-term  key  performance  indicators.  While  agency  officials  found  that  key 
performance indicators (KPIs) effectively evaluate the effectiveness of programme implementation, 
there are currently no KPIs in place for procurement or feedback to policy. They would recommend 
developing KPIs to monitor these areas.  

Some financial units also find the current indicators cumbersome and not reflective of on-the-ground 
realities. Interviewees emphasised the importance of monitoring the long-term impacts of projects, 
with a focus on indicators that assess the impact up to 3, 5, or 10 years after implementation.  

Additionally,  some  interviewees  suggested  the  use  of  “meta-analyses”  which  would  involve 
aggregating and analysing the impacts of multiple projects over a longer period of time. This approach 
makes sense at the level of a specific programme or instrument and should be more widely promoted.  

Effectiveness  of  review  process.  Overall,  experts  find  that  the  evaluation  process  is  smooth  and 
effective.  Most  of  the  interviewed  experts  found  the  fee  they  were  paid  reasonable,  although  one 
person indicated that this is low – but that it remains valuable, as it gains them market intelligence. 
One suggestion  made  during interviews was that experts  from outside the EU should  be found to 
review larger projects that have a global impact. 

Efficiency  

Application  procedure.  Interviewees  are  generally  satisfied  with  the  application  procedure.  One 
interviewed NCO mentioned that information was not always shared with competitors outside the 
Enterprise Europe Network, making it difficult to enter the network if you are not already part of it. 
One  beneficiary  also  note  that  they  couldn’t  have  made  the  application  without  the  help  of  a 
consultant, and that the process is too difficult for SMEs.  

Time  to  assess  the  application.  Generally,  interviewees  are  satisfied  with  the  timeliness  of  the 
evaluation process. One beneficiary, however, noted that the process is faster in the United States, 
where they got a response within 30 days.  

Feedback on application. Some interviewees mention that feedback and the evaluation process in 
general is too dependent on individual reviewers. Having a panel of reviewers is a good practice in 
this regard. One beneficiary found their reviewer incompetent.  

Payments. The great majority of interviewees were satisfied with the payments and their timeliness. 
The  advance  and  mid-term  payments  are  highly  appreciated  by  beneficiaries.  Nonetheless,  one 
interviewee  (beneficiary)  mentions  that  the  retention  of  funding  until  the  end  of  the  project  is 
problematic for cascade funding directed towards SMEs and suggests that this should not apply for 
third-party funding.  

This was confirmed by another beneficiary of the EIC Pilot, who noted that the payments come too 
late for SMEs. Another interviewee (beneficiary) mentioned there is too little flexibility in payments: 
when one partner in the project consortium is not delivering, this leads to no one getting paid, which 
is regarded as unfair.  

73 

 
 
Follow-up audits. Beneficiaries that had been audited were satisfied with this experience. Agency 
officials also mentioned that the audit recommendations resulted in an increased workload rather than 
simplification. They pointed that intensifying reviews is a good way to be more aware of potential 
risks and address them. Nevertheless, interviewees indicated that the discussions with DGs regarding 
audits were quite tense and that they had to make a lot of effort to contest some of the findings.  

Administrative  burden  and  reporting  requirements.  Interviewees  find  that  the  administrative 
burden  is  reasonable,  often  finding  it  lighter  than  national  funding  programmes  –  and  that  it  has 
improved  over  time.  However,  some  interviewees  mentioned  that  more  trust  from  the  European 
Commission, and fewer reporting requirements, would be welcome, especially for SMEs.  

IT  tools.  Generally,  interviewees  are  satisfied  with  the  IT  tools  for  applications  and  project 
management. The necessary information is there, and the IT tools perform as they should. However, 
improvements can be made regarding the user-friendliness of IT tools: applicants and beneficiaries 
that are not used to the tool can get lost.  

Moreover, several NCOs mention that there is a lack of continuity in their IT tools: although they 
improved during the course of the framework programme, new tools emerged during the EISMEA 
period and learnings were not transferred, resulting in very poorly performing IT tools in the current 
EISMEA period.  

For NCOs such as the Enterprise Europe Network, continuity is vital, because they rely on the CRM 
(customer relationship management) functions included in their IT tool. One NCO also indicated that 
the tools could be more customised, as they are not always effective in very specific situations (e.g. 
when  a  partner  leaves  the  consortium),  and  that  a  more  integrated  platform  that  contained  both 
internal consortium information and Commission requirements/information would be helpful.  

Commission officials also mentioned that to efficiently manage SMEs and numerous small projects, 
IT should be more  customisable. Currently, the reliance on manual  processes outside of the tools 
leads to an excessive workload, three times greater than that of digitally managed calls.  

When designing a call, consideration should be given to its compatibility with IT systems. The initial 
implementation of COSME in the eGrants portal was not optimal but has since been improved with 
EISMEA. Assistance from IT specialists helps matters. The e-grant platform required a more detailed 
and adaptive approach to enforce the correct regulations, and it lacked the necessary flexibility to 
apply the appropriate regulations. 

Human resources. According to interviewees, floating colleagues have been effective in handling 
workload spikes and should be reinstated with a clearly defined work allocation, as was the case under 
EASME. Despite a tripled finance budget and an increasing workload, the same workforce struggles 
with multiple calls from several DGs, hindering simplification efforts.  

The  management  of  over  1 000  yearly  phase  1  grants  under  COSME  was  simplified  from  15 
managers to 5, through streamlined reporting, IT tools, standardised communication and automated 
reminders, resulting in improved payment and communication processes for beneficiaries.  

Interviewees  also  mentioned  the  separation  of  financial  units  from  operational  units.  Increased 
autonomy, better morale and increased communication were highlighted as the key advantages of the 
separation.  Conversely,  the  separation  also  negatively  impacted  day-to-day  collaboration  with 
operational units.  

Some  interviewees  of  the  EASME  staff  also  pointed  to  a  lack  of  human  resources  for  certain 
programmes. One reason given for this lack of resources is the constant changes in management at 
the agency. 

74 

 
 
 
Coherence  

Clarity  of  delegated  tasks.  EASME  staff  highlighted  the  issue  of  inconsistent  programme 
management  across  the  organisation.  Each  programme  is  managed  differently,  with  differing 
agreements between EASME and its parent DGs. This causes confusion among external stakeholders 
about  the  assignment  of  tasks  and  where  exactly  funding  comes  from.  EASME  staff  therefore 
suggested that the role of the agency be clearly defined, and that it includes involvement in drafting 
work  programmes.  Finally,  according  EASME  staff,  DGs  can  be  reluctant  to  seek  advice  from 
agencies, as they think they may not possess the necessary qualifications, which is unfortunate.  

According to interviewees from parent DGs, during 2017-2020 the actual operations of the agency 
were aligned with the formal regulations defining its mandate and operating procedures. Moreover, 
EASME was found to have fulfilled its mandate without interfering with policy development (the 
role of the Commission). Interviewees did report that the dispersal of tasks across agencies was more 
blurred after 2021.  

Collaboration between parent DGs and EASME. The quality of collaboration between the parent 
DGs and EASME differs between EASME units and parent DGs. The managers of the financial unit 
maintained  frequent  communication  and  held  numerous  meetings  with  the  relevant  Commission 
departments. They are very aware of the difficulties posed by the programmes and have established 
tight coordination both within the agency and with the managers of the financial unit.  

On the other hand, certain DGs pointed out that there was insufficient time and attention devoted to 
comprehending mutual expectations. It was perceived that the interactions between the parent  DG 
and EASME mostly occurred when there were problems, instead of having a more regular flow of 
information sharing and mutual learning. On the other hand, some DGs expressed high satisfaction 
with  the  flow  of  information  to  and  from  EASME  and  noted  that  the  structure  of  EASME  was 
effective in utilising experts who effectively gather important information and deliver it promptly to 
the parent DG. They attribute this success to EASME’s dynamic and adaptable structure.  

2.3 Comparison of the results of consultation activities  

The table below presents the key results per consultation activity, organised by evaluation criteria, as 
well as by the level of consistency, complementarity and contradiction of results across consultation 
activities.  

Overall, as summarised in the table below, there was a large degree of convergence in the results of 
different consultation activities. 

75 

 
 
 
 
Key findings of the consultation activities and their level of consistency, complementarity and contradiction  

Evaluation criteria 

Survey 

Interview programme 

Consistency of 
results across 
consultation 
activities 

Complementarity 
of results across 
consultation 
activities 

Contradiction of 
results across 
consultation activities 

 High 

 High 

 Low 

Effectiveness  

Efficiency  

Overall, respondents were satisfied with 
the services provided by EASME and 
indicate they will apply for funding again 
in future.  

Generally, stakeholders found that 
EASME operates effectively and that its 
primary goals were met.  

Target stakeholders are also satisfied 
with the communication and clarity of 
information provided by the agency.  

In particular, the effective communication 
with EASME staff was appreciated by 
beneficiaries and national contact 
organisations.  

Respondents who underwent grant 
amendments or were audited were also 
satisfied with the follow-up, checks 
and/or audit procedures connected to the 
grant. 

To increase effectiveness, interviewees 
recommend the development of key 
performance indicators, to monitor the 
long-term impact of projects and enable 
feedback to policy. 

Stakeholders found that EASME operates 
efficiently overall.  

 High 

 High 

 Medium 

The administrative burden was also found 
to be reasonable.  

One area for improvement is the IT tools, 
especially their user-friendliness and 
customisation. 

Beneficiaries surveyed were satisfied 
with the timeliness of the contracting 
procedures and found monitoring 
activities to be useful.  

Payments were deemed to have been 
processed in time and simplification 
measures to have been effective.  

One third of respondents required 
external help from consultants to prepare 
applications, indicating cumbersome and 
highly complex administrative 
requirements, among other things.  

76 

 
 
 
 
 
Coherence  

Most respondents said they were aware 
of the general project implementation 
mandate given to EASME, and that 
projects are funded from the EU budget.  

However, less than half of potential 
applicants surveyed were aware of how 
tasks were divided between EASME and 
the Commission.  

There is a low level of coherence in the 
management of the different programmes 
and the agreements with different DGs.  

 High 

 High 

 Medium 

It was noted that there should be clearer 
and more aligned delineations in roles 
between the DGs and EASME, and that 
closer collaboration between the two could 
be beneficial. 

Unsuccessful applicants and beneficiaries 
said they did not to know who to contact 
at the agency for clarification and 
support questions.  

Source: Study supporting the triennial evaluation of the Agency 

77