REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL AND THE COURT OF AUDITORS Evaluation of the Consumers, Health, Agriculture and Food Executive Agency (CHAFEA), the Executive Agency for Small and Medium-Size Enterprises (EASME), the Innovation and Networks Executive Agency (INEA), the Education, Audiovisual and Culture Executive Agency (EACEA), the Research Executive Agency (REA) and the European Research Council Executive Agency (ERCEA).
Date: 2023-03-10
Institutions: Directorate-General for Research and Innovation, European Commission
Subjects: EU research policy, common agricultural policy, common transport policy, education policy, energy policy, food safety, health policy, rural development, young person, youth policy



IMMC.Ares%282023%291759177.ENG.xhtml.PART-2023-188606V1.docx











CALL FOR EVIDENCE


FOR AN EVALUATION / FITNESS CHECK 






This document aims to inform the public and stakeholders about the Commission’s work, so they can provide feedback and participate effectively in consultation activities.


We ask these groups to provide views on the Commission’s understanding of the problem and possible solutions and to share any relevant information that they may have.







Title of the evaluation




EU Executive Agencies – three-year evaluation 






Lead DG – responsible unit 




Directorate-General for Research and Innovation / Unit Common service for Executive Agencies & Funding Bodies






Indicative timetable


(planned start date and completion date)




Q3-2022- Q2-2023






Additional information




European Education and Culture Executive Agency (EACEA)


Executive Agency for Small and Medium-sized Enterprises |(EASME) 


European Research Executive Agency (REA)


European Research Council Executive Agency (ERCEA)


Consumers, Health, Agriculture and Food Executive Agency (CHAFEA)


Innovation and Networks Executive Agency (INEA)| 






This document is for information purposes only. It does not prejudge the final decision of the Commission on whether this initiative will be pursued or on its final content. All elements of the initiative described by the document, including its timing, are subject to change.






Political context, purpose and scope of the evaluation






Political context 






Under Article 25 of Council Regulation (EC) No 58/2003 laying down the statute for Executive Agencies (Agencies), an external evaluation of each agency must be carried out every 3 years and include a cost-benefit analysis. In line with Article 12 of Commission Implementing Decision (EU) 2021/173, CHAFEA, INEA, REA, ERCEA, EACEA and EASME will be evaluated in a coordinated way. These evaluations cover the second period of the 2014-2020 mandate of the Agencies that has not yet been evaluated, between 2017 and 2021.


The evaluation will give an insight into the operation of the Agencies, will help the Commission assess the performance of the executive Agencies and identify potential areas for improvement in the new programming period. The evaluation will be shared with the EA’s steering committees, the European Parliament, the Council and the Court of Auditors.


The current Agencies established under the EU’s 2021-2027 long-term budget (also known as the multiannual financial framework) are not in this evaluation’s scope. For 2021-2027, the Commission has created a new set of EU programmes
1
. This required a change to the distribution of portfolios among executive Agencies and resulted in the transfer of the activities of CHAFEA to Agencies in Brussels with similar thematic and programme activities. In order to ensure a smooth transition to the new EU programmes, the following Agencies were established on 1 April 2021: (a) the European Climate, Infrastructure and Environment Executive Agency; (b) the European Research Executive Agency; (c) the European Innovation Council and SMEs Executive Agency; (d) the European Education and Culture Executive Agency; (e) the European Research Council Executive Agency; and (f) the European Health and Digital Executive Agency. These newly established Agencies will be subject to the next 3-year evaluation.


 This initiative will evaluate the following six executive Agencies over the 2017-2021 period:


•
                              CHAFEA (1 January 2017 to 31 March 2021);


•
                              EACEA (1 January 2018 to 31 March 2021);


•
                              EASME (1 January 2017 to 31 March 2021);


•
                              ERCEA (16 July 2018 to 31 March 2021);


•
                              INEA 1 January 2017 until 31 March 2021;


•
                              REA (16 July 2018 until 31 March 2021).






Purpose and scope






In line with the Commission's better regulation principles
2
, the evaluation applies the standard evaluation criteria of effectiveness, efficiency, coherence
3
. The evaluation will assess whether the Agencies have fulfilled its tasks in an effective and efficient way, whether there are overlaps, gaps or inconsistencies in the management of the programme portfolio by the Agencies, and whether there is a clear delineation of tasks between the Agencies and with the parent DGs (coherence). The evaluation will be supported by a study carried out by an external contractor. 


The evaluation will help the Commission assess if the Agencies’ work has produced the expected results estimated in the cost-benefit analysis of 2013 for the delegation of tasks to the Agencies and will assess to what extent there are other areas for further improvement. The evaluation will aim to identify potential further synergies in implementing the delegated parts of the specific programmes entrusted to the Agencies
4
.


The main question to be answered is what more could be achieved in terms of financial savings, improved services and other synergies and efficiency gains by delegating implementation tasks to Agencies compared to carrying them out by the Commission. The evaluation will not focus on the operational achievements of the programmes which have been covered by the parent DGs’ mid-term and final programme evaluations.







B. Better regulation






Consultation strategy Why we are consulting?-






The consultation strategy aims to collect information, data and knowledge on how the Agencies perform their tasks. The targeted consultation will collect views and data from the identified stakeholder categories on the Agencies’ working methods for each programme. There will be a particular focus on the Agencies’ efficiency (access to and quality of information, time and quality of response to communications, time and quality of decisions, time and quality of reporting). A summary of the results of all the consultation activities will be annexed to the evaluation report. 


A survey, together with interviews, will be conducted among the funding beneficiaries and applicants to assess the quality of services delivered by the Agencies. The survey will be representative of the different programmes managed by each Agency and the individual programme strands in terms of their characteristics (e.g. annual/multiannual and small/large projects, type of beneficiaries). Furthermore, the evaluation approach and survey will ensure the results are comparable with the satisfaction surveys previously carried out and wherever possible, the evaluation will emphasise lessons learnt that can be applied to future programmes. Public consultations have been carried out to evaluate the underlying programmes and will therefore be complemented by targeted stakeholder consultations.







Target audience






The Commission plans to consult the six Agencies’ stakeholders as part of this evaluation and gather their opinions and views during the period being evaluated. The stakeholder consultation design comprises surveys of unsuccessful applicants, beneficiaries and external experts who assisted the Agencies during the evaluation period. There will also be extensive interviews with relevant Commission departments and staff of the Agencies.







Data collection and methodology






Taking into consideration the nature and purpose of this evaluation, the external contractor will carry out an evidence-based evaluation. The gathering and analysis of data will rely on the logic of triangulation
5
. The answers to each of the evaluation questions will rely on several sources of information. Conclusions and recommendations will be discussed with stakeholders to ensure a broad agreement and understanding of the evaluation process and results. The contractor will use a mix of different methods in the evaluation process to connect the qualitative evidence – based on documentary review, desk research, interviews, surveys and the qualitative cost-benefit analysis – with the quantitative evidence, based on administrative and monitoring data, surveys and the quantitative cost-based analysis.


The following data will be the key input to the evaluation: Agencies’ establishment acts and acts of delegation; strategic plans and annual work programmes of the parent DGs; annual activity reports of the Agencies and parent DGs and programme statements of the relevant programmes; memoranda of understanding between each Agency and its parent DGs; minutes of the regular management meetings of the Agencies; interim reports to the parent DGs; agendas and minutes of steering committee meetings; human resources data; annual work programmes of the Agencies for the period 2017-2021; annual work programmes for the underlying framework programmes delegated to the Agencies; the Agencies’ manuals of procedures; recent evaluation reports of framework programmes delegated to the Agencies; reports of the European Court of Auditors and the Commission’s Internal Audit Service (IAS); and the cost–benefit analysis.












(1)
 Commission Implementing Decision (EU) 2021/173 of 12 February 2021 set up the Agencies implementing EU programmes in the 2021-2027 multiannual financial framework,.



(2)
 
https://ec.europa.eu/info/files/better-regulation-guidelines_en
 
https://ec.europa.eu/info/better-regulation-toolbox_en



(3)
 The evaluation criteria are effectiveness, efficiency, coherence, relevance and EU added value. However, only three of them are applicable to this specific evaluation because relevance and EU added value are assessed as part of the evaluation of the programmes themselves and not their implementation by the Agencies.



(4)
 The Commission decided in the meantime, after a cost-benefit analysis carried out in 2020, to set up a new generation of Agencies.



(5)
 Triangulation involves examining an area from different points of view, e.g. corroborate interview findings with survey results and desk reviews to confirm a conclusion. It is used to improve the quality of evaluations and to make analyses more reliable.