Report on the 3-year (2015-2018) evaluation of operations of the European Research Council Executive Agency (ERCEA)
Date: 2018-03-13
Institutions: Directorate-General for Research and Innovation, European Commission
Subjects: EU research policy



IMMC.Ares%282018%291381211.ENG.xhtml.PART-2018-149625V1.docx













EVALUATION ROADMAP






title of the evaluation/fc




Triennial (16 July 2015 – 15 July 2018) evaluation of the European Research Centre Executive Agency (ERCEA).






lead dg – responsible unit




DG RTD R4 "New management modes" 




date of  this roadmap




03.2018






type of evaluation




Evaluation






planned start date


planned completion date




Q1 / 2018



Q2 / 2019












Planning calendar





http://ec.europa.eu/smart-regulation/evaluation/index_en.htm



(to be published)






This indicative roadmap is provided for information purposes only and is subject to change.










A. Purpose  






(A.1) Purpose 






The purpose of this periodical (every 3 years) evaluation of the operation of European Research Centre Executive Agency (ERCEA) is to assess ERCEA’s implementation of the parts of the EU funding programmes which have been entrusted to it.


ERCEA is governed by the 

Framework Regulation for executive agencies

, its 

Establishing Act

, which sets out its mandate; the 

Act of Delegation

, which specifies the tasks to be carried out by, and the powers delegated to the Agency in order to perform its mandate; and the decision establishing the ERCEA Steering Committee.


In line with the Commission's Better Regulation principles
1
, the evaluation applies several standard evaluation criteria. The evaluation will assess whether the Agency has fulfilled its tasks in an effective and efficient way, whether there are overlaps / gaps / inconsistencies in the management of the programme portfolio by the Agency, and whether there is a clear delineation of tasks between ERCEA and the parent DG or other executive agencies (coherence)
2
.  


The results of this evaluation will be summarised in an Evaluation Report to be established by the Commission and presented to the ERCEA Steering Committee, the European Parliament, the Council and the Court of Auditors in accordance with Article 25(1) of the Framework Regulation.


The evaluation will help the Commission services to assess whether the functioning of the Agency has yielded the expected positive results as estimated in the ex-ante Cost-Benefit Analysis for the delegation of tasks to the executive Agencies
3
 and identify potential areas of improvement. 


The results will feed into a future Cost-Benefit Analysis to assess the opportunity of expanding / modifying the executive Agency's mandate under the post-2020 Research and Innovation Framework Programme or after the expiry of the current mandate (31/12/2024).


Following the evaluation, the Commission and the Agency will take all appropriate steps to resolve any problems identified
4
. If, further to the evaluation, the Commission finds that the very existence of the Agency is no longer justified with a view to sound financial management, the Commission shall decide to wind up the Agency
5
 






(A.2) Justification






The triennial evaluation of ERCEA is a legal obligation stemming from Article 25(1) of the 

Framework Regulation for executive agencies

, requiring the Commission to carry out an external evaluation of the operation of each Executive Agency every three years.












B. Content and subject of the evaluation






(B.1) Subject area






The subject area is the functioning of ERCEA between 16 July 2015 and 15 July 2018.


ERCEA was set up by Commission Decision 2008/37/EC of 14 December 2007 for the management of the specific Community programme Ideas
                           6
 in the field of frontier research in application of Council Regulation (EC) No 58/2003 for a period beginning on 1st January 2008 and ending on 31st December 2017.


Further to the adoption of the Horizon 2020 Framework Programme for Research and Innovation
                           7
, the Commission re-established ERCEA by Decision 2013/779/EU to replace and succeed the Executive Agency established by Decision 2008/37/EC. 


ERCEA is responsible, under the supervision of the Commission, for the implementation of the following EU programmes in parts or fully:


Within the Horizon 2020 Framework Programme: the specific objective ‘Strengthening frontier research through the activities of the European Research Council’ of Part I ‘Excellence Science’, 


Within the Seventh Framework Programme, the implementation of the legacy of the Specific Programme ‘Ideas’.


ERCEA has one parent-DG, namely Directorate-General for Research and Innovation (DG RTD).


The Agency works closely with the Common Support Centre for Horizon 2020.






(B.2) Original objectives of the intervention







The outsourcing of certain management tasks to the executive agency according to the Framework Regulation and ERCEA's Establishing Act intends:


To allow the Commission to focus on its institutional tasks, i.e. tasks assigned to the institutions by the Treaty which require discretionary powers in translating political choices into action. Such institutional tasks should not be outsourced.


To enable the Commission to achieve the objectives of the delegated EU programmes more effectively and efficiently. According to the Cost-Benefit Analysis carried out in 2013, delegating tasks to ERCEA was estimated to be more cost-efficient than an in-house scenario. In addition the CBA showed that the Agency has high quality programme management and service delivery, visibility and existing communication outreach channels which have proved effective so far. Making use of the accumulated experience and expertise of the Agency would lead to efficiency gains. 







(B.3) How the objectives were to be achieved






In order for the objectives to be achieved, the Agency was made responsible, under the 2014-2020 Multi-Annual Financial Framework, for the implementation of the parts of the Union programmes outlined in section B.1.


ERCEA had to carry out the following tasks:


managing programme implementation and specific projects on the basis of the relevant work programmes established by the Scientific Council of the European Research Council (hereinafter referred to as the ERC Scientific Council) and adopted by the Commission, where the Commission has empowered it to do so in the instrument of delegation;


adopting the instruments of budget execution for revenue and expenditure and carrying out all the operations necessary for the management of the programme, where the Commission has empowered it to do so in the instrument of delegation; 


providing support in programme implementation where the Commission has empowered it to do so in the instrument of delegation; 


providing support to the ERC Scientific Council in the conduct of all of its tasks.


These tasks are further detailed in the ANNEXES to the Act of Delegation.












C. Scope of the evaluation






(C.1) Topics covered







A main topic will be to evaluate to which extent the original objectives of the delegation of Commission tasks to ERCEA have been met (see section B2 above). To this end the estimations of the Cost-Benefit Analysis of 2013 will be tested to provide evidence on the validity of the assumptions made in the ex-ante scenario by considering the actual costs and benefits of programme implementation by the Agency in a structured way. 


The aspects to be covered by the Cost-Benefit Analysis are specified in Article 3(1) of the Framework Regulation7 and the Commission Guidelines for the establishment and operation of executive agencies8.


The evaluation shall not cover the achievements of the programmes managed by ERCEA, which are subject to mid-term and ex-post evaluations themselves. The evaluation will nevertheless provide useful input for the preparation of the successor programmes post-2020, considering that the performance of the agency affects the efficiency and effectiveness of the programmes it manages. 


It will in particular look at whether the alignment of programme portfolio with the Agency’s core competences and the development of its brand identity brought the estimated qualitative benefits; whether the used instruments guaranteed consistent service delivery and whether there is scope for simplification and further efficiency gains.


The evaluation will cover the operation of ERCEA between 16 July 2015 and 15 July 2018.






(C.2) Issues to be examined







The below questions indicate possible issues this evaluation will address: 


Effectiveness – to what extent have the objectives of delegating programme management tasks to ERCEA been achieved? 



To what extent was ERCEA operating according to the legal framework establishing it?


To what extent has ERCEA achieved its objectives? What, if anything, could be done to render ERCEA more effective in achieving these objectives?


To what extent has ERCEA contributed to improved management of the programmes in terms of the elements assessed in the 2013 Cost-Benefit Analysis (see B2 above)?



Efficiency – to what extent has ERCEA carried out its tasks efficiently in line with estimations?



Did the actual costs (including cost of coordination and monitoring) of ERCEA correspond to the estimates of the 2013 CBA which was carried out in view of its establishment? If not, what are the reasons behind? 


Did the actual benefits correspond to the estimates of the 2013 CBA? If not, what are the reasons behind?


Was the management and execution of the programmes by the Agency cost-effective as compared to the alternative options?


To what extent has ERCEA contributed to improved management of the programmes in terms of simplification of the procedures and flexibility in the implementation of outsourced tasks (e.g. capacity to adapt to periods of high workload)?


Which further scope for simplification exists?


What aspects/means/actors or processes render ERCEA more or less efficient? What could be improved?


To what extent has ERCEA internal organisation and procedures been conducive to its efficiency? 


Is the size of ERCEA adequate? Is the balance between operational staff and administrative support appropriate?


Coherence – to what extent there is consistency between the tasks managed by ERCEA and those managed by the parent DG and other executive agencies, and has the Commission been able to maintain an adequate level of know-how in relation to the programmes entrusted to the Agency?


Are there overlaps/ gaps/ inconsistencies within the programme portfolio managed by ERCEA?


Is there a clear and appropriate delimitation of responsibilities and tasks between ERCEA and the Parent-DG? Are there overlaps or gaps?


To what extent has ERCEA enabled the Commission to better focus on its policy related tasks?


Are there any governance (financial and policy) issues in relation to the implementation of the tasks that are delegated to the executive agencies/ issues as regards the Commission's services ability to steer EU policy or budget implementation?


Are appropriate mechanisms and instruments put in place to ensure an adequate coordination and information flow between ERCEA and the Commission services?


Does ERCEA provide useful information in support of the policy process?







(C.3) Other tasks






N.A.






D. Evidence base






(D.1) Evidence from monitoring 






The following data will provide a key input for the evaluation:
                              


ERCEA's establishment act and act of delegation


Memorandum of Understanding


Strategic plans and annual management plans of the parent DG – DG RTD


Annual work programmes of ERC and ERCEA


Written Agreement between the European Research Council Executive Agency and the Directorate-General for Research and Innovation, Modalities and Procedures of Interaction, 17 December 2015


Operational Guidelines between the European Research Council Executive Agency and the Directorate-General for Research and Innovation of 21 May 2012


Annual activity reports of the parent DG 


Annual and bi-annual activity reports of the Agency


Minutes of the steering committee meetings


Minutes of ERC Board of Directors meetings


Minutes of the ERCEA Dir. A. meetings


Audit reports (Court of auditors)


Human resources related data







(D.2) Previous evaluations and other reports







Cost Benefit Analysis for the delegation of certain tasks regarding the implementation of Union Programmes 2014-2020 to the Executive Agencies, 19 August 2013, GFK in cooperation with Technopolis 




Mid-term evaluation of Horizon 2020 programme

 and its annexes


Evaluation of the Operation of ERCEA (2012-2015), PPMI (December 2015). This evaluation also included an update of the CBA of 2013.


Evaluation of the first three years of operation of ERCEA covering the period from July 2009 until July 2012, Deloitte (June 2013)






(D.3) Evidence from assessing the implementation and application of legislation  (complaints, infringement procedures)






N.A.






(D.4) Consultation






The Commission envisages a consultation of ERCEA's relevant stakeholders as part of this evaluation in the 2nd part of 2018.


The main stakeholders of the Agency are the Commission Services, the Member States, regional and national authorities, and the beneficiaries /applicants of the programmes managed by the Agency.


The targeted surveys and interviews to be performed in the context of the evaluation will collect views and data from the identified stakeholder categories on the modus operandi of the Agency in each programme, with particular focus on those aspects of the Agency's effectiveness, efficiency and coherence that are accessible to the stakeholders (access to and quality of information, time and quality of response to communications, time and quality of decisions, time and quality of reporting etc.)



Open public consultations

 have already been carried out for the finalised and ongoing evaluations of part of the programmes managed by the Agency. Relevant results for the Agency will be considered in this evaluation. 


A synopsis report in English, summarising the results of all consultation activities will be published on the public consultation page once all consultation activities are closed.






(D.5) Further evidence to be gathered






N.A.














E. Other relevant information/ remarks






An external independent contractor will be involved whose tasks will among others focus on the collection of data (including targeted surveys and interviews) and other factual information and their analysis for the purposes of this evaluation.






 




For internal use only – not for publication 






Planning 






Key milestones (indicative)






Steering Group set up




Q1 2018






Public Consultation




N/A






Other Stakeholder Consultation




Q3 2018






Submission to Regulatory Scrutiny Board 




If applicable






Final Report 




 December 2018






ISC launch




1Q 2019






Deadline for Report to Council and European Parliament




Q1 2019






Dissemination Plan




N/A






Action Plan




Q2 2019











(1)
 
                        Commission's Better Regulation Communication COM (2015) 215 final and its accompanying SWD (2015) 111 final and SWD (2015) 110 final.



(2)
 
                        The assessment of the evaluation criterion 'EU added-value', i.e. why the EU should act, is not perceived to be a relevant criterion for the evaluation of ERCEA as ERCEA carries out tasks which the European Commission has transferred to it. The EU added-value of the programmes that ERCEA manages is assessed in the context of the programmes evaluations. The previous needs which ERCEA were meant to address and whether they still are pertinent today (relevance) will be presented in the context of the evaluation.



(3)
 
                        Cost-Benefit analysis for the delegation of certain tasks regarding the implementation of Union Programmes 2014-2020 to the executive agencies - Final report for the Commission of 19/08/2013, GFK in cooperation with Technopolis.



(4)
 
                        Article 25(3) of Regulation 58/2003.



(5)
 
                        Article 25(4) of Regulation 58/2003.



(6)
      Cf. Council Decision 2006/972/EC of 19 December 2006 concerning the specific Programme ‘Ideas’ implementing the Seventh Framework Programme of the European Community for research, technological development and demonstration activities (2007-2013) (OJ L54, 22.2.2007) 



(7)
     Cf. Regulation (EU) N° 1291/2013 of 11/12/2013 of the European Parliament and of the Council establishing Horizon 2020 – The Framework Programme for Research and Innovation (2014-2020) (OJ L 347, 20.12.2013).