EUROPEAN COMMISSION Brussels, 30.8.2017 SWD(2017) 287 final COMMISSION STAFF WORKING DOCUMENT Interim Evaluation of the Union Civil Protection Mechanism (2014-2016) Accompanying the document Report from the Commission to the European Parliament and the Council on the Interim Evaluation of the Union Civil Protection Mechanism for the period 2014-2016

{COM(2017) 460 final}



DG SANTE Directorate-General for Health and Food Safety EASO European Asylum Support Office EC European Commission ECA European Court of Auditors ECDC European Centre for Disease Prevention and Control EEA European Economic Area EEAS European External Action Service EERC European Emergency Response Capacity EFAS European Flood Alert System EFFIS European Forest Fires Information System EFTA European Free Trade Association EMC European Medical Corps EMS Emergency Management System EMSA European Maritime Safety Agency ENI European Neighbourhood Instrument ENP European Neighbourhood Policy ERCC European Response Coordination Centre Evaluation Interim evaluation IFRC International Federation of Red Cross and Red Crescent Societies IOM International Organization for Migration

IPCR Integrated Political Crisis Response ISAA Integrated situational awareness and analysis ISG Inter-service Steering Group JRC Joint Research Centre Mechanism Union Civil Protection Mechanism MIC Monitoring and information centre ModEx Module exercise OECD Organisation for Economic Cooperation and Development OPC Open Public Consultation OPCW Organisation for the Prohibition of Chemical Weapons PPRD Prevention, Preparedness and Response to Natural and Man-made Disasters ToR Terms of Reference SWD Commission Staff Working Document UCPM Union Civil Protection Mechanism UN United Nations Union Mechanism Union Civil Protection Mechanism UNISDR United Nations Office for Disaster Risk Reduction 1INTRODUCTION 2Purpose of the evaluation This Staff Working Document presents the results of the interim evaluation (hereafter the 'Evaluation') of the implementation of Decision No. 1313/2013/EU of the European Parliament and the Council of 17 December 2013 (hereafter the 'Decision') on a Union Civil Protection Mechanism (hereafter 'the Union Mechanism' or 'UCPM'). The purpose of the Evaluation is to:

1) assess the performance of the Union Mechanism in order to determine progress towards the general and specific objectives set out in Article 1 and 3(1) of the Decision during the first half of its implementation period (i.e. 2014-2016); 2) identify any gaps or shortcomings in the current legislative framework; 3) put forward recommendations to increase the Mechanism's effectiveness and improve the implementation of the existing framework; 4) provide input for possible proposals to amend the Decision or implementing acts adopted thereof; 5) if appropriate, inform the review of the financial breakdown of the Union Mechanism as set out in Article 19(5) of the Decision. Furthermore, in accordance with point (a) of Article 34(2) of the Decision, the Commission is required to report to the European Parliament and the Council on the results obtained and the qualitative and quantitative aspects of the implementation of the Decision by no later than 30 June 2017, and to put forward, if appropriate, proposals for amendments to this Decision. To this end, the Directorate-General for European Civil Protection and Humanitarian Aid Operations (DG ECHO) contracted an external consultant to carry out an independent evaluation of the activities implemented under the Union Mechanism. As the reflections on the future of the Union Mechanism under the next Multi-annual Financial Framework (2020-2027) are starting, findings from the Evaluation could usefully contribute to developing a future impact assessment, as recommended by the EU Better Regulation Agenda. 3Scope of the evaluation As set out in the Interim Evaluation Roadmap, the Evaluation covers all activities carried out under the Union Mechanism in the timeframe spanning from January 2014 to December 2016. Actions targeting the UCPM Participating States, as well as those performed in relation to third countries, including in accordance with Article 28(2), have been covered by the Evaluation. Article 3 of the Decision spells out four specific objectives against which the Union Mechanism should be evaluated and their respective results indicators. These indicators encompass the spectrum of activities of the Union Mechanism across the fields of: (i) disaster prevention, (ii) preparedness, (iii) and response to natural and man-made disasters. In addition, cross-cutting activities relating to prevention, preparedness and response are included in the scope of the Evaluation. The Evaluation has addressed the five mandatory criteria set out in the European Commission's (“the Commission”) Better Regulations Guidelines: (i) Relevance; (ii) Effectiveness; (iii) Efficiency; (iv) Coherence; and (v) EU added value. In addition, one additional criteria, (vi) Sustainability, has been added to the scope of this evaluation. 4BACKGROUND TO THE INITIATIVE 5Baseline Prior to the entry into force of the Lisbon Treaty in 2009, EU civil protection actions and legislation were based on the catch-all provision of Article 308 of the EC Treaty, authorising the Council of the European Union (“the Council”) to act (by unanimity) where necessary in order to attain a Treaty objective in areas where there the EC Treaty provided no other legal basis. In line with the need to ensure better protection in disasters occurring both inside and outside the EU, the Union Mechanism was established in 2001 by the Council and subsequently recast in 2007. With the advent of the Treaty of Lisbon in 2009, civil protection became a self-standing policy with its own legal basis provided by Article 196 of the Treaty on the Functioning of the EU, with the aim to: a.Support and complement Member States' action at national, regional and local level in risk prevention, in preparing their civil-protection personnel and in responding to natural or man-made disasters within the Union; b.Promote swift, effective operational cooperation within the Union between national civil protection services; c.Promote consistency in international civil protection work.

Several challenges needed to be addressed at the time. The Impact Assessment of the civil protection regulatory framework carried out in the context of the preparation of the Decision had highlighted, among others, the following shortcomings. ·Prevention: Lack of coordination between sector specific prevention policies; under-developed organisation for the sharing of experience; insufficient links between prevention, preparedness and response; limited awareness, vision and strategic investments. ·Preparedness: Lack of consistency among national training programmes hindering the sharing of experience; limited scope of EU preparedness action and limited links with national programmes, leading to lower preparedness levels; uncertain level of preparedness for major emergencies. ·Response: Reactive and ad-hoc mandate of the Mechanism limiting the effectiveness, efficiency and coherence of the European disaster response; some needs not being met due to the lack of specific assets (capacity gap); limited financial and logistical transport solutions slowing down or hindering optimal response; burdensome administrative procedures jeopardising a swift and efficient response. Based on a Commission proposal, the European Parliament and the Council repealed previous legislation and adopted Decision No 1313/2013/EU of the European Parliament and of the Council of 17 December 2013 on a "Union Civil Protection Mechanism". This Decision is currently in force and governs actions and activities in the field of civil protection. 6Description of the initiative and its objectives Decision 1313/2013 added to the Mechanism's original focus on disaster response, a greater emphasis on cooperation for disaster prevention and preparedness with particular focus on national risk assessment and risk management planning. To enhance preparedness of European countries to respond to disasters, the Mechanism created a Voluntary Pool of response capacities pre-committed by the Participating States for use in European civil protection operations (see further the paragraph below on Disaster Preparedness). Article 1(2) clarifies the scope of the UCPM, namely that protection ensured by the Union Mechanism shall cover primarily people, but also the environment and property, including cultural heritage, against all kinds of natural and man-made disasters, including the consequences of acts of terrorism, technological, radiological or environmental disasters, marine pollution, and acute health emergencies, occurring inside or outside the Union. In the case of consequences of acts of terrorism or radiological disasters, the UCPM covers only preparedness and response actions. The Union Mechanism is open to the participation of the European Free Trade Association (EFTA) countries which are members of the European Economic Area (EEA) and other European countries (when agreements and procedures so provide), as well as to acceding countries, candidate countries and potential candidate in accordance with the general principles and terms laid down in framework agreements or similar arrangements. In addition, candidate countries and potential candidates not participating in the Union Mechanism, and countries that are part of the European Neighbourhood Policy (ENP), may benefit from some of the activities financed under the Decision. Disaster prevention Under the UCPM, Member States shall develop and make available to the Commission their national risk assessments and national risk management capability assessments. The Commission shall provide guidance to Member States in carrying out risk assessments and facilitate the dissemination and exchange of information of good practices. This includes, in particular, improving the knowledge base on disaster risks (e.g. by developing an overview of risks the EU may face which takes into account the impact of climate change) and developing an integrated approach to disaster management of natural and man-made hazards. The Commission shall facilitate the sharing of knowledge, best practices and information related to prevention, taking due account of existing international commitments (such as the Sendai Framework for Disaster Risk Reduction 2015-2030, and the Paris Agreement), including through voluntary peer reviews. The Commission shall also finance cooperation projects in the field of disaster prevention based on yearly calls for proposals, which are published on the DG ECHO website. Disaster preparedness The creation of a European Emergency Response Capacity (EERC) in the form of a "Voluntary Pool" is one of the main innovations of the UCPM. Participating States can pre-commit different types of response capacities for participation in EU Civil Protection missions. The European Medical Corps (EMC) is a sub-set of the EERC through which teams and equipment from EU Member States can be rapidly deployed to provide medical assistance and public health expertise in response to emergencies inside and outside the EU. Through the Common Emergency Communication and Information System (CECIS), a web-based alert and notification application enabling real time exchange of information, the UCPM Participating States and the Commission share a common information system for mobilizing and deploying these pre-committed assets. This increases predictability of assistance while significantly reducing deployment times. Participating States and the Commission work closely together to develop quality requirements and a certification process for different modules and other response capacities, thus ensuring that all teams meet high quality and inter-operability standards and can effectively work together in the field. The UCPM also supports trainings and disaster preparedness exercises to test and further improve performance and coordination. Costs to upgrade existing national response capacities and certification costs, trainings and workshops can be financed by the EU. Once certified and part of the Voluntary Pool, the transport costs of teams in response to emergencies is eligible for EU co-financing of up to 85%. Additionally, the Union Mechanism works via its Emergency Response Coordination Centre (ERCC – see further paragraph below) to: (i) monitor hazards, collect, analyse and disseminate information on disasters; (ii) map available assets in close cooperation with the UCPM's Participating States; and (iii) prepare plans for the deployment of individual experts, teams and/or assets, providing emergency communications and monitoring tools. Disaster Response

The operational heart of the UCPM is the 24/7 Emergency Response Coordination Centre (ERCC), which is located within the Commission's DG ECHO in Brussels. The ERCC supports a coordinated and rapid response to disasters both inside and outside Europe using resources from the countries participating in the UCPM. With a capacity to deal with several simultaneous emergencies on a 24/7 basis, the ERCC facilitates a coherent and efficient European response to emergencies, aiming to reduce unnecessary and inefficient duplication of efforts. Whenever the scale of an emergency overwhelms the response capabilities of a country, the UCPM may be activated upon official request of that country or the United Nations (UN) and its agencies, as well as the International Organization for Migration (IOM), International Federation of Red Cross and Red Crescent Societies (IFRC) and the Organisation for the Prohibition of Chemical Weapons (OPCW). When this occurs, the ERCC acts as an operational hub, facilitating the coordination of assistance made available by the UCPM's Participating States to the affected country. Requests for assistance through the UCPM can concern disasters of any type, both within and outside Europe. The UCPM can also be activated in response to marine pollution and other maritime disasters, on which it coordinates closely with the European Maritime Safety Agency (EMSA). Assistance made available via the UCPM may consist of operational assets, in-kind assistance (relief items) and experts . The latter can be deployed for needs assessment and coordination missions as well as to provide advice to a requesting country on prevention and preparedness measures. The Commission can also co-finance the transport costs of assistance delivered via the UCPM. This enables a rapid distribution of assistance to the affected country and results in a reduced budgetary impact on the UCPM's Participating State(s) offering the assistance. Pooling and consolidating shipments of assistance for the affected country with the facilitation of the ERCC, allows to further boost the efficiency of the European response. The ERCC fosters coordination between civil protection and humanitarian aid operations. Maintaining direct links to the civil protection and humanitarian aid authorities in Member States it enables a smooth and real-time exchange of information. It ensures deployment of coordination and assessment teams composed of humanitarian aid and civil protection experts to conduct joint needs assessments. Cross-cutting activities Prevention and preparedness ("advisory") missions support Participating States or third countries with technical assistance on prevention and preparedness measures through the deployment of a team of experts. These missions can be requested by a Participating State, international organisation or third country. The exchange of expert programme allows for the secondment of civil protection experts from one Participating State to another, with some third countries also being eligible to participate. This exchange allows participants to gain direct knowledge and experience on all aspects of an emergency intervention and the different approaches of national systems. The peer review programme is addressed to all the Participating States (with their different systems and backgrounds) to identify – in a spirit of mutual support and peer-to-peer learning – areas for improvement. Finally, the Decision promotes the identification, analysis and implementation of lessons learnt distilled from the entire disaster cycle. The intervention logic (Annex 1) presents all the activities carried out by the Commission and the Participating States. Activities are linked to the respective objectives/indicators (as set out by Art. 3 of the Decision) they contribute to and categorised under the three thematic pillars of disaster prevention, preparedness, and response. 7EVALUATION QUESTIONS In order to assess the implementation and performance of the Union Mechanism and to evaluate whether it is on course to achieve its objectives, the following general questions were used to guide the analysis. The full list of evaluation questions is provided in Annex 2. Criteria Evaluation question 1. Relevance Are the objectives the UCPM is addressing still relevant to Europe's current problems and needs? 2. Effectiveness To what extent have the changes/effects generated by UCPM activities contributed to achieving general and specific objectives set out by Decision 1313/2013/EU? 3. Efficiency Could similar/greater benefits have been achieved at lower/similar costs? 4. Coherence (Internal) To what extent are the UCPM pillars and activities well-articulated, interconnected and mutually supportive?

(External) To what extent has the UCPM developed synergies with other policy areas and initiatives? 5. EU added value What results of the intervention could not have been achieved (or would have been less effective/efficient) without coordinated effort at EU level? 6. Sustainability To what extent are results achieved by the UCPM likely to last beyond the timeframe of the intervention? 8METHOD This Evaluation builds on the independent evaluation conducted by an external contractor, as well as certain recent inputs received during discussions with Participating States, including during Civil Protection Committee, Civil Protection Council Working Party, and Directors-General meetings. Overall, the design of the external evaluation was satisfactory and used a sound methodological approach. The evaluation relied on a large amount of data collected from reliable sources (agreed with the Commission during the inception phase). The combination of evidence provided by different research tools (online surveys/questionnaire, semi-structured interviews, case studies, etc.) was balanced. In addition to civil protection authorities from Participating States, other stakeholders (e.g. other ministries from Participating States, international agencies, EU services, etc.) are well represented, which adds to the credibility of findings. Generally, the external evaluation is in line with what was foreseen in the Roadmap. The evaluator has followed the guidelines provided in the Terms of Reference (ToR), which were adopted by the Inter-service Steering Group (ISG), as indicated in the Roadmap. The external evaluation was carried out between September 2016 and June 2017 and took account of the inputs received from the Open Public Consultation (OPC). The European Court of Auditors (ECA) special report on the UCPM, and the Commission's Report on progress made and gaps remaining in the EERC have also provided useful evidence to some areas of this Evaluation. The external evaluation consisted of three phases (see figure 1): (I)inception, including a desk review of UCPM activities, scoping interviews with desk officers and conception of research tools; (II) research, during which a targeted consultation reached out to all the main stakeholders of the UCPM through online surveys, face-to-face and telephone interviews; three case studies (Cyprus Forest Fire response; Estonia Module Exercise (ModEx); Balkan route migration response) were conducted. (III) analysis and synthesis, including analysis of UCPM performance, data triangulation, elaboration of conclusions and recommendations, and drafting an external Evaluation Report. The above-mentioned OPC was carried out online for three months (November 2016 – February 2017), with some 130 responders from all over Europe providing feedback on the UCPM. The report on the outcome of the OPC was published at the end of March 2017.

However, two methodological limitations should be noted: (i) The current UCPM monitoring system does not systematically capture quantitative data of performance. As already highlighted by the recent ECA audit, the lack of robust indicators (e.g. baseline and target values) systematically monitored by the UCPM partly hindered the collection of evidence needed to formulate more definitive findings. (ii) Limited statistical base of the external evaluation. Most of the data gathered by the contractor during the Evaluation timeframe are of qualitative nature and they refer to "perceptions" of stakeholders on the UCPM performance. While consultations have been extensive across all the stakeholders, the external evaluation did not manage to introduce and monitor additional indicators to fill this gap. To mitigate the shortage of quantitative data monitored by the UCPM, the contractor was encouraged to collect primary data and carry out quick analyses for a sample of activities (e.g. comparing timeframes of different response operations, cost analysis for forest fire fighting missions). While the statistic sample is too limited to draw any solid conclusion, these analyses point to a number of monitoring gaps which should be addressed in the second half of the UCPM implementation. Where the contractor failed to provide sufficient indicators/data, the Commission integrated the evidence obtained via alternative channels, such as bilateral meetings with Member States and operational lessons learned (e.g. feedback from module exercises, training programme, etc.). Within the limits of data and analysis, the conclusions in this document are well balanced, follow logically from the findings and are linkable to facts and data. 9IMPLEMENTATION STATE OF PLAY Financial data shows that nearly EUR 131 million was allocated to the UCPM over 2014-2016. The UCPM budget components are presented in table 1 below. Table 1: Budget allocated for the UCPM 2014-2016 (million EUR) Source: DG ECHO Budget line 2014 2015 2016 EUR 23030101 - prevention and preparedness - Inside the EU 27.6 29.1 28.0 EUR 23030102 - prevention and preparedness - Third countries (outside the EU) 4.0 5.5 5.6 EUR 23030201 - response inside the EU 1.1 0.7 2.5 EUR 23030202 - response outside the EU 11.1 13.6 3.09 Total 43.8 49.09 38.9 The evaluators compared the average annual allocation by activity during the periods of 2007-2013 and 2014-2016. Whilst a direct comparison is difficult between the years due to differences in budget structures, this analysis gives a rough overview of the shift in priorities for the Mechanism over different periods. Prevention All 28 EU Member States complied with Article 6 of Decision 1313/2013 and made available to the Commission a summary of their National or Sub-national level Risk Assessment by the deadline set. Based on such summaries the Commission developed an EU-wide Overview of Risks. The European Commission also published Risk Management Capability Assessment Guidelines to facilitate the national capability assessments that are due by end of 2018. The total funds committed for Prevention Projects amounted to over EUR 22 million for the period 2014-2016, compared to a budget of EUR 14 million for the period 2007-2013. The average project value was EUR 610 000 (with EU funding averaging EUR 470 000), which was 37% higher than for 2007-2013. The success rate of applicants for prevention projects during 2014-2016 was 39%, decreasing from approximately 50% in 2014 and 2015 to 27% in 2016. Financed projects include the following activities: knowledge sharing (meeting, workshops, conferences, exchange programmes), studies (e.g. new methodologies and tools for risk reduction), and awareness raising activities (websites, awareness campaigns, etc.). Preparedness

Established in 2013 to provide a coordinated and quicker response to disasters both inside and outside Europe, the ERCC replaced the previous Monitoring and Information Centre (MIC). The period 2014-2016 marked its full maturation, coordinating European assistance to West African countries in the context of the Ebola outbreak (2014-2015), and serving as an operational hub for logistics and transport in the response to the 2014 Balkan floods as well as the devastating earthquake in Nepal in 2015. Furthermore, between 2014 and 2016 the ERCC’s supported the political decision-making in the Council, particularly in the context of Integrated Political Crisis Response (IPCR) arrangements that are triggered automatically by the invocation of the Solidarity Clause or in cases of 'stand-alone' activations of the IPCR. In accordance with Article 9(7) of the Decision, Participating States designated national contact points to enable communication with the ERCC in the event of occurring or imminent disasters. Between 2014 and 2016, CECIS registered 64 Requests for Assistance, 14 pre-alerts and 15 monitoring requests. The current platform, created in 2007, will undergo several improvements between 2017 and 2019 in order to improve user-friendliness and enhance its analysis and monitoring capabilities. The European Emergency Response Capacity (EERC) or 'Voluntary Pool' was launched in October 2014 and, by the end of 2016, included a total of 77 response capacities from 16 Participating States. The UCPM has taken advantage of the integration of the European Flood Alert System (EFAS) and the European Forest Fires Information System (EFFIS) under the umbrella of Copernicus' Emergency Management System (EMS) in 2012. In close cooperation with the Joint Research Centre (JRC) (responsible for the design of the systems themselves) and DG GROW/Copernicus (managing the early warning systems) the UCPM has provided the link with the end-users, namely civil protection authorities from Participating States and the ERCC. Similarly, the rapid mapping services allow on-demand and quick provision of geospatial information (e.g. damage assessment maps) to inform emergency response and recovery. During the 2014-2016 period, more than 30 requests for maps have been received each year. 1 680 individual experts have undergone at least one course of the UCPM Training Programme between 2014 and 2016, whereas there were 2 255 trained experts over the entire 7-year period of the previous Mechanism. The yearly average has been 740 for the current financial framework, compared to 322 during the previous one. Around 15% of the total trained experts over the evaluation period are women. A European Disaster Management Training Network was set up at the end of 2016 to improve all phases of disaster management, create synergies and develop guidance on EU and international civil protection training. A number of disaster simulation exercises are financed every year through a Call for Proposals for full-scale exercises and for module exercises (ModEx). Between 2014 and 2016 8 ModEx in 7 Participating States and 13 full scale exercises in 10 Participating States have been organised. Differently from the previous Mechanism (2007-2013), ModEx exercises under the UCPM have become an integral part of the process of certification for modules which will become part of the EERC. Overall, EUR 26.1 million were allocated to exercises. During the first three years of the UCPM (2014-2016) the committed amount for Preparedness Projects reached nearly EUR 17 million, a significant increase from the budget of the previous Mechanism (nearly EUR 10 million for the entire framework 2007-2013). The average project value was EUR 691 000 (with EU funding averaging EUR 519 000), approximately 55% higher than the average figure during 2007-2013. The success rate of applicants for preparedness projects was 42% for the period spanning from 2014 to 2016. Financed projects included the following activities: training and exercises; knowledge management (best practices; study visits; workshops; etc.); new tools and procedures for preparedness. Response The UCPM was activated 19 times in 2014 and 2015, and 26 times in 2016. During the Evaluation timeframe (2014-2016), on average 33% (or 21 out of 62) of requests came from inside the EU and the remaining from third-countries. A total of EUR 28.4 million in transport grants were provided for operations inside (48 grants) and outside the EU (247 grants). Figure 2: Number of activations inside and outside EU, 2008-2016. Source: DG ECHO, last update 24 February 2017 Figure 3: Number of experts deployed per mission, 2008-2016 Source: DG ECHO, last update 24 February 2017 Although the number of disaster response missions has not changed significantly, in 2016 the total number of experts deployed (86) increased when compared to first two years of the UCPM. On average, 4.8 experts were deployed per mission in 2016. Regarding modules, statistical data shows that over the evaluation period (and before) the majority of modules were deployed outside the EU (see figure 4). Figure 4: Number of UCPM modules deployed inside and outside of EU, 2008-2016 Source: DG ECHO, last update 24 February 2017

Cross-cutting activities Peer reviews: Two "pilot" peer reviews were carried out in the UK and Finland in 2013 and 2014 in cooperation with the United Nations Office for Disaster Risk Reduction (UNISDR) and the Organisation for Economic Cooperation and Development (OECD). While the two pilots focused on assessing progress in national implementation of the Hyogo Framework for Action, the subsequent six peer reviews focused on more specific issues of disaster risk management relevant to the selected countries. A total of EUR 1.3 million was allocated for peer reviews over 2014-2016. Prevention and Preparedness ("advisory") missions: Out of thirteen of such missions undertaken between 2014 and 2016, six were carried out in a post-emergency context; the remainder focused on capacity building for recovery planning (two of them were embedded in two Post-Disaster Needs Assessments, i.e. in Albania and FYRoM); and ex ante prevention measures. A total of 13 missions were carried out over 2014-2016, with EUR 230 000 allocated. Lessons learnt: DG ECHO implemented lessons learnt and technical workshops following the activation of the UCPM on a case-by-case basis, depending on the nature of the disaster. A total of EUR 1.85 million was allocated for lessons learned over 2014-2016 of which EUR 598 358 were disbursed as of June 15, 2017. Exchange of experts: The programme was implemented by an external organisation for the 2014-2016 period. Overall, 375 experts participated in exchanges within the EU and 47 experts from 11 partner countries joined the programme. Budget allocated to this activity was: EUR 900 000 for experts from Participating States and EUR 196 025 for experts from third countries. Monitoring The external evaluation found that existing monitoring arrangements are rather limited and may not fully capture the UCPM performance nor the follow-up required. The Commission shares this view, which applies to all of the three pillars of the UCPM. Monitoring of results is based on the four indicators listed in Article 3 of the Decision. However for most of these indicators baseline/target values are not indicated in the Decision 1313/2013 nor in the 2014 Implementing Decision, which makes it difficult to assess progress. For instance, assessing "speed of intervention under the Union Mechanism and the extent to which the assistance contributes to the needs on the ground" (indicator for Objective 3) is hindered by not recording systematically via CECIS time milestones for all response operations. Similar considerations apply to the lack of a standardized format for needs assessments underpinning response operations, which in some cases can hinder the assessment of whether and how assistance (especially in-kind) matched needs on the ground. The preparedness indicator set by the Decision for Objective 2, ("quantity of response capacities included in the Voluntary Pool in relation to the capacity goals and the number of modules registered in CECIS") remains vague about including the type/number of individual experts registered in the system. At the moment this analysis cannot be fully carried out on CECIS (not all UCPM trained experts are in CECIS and its research filters are very limited, e.g. not allowing research by "tags" or keywords). This makes it challenging to comprehensively map Europe's response capacity in terms of experts. In addition, the Objective 4 (risk awareness) indicator ("progress in increasing public awareness and preparedness for disasters") has not been systematically measured by the UCPM. Although the Eurobarometer surveys do support DG ECHO in its ability to assess the general level of awareness of civil protection among European citizens, this cannot be fully attributable to the UCPM (furthermore it is only limited to EU Member States and several Participating States are not covered). The section below on effectiveness (6.2) elaborates further on the limitation of the current monitoring system. The external evaluation presents additional analytical limitations of CECIS, e.g. for producing automatic reports on response operations (e.g. speed, cost, etc.). Several of these shortcomings were already highlighted in the European Court of Auditors' report on the UCPM. 10ANSWERS TO THE EVALUATION QUESTIONS 11Relevance The general and specific objectives of Decision 1313/2013 continue to be relevant to Europe's current social and economic context. Consultations with stakeholders in the context of the Evaluation have highlighted that, over the past few years, civil protection authorities have been confronted with a changing risk landscape, which calls for new response capacities and an increased level of coordination with a much broader array of stakeholders. Within Europe, new response scenarios can range from supporting traditional police functions in the aftermath of a terrorist attack (e.g. victim identification) or dealing with the consequence of a cyber-attack affecting a lifeline infrastructure, to providing relief items to an exceptionally large number of migrants. Outside Europe, the UCPM is increasingly being activated to respond to rapid onset disasters, including a significant number of complex emergencies (see figure 5) in which civil protection assets are being used to address acute health/shelter/water and sanitation needs. As the UCPM becomes increasingly in demand, the issue arises as to its core mandate. In particular, the challenge for the UCPM is to find the right balance between being a framework of mutual assistance between European countries in the aftermath of natural and man-made disasters and evolving towards a more complex instrument capable of addressing crises of a multiple nature and with a global reach. Figure 5: Hazards having led to UCPM activation: comparison between 2007-2013 and 2014- 2016 (annual average) Source: Emergency Response Coordination Centre (ERCC), elaborated by ICF The UCPM's flexibility in addressing emerging or unanticipated developments and needs in the civil protection/disaster risk management arena in Europe and beyond has also been recognized by many stakeholders. The activation of the Mechanism in response to the Ebola outbreak (March 2014) and to support Participating States facing an exceptional influx of migrants (2015-2017) are probably the examples that best demonstrate the UCPM's flexibility in responding to non-traditional civil protection emergencies. However, stakeholders have voiced concerns about whether the UCPM is the right tool to address emergencies that are not fully covered by Decision 1313/2013. The relevance of the UCPM is also confirmed by:

·Several recent resolutions of the European Parliament calling on the Commission and the Member States to foster cooperation among civil protection authorities across Europe and to further simplify the procedures for the activation of the Mechanism in order to make it rapidly available and effective in the immediate aftermath of a disaster. ·The 2017 Eurobarometer Report on Civil Protection, according to which 90% of European citizens think that the EU should coordinate the response to disasters in any of the Member States and 87% believe there should be a common civil protection policy. There is also strong agreement (86%) that the EU should help any country worldwide hit by disasters in a coordinated manner, by sending experts and equipment to affected areas. ·The Overview of Risks in the EU report stresses that disasters ‘know no borders’ and there is a need to further address cross-border and "cascading"/inter-connected risks. As far as preparedness is concerned, the following challenges have emerged from consultations with national civil protection authorities and other key stakeholders (national training coordinators and international partners). ·The training and exercise programmes require a needs-based approach which focuses on the competences necessary to implement the specific activities of the UCPM, while incorporating relevant lessons learnt. It is therefore critical to thoroughly evaluate response missions, as well as prevention and preparedness activities, in order to distil specific training needs, with a view to develop a "competence curriculum" for European civil protection experts. The UCPM's Training Policy Group has recently (March 2017) developed a new strategic framework for training and exercises. Figure 6: Number of individuals attending at least one training and number of individuals deployed 2014-2016 Source: Data from ECHO, elaborated by ICF ·Further developing the cross-border early warning systems platforms would increase its relevance for the entire Europe. Forest fires continue to take a significant toll on Europe's society, economy and environment. Recent analyses from JRC have put the average annual losses caused by forest fires to EU Member States at nearly EUR 2.7 billion/year. The case study carried out by the external evaluators (Cyprus forest fires of June 2016) underlined the importance of EFFIS as a tool to assess the likelihood of locating available response capacities in other Participating States during the forest fire season. Currently the ERCC is working on the possibility of mapping such capacities. One important caveat about the relevance of the trans-boundary early warning systems for forest fires and floods concerns the different needs (and use) across Participating States. The countries that already manage comprehensive early warning systems at the national level consider the EU-level system as mainly relevant for informing disaster preparedness and response in other Participating States and third countries. Countries which are still in the process of consolidating their national early warning systems tend to see it as directly relevant for their national needs, in addition to response in third countries. Furthermore, interviews with Participating States and Commission officials have shown that the end-user side of early warning systems platforms such as EFFIS and EFAS should be further developed, so as to create a community of practitioners that reflects on operational evidence emerging from disaster events and which shares knowledge, with the objective of improving practice. Such a development would increase relevance of UCPM early warning platforms for most countries within Europe, regardless of their level of preparedness. Better linking prevention with preparedness and response activities was already recommended in 2014 by the ex-post evaluation of the previous Civil Protection Mechanism (2007-2013). This need remains relevant at the mid-term mark of the 2014-2020 Mechanism, based on what the main stakeholders have expressed. The need to match response capacities with the major risks in Europe will represent the key challenge to ensure continued relevance of UCPM. In particular, an effort for forward looking assessments that take into account "emerging" threats (e.g. cyber-attacks, CBRN, critical infrastructure failure, etc.) is required. 12Effectiveness The UCPM is making good progress towards achieving the objectives set out by the legislation, most notably in improving cooperation and coordination between stakeholders in the civil protection sector. Article 3 of Decision 1313/2013 spells out four specific objectives which, in turn, are expected to contribute to achieving the two overarching general objectives (See Annex 1: Intervention Logic). With regards to the specific objectives of the Decision: Specific objective 1: To achieve a high level of protection against disasters by preventing or reducing their potential effects, by fostering a culture of prevention and by improving cooperation between the civil protection and other relevant services. The Mechanism is on track to achieve this objective. The target for the first result indicator established by the legislation ("a National Risk Assessment submitted by all Member States by end 2015") was fully achieved. As far as the second indicator is concerned ("an assessment of Member States' risk management capability"), the target is due by end 2018 and guidelines were issued by the Commission. From the stakeholder consultation, a consensus emerged both among civil protection authorities and other national ministries from the Participating States consulted that the UCPM has contributed towards a higher level of protection against disasters. For instance, the views of stakeholders consulted by the evaluators indicate that the national risk assessments were helpful to inform risk reduction policies and plans as well as improving the understanding of risks. However, while the Decision's indicators capture outputs (National Risk Assessment and National Risk Management Capability reports), additional evidence would be needed to describe the UCPM progress in achieving outcomes (preventing or reducing potential effects and fostering a culture of prevention). For instance, this would require monitoring the uptake of outputs delivered by UCPM prevention projects (e.g. how do risk assessment studies or knowledge sharing activities result in more effective cross-border policies and investments for risk reduction once the project is closed?). The external evaluation did not conduct such an analysis nor pointed to possible measures to carry it out in future. As far as the second part of the objective (improving cooperation between civil protection and other services) is concerned, the Decision does not set any specific indicator. However, the survey carried out with national civil protection authorities in the context of the evaluation shows a significant number of agencies/ministries involved by civil protection authorities in the preparation of NRAs. This information can be considered as a proxy of the good level of cooperation between civil protection and other services on disaster prevention, at least in sharing risk information. Figure 7: Survey of national civil protection authorities. Question: Which ministries and/or public agencies were consulted during the development of your NRA?

Source: ICF, Survey of national civil protection authorities Activities that stakeholders consider as having contributed the most to this specific objective are: (i) the introduction of the national disaster risk assessment provision in the legislation and (ii) the financing of prevention projects on a yearly basis. When compared with preparedness and response, the perception of the public is that results for the prevention pillar of the UCPM are less evident. A factor that needs to be taken into account is that annual financial resources allocated to the prevention pillar are only 20% of the UCPM total budget. The impact generated by UCPM prevention activities should also include what these activities have managed to leverage (e.g. additional financing, technical resources, etc.) outside of the Mechanism. For instance, prevention (and preparedness) activities financed by the UCPM could work as 'seed funds' that national and sub-national authorities could use to mobilise and/or steer additional resources for disaster risk management from other EU (e.g. Cohesion/European Regional Development Fund, Horizon 2020, LIFE programme, etc.) and international (European Investment Bank; World Bank; UN agencies; etc.) programmes. Currently, from the reporting carried out by the UCPM-funded prevention activities, it is not possible to appreciate this type of impact.

Consultations with Participating States and Commission officials also highlighted a number of challenges relating to fully capturing and deepening the impacts of the prevention activities, which will be critical for deciding how to allocate future financing. These include: ·Increasing the focus of the prevention (and preparedness) projects that the UCPM finances every year. The programme, which has been running since 2000, has financed a large number of projects without always capitalising on their results. Given the relatively small size of financing (between 2014 and 2016 average funding was EUR 470 000), projects need to be targeting the needs of the end-users and need to incorporate measures for scaling up their results. ·Better articulating the linkages between the UCPM prevention (and preparedness) projects and broader EU programmes (e.g. Cohesion/European Regional Development Fund, etc.). A more strategic link with such programmes could help maximize the uptake of projects results and leverage additional technical and financial resources for prevention.



Specific objective 2: To enhance preparedness at Member State and Union Level to respond to disasters. The UCPM has been very effective in achieving this objective, based on the indicator set by the Decision ("quantity of response capacities included in the Voluntary Pool in relation to the capacity goals and the number of modules registered in the CECIS"). As highlighted by a recently published Commission report, substantial progress was made with the establishment and the functioning of the EERC and, in particular, by exceeding the initial capacity goals for most of the modules (see Figure 8) during its first three years. As far as the "other response capacities" set by the Implementing Decision, after a somewhat slow start (2 capacities registered by July 2015), significant progress was recorded with 14 other capacities registered in the EERC as of May 2017. In addition to meeting these quantitative targets, the Decision has also resulted in higher quality standards for response capacities across Europe.



Figure 8: ERCC capacity goals compared with modules registered in the EERC as of 31 May, 2017 Source: DG ECHO and the Commission Implementing Decision of 16 October 2014 (C(2014) 7489) This assessment is also confirmed by the results of the stakeholder consultation. Nearly all of the civil protection authorities and more than two thirds of the national ministries consulted agreed that the UCPM has been effective in enhancing preparedness at Participating State and Union levels to respond to disasters. In addition to the EERC or Voluntary Pool, this pillar, which received 50% of the total UCPM budget, included some of the flagship activities of the UCPM, such as the training programme; the exercise programme; the functioning of the ERCC and the CECIS. Although the Decision does not set specific results indicators for training and exercises, these were recognized among the activities that contributed the most to achieving the preparedness objective of the UCPM. National civil protection authorities consider training and exercises as being generally fit for purpose, improving the capacities of Participating States, establishing common approaches and facilitating cooperation. The stakeholder consultation also highlighted the need to assess the number of slots available to Participating States for training and exercises on operational criteria (e.g. number of pooled capacities and experts, level of national capacity, etc.) other than the population.



The external evaluation and its stakeholders' consultation point to a number of challenges that may affect the full achievement of this objective by the end of the UCPM implementation period. These include: ·A more evidence-based approach to preparedness is required. The capacity goals set by the Implementing Decision should be revised against the main/critical risks Europe is facing and prioritized accordingly. This is fully in line with Art. 14 (2) of the UCPM Implementing Decision which requires the Commission to regularly assess and revise the suitability of capacity goals "on the basis of risks identified in national risk assessments or other appropriate national or international source of information." A close dialogue with Participating States is critical in order to capture all national capacities, including those which are not registered in the EERC but may still be deployable via the UCPM. Similarly, the spatial distribution of (natural and man-made) risks across Europe should inform the geographical distribution of response capacities. ·The need to re-assess the system of incentives for pooling assets in the EERC based on lessons learnt and other knowledge emerging from operations. A significant part of these incentives concern the level of co-financing for the deployment of EERC capacities or the mobilisation of buffer capacities; providing health and third-party liability insurance coverage for experts deployed. The deployment of EERC capacities, particularly for long-term missions outside the EU, would certainly benefit from the availability of more ‘robust’ EU co-financing possibilities. The administrative burden involved in registering and certifying modules also ranks among the main concerns of Participating States, although this is being addressed by the Commission with a more streamlined and coherent planning process for exercises, certification and registration of assets in the EERC. ·The training and exercise activities of the UCPM require a strategic approach focusing on results. Some of the consulted Participating States have highlighted the need to move from a system focused on outputs (i.e. number of individuals trained) to addressing operational needs that stem from operations. This is currently being addressed with the Strategic Framework for Training and Exercises, recently launched by the Commission, which is focused on competences. Furthermore, a robust framework for evaluating the training courses administered as well as the individuals trained is considered as a priority in order to match the needs on the ground with the best technical resources available. Similar considerations were raised with regards to the exercises. Specific objective 3: To facilitate rapid and efficient response in the event of disasters or imminent disasters

Overall, the UCPM has made significant progress towards achieving a rapid and efficient response to disasters. Stakeholders across all categories, including those who participated in the Open Public Consultation, noted as one of the key achievements the improved coordination and flow of information at EU-level through the scaling-up of the ERCC and CECIS during the period 2014-2016. Civil protection authorities were generally satisfied with the ability of the UCPM to respond to disasters within the Union, a vast majority considering that information provided to countries for deploying assistance was comprehensive, sufficiently detailed, based on the needs on the ground and provided in a timely manner. Three quarters of civil protection authorities consulted also considered response coordination to be smooth and timely. Outside of the EU, UCPM response was considered "broadly effective" by the recent audit of the ECA, which focused on three disasters (floods in Bosnia and Herzegovina, Ebola outbreak in West Africa and Nepal earthquake) to which the UCPM responded. According to the ECA, the UCPM activation was timely and, in each of the three cases observed, the EU Civil Protection Team facilitated coordination on the ground with the Participating States' response teams. The coordination among different Commission services, as well as with other EU and non-EU bodies, was inclusive (see also Section 6.4 on coherence). The Commission also respected the UN’s overall lead, and took steps to ensure a smooth transition into the recovery phase. Response missions outside the Union were deemed effective also by international actors (UN OCHA, WFP, IOM, UNHCR, etc.) participating in the stakeholders' consultation. The importance of gaining a better/broader understanding of the context and partners on the ground was highlighted by some of the international stakeholders. This goes in the same direction as the ECA's recommendation to involve more systematically DG ECHO personnel in the field during UCPM response missions. In order to complement the perception of stakeholders with statistical evidence on the speed of UCPM response to comparable disasters inside Europe (e.g. forest fires), the external evaluation looked at response times for selected emergencies based on data recorded in CECIS. While the analysis carried out in the external evaluation report is too limited to allow any solid conclusion on the speed of response, it pointed to a discrepancy between the indicators set by the UCPM Implementing Decision and the data that are currently recorded (in some cases manually entered) in CECIS. In particular, while the only speed indicator that the legislation (Implementing Decision 16 October 2014) refers to is the "availability" of each response module, the parameter monitored by the ERCC is the "departure time". Therefore, there is a need to harmonize indicators in order to allow comparisons and measure whether there is any progress in the speed of response. In addition to identifying and harmonizing the indicators, as already highlighted by a recommendation of the ECA, statistics and reports based on CECIS data should be automatically generated and analysed. Moreover, the lack of baseline and target values (i.e. what can be considered as a "rapid response" to a forest fire/earthquake/flash flood/etc.?) is an obstacle to appreciating effective progress in making disaster response speedier. Similar challenges exist for the second dimension of this indicator ("efficient response"), as explained in section 6.3 on Efficiency. In addition to adequately monitoring the speed of response, the external evaluation raised other challenges to fully measuring and, possibly, achieving the results under this specific objective: ·Needs assessments in the Requests for Assistance are often not available and/or not collected in a standardised format. Due to this, the second indicator set by the Decision for assessing disaster response, ("extent to which the assistance contributes to the needs on the ground") can only be measured based on perceptions and other qualitative input provided by the stakeholders. In this context the external evaluation noted a low level of awareness among Participating States of the common template for request for assistance, which was agreed in the context of the Host Nations Support Guidelines. ·When facing imminent disasters, the pre-deployment of a small UCPM team ahead of the formal activation should be considered, as this could prepare the ground for the following deployment and make a significant difference to accelerate UCPM response. ·Further simplifying administrative and financial procedures for engaging response capacities under the UCPM which, according to some Participating States, in some cases is less rapid than bilateral channels. A number of operational 'quick fixes' have already been identified by the users (e.g. integrating in CECIS the transport co-financing request form; revising the minimum threshold for individual transport grant applications; etc.) and could be addressed rapidly. ·Ensuring the best match between operational needs and experts deployed in the EU Civil Protection Team. Consulted civil protection authorities found the Terms of Reference produced by the ERCC for selecting experts (their use became increasingly systematic since 2015) helpful but raised some concerns on the selection process of the experts, which could still be improved. Specific Objective 4: "To increase public awareness and preparedness for disasters" The UCPM has contributed to this objective mainly with the dissemination of good practices sharing and other knowledge management activities (e.g. Civil Protection Forum 2015, 3rd edition of the World Reconstruction Conference, establishment of the online-based Disaster Risk Management Knowledge Centre, Guidelines on Risk Communication, etc.). When compared to the other three, stakeholders consulted in the context of the external evaluation feel that progress under this objective has been less advanced. This seems also in line with the trend showed by the 2017 Eurobarometer survey, whereby over the past years a significant part of the surveyed population (42%) is not aware of EU civil protection work. Boosting the UCPM visibility (for instance during exercises, as observed by the evaluators during one of the case studies) to European citizens would also contribute to increasing the profile of the public awareness activities supported by the UCPM. Finally, there is another factor related to the achievement of this specific objective which should be taken into account in the period leading up to the end of the UCPM implementation framework. The activities carried out by the UCPM (mainly complementing the work carried out at the national level by civil protection authorities) and the UCPM indicator for this objective (level of awareness of Union citizens of the risks in their region) are on two different levels. This mismatch can make it difficult to find a direct link between the awareness raising and knowledge management activities financed by the UCPM and the increased level of awareness among the population at the local level. No baseline or target has been set for this risk awareness indicator which, moreover, has not been surveyed by the 2017 Eurobarometer report on Civil Protection. General Objectives: ·To strengthen cooperation between the Union and Member States and facilitate coordination in the field of civil protection in order to improve the effectiveness of systems for preventing, preparing for and responding to natural and man-made disasters; and ·To promote solidarity between the Member States and other participating countries through practical cooperation and coordination Overall, based on the extent to which the specific objectives are being achieved, the UCPM has been successful in progressing towards the general objectives of the Decision. National civil protection authorities and other stakeholders agree that the UCPM contributed to improving cooperation in the civil protection sector within Europe, in particular with regards to disaster response and preparedness. The creation of a well-established system of civil protection authorities and a network of colleagues who know each other across all the Participating States was instrumental to enhance communication and information exchange. Survey results and interviews also suggest that the UCPM contributed to improving the effectiveness of systems for preventing, preparing for and responding to disasters. The legislation does not set specific result indicators for the General Objectives and the external evaluation did not suggest potential indicators to use as proxies.

13Efficiency Generally speaking, the stakeholders' perception of the UCPM's efficiency is very positive, particularly with regards to emergency response and, overall, administrative costs associated to implementing the UCPM decision are considered acceptable by most Participating States. While the efficiency question encompasses the entire spectrum of UCPM activities, it is particularly critical for the response pillar since one of the specific results indicators set by the Decision refers to "promoting rapid and efficient response". The external evaluators' analysis of efficiency focused on transport provision under the UCPM. In particular, the external evaluator attempted to carry out a cost analysis of transport grants requested by Participating States for comparable response operations, namely aerial forest fire fighting, water purification in the aftermath of floods and in-kind assistance in the context of the refugee crisis. Preliminary evidence collected in the context of the evaluation points to some discrepancies in the net contribution requested by Participating States for comparable types of services (cost/mile ratio). This is the case for Canadair aircraft operations carried out by different Participating States, where the cost per mile ratio seems to vary significantly (even though the aircraft used is similar). Similar discrepancies also emerge from the net contribution granted to two Participating States providing a water purification module during a flood response operation inside Europe and from transport of in-kind resources by truck or plane. Although the analysis carried out by the external evaluators remains preliminary due to the limited data sample considered, they point to the need to systematically collect cost data and run analysis of comparability of transport reimbursement. Similarly, a robust analysis of transport costs for in-kind assistance goods and/or capacities via the EC broker or framework contractors could help in order to look for any significant variation and determine whether EU transport co-financing is always the most cost-efficient option. Similarly, the external evaluation highlights that useful data concerning costs remain locked in paper forms instead of being recorded in CECIS and analysed. Such analysis could lead to determining benchmarks (lower/higher bounds or ranges) for cost of deploying certain assets. In some cases this information could lead to quicker approval of deployments and to faster reimbursement processes. As far as efficiency of UCPM-financed activities, the following challenge emerged from the external evaluation: ·The UCPM needs to monitor more systematically and consistently cost and speed of response missions. This can help establish baselines/benchmarks (lower/higher bounds or a range) allowing comparisons between emergencies responses and produce analyses needed to improve efficiency of future performance. Moreover, rigorous monitoring arrangements could also help speed up the reimbursement process (for instance via pre-agreed ceilings). 14Coherence The UCPM has achieved internal coherence between its different components and a good level of cooperation with other policy areas. Internally, a number of cross-cutting activities (e.g. lessons learned, advisory missions, peer reviews, exchange of experts, etc.) within the UCPM provide for an integrated prevention-preparedness-response approach and therefore help different activities of the UCPM reinforce each other. This is confirmed by the views collected through the stakeholder consultations with Participating States (see figures 9-10), where a vast majority of the responding national civil protection authorities and ministries agreed that the UCPM promoted sufficient linkages and synergies across its three pillars. Figure 9: Stakeholder consultations. Source: ICF, Survey of national civil protection authorities from 27 Participating States

Figure 10: Stakeholder consultations. Source: ICF, Survey of National Ministries (other than civil protection) from 13 Participating States Consultations with stakeholders in the context of the external evaluation also highlighted a number of significant challenges for strengthening the internal coherence of the UCPM through its cross-cutting activities. ·There needs to be a continuous feedback loop between lessons learnt and the implementation of activities (including prevention and preparedness ones, in addition to response). Participating States have highlighted that some specific activities (e.g. training and exercises) would benefit from a stronger link with UCPM operations, for instance focusing on specific challenges arising from the deployment of modules in the field (e.g. custom clearance in fragile and conflict-affected states; cooperation with host countries; integration in broader international frameworks for disaster response and recovery planning; etc.). To this end, there is a shared feeling that the lessons learnt process of the UCPM needs to be more structured and systematic from identification and throughout implementation (on both the Commission and Participating States sides), with regular monitoring and transparent reporting on the progress. ·The UCPM training and exercise programme should promote a more coherent approach between training and deployment. Training should be focused on the competences needed during deployment (including for preparedness and prevention activities). In this respect, as confirmed by one of the case studies, the link between training and exercises is critical and should be strengthened. Consultations with Participating States suggested potential synergies between training (particularly the most advanced courses) and ModEx or full-scale exercises, in order to enhance operational preparedness of experts. Some of these recommendations are already being addressed by the Commission. ·Peer reviews should ensure a good match between reviewers and the needs of reviewed countries (i.e. between profile/competences of the reviewer and specificities of the country/sector to be reviewed). Additional linkages could be explored between training and the deployment of experts on peer review missions. This may include a process to certify experts' technical and operational competence in order to better address the needs of the reviewed countries.

·Prevention and preparedness ("advisory") missions should build on a well-formulated, result-oriented approach. A structured process needs to be put in place to prepare and implement these missions, be it in preparation for potential disasters (e.g. assessment of national emergency management capacity) or, in the aftermath of a disaster, to inform recovery and reconstruction strategies. Synergies with other EU-funded programmes (in particular the Prevention Preparedness and Response to Natural and man-made Disasters (PPRD) and the Instrument for Pre-Accession (IPA) programmes) and/or Participating States engaged in third countries should be strategically considered, with a view to provide additional resources (technical and financial) needed for implementing recommendations. In general, the consultations highlighted the need for a more result-oriented focus for the cross-cutting activities of the Mechanism (e.g. lessons learned, peer reviews, advisory missions, exchange of experts). This includes focusing on measurable policy and/or operational results which should be achieved, in a given timeframe by the beneficiary. In some cases, activities conducted in response to a request from a Participating States or a third country (e.g. advisory missions) could be conditioned to the requesting country meeting some policy/capacity targets prior to the UCPM fielding a technical assistance mission. As regards the external coherence, the external evaluation mapped the extent to which the UCPM has been pursuing synergies with other policy areas and instruments in the period 2014-2016. When compared to the final evaluation of the Civil Protection Mechanism (2007-2013), the evaluators noted that the current Union Mechanism is cutting across a larger number of policy areas. Although the degree of coordination could only be assessed on the basis of qualitative input provided during interviews and mostly at Commission level, a higher degree of coordination with other areas was observed, compared with the previous Mechanism. The level of cooperation between the UCPM and other policy areas related to civil protection, based on the external evaluation (online survey of national civil protection authorities and interviews with several EC/EU services) is presented in figure 11 below. Figure 11: Current status of coordination between UCPM and other policy areas. Red arrows show growing importance of the policy area and need for inter-institutional cooperation. Green arrows mean that inter-institutional cooperation has grown over the period 2014-2016.

Source: ICF External Evaluation (based on stakeholder consultation) During the first half of its implementation, the UCPM strengthened its linkages with key policy areas. These include: research and innovation (including the Disaster Risk Management Knowledge Centre and the Copernicus rapid mapping services); regional policy (ex-ante conditionality to cohesion funds on disaster risk management and policy coordination with the cohesion policy instrument); environment (integration of hydrological and civil protection early warning systems for floods under the Flood Directive and, to a lesser extent, promoting the UCPM to support implementation of the Seveso Directive), climate adaptation (sharing of knowledge under the Disaster Risk Management Knowledge Centre and, to a lesser extent, integration of climate change in the national risk assessments); health (cooperation with DG SANTE and the European Centre for Disease Prevention and Control (ECDC) on health-related activities, including the European Medical Corps), neighbourhood and enlargement policy (in cooperation with DG NEAR) as well as coherence and complementarity between CP, humanitarian and development assistance. As far as the neighbourhood policy is concerned, further coherence between UCPM and DG NEAR activities in Partner countries is needed. In particular, coordination between the yearly Prevention and Preparedness Calls for Proposals of the UCPM and the disaster risk management related activities of the PPRD was found to be insufficient by the 2017 final evaluation of the PPRD-East II. The evaluation called for enhancing coordination between DG ECHO and DG NEAR on their respective programmes. Similarly the 2017 final evaluation of the PPRD-South Programme underscored the need to provide to both decision- makers and practitioners in partner countries with more knowledge on the UCPM and the opportunities it offers them in prevention/preparedness/response. The IPA programme, would benefit from further integration in the UCPM. This includes, among others, participation in UCPM exercises and increased awareness of UCPM procedures; integration of IPA-supported multi-countries response modules in the EERC; and joint efforts with UCPM to strengthen the governance of the prevention activities (e.g. on the Disaster Risk Assessment and Mapping initiative). Recent developments affecting Europe's security (mainly related to terrorist attacks) have also prompted the security policy as a priority area for cooperation with civil protection. The "Security Union" initiative launched towards the end of 2016 aims at providing a platform for cooperation across several EU instruments, including the UCPM and is expected to take on a primary role in the second half of the UCPM implementation. The approach taken by the UCPM-Security Union coordination goes beyond the exchange of information and focuses on a common agenda with specific milestones. This result-oriented approach could possibly be replicated in other areas. Finally, as far as the response outside of Europe is concerned, the comparative advantage of civil protection intervention vis-a-vis humanitarian assistance needs to be carefully considered as the UCPM becomes increasingly activated for dealing with humanitarian and complex emergencies (see section 6.1). In particular, the UCPM can fill important emergency gaps in cases where it is not possible to fully deploy humanitarian teams or where specific technical expertise and/or assets are needed. In the same vein, linkages with other prevention, preparedness or response activities outside of Europe should be ensured from early stages of intervention under the UCPM. For example, in the aftermath of the Nepal earthquake the support offered through the UCPM (e.g. structural engineering, water purification systems) was followed by the development of a Post Disaster Needs Assessment (PDNA), which became the recovery and rehabilitation framework for the country. In such situations, the assessments carried out by civil protection experts deployed by EU Member States or the Commission should feed into wider recovery frameworks conducted by partner countries governments, with support from the international community. 15EU Added Value The UCPM provides Participating States with cooperation opportunities and increased capacity – and predictability – in terms of expertise and resources that would not be possible without coordinated effort at EU level. The vast majority of Participating States consulted see an added value in EU-level action under the UCPM as opposed to bilateral actions or to the action of international organisations or NGOs. Prevention: Consulted stakeholders noted the importance of the Decision's provisions on prevention (e.g. National Risk Assessments) as a "pull factor" to develop throughout all the Member States policies and strategies addressing disaster prevention. Furthermore, the introduction of European Commission guidelines for risk assessments had a very significant impact on this pillar of the UCPM, contributing to the first-ever overview of risks in the EU. Prevention and preparedness projects, which must necessarily involve three or more Participating States, deliver an inherent cross-border value by facilitating the establishment of informal channels between national authorities having to manage common risks. Most of the Participating States consulted by the external evaluators do not believe these projects would have been carried out in the absence of UCPM funding or, if they had been, they would have involved duplications and fragmentations.



Figure 12: Stakeholder consultations. Source: ICF, Survey of Project Leads of Prevention and Preparedness Projects from 12 Participating States. A more robust UCPM monitoring framework (as explained in section 6.2 on Effectiveness) that, for instance, allowed to keep better track of outputs and outcomes of cross-border prevention projects (e.g. number of cross-border risk assessments developed by the UCPM, or number of macro-regional development plans that take into account findings of risk analysis developed by the UCPM) would also help to better appreciate and communicate to stakeholders and the public the added value of EU action in the area of civil protection and disaster risk management. Preparedness: EU assistance to Participating States in organising resources through the creation of the EERC and the intervention modules is considered by stakeholders as the most tangible (and visible) added value generated by the UCPM. The overall capacity and deployment capabilities (e.g. training and inter-operability) of the EERC is larger than what would have been possible through the sum of the national capacities. Voluntary pooling of Participating States' assets on standby has facilitated the planning of operations and added an element of predictability, thus enhancing EU-wide contingency planning. In addition to making response capacities more adaptable to the international context, most of the consulted countries agree that promoting common European standards for modules that join the EERC has contributed to enhance their quality and robustness. Similar considerations apply to training and exercises, which have contributed to create a network of EU civil protection professionals sharing common operating procedures and standards throughout Europe. To this end, the exchange of expert program is also considered as being a valuable instrument to provide to a large number of civil protection experts opportunities to share knowledge and acquire new skills while belonging to an international network of civil protection professionals. Response: The evolution of the MIC into a single "hub" for information sharing and operational coordination, the ERCC, was considered by stakeholders to be the most tangible added value for European and third countries. Managing multiple bilateral offers at the same time and without ERCC coordination would be more cumbersome/costly and would bring lower visibility. Another significant value brought by EU-level action in the area of civil protection is the possibility to leverage satellite systems (e.g. Copernicus) for making available trans-boundary hazard early warning systems and to inform disaster response operations (e.g. providing damage assessment maps). Preliminary recommendations by the external contractor to further increase the added value of the UCPM interventions echo those enshrined in the recent ECA report, i.e. shortening the process for selecting experts for deployment and better capture UCPM performance by systematically monitoring quantitative indicators for speed (e.g. time needed to respond to a request for assistance) of response. Tracking the cost-effectiveness of EU-level action (e.g. savings generated by well-coordinated response operations) could also demonstrate another significant added value.

16Sustainability The sustainability of the UCPM's results is being ensured to a certain extent through the acquisition of knowledge and skills by experts participating in different UCPM activities. Consultations with Participating States highlighted the importance of training, exercises and exchange of experts and of fostering cross-border prevention and preparedness. A critical mass of trained European civil protection experts and practitioners enables the uptake of knowledge and tools made available by the UCPM. Sustainability of prevention and preparedness projects financed under the UCPM had been flagged as a potential issue by the ex-post evaluation of the Civil Protection Mechanism (2007-2013). Lack of trans-national coordination mechanisms and insufficient follow-up funding very often limited the uptake of project results at national level. The survey of stakeholders carried out for the external evaluation of the UCPM shows a better perception of sustainability of such projects which, according to the views of the project leads, are likely to be continued and/or scaled up with national, regional or other EU funding. Compared to past years, progress was also helped by more systematic publication of project results via the UCPM website managed by DG ECHO. However statistical evidence is very limited and uncertainties remain on the effective use of results by end-users as well as on the public/private sector hand-over after project closure. The following issues were highlighted by the stakeholders consulted in the context of the external evaluation and during discussions with the Commission as critical challenges that may affect the sustainability of UCPM activities and their results in the coming years. ·A viable financial model to ensure availability of capacities in the EERC. Providing realistic economic incentives to Participating States for pooling their assets and developing new response capacities is critical to ensuring a durable pool of capacities in the EERC. At the same time, creating bigger incentives to pooling assets should not come at the detriment of Participating States ' investments in national response capacities. ·An efficient strategy to build the European civil protection human resource capacity. New ideas and approaches will be needed to ensure that UCPM foundation courses be rolled out extensively without creating duplications with the training program of Participating States, while more specialized/advanced training (e.g. training for deployment) be delivered to well-identified groups of experts. Matching competences with needs is critical to make sure the UCPM fulfil its mission of supporting its Participating States. ·The role of the private sector in the implementation of UCPM activities. Several Participating States have highlighted the potential risks arising from private companies carrying out needs assessments, advisory missions, peer reviews and other activities that provide commercial actors with access to potential market opportunities. Measures should be introduced to guarantee that such companies are not in an advantageous position vis-a-vis other private competitors or the public sector itself when participating in a call for tenders. Furthermore, there may be a need to protect confidentiality and secrecy of sensitive information in some sectors (e.g. national security threats; critical infrastructure; critical capacity gaps; etc.). ·Bringing the UCPM closer to the European citizens while guaranteeing support to the national level. Subsidiarity is one of the core principles of the UCPM, whose primary function is to support Participating States' responsibility to protect their citizens and assets. Being relevant to the civil protection/disaster risk management actors in the front line (regions and other territorial authorities) while strengthening the coordinator/broker role of national civil protection authorities is a critical challenge for the sustainability of the UCPM. Similarly, the UCPM needs to promote the bi-lateral, tripartite dimension of cooperation between countries (e.g. neighbouring countries) while ensuring its pan-European approach and values. ·Maximizing the uptake of projects results (e.g. prevention and preparedness projects, exercises). UCPM-financed projects need to be designed and monitored with a view to focusing on the uptake of project results by end users. There is a need to shift the focus from the output (e.g. the delivery of a risk assessment, a workshop or a new training methodology) to the outcome (the actual change generated by that product, e.g. a new infrastructure project that takes into account the results of that risk assessment or workshop, or the use of that methodology by the trained experts in a real-life situation). 17CONCLUSIONS Based on the findings of the external evaluation and taking into account the results of recent analytical work on the UCPM and its own operational knowledge, the Commission's overall conclusion is that the implementation of UCPM activities is well underway and on track to achieve its objectives by the end of the legislative timeframe (December 2020). The performance of the UCPM during the first half of its timeframe has showed encouraging results in each of the three pillars (prevention/preparedness/response) and overall positive views from the Participating States and the main stakeholders. At the same time, the Evaluation has also highlighted critical challenges that, if addressed promptly, may further increase the impact of the UCPM. A general area for improvement is the strengthening of the UCPM results monitoring framework, through the possible introduction of quantitative indicators which capture more adequately progress achieved under the three pillars of the UCPM. There is sufficient evidence to demonstrate the continued relevance of UCPM objectives to Europe's needs. The results of the 2017 Eurobarometer survey on Civil Protection show that a vast majority of European citizens support a common EU policy in the field of civil protection and agree that the EU should lead the coordination of disaster response to affected countries, both in Europe and in the rest of the world. Recent resolutions of the European Parliament have also called on the Commission to further support cooperation among European civil protection authorities. The stakeholders consulted also highlighted the flexibility of the UCPM to rapidly adapt and address newly emerging crises which were not "traditional" civil protection emergencies (e.g. the Ebola outbreak in West Africa and the migrant crisis in Europe). The increasing number and scope of UCPM activations outside of Europe pose, however, the challenge of finding the right balance between being a framework of mutual assistance between European countries in the aftermath of natural and man-made disasters and evolving towards a more complex instrument capable of addressing crises of a multiple nature and with a global reach. An analysis of the UCPM effectiveness highlighted encouraging progress under each of the specific objectives. The UCPM is fully on track to achieve objectives pertaining to disaster prevention framework (objective 1) and readiness for disasters (objective 2), based on the indicators set by the legislation and on feedback received by key stakeholders consulted. The UCPM's response to disasters (objective 3) has been timely and effective according to the vast majority of stakeholders consulted. Confirming one of the findings of the ECA's review of the UCPM, the external evaluation struggled to find robust and consistent quantitative data (e.g. speed, needs assessments) to corroborate the stakeholders' very positive views on UCPM disaster response. Lastly, public awareness and preparedness to disasters (objective 4) have received slightly less satisfactory views from stakeholders when compared to the other objectives. In this respect, it is questionable whether the specific indicator ("level of awareness of Union citizens of risks in their regions") set by the Decision for this objective can be fully attributable to the level of activities carried out by the UCPM. The external evaluation also underscored several challenges which, if promptly addressed, could further increase the effectiveness of the UCPM. These include, among others: establishing a baseline/benchmarks (e.g. speed, cost/effectiveness, etc.) for comparable types of emergencies; developing a more evidence-based process to set capacity goals for disaster preparedness; revisiting the system of incentives to pool response assets and develop new capacities in the EERC. Disaster prevention is considered a critical area that may require additional resources in future (currently only 20% of UCPM budget) although a more robust impact evaluation may be required to establish where the UCPM action to date has been most effective in assessing/reducing disaster and climate risks. Cooperation and coherence between the UCPM and the EU climate change adaptation agenda should be enhanced. Overall, closer linkages between UCPM activities in prevention and preparedness and broader EU-wide programmes (e.g. Cohesion/European Regional Development Fund, European Investment Bank, etc.) should be explored. Generally speaking, the stakeholders' perception of the UCPM's efficiency is positive; particularly that emergency response and administrative costs associated to implementing the UCPM decision are considered acceptable by most Participating States. The external evaluation has attempted to carry out analysis of costs for comparable response missions (e.g. aerial forest fires missions; transport of modules and in-kind assistance; etc.). Although evidence cannot be considered sufficient to draw final conclusions, this confirmed the need to systematically carry out cost analyses and, possibly, determine cost benchmarks (or ranges) for certain type of missions/transport.

Internal coherence is deemed satisfactory by the external evaluation. Several cross-cutting activities of the Mechanism (e.g. lessons learned, advisory missions, peer reviews, exchange of experts, etc.) provide for an integrated prevention-preparedness-response approach. Externally, linkages between the UCPM and other policy areas have increased compared to the previous framework (2007-2013). This was confirmed by interviews with a large number of EU services. Focusing on a result-based agenda and milestones is seen as conducive to a more effective cooperation and it is felt that enhanced coordination mechanisms need to be considered in order to improve synergies between instruments funding prevention and preparedness activities in Neighbourhood and Pre-Accession countries. As far as response outside of Europe is concerned, the comparative advantage of civil protection intervention àa-vis humanitarian assistance needs to be carefully considered as the UCPM becomes increasingly activated for dealing with humanitarian and complex emergencies. The existence of a single European "hub" for information sharing and operational coordination, the introduction of common European standards for disaster response capacities and common guidelines on risk assessments, which have contributed to the first-ever overview of risks in the EU, are perceived as having delivered high EU added value in the areas of disaster response, preparedness and prevention, respectively. Increased capacity and predictability of emergency response capacity provided by the UCPM would not have been possible without coordinated effort at EU level. Over the next years, introducing indicators to monitor the effectiveness and efficiency of the UCPM more adequately may provide additional evidence to appreciate (and communicate) the added value of EU-level action in civil protection. Finally, knowledge-sharing and training activities are critical to creating a community of European civil protection practitioners and, ultimately, to ensuring sustainability of the UCPM's results. However, at a system level, several issues are considered as critical for the sustainability of the UCPM going forward. A viable financial model needs to ensure availability of capacities in the Voluntary Pool; an efficient strategy to build the European civil protection human resource capacity needs to be rolled out; the role of the private sector in the implementation of UCPM activities needs to be clarified; end-users needs need to inform design and monitoring of prevention and preparedness projects financed by the UCPM, in order to maximize their impact beyond the project closure (impact should be evaluated one or two years after funding has ended). The UCPM needs to get closer to the territorial actors while supporting the role of the national civil protection authorities in cementing the risk management governance all along the European-national-regional-local chain. The results of this Evaluation provide a good analytical basis to start reflection on a future framework. The Commission invites the European Parliament and the Council to take note of these evaluation findings. ANNEX 1: INTERVENTION LOGIC OF THE UCPM







ANNEX 2: PROCEDURAL INFORMATION A) Lead DG: European Civil Protection and Humanitarian Aid Operations (ECHO) B) Reference in Decide: 2016/ECHO/007 C) Organisation: The external evaluation was supervised by an Inter-service Steering Group (ISG), composed of representatives of the EEAS and the following European Commission (EC) DGs: CLIMA, DEVCO, ECHO, ENER, ENV, HOME, JRC, MOVE, NEAR, RTD, SANTE and SG. The external evaluation contract had a length of 9 months, from October 2016 to June 2017. Four meetings were held: ·9 November 2016: Kick-off meeting ·8 December 2016: Inception report meeting ·3 March 2017: Interim report meeting ·20 April 2017: Draft final report meeting D) Quality assurance: ·The ISG assessed the external draft Final Report against the following criteria: compliance with technical specifications; accuracy and completeness of data; relevance and soundness of analysis; robustness of conclusions; usefulness of recommendations; general clarity of the report. ·The ISG provided the following main comments to the external draft Final Report: oCertain missing elements should be provided: a chapter explaining the intervention, a chapter on the evaluation methodology, the case study reports, a proposal on the dissemination plan…; oThe soundness of the arguments should be improved, and triangulation should be more evident;

oConclusions on cost-effectiveness should be better supported on quantitative data; oConclusions on effectiveness, coherence and EU added value should be better elaborated; oSome links between conclusions and recommendations should be strengthened; oRecommendations should be further tailored and detailed; oThe general clarity of the report should be improved, and the report should be shortened. These comments were fully taken into account in the final version of the report. E) Evidence: Several information sources were used to reach triangulated conclusions, i.e. document review; analysis of the EC's project database (‘HOPE’); face to face and telephone interviews; four surveys (to national ministries, civil protection authorities, Trainers and national training coordinators, civil protection experts and project coordinators); an Open Public Consultation, and case studies. F) External expertise: The evaluation was carried out by an external consultant, ICF Consulting Services Limited, who was selected through a competitive process under DG ECHO's evaluation framework contract. The current Staff Working Document aims at providing a self-reflected summary of all sources of evidence, including the evaluation carried out by the external consultant, about the first years of existence of the current Mechanism. The report is available at: http://bookshop.europa.eu . LIST OF EVALUATION QUESTIONS Effectiveness ØQuestion 1. To what extent have the objectives of the Decision been achieved? To what extent has the Union Mechanism contributed to achieving the general objective of the Decision? To what extent has progress been made in relation to the four specific objectives? Given the current timeframe, is the Union Mechanism on track to achieve the specific objectives of the Decision? To what extent have external factors influenced the observed progress towards the objectives? Efficiency ØQuestion 2. To what extent are the costs of the activities performed under the UCPM justified when compared to their benefits? To what extent did ECHO achieve cost-effectiveness in its actions under the UCPM? To address this question the evaluator is invited to propose a dedicated, overall approach. What main factors influenced cost-effectiveness of the actions? Were there significant differences in costs and objective achievement between the Participating States? If so, what are the underlying causes? Which decisions could contribute to improve cost-effectiveness? ØQuestion 3: To what extent have the monitoring arrangements contributed to an efficient and effective implementation of the intervention? Are the indicators currently set by the Decision adequate and sufficient to capture to monitor successful implementation of the Decision? Which, if at all, additional indicators should be considered? Has data been properly collected and monitored? Is data generally available? Relevance ØQuestion 4: To what extent are the Decision's objectives still relevant to the needs identified and to the European Commission's priorities for 2014-2020?

To what extent do the general and specific objectives of Decision still correspond to current needs within the EU? To what extent have the general and specific objectives of the Decision proved to be appropriate to address the issues highlighted in the Impact Assessment (2011) Were the main findings and lessons learnt presented in the Ex-Post Evaluation of the Community Civil Protection Mechanism (2007-2013) taken into account for the implementation of the UCPM? Has the Mechanism been flexible enough to address emerging/unanticipated critical issues in the disaster risk management arena? To what extent has the UCPM integrated in its functioning as well as in its activities scientific and technological research and development that has become available since its creation? Coherence ØQuestion 5. To what extent is this intervention internally and externally coherent? How well are the different activities of the UCPM articulated, interlinked, and mutually supportive? What examples of synergies exist between different activities and areas (i.e. prevention/preparedness/response) of the UCPM? Can further synergies be sought? Are there any missed opportunities? To what extent are the potential synergies between the UCPM and other EC policy areas as well as international frameworks/initiatives being exploited? EU added value ØQuestion 6. What is the EU Added Value of the UCPM, including the added value to other participating states? What results of the intervention can be identified that could not have been achieved without a coordinated effort at the EU level? Which criteria could be considered to demonstrate EU added value of the UCPM-supported interventions in prevention, preparedness, and response? Sustainability ØQuestion 7. To what extent are the results achieved by the UCPM sustainable? To what extent are the results achieved by the UCPM likely to last beyond the timeframe of the intervention? Are adequate arrangements in place to ensure (financial, capacity, technical, etc.) sustainability of current and future interventions?



ANNEX 3: OVERVIEW OF ACTIVITIES CARRIED OUT UNDER THE UCPM (2014-2016) Prevention Preparedness Response Improving the knowledge base for disaster prevention Emergency Response Coordination Centre (ERCC) Grants for transport and related logistics, inside the Union EU overview of risks Early warning and analysis Grants for procurement, inside the Union: (1) Dispatching expertise (2) Transport and related logistics Prevention projects Preparedness projects Grants for transport and related logistics, outside the Union Training Grants for procurement, inside the Union: (1) Dispatching expertise (2) Transport and related logistics Training Network Civil Protection Mechanism Exercises Exercises on civil protection modules, technical assistance and support teams and EU Civil Protection Teams (EUCPT) The European Emergency Response Capacity (EERC) Voluntary Pool Maintenance and further development of the Common Emergency Communication and Information System (CECIS) Addressing temporary shortcomings Exchange of experts Programme Lessons Learned Programme Advisory missions for prevention and preparedness in and outside the Union Peer reviews Awareness raising and dissemination of good practices CECIS hosting, ERCC security and resilience ANNEX 4: SYNOPSIS REPORT ON STAKEHOLDER CONSULTATION Table of Content 1. INTRODUCTION

6.6 EU added value (only for in-de pth knowledge) 7. CONCLUSIONS 1.INTRODUCTION The general objective of the consultation activities carried out in the context of the Interim Evaluation of the UCPM was to ensure that all concerned stakeholders had the opportunity to express their views on the performance of the UCPM during the first half of its implementation. In line with this, the following specific objectives were set out: a)Collect views and opinions of the European citizens on the results achieved by the UCPM to date; b)Determine the importance of different priorities for civil protection and disaster risk management as they are perceived across the EU; c)Obtain targeted feedback and technical advice from the primary stakeholders of the UCPM; d)Gather evidence to support the interim evaluation of the UCPM for each of the main evaluation criteria. The Commission contracted an external consultant to carry out the consultations, prepare a report, and integrate the main findings into the final evaluation report. 2.