4. The Commission Staff Working Document on the evaluation builds on the evidence provided therein as well as feedback received during discussions with Participating States and the OPC. It also takes into account the results of the European Court of Auditors' (ECA) review of the UCPM, the Commission's Report on progress made and gaps remaining in the Voluntary Pool, as well as other reports publicly available. Overall, the Commission finds the design of the external evaluation adequate to the research questions and based on a satisfactory methodological approach. One limitation should be noted, namely the limited statistical base of the external evaluation. Most of the data gathered by the contractor during the independent evaluation's timeframe are of qualitative nature and refer to "perceptions" of stakeholders on the UCPM performance. This limitation points also to the need to strengthen the monitoring system of the UCPM in particular in relation to including additional quantitative indicators of performance (as highlighted by the recent ECA report). The external evaluation has not proposed specific additional indicators to fill this gap. Nevertheless, within the limits of data-gathering and analysis, the evaluation findings are credible. The conclusions are well balanced and logical consequences of findings are linked to the available evidence.

3. An independent evaluation of the activities implemented under the Union Mechanism was carried out between November 2016 and June 2017. The fact-finding phase included a comprehensive document review and a targeted stakeholders' consultation involving, among others, civil protection authorities from 26 Participating States, 14 international partners (e.g. UN agencies, OECD, World Bank, Red Cross and Red Crescent, NGOs etc.) and officials from 15 Directorates-General and services of the European Commission. Three case studies were also conducted (two in-country and one desk review). In addition, an Open Public Consultation (OPC) was carried out online for three months (November 2016 – February 2017), with some 130 responders from all over Europe providing feedback on the UCPM. A report on the outcome of the OPC was published at the end of March 2017.

2. Under Decision No 1313/2013/EU of 17 December 2013 on a Union Civil Protection Mechanism (hereafter 'the Decision'), the Commission is required to evaluate the performance of the UCPM and report to the European Parliament and the Council on the results obtained and the qualitative and quantitative aspects of the implementation of the Decision by no later than 30 June 2017. This report presents the results of the Interim Evaluation (hereafter 'the Evaluation') and is accompanied by a Staff Working Document covering the detailed findings of that evaluation.

1. The Union's Civil Protection Mechanism (hereafter: the 'UCPM') constitutes a visible expression of European solidarity by ensuring a practical and timely contribution to prevention, preparedness and response to disasters of all kinds occurring inside or outside the Union. The UCPM facilitates the mobilisation and coordination of assistance provided by the Participating States with the overarching goal of protecting Europe's citizens, economy and the environment. The Emergency Response Coordination Centre (ERCC) and the European Emergency Response Capacities (EERC, also referred to as the 'Voluntary Pool' of pre-committed capacities and trained experts from Member States) are core elements of the functioning of the Mechanism.

5. The evaluation addresses the five mandatory criteria set out in the Commission's Better Regulation Guidelines: (i) Relevance; (ii) Effectiveness; (iii) Efficiency; (iv) Coherence; and (v) EU added value. An additional criterion, (vi) Sustainability, was added to the scope of this evaluation. The main findings are summarized in the paragraphs below.

6. There is sufficient evidence to demonstrate the continued relevance of UCPM objectives to Europe's needs. The 2017 Eurobarometer survey on civil protection shows that a vast majority of European citizens support a common EU policy in the field of civil protection and agree that the EU should lead the coordination of disaster response to affected countries, both in Europe and in the rest of the world. Recent resolutions of the European Parliament have also called on the Commission to further support cooperation among European civil protection authorities. The stakeholders consulted also highlighted the flexibility of the UCPM to rapidly adapt and address newly emerging crises which were not "traditional" civil protection emergencies (e.g. the Ebola outbreak in West Africa and the migrant crisis in Europe). The increasing number and scope of UCPM activations outside of Europe pose, however, the challenge of finding the right balance between being a framework of mutual assistance between European countries in the aftermath of natural and man-made disasters and evolving towards a more complex instrument capable of addressing crises of a multiple nature and with a global reach.

7. An analysis of the UCPM effectiveness showed encouraging progress under each of the specific objectives. The UCPM is fully on track to achieve objectives pertaining to disaster prevention framework (objective 1) and readiness for disasters (objective 2), based on the indicators set by the legislation and on feedback received by key stakeholders consulted. The UCPM's response to disasters (objective 3) has been timely and effective according to the vast majority of stakeholders consulted. Confirming one of the findings of the ECA review of the UCPM, the external evaluation found limitations in respect of consistent and robust quantitative data (e.g. speed, cost, needs assessments) to corroborate the stakeholders' very positive views on UCPM disaster response. Lastly, public awareness and preparedness to disasters (objective 4) have received slightly less satisfactory views from stakeholders when compared to the other objectives. In this respect, a directly attributable link between awareness of Union citizens of risks in their regions as the objective set by the Decision and activities carried out by the UCPM would be difficult to establish. Nonetheless, the Commission recognises that given the changing risk landscape in Europe, reinforcement of action related to citizen awareness would be relevant and timely.

8. Generally speaking, the stakeholders' perception of the UCPM's efficiency is positive. Emergency response and administrative costs associated to implementing the UCPM decision are considered acceptable by most Participating States. The external evaluation attempted to carry out analyses of costs for comparable response missions (e.g. aerial forest fires missions; transport of modules and in-kind assistance; etc.). The evidence cannot be considered fully sufficient to draw definitive conclusions. However, this confirmed the desirability of carrying out more detailed cost analyses and, possibly, determining benchmarks (or range of costs) for deployment and transport of assistance provided under the UCPM.

9. Internal coherence, that is coherence within the different pillars of the Mechanism, is deemed satisfactory by the external evaluation. Several cross-cutting activities of the UCPM (e.g. lessons learned, advisory missions, peer reviews, exchange of experts, etc.) provide for an integrated prevention-preparedness-response approach. External coherence, that is the linkages between the UCPM and other Union policy areas, has increased as compared to the previous framework (2007-2013). This was confirmed by interviews with a large number of Commission services. Focusing on a result-based agenda and milestones is seen as conducive to a more effective cooperation and it is felt that enhanced coordination mechanisms need to be considered in order to improve synergies between instruments funding prevention and preparedness activities, for instance in the Neighbourhood and Pre-Accession countries. As far as response outside of Europe is concerned, the comparative advantage of civil protection intervention vis-a-vis humanitarian assistance needs to be carefully considered as the UCPM becomes increasingly activated for dealing with humanitarian and complex emergencies. In the same vein, a better articulation with other EU external policies (e.g. development aid) and tools is deemed necessary considering that the UCPM is increasingly intervening outside the EU (e.g. Ebola outbreak, Nepal earthquake, migration crisis) where recurrent disasters might interact with other risk drivers, such as conflict and fragility.

10. The existence of a single European "hub" for information sharing and operational coordination, the introduction of common European standards for disaster response capacities and common guidelines on risk assessments, which have contributed to the development of an overview of risks in the EU in the framework of the UCPM legislation, are perceived as having delivered high EU added value in the areas of disaster response, preparedness and prevention, respectively. Increased capacity and predictability of the EERC provided by the UCPM in the form of the 'Voluntary Pool' would not have been possible without coordinated effort at EU level. Over the next years, introducing indicators to monitor the effectiveness and efficiency of the UCPM more adequately may provide additional evidence to appreciate (and communicate) the added value of EU-level action in civil protection.

11. Knowledge-sharing, expert exchange and training activities are seen as essential to creating a community of European civil protection practitioners and, ultimately, to ensuring sustainability of the UCPM's contribution to supporting safer and more resilient societies. The Disaster Risk Management Knowledge Centre, an online hub launched at the end of 2015, is providing services to enhance knowledge sharing and capacity building across the UCPM Participating States. However, at a system level, several issues are considered key to the future sustainability of the UCPM contribution. The financial model needs re-examining to ensure availability of capacities in the Voluntary Pool adapted to risk-based needs; an efficient strategy to build the European civil protection human resource capacity needs to be rolled out; and the role of the private sector in the implementation of UCPM activities should be clarified. In order to maximize the impact of prevention and preparedness projects financed by the UCPM, end-users' needs should inform design and monitoring of the projects. Further consideration should be given to evaluating their impact over a longer timeframe. Finally, recognising the importance of the role of regional and local authorities in disaster management, the work of the UCPM needs to be brought closer to local actors, while primarily supporting the role of the national civil protection authorities. Due attention would also need to be given to cross-border prevention and preparedness, thereby strengthening the risk management governance along the European-national-regional-local chain.

3.Conclusions

12. Taking into account the findings of the external evaluation report, its own assessment of the qualitative and quantitative evidence provided as well as operational experience and lessons learned, the Commission's overall conclusion is that the Mechanism is on a solid footing to achieve the specific and general objectives of the Decision. For the period 2014-2016 under evaluation, the performance of the UCPM has showed encouraging results in each of the three pillars (prevention/preparedness/response) and overall is viewed positively by the Participating States and the main stakeholders. At the same time, the evaluation also highlighted a number of challenges that, if addressed promptly, could increase even further the impact of UCPM supported action. These aspects are further highlighted in the accompanying Staff Working Document with recommendations (summarized below) encompassing the prevention, preparedness and response pillars of the UCPM. A general area identified for improvement is the strengthening of the UCPM results monitoring framework, including better measurability through the possible introduction of quantitative indicators and baselines, in support of a stronger focus on impact over time.

3.1.Prevention

13. Increase the focus of the annual prevention (and preparedness) projects that the UCPM finances every year. The programme, which has been running since 2000, has financed a large number of projects without always capitalising on their results. Given the relatively small size of financing (between 2014 and 2016 average EU funding was EUR 470 000), projects need to: target the needs of the end-users, incorporate measures for scaling up their results and achieve bigger impact. Cooperation between the UCPM and the EU climate change adaptation agenda should be enhanced.

14. Better articulate the linkages between the UCPM prevention (and preparedness) projects and broader EU programmes (e.g. Cohesion/European Regional Development Fund, European Investment Bank, etc.) building on work started with support of the Disaster Risk Management Knowledge Centre. A more strategic link with such programmes could help maximize the uptake of project results and leverage additional technical and financial resources for achieving more significant results in prevention.

15. Strengthen the link between prevention and preparedness/response activities. This recommendation, which had already been highlighted in 2014 by the ex-post evaluation of the previous Civil Protection Mechanism (2007-2013), remains relevant at the mid-term mark of the 2014-2020 Mechanism. The need to match response capacities with the major risks in Europe and global emergencies will represent the key challenge to ensure continued relevance of UCPM. The 'Overview of Risks in the EU' underscored that forward looking assessments will need to increasingly take into account "emerging" risks (e.g. cascading risks, such as technological accidents resulting from natural events as well as the loss of critical infrastructure, nuclear and industrial accidents due to the increased risks of terrorism and cyber-attacks). A performance-based approach to risk assessment could help go beyond the estimated physical damage to infrastructure and focus on the level of vital services required in the aftermath of a disaster, enhancing contingency planning and response operations by civil protection authorities.

3.2.Preparedness

16. Revise the capacity goals for the Voluntary Pool with an evidence-based approach. The capacity goals for modules and other response assets are defined by the UCPM Implementing Decision and should be revised against the main/critical risks Europe or other beneficiary countries are facing and prioritized accordingly. Capacity goals for different profiles of individual experts (currently not set by the Decision) would be required in order to map out more clearly technical resources and gaps in Europe. A closer dialogue with the Participating States is critical in order to capture all national capacities, including those which are not registered in the Voluntary Pool but may still be deployable via the UCPM. Similarly, the spatial distribution of (natural and man-made) risks across and beyond Europe should inform the geographical distribution of certain response capacities, whilst ensuring the ability to provide adequate assistance under the auspices of the UCPM wherever it is needed.

17. Re-assess the system of incentives for pooling assets in the Voluntary Pool based on lessons learnt and other knowledge emerging from operations. The level of co-financing for the deployment of Voluntary Pool capacities or the mobilisation of buffer capacities (particularly outside of the EU); support to multi-national modules and response capacities; insurance coverage for health and third-party liability for experts deployed are, among others, incentives needed to strengthen the UCPM capacity to respond. The administrative burden involved in registering and certifying modules also requires further improvement, although this is being addressed by the Commission with a more streamlined and coherent planning process for exercises, certification and registration of assets in the Voluntary Pool.

18. Develop a more need-based, efficient approach for the training and exercise programmes. Shifting the focus of training from outputs (i.e. number of individuals trained) to competences allows to address needs that stem from operations. This is currently being addressed by the Commission with the forthcoming 'Strategic Framework for Training and Exercises', which is built around competences. Furthermore, a robust framework for evaluating the training courses administered as well as the individuals trained is considered as a priority in order to match the needs on the ground with the best technical resources available. Similar considerations were raised with regards to the exercises. In addition, considering different implementation models for the UCPM "basic" courses, such as the Union Civil Protection Mechanism Introduction Course (CMI) targeting large audiences and specialized/advanced training (e.g. training for deployment; assessments; medical teams etc.), which are delivered to well-identified groups of experts, could help increase value for money. Finally, the Strategic Framework for Training and Exercises should also take into account needs stemming from emerging and evolving threats, such as CBRN and critical infrastructure risks or terrorist attacks, while avoiding duplication with existing training initiatives supported by the EU.

19. Expand the end-user side of trans-national early warning systems platforms, such as the European Forest Fire Information System (EFFIS) and the European Flood Awareness System (EFAS). These systems bring a very high EU added value by consolidating national data into Europe-wide forecasting tools that provides support to decision-making of emergency managers in the ERCC and in the Participating States. Developing the community of end-users would help complete the technology platform, allowing experts from Participating States to evaluate the performance of early warning systems and response operations with the objective of improving the practice. Such a development would make the tools more relevant for all countries within Europe, regardless of their level of preparedness.

3.3.Response

20. Promote a consistent collection of quantitative data and produce systematic analysis of response missions to improve future performance. This could help establish baselines/benchmarks (lower/higher bounds or a range) for speed and cost, allowing comparisons between emergency responses and contributing to enhance the efficiency of response missions. Needs assessments in the Requests for Assistance also need to be made available on regular basis and provided in a more standardised format.