COMMISSION STAFF WORKING DOCUMENT Interim Evaluation of the Union Civil Protection Mechanism (2014-2016) Accompanying the document Report from the Commission to the European Parliament and the Council on the Interim Evaluation of the Union Civil Protection Mechanism for the period 2014-2016
Date: 2017-08-30
Institutions: Directorate-General for European Civil Protection and Humanitarian Aid Operations (ECHO), European Commission
Subjects: EU financial instrument, European cooperation, aid to disaster victims, civil defence, emergency aid, environmental risk prevention, man-made disaster, natural disaster, project evaluation

EN 

   EN 

   EUROPEAN COMMISSION  Brussels, 30.8.2017  SWD(2017) 287 final   COMMISSION STAFF WORKING DOCUMENT Interim Evaluation of the Union Civil Protection Mechanism (2014-2016) Accompanying the document Report from the Commission to the European Parliament and the Council on the Interim Evaluation of the Union Civil Protection Mechanism for the period 2014-2016  {COM(2017) 460 final}  
 
 
TABLE OF CONTENTS 

LIST OF ABBREVIATIONS AND ACRONYMS ............................................................................. 4 

1 

INTRODUCTION ......................................................................................................................... 5 

1.1 

1.2 

Purpose of the evaluation ........................................................................................................ 5 

Scope of the evaluation ........................................................................................................... 6 

2  BACKGROUND TO THE INITIATIVE .................................................................................... 6 

2.1 

2.2 

Baseline ................................................................................................................................... 6 

Description of the initiative and its objectives ........................................................................ 7 

3  EVALUATION QUESTIONS .................................................................................................... 10 

4  METHOD ..................................................................................................................................... 11 

5 

IMPLEMENTATION STATE OF PLAY ................................................................................ 13 

6  ANSWERS TO THE EVALUATION QUESTIONS ............................................................... 19 

6.1 

6.2 

6.3 

6.4 

6.5 

6.6 

Relevance .............................................................................................................................. 19 

Effectiveness ......................................................................................................................... 22 

Efficiency .............................................................................................................................. 30 

Coherence .............................................................................................................................. 31 

EU Added Value ................................................................................................................... 35 

Sustainability ......................................................................................................................... 37 

7  CONCLUSIONS .......................................................................................................................... 39 

ANNEX 1: INTERVENTION LOGIC OF THE UCPM ................................................................. 42 

ANNEX 2: PROCEDURAL INFORMATION ................................................................................. 43 

ANNEX 3: OVERVIEW OF ACTIVITIES CARRIED OUT UNDER THE UCPM (2014-2016)

46 

ANNEX 4: SYNOPSIS REPORT ON STAKEHOLDER CONSULTATION .............................. 47 

1. 

INTRODUCTION ....................................................................................................................... 48 

2.  SCOPE OF THE CONSULTATION ........................................................................................ 48 

3.  STAKEHOLDER GROUPS COVERED BY THE CONSULTATION ACTIVITIES ........ 48 

4.  CONSULTATION METHODS AND TOOLS ......................................................................... 49 

4.1 

4.2 

4.3 

4.4 

Interviews .............................................................................................................................. 49 

Online surveys ....................................................................................................................... 50 

Open Public Consultation ...................................................................................................... 50 

Case Studies .......................................................................................................................... 51 

5.  RESULTS OF EACH OF THE CONSULTATION ACTIVITIES (TARGETED) .............. 51 

5.1 

Relevance .............................................................................................................................. 51 

2 

 
 
 
 
 
5.2 

5.3 

5.4 

5.5 

5.6 

Effectiveness ......................................................................................................................... 52 

Efficiency .............................................................................................................................. 52 

Coherence .............................................................................................................................. 52 

EU Added Value ................................................................................................................... 53 

Sustainability ......................................................................................................................... 53 

6.  RESULTS OF THE CONSULTATION IN THE FRAMEWORK OF THE OPC ............... 54 

6.1 

6.2 

6.3 

6.4 

6.5 

6.6 

General Questions (for both questionnaires) ......................................................................... 54 

Relevance (only for in-depth knowledge) ............................................................................. 54 

Effectiveness (only for in-depth knowledge) ........................................................................ 55 

Efficiency (only for in-depth knowledge) ............................................................................. 55 

Coherence (only for in-depth knowledge) ............................................................................. 55 

EU added value (only for in-depth knowledge) .................................................................... 55 

7.  CONCLUSIONS .......................................................................................................................... 56 

3 

 
 
LIST OF ABBREVIATIONS AND ACRONYMS  

CECIS   
CPFI 
Decision 

DG ECHO 
DG DEVCO 
DG GROW 
DG NEAR 
DG RTD 
DG SANTE 
EASO 
EC 
ECA 
ECDC 
EEA 
EEAS 
EERC 
EFAS 
EFFIS 
EFTA 
EMC 
EMS 
EMSA   
ENI 
ENP 
ERCC 
Evaluation 
IFRC 
IOM 
IPCR 
ISAA 
ISG 
JRC 
Mechanism 
MIC 
ModEx   
OECD   
OPC 
OPCW   
PPRD 
ToR 
SWD 
UCPM   
UN 
Union Mechanism 
UNISDR 

Common Emergency Communication and Information System 
Civil Protection Financial Instrument 
Decision No. 1313/2013/EU of the European Parliament and the Council of 17 
December 2013 
Directorate-General European Civil Protection and Humanitarian Aid Operations 
Directorate-General for International Cooperation and Development 
Directorate-General for Internal Market, Industry, Entrepreneurship and SMEs 
Directorate-General for Neighbourhood and Enlargement Negotiations 
Directorate-General for Research and Innovation 
Directorate-General for Health and Food Safety 
European Asylum Support Office 
European Commission 
European Court of Auditors 
European Centre for Disease Prevention and Control 
European Economic Area 
European External Action Service 
European Emergency Response Capacity 
European Flood Alert System 
European Forest Fires Information System 
European Free Trade Association  
European Medical Corps 
Emergency Management System 
European Maritime Safety Agency 
European Neighbourhood Instrument 
European Neighbourhood Policy  
European Response Coordination Centre 
Interim evaluation 
International Federation of Red Cross and Red Crescent Societies 
International Organization for Migration 
Integrated Political Crisis Response 
Integrated situational awareness and analysis 
Inter-service Steering Group 
Joint Research Centre 
Union Civil Protection Mechanism 
Monitoring and information centre 
Module exercise 
Organisation for Economic Cooperation and Development 
Open Public Consultation 
Organisation for the Prohibition of Chemical Weapons 
Prevention, Preparedness and Response to Natural and Man-made Disasters 
Terms of Reference 
Commission Staff Working Document 
Union Civil Protection Mechanism 
United Nations 
Union Civil Protection Mechanism 
United Nations Office for Disaster Risk Reduction

4 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1 

INTRODUCTION 

1.1 

Purpose of the evaluation 

This  Staff  Working  Document  presents  the  results  of  the  interim  evaluation  (hereafter  the 
'Evaluation')  of  the  implementation  of  Decision  No.  1313/2013/EU  of  the  European 
Parliament and the Council of 17 December 20131 (hereafter the 'Decision') on a Union Civil 
Protection  Mechanism  (hereafter  'the  Union  Mechanism'  or  'UCPM').  The  purpose  of  the 
Evaluation is to: 

1) assess the performance of the Union Mechanism in order to determine progress towards 
the general and specific objectives set out in Article 1 and 3(1) of the Decision during the 
first half of its implementation period (i.e. 2014-2016); 

2) identify any gaps or shortcomings in the current legislative framework2; 

3)  put  forward  recommendations  to  increase  the  Mechanism's  effectiveness  and  improve 
the implementation of the existing framework; 

4)  provide  input  for  possible  proposals  to  amend  the  Decision  or  implementing  acts 
adopted thereof; 

5) if appropriate, inform the review of the financial breakdown of the Union Mechanism as 
set out in Article 19(5) of the Decision3. 

Furthermore, in accordance with point (a) of Article 34(2) of the Decision, the Commission is 
required to report to the European Parliament and the Council on the results obtained and the 
qualitative and quantitative aspects of the implementation of the Decision by no later than 30 
June 2017, and to put forward, if appropriate, proposals for amendments to this Decision. 

To  this  end,  the  Directorate-General  for  European  Civil  Protection  and  Humanitarian  Aid 
Operations  (DG  ECHO)  contracted  an  external  consultant  to  carry  out  an  independent 
evaluation of the activities implemented under the Union Mechanism4. As the reflections on 
the future of the Union Mechanism under the next Multi-annual Financial Framework (2020-
2027)  are  starting,  findings  from  the  Evaluation  could  usefully  contribute  to  developing  a 
future impact assessment, as recommended by the EU Better Regulation Agenda5. 

1 OJ L 347, 20.12.2013, p. 924 
Available at: http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2013:347:0924:0947:EN:PDF  
2 Namely, Decision 1313/2013/EU (see above) and Commission Implementing Decision 2014/762/EU of 16 October 2014 
laying down the rules for the implementation of Decision 1313/2013/EU (herein 'implementing rules'), OJ L 320, 6.11.2014, 
p. 1 
3 The financial envelope for the implementation of the Union Mechanism's actions in the period of 2014-2020 is EUR 368 
428  000.  Article  19(4)  specifies  that  this  financial  envelope  is  allocated  according  to  the  following  breakdown:  20% 
Prevention, 50% preparedness, 30% response, with the possibility to vary each by ± 8 percentage points. 
4 Final report available at: https://ec.europa.eu/echo/sites/echo-site/files/ucpm_final_report.pdf  
5  European  Commission,  Better  Regulation  for  Better  Results  —  An  EU  agenda,  COM  (2015)215,  and  Better  Regulation 
Guidelines (Commission Staff Working Document), SWD (2015) 111. 

5 

 
 
                                                            
1.2 

Scope of the evaluation 

As set out in the Interim Evaluation Roadmap6, the Evaluation covers all activities carried out 
under  the  Union  Mechanism  in  the  timeframe  spanning  from  January  2014  to  December 
2016. Actions targeting the UCPM Participating States7, as well as those performed in relation 
to  third  countries,  including  in  accordance  with  Article  28(2)8,  have  been  covered  by  the 
Evaluation. 

Article  3  of  the  Decision  spells  out  four  specific  objectives  against  which  the  Union 
Mechanism  should  be  evaluated  and  their  respective  results  indicators.  These  indicators 
encompass the spectrum of activities of the Union Mechanism across the fields of: (i) disaster 
prevention,  (ii)  preparedness,  (iii)  and  response  to  natural  and  man-made  disasters.  In 
addition,  cross-cutting  activities  relating  to  prevention,  preparedness  and  response  are 
included in the scope of the Evaluation. 

The  Evaluation  has  addressed  the  five  mandatory  criteria  set  out  in  the  European 
Commission's  (“the  Commission”)  Better  Regulations  Guidelines:  (i)  Relevance;  (ii) 
Effectiveness;  (iii)  Efficiency;  (iv)  Coherence;  and  (v)  EU  added  value.  In  addition,  one 
additional criteria, (vi) Sustainability, has been added to the scope of this evaluation. 

2 

BACKGROUND TO THE INITIATIVE 

2.1  Baseline 

Prior  to  the  entry  into  force  of  the  Lisbon  Treaty  in  2009,  EU  civil  protection  actions  and 
legislation were based on the catch-all provision of Article 308 of the EC Treaty, authorising 
the Council of the European Union (“the Council”) to act (by unanimity) where necessary in 
order to attain a Treaty objective in areas where there the EC Treaty provided no other legal 
basis. In line with the need to ensure better protection in disasters occurring both inside and 
outside  the  EU,  the  Union  Mechanism  was  established  in  2001  by  the  Council9  and 
subsequently recast in 200710. 

With  the  advent  of  the  Treaty  of  Lisbon  in  2009,  civil  protection  became  a  self-standing 
policy with its own legal basis provided by Article 196 of the Treaty on the Functioning of the 
EU, with the aim to: 

6 The Roadmap of the Interim Evaluation can be consulted here: http://ec.europa.eu/smart-
regulation/roadmaps/docs/2016_echo_007_evaluation_ucpm_en.pdf 
7 As of June 2017, the Participating States of the Union Mechanism include all EU Member States, Iceland, Norway, Serbia, 
the Former Yugoslav Republic of Macedonia, Montenegro and Turkey. 
8  "Financial  assistance  may  also  be  granted  to  candidate  countries  and  potential  candidates  not  participating  in  the  Union 
Mechanism as well as to countries that are part of the ENP (…)" 
9 Council Decision 2001/729/EC, Euratom of 23 October 2001 establishing a Community mechanism to facilitate reinforced 
cooperation in civil protection assistance interventions (OJ L 297, 15.11.2001, p.7) 
10 Council Decision 1007/779/EC, Euratom of 8 November 2007 establishing a Community Civil Protection Mechanism (OJ 
L 314, 1.12.2007, p.9) 

6 

 
 
 
 
                                                            
a.  Support and complement Member States' action at national, regional and local 
level in risk prevention, in preparing their civil-protection personnel and in 
responding to natural or man-made disasters within the Union; 

b.  Promote swift, effective operational cooperation within the Union between 

national civil protection services; 

c.  Promote consistency in international civil protection work. 

Several challenges needed to be addressed at the time. The Impact Assessment11 of the civil 
protection regulatory framework carried out in the context of the preparation of the Decision 
had highlighted, among others, the following shortcomings. 

  Prevention:  Lack  of  coordination  between  sector  specific  prevention  policies; 
under-developed  organisation  for  the  sharing  of  experience;  insufficient  links 
between  prevention,  preparedness  and  response;  limited  awareness,  vision  and 
strategic investments. 

  Preparedness: Lack of consistency among national training programmes hindering 
the  sharing  of  experience;  limited  scope  of  EU  preparedness  action  and  limited 
links  with  national  programmes,  leading  to  lower  preparedness  levels;  uncertain 
level of preparedness for major emergencies. 

  Response:  Reactive  and  ad-hoc  mandate  of  the  Mechanism  limiting  the 
effectiveness,  efficiency  and  coherence  of  the  European  disaster  response;  some 
needs  not  being  met  due  to  the  lack  of  specific  assets  (capacity  gap);  limited 
financial  and  logistical  transport  solutions  slowing  down  or  hindering  optimal 
response; burdensome administrative procedures jeopardising a swift and efficient 
response. 

Based on a Commission proposal, the European Parliament and the Council repealed previous 
legislation  and  adopted  Decision  No  1313/2013/EU  of  the  European  Parliament  and  of  the 
Council  of  17  December  2013  on  a  "Union  Civil  Protection  Mechanism".  This  Decision  is 
currently in force and governs actions and activities in the field of civil protection. 

2.2  Description of the initiative and its objectives 

Decision 1313/2013 added to the Mechanism's original focus on disaster response, a greater 
emphasis  on  cooperation  for  disaster  prevention  and  preparedness  with  particular  focus  on 
national  risk  assessment  and  risk  management  planning.  To  enhance  preparedness  of 
European  countries  to  respond  to  disasters,  the  Mechanism  created  a  Voluntary  Pool  of 
response  capacities  pre-committed  by  the  Participating  States  for  use  in  European  civil 
protection operations (see further the paragraph below on Disaster Preparedness). 

11 Impact Assessment (2011) Review of the civil protection regulatory framework, SEC(2011) 1632 final 

7 

 
 
 
 
 
 
 
                                                            
Article  1(2)  clarifies  the  scope  of  the  UCPM,  namely  that  protection  ensured  by  the  Union 
Mechanism  shall  cover  primarily  people,  but  also  the  environment  and  property,  including 
cultural  heritage,  against  all  kinds  of  natural  and  man-made  disasters,  including  the 
consequences  of  acts  of  terrorism,  technological,  radiological  or  environmental  disasters, 
marine pollution, and acute health emergencies, occurring inside or outside the Union. In the 
case  of  consequences  of  acts  of  terrorism  or  radiological  disasters,  the  UCPM  covers  only 
preparedness and response actions. 

The  Union  Mechanism  is  open  to  the  participation  of  the  European  Free  Trade  Association 
(EFTA)  countries  which  are  members  of  the  European  Economic  Area  (EEA)  and  other 
European  countries  (when  agreements  and  procedures  so  provide),  as  well  as  to  acceding 
countries,  candidate  countries  and  potential  candidate  in  accordance  with  the  general 
principles  and  terms  laid  down  in  framework  agreements  or  similar  arrangements12.  In 
addition,  candidate  countries  and  potential  candidates  not  participating  in  the  Union 
Mechanism,  and  countries  that  are  part  of  the  European  Neighbourhood  Policy  (ENP),  may 
benefit from some of the activities financed under the Decision13. 

Disaster prevention 

Under the UCPM, Member States shall develop and make available to the Commission their 
national  risk  assessments  and  national  risk  management  capability  assessments.  The 
Commission  shall  provide  guidance  to  Member  States  in  carrying  out  risk  assessments  and 
facilitate the dissemination and exchange of information of good practices. This includes, in 
particular, improving the knowledge base on disaster risks (e.g. by developing an overview of 
risks the EU may face which takes into account the impact of climate change) and developing 
an  integrated  approach  to  disaster  management  of  natural  and  man-made  hazards.  The 
Commission shall facilitate the sharing of knowledge, best practices and information related 
to prevention, taking due account of existing international commitments (such as the Sendai 
Framework  for  Disaster  Risk  Reduction  2015-2030,  and  the  Paris  Agreement),  including 
through  voluntary  peer  reviews.  The  Commission  shall  also  finance  cooperation  projects  in 
the field of disaster prevention based on yearly calls for proposals, which are published on the 
DG ECHO website. 

Disaster preparedness 

The  creation  of  a  European  Emergency  Response  Capacity  (EERC)  in  the  form  of  a 
"Voluntary Pool" is one of the main innovations of the UCPM. Participating States can pre-
commit  different  types  of  response  capacities  for  participation  in  EU  Civil  Protection 
missions. The European Medical Corps (EMC) is a sub-set of the EERC through which teams 
and equipment from EU Member States can be rapidly deployed to provide medical assistance 
and public health expertise in response to emergencies inside and outside the EU. Through the 
Common  Emergency  Communication  and  Information  System  (CECIS),  a  web-based  alert 
and  notification  application  enabling  real  time  exchange  of  information,  the  UCPM 

12 Article 28(1) of Decision No 1313/2013/EU 
13 Article 28(2) of Decision No 1313/2013/EU 

8 

 
 
                                                            
Participating States and the Commission share a common information system for mobilizing 
and  deploying  these  pre-committed  assets.  This  increases  predictability  of  assistance  while 
significantly  reducing  deployment  times.  Participating  States  and  the  Commission  work 
closely  together  to  develop  quality  requirements  and  a  certification  process  for  different 
modules14 and other response capacities15, thus ensuring that all teams meet high quality and 
inter-operability standards and can effectively work together in the field. 

The  UCPM  also  supports  trainings  and  disaster  preparedness  exercises  to  test  and  further 
improve performance and coordination. Costs to upgrade existing national response capacities 
and certification costs, trainings and workshops can be financed by the EU. Once certified and 
part of the Voluntary Pool, the transport costs of teams in response to emergencies is eligible 
for  EU  co-financing  of  up  to  85%.  Additionally,  the  Union  Mechanism  works  via  its 
Emergency  Response  Coordination  Centre  (ERCC  –  see  further  paragraph  below)  to:  (i) 
monitor hazards, collect, analyse and disseminate information on disasters; (ii) map available 
assets in close cooperation with the UCPM's Participating States; and (iii) prepare plans for 
the  deployment  of 
teams  and/or  assets,  providing  emergency 
communications and monitoring tools. 

individual  experts, 

Disaster Response 

The  operational  heart  of  the  UCPM  is  the  24/7  Emergency  Response  Coordination  Centre 
(ERCC),  which  is  located  within  the  Commission's  DG  ECHO  in  Brussels.  The  ERCC 
supports a coordinated and rapid response to disasters both inside and outside Europe using 
resources from the countries participating in the UCPM. With a capacity to deal with several 
simultaneous  emergencies  on  a  24/7  basis,  the  ERCC  facilitates  a  coherent  and  efficient 
European response to emergencies, aiming to reduce unnecessary and inefficient duplication 
of  efforts.  Whenever  the  scale  of  an  emergency  overwhelms  the  response  capabilities  of  a 
country,  the  UCPM  may  be  activated  upon  official  request  of  that  country  or  the  United 
Nations (UN) and its agencies, as well as the International Organization for Migration (IOM), 
International  Federation  of  Red  Cross  and  Red  Crescent  Societies  (IFRC)  and  the 
Organisation for the Prohibition of Chemical Weapons (OPCW). When this occurs, the ERCC 
acts  as  an  operational  hub,  facilitating  the  coordination  of  assistance  made  available  by  the 
UCPM's  Participating  States  to  the  affected  country.  Requests  for  assistance  through  the 
UCPM can concern disasters of any type, both within and outside Europe16. The UCPM can 
also  be  activated  in  response  to  marine  pollution  and  other  maritime  disasters,  on  which  it 
coordinates closely with the European Maritime Safety Agency (EMSA). 

Assistance made available via the UCPM may consist of operational assets, in-kind assistance 
(relief items) and experts17. The latter can be deployed for needs assessment and coordination 
missions as well as to provide advice to a requesting country on prevention and preparedness 

14 Defined in Annex II of  the Commission Implementing Decision of 16 October 2014 (C(2014) 7489) 
15 Listed in Annex III of the Commission Implementing Decision of 16 October 2014 (C(2014) 7489) 
16Recent examples include: the Ebola outbreak in West Africa (2014), the floods in the Western Balkans (2014), the conflict 
in  Eastern  Ukraine  (2015),  the  voluntary  evacuation  of  EU  citizens  from  Yemen  (2015),  and  the  ongoing  refugee  crisis 
(2015-16). 
17 When civil protection assistance is requested by third countries, it is often combined with humanitarian aid. 

9 

 
 
                                                            
measures. The Commission can also co-finance the transport costs of assistance delivered via 
the UCPM. This enables a rapid distribution of assistance to the affected country and results 
in  a reduced budgetary  impact on the UCPM's Participating State(s)  offering  the assistance. 
Pooling  and  consolidating  shipments  of  assistance  for  the  affected  country  with  the 
facilitation of the ERCC, allows to further boost the efficiency of the European response. The 
ERCC  fosters  coordination  between  civil  protection  and  humanitarian  aid  operations. 
Maintaining  direct  links  to  the  civil  protection  and  humanitarian  aid  authorities  in  Member 
States  it  enables  a  smooth  and  real-time  exchange  of  information.  It  ensures  deployment  of 
coordination and assessment teams composed of humanitarian aid and civil protection experts 
to conduct joint needs assessments. 

Cross-cutting activities 

Prevention  and  preparedness  ("advisory")  missions  support  Participating  States  or  third 
countries  with  technical  assistance  on  prevention  and  preparedness  measures  through  the 
deployment  of  a  team  of  experts.  These  missions  can  be  requested  by  a  Participating  State, 
international organisation or third country. The exchange of expert programme allows for the 
secondment  of  civil  protection  experts  from  one  Participating  State  to  another,  with  some 
third  countries  also  being  eligible  to  participate.  This  exchange  allows  participants  to  gain 
direct knowledge and experience on all aspects of an emergency intervention and the different 
approaches  of  national  systems.  The  peer  review  programme  is  addressed  to  all  the 
Participating States (with their different systems and backgrounds) to identify – in a spirit of 
mutual  support  and  peer-to-peer  learning  –  areas  for  improvement.  Finally,  the  Decision 
promotes  the identification, analysis and implementation  of  lessons learnt distilled from the 
entire disaster cycle. 

The intervention logic (Annex 1) presents all the activities18 carried out by the Commission 
and the Participating States. Activities are linked to the respective objectives/indicators (as set 
out  by  Art.  3  of  the  Decision)  they  contribute  to  and  categorised  under  the  three  thematic 
pillars of disaster prevention, preparedness, and response. 

3 

EVALUATION QUESTIONS 

In  order  to  assess  the  implementation  and  performance  of  the  Union  Mechanism  and  to 
evaluate whether it is on course to achieve its objectives, the following general questions were 
used to guide the analysis. The full list of evaluation questions is provided in Annex 2. 

Criteria 
1.    Relevance 

2.    Effectiveness 

Evaluation question 
Are the objectives the UCPM is addressing still relevant to 
Europe's current problems and needs? 
To what extent have the changes/effects generated by UCPM 
activities contributed to achieving general and specific objectives 

18 As per Commission Decision on the Union Mechanism Annual Work Programmes (2014, 2015, 2016) and Commission 
Decision on financing the UCPM emergency response actions (2014, 2015, 2016). 

10 

 
 
 
                                                            
 
3.    Efficiency 

4.    Coherence 

set out by Decision 1313/2013/EU? 
Could similar/greater benefits have been achieved at 
lower/similar costs? 
(Internal) To what extent are the UCPM pillars and activities 
well-articulated, interconnected and mutually supportive? 
(External) To what extent has the UCPM developed synergies 
with other policy areas and initiatives? 

5.    EU added value  What results of the intervention could not have been achieved (or 

would have been less effective/efficient) without coordinated 
effort at EU level? 
To what extent are results achieved by the UCPM likely to last 
beyond the timeframe of the intervention? 

6.    Sustainability 

4 

METHOD 

This Evaluation builds on the independent evaluation conducted by an external contractor, as 
well as certain recent inputs received during discussions with Participating States, including 
during  Civil  Protection  Committee,  Civil  Protection  Council  Working  Party,  and  Directors-
General meetings. 

Overall,  the  design  of  the  external  evaluation  was  satisfactory  and  used  a  sound 
methodological  approach.  The  evaluation  relied  on  a  large  amount  of  data  collected  from 
reliable sources (agreed with the Commission during the inception phase). The combination 
of  evidence  provided  by  different  research  tools  (online  surveys/questionnaire,  semi-
structured  interviews,  case  studies,  etc.)  was  balanced.  In  addition  to  civil  protection 
authorities  from  Participating  States,  other  stakeholders  (e.g.  other  ministries  from 
Participating  States,  international  agencies,  EU  services,  etc.)  are  well  represented,  which 
adds to the credibility of findings. 

Generally,  the  external  evaluation  is  in  line  with  what  was  foreseen  in  the  Roadmap.  The 
evaluator has followed the guidelines provided in the Terms of Reference (ToR), which were 
adopted by the Inter-service Steering Group (ISG), as indicated in the Roadmap19. 

The  external  evaluation  was  carried  out  between  September  2016  and  June  2017  and  took 
account  of  the  inputs  received  from  the  Open  Public  Consultation  (OPC)20.  The  European 
Court  of  Auditors  (ECA)  special  report  on  the  UCPM21,  and  the  Commission's  Report  on 
progress made and gaps remaining in the EERC22 have also provided useful evidence to some 
areas of this Evaluation.  

19 See footnote 6 
20 The Report on the Open Public Consultation can be consulted here: http://ec.europa.eu/echo/sites/echo-site/files/ucpm_-
_opc_report_0.pdf  
21 Special report no 33/2016: Union Civil Protection Mechanism: the coordination of responses to disasters outside the EU 
has been broadly effective, January 2017 
22  Report  from  the  Commission  to  the  European  Parliament  and  the  Council  on  progress  made  and  gaps  remaining  in  the 
European Emergency Response Capacity, 17.02.2017 

11 

 
 
 
                                                            
The external evaluation consisted of three phases (see figure 1): 

(i)  inception, including a desk review of UCPM activities, scoping interviews with desk 

officers and conception of research tools; 

(ii)  research,  during  which  a  targeted  consultation  reached  out  to  all  the  main 
stakeholders  of  the  UCPM23  through  online  surveys,  face-to-face  and  telephone 
interviews; three case studies (Cyprus Forest Fire response; Estonia Module Exercise 
(ModEx); Balkan route migration response) were conducted. 

(iii) analysis and synthesis, including analysis of UCPM performance, data triangulation, 
elaboration of conclusions and recommendations, and drafting an external Evaluation 
Report. 

The  above-mentioned  OPC  was  carried  out  online  for  three  months  (November  2016  – 
February  2017),  with  some 130 responders from all  over Europe providing feedback on the 
UCPM. The report on the outcome of the OPC was published at the end of March 201724. 
Figure 1: Overview of the external evaluation process 

23 Stakeholders that  responded  include: civil  protection authorities  from  26  Participating  States (PS)  (online  surveys); civil 
protection  authorities  from  17  PS  (interviews);  National  Training  Coordinators  from  21  PS  (online  surveys);  137  experts 
having participated in the exchange of expert programme and 23 leads of projects funded via the UCPM from 13 PS (online 
surveys); 14 international partners including UN agencies, OECD, World Bank, Red Cross, etc.(interviews); 15 Directorates-
General and services of the EU (interviews) 
24 See footnote 20 

12 

 
 
 
 
                                                            
However, two methodological limitations should be noted: 

(i) The current UCPM monitoring system does not systematically capture quantitative data of 
performance.  As  already  highlighted  by  the  recent  ECA  audit,  the  lack  of  robust  indicators 
(e.g. baseline and  target values)  systematically monitored by the UCPM partly hindered the 
collection of evidence needed to formulate more definitive findings. 

(ii)  Limited  statistical  base  of  the  external  evaluation.  Most  of  the  data  gathered  by  the 
contractor  during  the  Evaluation  timeframe  are  of  qualitative  nature  and  they  refer  to 
"perceptions"  of  stakeholders  on  the  UCPM  performance.  While  consultations  have  been 
extensive across all the stakeholders, the external evaluation did not manage to introduce and 
monitor additional indicators to fill this gap. 

To  mitigate  the  shortage  of  quantitative  data  monitored  by  the  UCPM,  the  contractor  was 
encouraged to collect primary data and carry out quick analyses for a sample of activities (e.g. 
comparing  timeframes  of  different  response  operations,  cost  analysis  for  forest  fire  fighting 
missions).  While  the  statistic  sample  is  too  limited  to  draw  any  solid  conclusion,  these 
analyses point to a number of monitoring gaps which should be addressed in the second half 
of 
to  provide  sufficient 
indicators/data,  the  Commission  integrated  the  evidence  obtained  via  alternative  channels, 
such as bilateral meetings with Member States and operational lessons learned (e.g. feedback 
from module exercises, training programme, etc.). 

implementation.  Where 

the  contractor  failed 

the  UCPM 

Within  the  limits  of  data  and  analysis,  the  conclusions  in  this  document  are  well  balanced, 
follow logically from the findings and are linkable to facts and data. 

5 

IMPLEMENTATION STATE OF PLAY 

Financial  data  shows  that  nearly  EUR  131  million  was  allocated  to  the  UCPM  over  2014-
2016. The UCPM budget components are presented in table 1 below. 

Table 1: Budget allocated for the UCPM 2014-2016 (million EUR) 

Source: DG ECHO 

Budget line 

2014 

EUR 23030101 - prevention and preparedness - Inside the EU  27.6  

EUR 23030102 - prevention and preparedness - Third countries 
(outside the EU) 

4.0 

EUR 23030201 - response inside the EU 

EUR 23030202 - response outside the EU 

Total 

1.1 

11.1  

43.8 

2015 

29.1 

5.5  

0.7 

13.6  

49.09  

2016 

28.0  

5.6 

2.5 

3.09 

38.9 

The evaluators compared the average annual allocation by activity during the periods of 2007-
2013  and  2014-2016.  Whilst  a  direct  comparison  is  difficult  between  the  years  due  to 

13 

 
 
 
differences in budget structures, this analysis gives a rough overview of the shift in priorities 
for the Mechanism over different periods. 

Table  2:  Planned  budget  for  the  periods  2014-2016  and  2007-2013  according  to  budget  lines 

 (million EUR) 

Prevention 

All 28 EU Member States complied with Article 6 of Decision 1313/2013 and made available 
to the Commission a summary of their National or Sub-national level Risk Assessment by the 
deadline set25. Based on such summaries the Commission developed an EU-wide Overview of 
Risks26. The European Commission also published Risk Management Capability Assessment 
Guidelines27 to facilitate the national capability assessments that are due by end of 2018. 

The total funds committed for Prevention Projects amounted to over EUR 22 million for the 
period  2014-2016,  compared  to  a  budget  of  EUR  14  million  for  the  period  2007-2013.  The 
average project value was EUR 610 000 (with EU funding averaging EUR 470 000), which 
was  37%  higher  than  for  2007-2013.  The  success  rate  of  applicants  for  prevention  projects 
during 2014-2016 was 39%, decreasing from approximately 50% in 2014 and 2015 to 27% in 
2016.  Financed  projects  include  the  following  activities:  knowledge  sharing  (meeting, 
workshops,  conferences,  exchange  programmes),  studies  (e.g.  new  methodologies  and  tools 
for risk reduction), and awareness raising activities (websites, awareness campaigns, etc.). 

25  Council  of  the  EU,  Report  of  the  Netherlands  Presidency  on  the  main  achievements  at  EU  level  in  the  field  of  civil 
protection, Document 10019/16, Brussels, 17 June 2016, available at: 
http://data.consilium.europa.eu/doc/document/ST-10019-2016-INIT/en/pdf 
26 Staff Working Document, Overview of Natural and Man-made Disaster Risks the European Union may face, 23.05.2017 
SWD(2017)176 final 
27  Commission  Notice,  Risk  Management  Capability  Assessment  Guidelines,  (2015/C  261/03).  Available  at:  http://eur-
lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A52015XC0808%2801%29 

14 

 
 
  
 
                                                            
Preparedness 

Established in 2013 to provide a coordinated and quicker response to disasters both inside and 
outside Europe, the ERCC replaced the previous Monitoring and Information Centre (MIC). 
The period 2014-2016 marked its full maturation, coordinating European assistance to West 
African  countries  in  the  context  of  the  Ebola  outbreak  (2014-2015),  and  serving  as  an 
operational hub for logistics and transport in the response to the 2014 Balkan floods as well as 
the  devastating  earthquake  in  Nepal  in  2015.  Furthermore,  between  2014  and  2016  the 
ERCC’s supported the political decision-making in the Council, particularly in the context of 
Integrated Political Crisis Response (IPCR) arrangements that are triggered automatically by 
the invocation of the Solidarity Clause28 or in cases of 'stand-alone' activations of the IPCR. In 
accordance with Article 9(7) of the Decision, Participating States designated national contact 
points  to  enable  communication  with  the  ERCC  in  the  event  of  occurring  or  imminent 
disasters. 

Between 2014 and 2016, CECIS registered 64 Requests for Assistance, 14 pre-alerts and 15 
in  2007,  will  undergo  several 
monitoring  requests.  The  current  platform,  created 
improvements between 2017 and 2019 in order to improve user-friendliness and enhance its 
analysis and monitoring capabilities. 

The  European  Emergency  Response  Capacity  (EERC)  or  'Voluntary  Pool'  was  launched  in 
October  2014  and,  by  the  end  of  2016,  included  a  total  of  77  response  capacities  from  16 
Participating States. 

The  UCPM  has  taken  advantage  of  the  integration  of  the  European  Flood  Alert  System 
(EFAS)  and  the  European  Forest  Fires  Information  System  (EFFIS)  under  the  umbrella  of 
Copernicus' Emergency Management System (EMS)29 in 2012. In close cooperation with the 
Joint Research Centre (JRC) (responsible for the design of the systems themselves) and DG 
GROW/Copernicus  (managing  the  early  warning  systems)  the  UCPM  has  provided the  link 
with  the  end-users,  namely  civil  protection  authorities  from  Participating  States  and  the 
ERCC.  Similarly,  the  rapid  mapping  services  allow  on-demand  and  quick  provision  of 
geospatial  information  (e.g.  damage  assessment  maps)  to  inform  emergency  response  and 
recovery. During the 2014-2016 period, more than 30 requests for maps have been received 
each year. 

1  680  individual  experts  have  undergone  at  least  one  course  of  the  UCPM  Training 
Programme between 2014 and 2016, whereas there were 2 255 trained experts over the entire 
7-year  period  of  the  previous  Mechanism.  The  yearly  average  has  been  740  for  the  current 
financial  framework,  compared  to  322  during  the  previous  one.  Around  15%  of  the  total 
trained  experts  over  the  evaluation  period  are  women.  A  European  Disaster  Management 
Training  Network  was  set  up  at  the  end  of  2016  to  improve  all  phases  of  disaster 

28 Article 222 TFEU 
29  Copernicus  EMS  provides  information  for  emergency  response  to  different  types  of  disasters,  including  meteorological 
hazards,  geophysical  hazards,  deliberate  and  accidental  man-made  disasters  and  other  humanitarian  disasters  as  well  as 
prevention, preparedness, response and recovery activities. For further information see http://emergency.copernicus.eu/. 

15 

 
 
                                                            
management, create synergies and develop guidance on EU and international civil protection 
training. 

A  number  of  disaster  simulation  exercises  are  financed  every  year  through  a  Call  for 
Proposals  for  full-scale exercises30  and  for  module  exercises31  (ModEx).  Between  2014  and 
2016 8 ModEx in 7 Participating States and 13 full scale exercises in 10 Participating States 
have  been  organised.  Differently  from  the  previous  Mechanism  (2007-2013),  ModEx 
exercises  under  the  UCPM  have  become  an  integral  part  of  the  process  of  certification  for 
modules which will become part of the EERC. Overall, EUR 26.1 million were allocated to 
exercises. 

During  the  first  three  years  of  the  UCPM  (2014-2016)  the  committed  amount  for 
Preparedness Projects reached nearly EUR 17 million, a significant increase from the budget 
of the previous Mechanism (nearly EUR 10 million for the entire framework 2007-2013). The 
average  project  value  was  EUR  691  000  (with  EU  funding  averaging  EUR  519  000), 
approximately  55%  higher  than  the  average  figure  during  2007-2013.  The  success  rate  of 
applicants  for  preparedness  projects  was  42%  for  the  period  spanning  from  2014  to  2016. 
Financed  projects  included  the  following  activities:  training  and  exercises;  knowledge 
management  (best  practices;  study  visits;  workshops;  etc.);  new  tools  and  procedures  for 
preparedness. 

Response 

The  UCPM  was  activated  19  times  in  2014  and  2015,  and  26  times  in  2016.  During  the 
Evaluation timeframe (2014-2016), on average 33% (or 21 out of 62) of requests came from 
inside the EU and the remaining from third-countries. A total of EUR 28.4 million in transport 
grants were provided for operations inside (48 grants) and outside the EU (247 grants). 

Figure 2: Number of activations inside and outside EU, 2008-2016.  

Source: DG ECHO, last update 24 February 2017 

Figure 3: Number of experts deployed per mission, 2008-2016 

Source: DG ECHO, last update 24 February 2017 

30 Full Scale Exercises are designed to replicate one or several phases of an emergency with the commitment of all the public 
bodies and authorities that would be committed in a real situation. 
31  Modules  field  exercises  aim  to  provide  an  opportunity  for  testing  specific  response  capacities,  as  well  as  the  self-
sufficiency, interoperability, coordination and procedures of response teams and equipment. 

16 

 
 
 
 
                                                            
Although the number of disaster response missions has not changed significantly, in 2016 the 
total  number  of  experts  deployed32  (86)  increased  when  compared  to  first  two  years  of  the 
UCPM.  On  average,  4.8  experts  were  deployed  per  mission  in  2016.  Regarding  modules, 
statistical  data  shows  that  over  the  evaluation  period  (and  before)  the  majority  of  modules 
were deployed outside the EU (see figure 4). 

Figure 4: Number of UCPM modules deployed inside and outside of EU, 2008-2016  

Source: DG ECHO, last update 24 February 2017 

Cross-cutting activities 

Peer reviews: Two "pilot" peer reviews were carried out in the UK and Finland in 2013 and 
2014 in cooperation with the United Nations Office for Disaster Risk Reduction (UNISDR) 
and  the  Organisation  for  Economic  Cooperation  and  Development  (OECD).  While  the  two 
pilots focused on assessing progress in national implementation of the Hyogo Framework for 
Action,  the  subsequent  six  peer  reviews  focused  on  more  specific  issues  of  disaster  risk 
management relevant to the selected countries. A total of EUR 1.3 million was allocated for 
peer reviews over 2014-2016. 

Prevention  and  Preparedness  ("advisory")  missions:  Out  of  thirteen  of  such  missions 
undertaken  between  2014  and  2016,  six  were  carried  out  in  a  post-emergency  context;  the 
remainder focused on capacity building for recovery planning (two of them were embedded in 
two Post-Disaster Needs  Assessments,  i.e.  in Albania and FYRoM);  and  ex  ante  prevention 
measures.  A  total  of  13  missions  were  carried  out  over  2014-2016,  with  EUR  230  000 
allocated. 

32 This includes: 46 experts deployed to 11 response missions and 38 experts deployed to 7 advisory missions. 

17 

 
 
 
 
 
                                                            
Lessons learnt: DG ECHO implemented lessons learnt and technical workshops following the 
activation of the UCPM on a case-by-case basis, depending on the nature of the disaster. A 
total  of  EUR  1.85  million  was  allocated  for  lessons  learned  over  2014-201633  of  which 
EUR 598 358 were disbursed as of June 15, 2017. 

Exchange  of  experts:  The  programme  was  implemented  by  an  external  organisation  for  the 
2014-2016  period.  Overall,  375  experts  participated  in  exchanges  within  the  EU  and  47 
experts from 11 partner countries joined the programme. Budget allocated to this activity was: 
EUR 900 000 for experts from Participating States and EUR 196 025 for experts from third 
countries. 

Monitoring 

The  external  evaluation  found  that  existing  monitoring  arrangements  are  rather  limited  and 
may not fully capture the UCPM performance nor the follow-up required. The Commission 
shares this view, which applies to all of the three pillars of the UCPM. Monitoring of results is 
based  on  the  four  indicators  listed  in  Article  3  of  the  Decision.  However  for  most  of  these 
indicators baseline/target values are not indicated in the Decision 1313/2013 nor in the 2014 
Implementing Decision34, which makes it difficult to assess progress. For instance, assessing 
"speed  of  intervention  under  the  Union  Mechanism  and  the  extent  to  which  the  assistance 
contributes  to  the  needs  on  the  ground"  (indicator  for  Objective  3)  is  hindered  by  not 
recording  systematically  via  CECIS  time  milestones  for  all  response  operations35.  Similar 
considerations apply to the lack of a standardized format for needs assessments underpinning 
response  operations,  which  in  some  cases  can  hinder  the  assessment  of  whether  and  how 
assistance (especially in-kind) matched needs on the ground. The preparedness indicator  set 
by the Decision for Objective 2, ("quantity of response capacities included in the Voluntary 
Pool  in  relation  to  the  capacity  goals  and  the  number  of  modules  registered  in  CECIS") 
remains vague about including the type/number of individual experts registered in the system. 
At  the  moment  this  analysis  cannot  be  fully  carried  out  on  CECIS  (not  all  UCPM  trained 
experts  are  in  CECIS  and  its  research  filters  are  very  limited,  e.g.  not  allowing  research  by 
"tags"  or  keywords).  This  makes  it  challenging  to  comprehensively  map  Europe's  response 
capacity in terms of experts. In addition, the Objective 4 (risk awareness) indicator ("progress 
in  increasing  public  awareness  and  preparedness  for  disasters")  has  not  been  systematically 
measured  by  the  UCPM.  Although  the  Eurobarometer  surveys  do  support  DG  ECHO  in  its 
ability to assess the general level of awareness of civil protection among European citizens, 
this cannot be fully attributable to the UCPM (furthermore it is only limited to  EU Member 
States  and  several  Participating  States  are  not  covered).  The  section  below  on  effectiveness 
(6.2)  elaborates  further  on  the  limitation  of  the  current  monitoring  system.  The  external 
evaluation  presents  additional  analytical  limitations  of  CECIS,  e.g.  for  producing  automatic 

33 This budget line of the Work Programme also include other meetings with Participating States, such as the bi-annual civil 
protection Directors-General meeting. 
34 European Commission, Implementing Decision of 16 October 2014 (C(2014) 7489) 
35 ICF, Interim Evaluation of the Union Civil Protection Mechanism, 2014-2016, Final Report, July 2017, (hereafter: 
External Evaluation Report), p. 51 

18 

 
 
                                                            
reports  on  response  operations  (e.g.  speed,  cost,  etc.)36.  Several  of  these  shortcomings  were 
already highlighted in the European Court of Auditors' report on the UCPM37. 

6 

ANSWERS TO THE EVALUATION QUESTIONS 

6.1  Relevance 

The general and specific objectives of Decision 1313/2013 continue to be relevant to Europe's 
current social and economic context. 

Consultations with  stakeholders in  the context of the Evaluation  have  highlighted that, over 
the  past  few  years,  civil  protection  authorities  have  been  confronted  with  a  changing  risk 
landscape, which calls for new response capacities and an increased level of coordination with 
a  much  broader  array  of  stakeholders38.  Within  Europe,  new  response  scenarios  can  range 
from supporting traditional police functions in the aftermath of a terrorist attack (e.g. victim 
identification)  or  dealing  with  the  consequence  of  a  cyber-attack  affecting  a  lifeline 
infrastructure, to providing relief items to an exceptionally large number of migrants. 

Outside Europe, the UCPM is increasingly being activated to respond to rapid onset disasters, 
including  a  significant  number  of  complex  emergencies  (see  figure  5)  in  which  civil 
protection assets are being used to address acute health/shelter/water and sanitation needs. As 
the  UCPM  becomes  increasingly  in  demand,  the  issue  arises  as  to  its  core  mandate.  In 
particular, the challenge for the UCPM is to find the right balance between being a framework 
of  mutual  assistance  between  European  countries  in  the  aftermath  of  natural  and man-made 
disasters and evolving towards a more complex instrument capable of addressing crises of a 
multiple nature and with a global reach39. 

Figure 5: Hazards having led to UCPM activation: comparison between 2007-2013 and 2014- 

2016 (annual average) 
Source: Emergency Response Coordination Centre (ERCC), elaborated by ICF 

36 External Evaluation Report, p. 18 
37 European Court of Auditors, Special Report (2016), Union Civil Protection Mechanism: the coordination of responses to 
disasters outside the EU has been broadly effective. Available at: 
http://www.eca.europa.eu/Lists/ECADocuments/SR16_33/SR_DISASTER_RESPONSE_EN.pdf 
38 See Annex 4: "Synopsis of Stakeholder Consultation". Also, External Evaluation Report, p. 69 
39 External Evaluation Report, p. 73 

19 

 
 
 
 
                                                            
The UCPM's flexibility in addressing emerging or unanticipated developments and needs in 
the  civil  protection/disaster  risk  management  arena  in  Europe  and  beyond  has  also  been 
recognized by many stakeholders40. The activation of the Mechanism in response to the Ebola 
outbreak  (March  2014)  and  to  support  Participating  States  facing  an  exceptional  influx  of 
migrants (2015-2017) are probably the examples that best demonstrate the UCPM's flexibility 
in  responding  to  non-traditional  civil  protection  emergencies.  However,  stakeholders  have 
voiced concerns about whether the UCPM is the right tool to address emergencies that are not 
fully covered by Decision 1313/201341. 

The relevance of the UCPM is also confirmed by: 

  Several  recent  resolutions42  of  the  European  Parliament  calling  on  the  Commission 
and the Member States to foster cooperation among civil protection authorities across 
Europe and to further simplify the procedures for the activation of the Mechanism in 
order  to  make  it  rapidly  available  and  effective  in  the  immediate  aftermath  of  a 
disaster. 

  The  2017  Eurobarometer  Report  on  Civil  Protection43,  according  to  which  90%  of 
European citizens think that the EU should coordinate the response to disasters in any 
of  the  Member  States  and  87%  believe  there  should  be  a  common  civil  protection 
policy.  There  is  also  strong  agreement  (86%)  that  the  EU  should  help  any  country 
worldwide hit by disasters in a coordinated manner, by sending experts and equipment 
to affected areas. 

  The Overview of Risks in the EU report44 stresses that disasters ‘know no borders’ and 
there is a need to further address cross-border and "cascading"/inter-connected risks. 

As  far  as  preparedness  is  concerned,  the  following  challenges  have  emerged  from 
consultations  with  national  civil  protection  authorities  and  other  key  stakeholders  (national 
training coordinators and international partners)45. 

  The  training  and  exercise  programmes  require  a  needs-based  approach  which 
focuses  on  the  competences  necessary  to  implement  the  specific  activities  of  the 
UCPM,  while  incorporating  relevant  lessons  learnt.  It  is  therefore  critical  to 
thoroughly  evaluate  response  missions,  as  well  as  prevention  and  preparedness 
activities,  in  order  to  distil  specific  training  needs,  with  a  view  to  develop  a 
"competence  curriculum"  for  European  civil  protection  experts.  The  UCPM's 
Training  Policy  Group  has  recently  (March  2017)  developed  a  new  strategic 
framework for training and exercises46. 

40 External Evaluation Report, p. 305 
41 External Evaluation Report, p. 73 
42  Among  others,  European  Parliament  resolution  of  1  December  2016  on  the  situation  in  Italy  after  the  earthquakes 
(2016/2988(RSP)) 
43 European Commission, Special Eurobarometer 454, Civil Protection, May 2017 
44 See footnote 25 
45 Based on conversations with PS representatives during meetings (CPC, lessons learned, experts' meetings, etc.). 
46 European Commission, Towards a Strategic Framework for Training and Exercises within the Union Civil Protection 
Mechanism (UCPM) 2017-2020, not available online. 

20 

 
 
                                                            
Figure 6: Number of individuals attending at least one training and number of individuals deployed 

2014-2016 
Source: Data from ECHO, elaborated by ICF 

  Further  developing  the  cross-border  early  warning  systems  platforms  would 
increase  its  relevance  for  the  entire  Europe.  Forest  fires  continue  to  take  a 
significant  toll  on  Europe's  society,  economy  and  environment.  Recent  analyses 
from JRC have put the average annual losses caused by forest fires to EU Member 
States  at  nearly  EUR  2.7  billion/year47.  The  case  study  carried  out  by  the  external 
evaluators (Cyprus forest fires of June 2016) underlined the importance of EFFIS as 
a  tool  to  assess  the  likelihood  of  locating  available  response  capacities  in  other 
Participating States during the forest fire season48. Currently the ERCC is working 
on  the  possibility  of  mapping  such  capacities.  One  important  caveat  about  the 
relevance  of  the  trans-boundary  early  warning  systems  for  forest  fires  and  floods 
concerns the different needs (and use) across Participating States. The countries that 
already manage comprehensive early warning systems at the national level consider 
the  EU-level  system  as  mainly  relevant  for  informing  disaster  preparedness  and 
response in other Participating States and third countries. Countries which are still in 
the  process  of  consolidating  their  national  early  warning  systems  tend  to  see  it  as 
directly relevant for their national needs, in addition to response in third countries. 
Furthermore,  interviews  with  Participating  States  and  Commission  officials  have 
shown that the end-user side of early warning systems platforms such as EFFIS and 
EFAS should be further developed, so as to create a community of practitioners that 
reflects  on  operational  evidence  emerging  from  disaster  events  and  which  shares 
knowledge, with the objective of improving practice49. Such a development would 
increase  relevance  of  UCPM  early  warning  platforms  for  most  countries  within 
Europe, regardless of their level of preparedness. 

Better linking prevention with preparedness and response activities was already recommended 
in 2014 by the ex-post evaluation of the previous Civil Protection Mechanism (2007-2013). 
This need remains relevant at the mid-term mark of the 2014-2020 Mechanism, based on what 

47 Source: European Forest Fires Information System (EFFIS). This figure is based on a conservative assessment of economic 
losses (does not include, for instance, the high cost of firefighting equipment). 
48 External Evaluation Report, p. 153 
49 External Evaluation Report, p. 97 

21 

 
 
 
                                                            
the main stakeholders have expressed50. The need to match response capacities with the major 
risks in Europe will represent the key challenge to ensure continued relevance of UCPM. In 
particular, an effort for forward looking assessments that take into account "emerging" threats 
(e.g. cyber-attacks, CBRN, critical infrastructure failure, etc.) is required. 

6.2  Effectiveness 

The  UCPM  is  making  good  progress  towards  achieving  the  objectives  set  out  by  the 
legislation, most notably in improving cooperation and coordination between stakeholders in 
the civil protection sector. 

Article  3  of  Decision  1313/2013  spells  out  four  specific  objectives  which,  in  turn,  are 
expected  to  contribute  to  achieving  the  two  overarching  general  objectives  (See  Annex  1: 
Intervention Logic). 

With regards to the specific objectives of the Decision: 

Specific objective 1: To achieve a high level of protection against disasters by preventing or 
reducing  their  potential  effects,  by  fostering  a  culture  of  prevention  and  by  improving 
cooperation between the civil protection and other relevant services. 

The Mechanism is on track to achieve this objective. The target for the first result indicator 
established by the legislation ("a National Risk Assessment submitted by all Member States 
by  end  2015")  was  fully  achieved51.  As  far  as  the  second  indicator  is  concerned  ("an 
assessment of Member States' risk management capability"), the target is due by end 2018 and 
guidelines were issued by the Commission52. From the stakeholder consultation, a consensus 
emerged  both  among  civil  protection  authorities  and  other  national  ministries  from  the 
Participating  States  consulted  that  the  UCPM  has  contributed  towards  a  higher  level  of 
protection  against  disasters53.  For  instance,  the  views  of  stakeholders  consulted  by  the 
evaluators  indicate  that  the  national  risk  assessments  were  helpful  to  inform  risk  reduction 
policies  and  plans  as  well  as  improving  the  understanding  of  risks54.  However,  while  the 
Decision's  indicators  capture  outputs  (National  Risk  Assessment  and  National  Risk 
Management Capability reports), additional evidence would be needed to describe the UCPM 
progress  in  achieving  outcomes  (preventing  or  reducing  potential  effects  and  fostering  a 
culture  of  prevention).  For  instance,  this  would  require  monitoring  the  uptake  of  outputs 
delivered  by  UCPM  prevention  projects  (e.g.  how  do  risk  assessment  studies  or  knowledge 
sharing  activities  result  in  more  effective  cross-border  policies  and  investments  for  risk 
reduction  once  the  project  is  closed?).  The  external  evaluation  did  not  conduct  such  an 
analysis nor pointed to possible measures to carry it out in future. As far as the second part of 
the  objective  (improving  cooperation  between  civil  protection  and  other  services)  is 
concerned, the Decision does not set any specific indicator. However, the survey carried out 

50 External Evaluation Report, p. 114 
51 Overview of Risks Report, p. 3 
52 Commission Notice, Risk Management Capability Assessment Guidelines (2015/C 261/03), available at: http://eur-
lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:52015XC0808(01)&from=EN 
53 External Evaluation Report, p. 22 
54 External Evaluation Report, p. 23 

22 

 
 
                                                            
with national civil protection authorities in the context of the evaluation shows a significant 
number  of  agencies/ministries  involved  by  civil  protection  authorities  in  the  preparation  of 
NRAs55.  This  information  can  be  considered  as  a  proxy  of  the  good  level  of  cooperation 
between  civil  protection  and  other  services  on  disaster  prevention,  at  least  in  sharing  risk 
information. 

Figure 7: Survey of national civil protection authorities. Question: Which ministries and/or public 
agencies were consulted during the development of your NRA? 
Source: ICF, Survey of national civil protection authorities 

Activities that stakeholders consider as having contributed the most to this specific objective 
are: (i) the introduction of the national disaster risk assessment provision in the legislation and 
(ii) the financing of prevention projects on a yearly basis. When compared with preparedness 
and  response,  the  perception  of  the  public56  is  that  results  for  the  prevention  pillar  of  the 
UCPM are less evident. A factor that needs to  be taken into account is that annual financial 
resources  allocated  to  the  prevention  pillar  are  only  20%  of  the  UCPM  total  budget.  The 
impact  generated  by  UCPM  prevention  activities  should  also  include  what  these  activities 
have managed to leverage (e.g. additional financing, technical resources, etc.) outside of the 
Mechanism.  For  instance,  prevention  (and  preparedness)  activities  financed  by  the  UCPM 
could  work  as  'seed  funds'  that  national  and  sub-national  authorities  could  use  to  mobilise 
and/or  steer  additional  resources  for  disaster  risk  management  from  other  EU  (e.g. 
Cohesion/European Regional Development Fund, Horizon 2020, LIFE programme, etc.) and 
international  (European  Investment  Bank;  World  Bank;  UN  agencies;  etc.)  programmes. 
Currently, from the reporting carried out by the UCPM-funded prevention activities, it is not 
possible to appreciate this type of impact. 

Consultations  with  Participating  States  and  Commission  officials  also  highlighted  a  number 
of  challenges  relating  to  fully  capturing  and  deepening  the  impacts  of  the  prevention 
activities, which will be critical for deciding how to allocate future financing57. These include: 

 

Increasing  the  focus  of  the  prevention  (and  preparedness)  projects  that  the  UCPM 
finances  every  year.  The  programme,  which  has  been  running  since  2000,  has 

55 See footnote 54 
56 This includes the 2017 Eurobarometer survey on civil protection, where about half of all respondents think that not enough 
is being done for disaster prevention by their region and country (49%) and at EU level (43%). 
57 External Evaluation Report, p. 25, 93-94 

23 

 
 
 
                                                            
financed a large number of projects without always capitalising on their results. Given 
the  relatively  small  size  of  financing  (between  2014  and  2016  average  funding  was 
EUR 470 000), projects need to be targeting the needs of the end-users and need to 
incorporate measures for scaling up their results.  

 

Integrate  more  systematically  climate  change  adaptation  into  the  National  Risk 
Assessments across the Participating States. The 2010 Risk Assessment and Mapping 
(voluntary)  Guidelines  for  Disaster  Management58  issued  by  the  Commission  has 
asked Participating States to adopt a longer term perspective to adequately capture the 
potential  impacts  of  climate  change  on  certain  types  of  disasters  such  as  floods  and 
droughts.  

  Better  articulating  the  linkages  between  the  UCPM  prevention  (and  preparedness) 
projects  and  broader  EU  programmes 
(e.g.  Cohesion/European  Regional 
Development  Fund,  etc.).  A  more  strategic  link  with  such  programmes  could  help 
maximize  the  uptake  of  projects  results  and  leverage  additional  technical  and 
financial resources for prevention.  

Specific objective 2: To enhance preparedness at Member State and Union Level to respond 
to disasters. 

The UCPM has been very effective in achieving this objective, based on the indicator set by 
the Decision ("quantity of response capacities included in the Voluntary Pool in relation to the 
capacity  goals  and  the  number  of  modules  registered  in  the  CECIS").  As  highlighted  by  a 
recently  published  Commission  report59,  substantial  progress  was  made  with 
the 
establishment  and  the  functioning  of  the  EERC  and,  in  particular,  by  exceeding  the  initial 
capacity goals60 for most of the modules (see Figure 8) during its first three years. As far as 
the "other response capacities" set by the Implementing Decision, after a somewhat slow start 
(2  capacities  registered  by  July  2015),  significant  progress  was  recorded  with  14  other 
capacities registered in the EERC as of May 201761. In addition to meeting these quantitative 
targets,  the  Decision  has  also  resulted  in  higher  quality  standards  for  response  capacities 
across Europe. 

58 Commission Staff Working Paper, Risk Assessment and Mapping Guidelines for Disaster Management, 2010 
59  Report  from  the  Commission  to  the  European  Parliament  and  the  Council  on  progress  made  and  gaps  remaining  in  the 
European Emergency Response Capacity, 17.02.2017 
60 Such goals were defined by the Commission Implementing Decision of 16 October 2014 (C(2014) 7489) 
61 Source: DG ECHO 

24 

 
 
 
 
 
                                                            
Figure 8: ERCC capacity goals compared with modules registered in the EERC62 as of 31 May, 2017 

Source: DG ECHO and the Commission Implementing Decision of 16 October 2014 (C(2014) 7489) 

15
14
13
12
11
10
9
8
7
6
5
4
3
2
1
0

6 

6 

2 

2 

2 

2 

2 

2 

2 

2 

1 

1 

2 

2 

1 

2 

2 

2 

Resources registered (Modules)

ERCC capacity goals (Modules)

This assessment is also confirmed by the results of the stakeholder consultation63. Nearly all 
of the civil protection authorities and more than two thirds of the national ministries consulted 
agreed that the UCPM has been effective in enhancing preparedness at Participating State and 
Union levels to respond to disasters. In addition to the EERC or Voluntary Pool, this pillar, 
which received 50% of the total UCPM budget, included some of the flagship activities of the 
UCPM,  such  as  the  training  programme;  the  exercise  programme;  the  functioning  of  the 
ERCC  and  the  CECIS.  Although  the  Decision  does  not  set  specific  results  indicators  for 
training and exercises, these were recognized among the activities that contributed the most to 
achieving  the  preparedness  objective  of  the  UCPM.  National  civil  protection  authorities 
consider training and exercises as being generally fit for purpose, improving the capacities of 
Participating  States,  establishing  common  approaches  and  facilitating  cooperation.  The 
stakeholder consultation also highlighted the need to assess the number of slots available to 
Participating States  for training  and exercises on operational criteria (e.g. number of  pooled 
capacities and experts, level of national capacity, etc.) other than the population64. 

62  MUSAR  -  Medium  urban  search  and  rescue,  GFFFV  -  Ground  Forest  Fire  Fighting  Using  Vehicles,  CBRNDET  -
Chemical,  biological,  radiological  and  nuclear  detection  and  sampling  (CBRN),  HCP  -  High  capacity  pumping,  HUSAR  - 
Heavy urban search and rescue, AMP - Advanced medical post, GFF - Ground Forest Fire Fighting, USARCBRN - Search 
and  rescue  in  CBRN  conditions,  WP  -  Water  purification,  FRB  -  Flood  Rescue  Using  Boats,  AMP-S  -  Advanced  medical 
post with surgery, MEVAC - Medical aerial evacuation of disaster victims, FFFP - Aerial forest fire fighting module using 
airplanes, FFFH – Aerial forest firefighting module using helicopters, FHOS – Field hospital, FC - Flood Containment, ETS - 
Emergency temporary shelter, TAST – Technical Assistance and Support Teams 
63 Externarl Evaluation Report, p. 26-28 
64 External Evaluation Report, p. 32 

25 

 
 
 
 
 
 
                                                            
The external evaluation and its stakeholders' consultation point to a number of challenges that 
may  affect  the  full  achievement  of  this  objective  by  the  end  of  the  UCPM  implementation 
period65. These include: 

  A more evidence-based approach to preparedness is required. The capacity goals set 
by the Implementing Decision should be revised against the main/critical risks Europe 
is facing and prioritized accordingly. This is fully in line with Art. 14 (2) of the UCPM 
Implementing  Decision66,  which  requires  the  Commission  to  regularly  assess  and 
revise the suitability of capacity goals "on the basis of risks identified in national risk 
assessments  or  other  appropriate  national  or  international  source  of  information."  A 
close  dialogue  with  Participating  States  is  critical  in  order  to  capture  all  national 
capacities,  including  those  which  are  not  registered  in  the  EERC  but  may  still  be 
deployable  via  the  UCPM.  Similarly,  the  spatial  distribution  of  (natural  and  man-
made)  risks  across  Europe  should  inform  the  geographical  distribution  of  response 
capacities. 

  The need to re-assess the system of incentives for pooling assets in the EERC based on 
lessons  learnt  and  other  knowledge  emerging  from  operations.  A  significant  part  of 
these  incentives  concern  the  level  of  co-financing  for  the  deployment  of  EERC 
capacities  or  the  mobilisation  of  buffer  capacities;  providing  health  and  third-party 
liability  insurance  coverage  for  experts  deployed.  The  deployment  of  EERC 
capacities, particularly for long-term missions outside the EU, would certainly benefit 
the  availability  of  more  ‘robust’  EU  co-financing  possibilities.  The 
from 
administrative burden involved in registering and certifying modules also ranks among 
the  main  concerns  of  Participating  States,  although  this  is  being  addressed  by  the 
Commission  with  a  more  streamlined  and  coherent  planning  process  for  exercises, 
certification and registration of assets in the EERC. 

  The  training  and  exercise  activities  of  the  UCPM  require  a  strategic  approach 
focusing  on  results.  Some  of  the  consulted  Participating  States  have  highlighted  the 
need to move from a system focused on outputs (i.e. number of individuals trained) to 
addressing  operational  needs  that  stem  from  operations.  This  is  currently  being 
addressed with the Strategic Framework for Training and Exercises, recently launched 
by  the  Commission,  which  is  focused  on  competences.  Furthermore,  a  robust 
framework for evaluating the training courses administered as well as the individuals 
trained is considered as a priority in order to match the needs on the ground with the 
best technical resources available. Similar considerations were raised with regards to 
the exercises. 

Specific  objective  3:  To  facilitate  rapid  and  efficient  response  in  the  event  of  disasters  or 
imminent disasters 

65 See footnote 45 
66 European Commission, Implementing Decision of 16 October 2014 (C(2014) 7489) 

26 

 
 
 
 
 
                                                            
Overall,  the  UCPM  has  made  significant  progress  towards  achieving  a  rapid  and  efficient 
response to disasters67. Stakeholders across all categories, including those who participated in 
the  Open  Public  Consultation,  noted  as  one  of  the  key  achievements  the  improved 
coordination  and  flow  of  information  at  EU-level  through  the  scaling-up  of  the  ERCC  and 
CECIS during the period 2014-2016. Civil protection authorities were generally satisfied with 
the ability of the UCPM to respond to disasters within the Union, a vast majority considering 
that  information  provided  to  countries  for  deploying  assistance  was  comprehensive, 
sufficiently  detailed,  based  on  the  needs  on  the  ground  and  provided  in  a  timely  manner68. 
Three quarters of civil protection authorities consulted also considered response coordination 
to be smooth and timely. 

Outside of the EU, UCPM response was considered "broadly effective" by the recent audit of 
the  ECA69,  which  focused  on  three  disasters  (floods  in  Bosnia  and  Herzegovina,  Ebola 
outbreak in West Africa and Nepal earthquake) to which the UCPM responded. According to 
the ECA, the UCPM activation was timely and, in each of the three cases observed, the EU 
Civil  Protection  Team  facilitated  coordination  on  the  ground  with  the  Participating  States' 
response teams. The coordination among different Commission services, as well as with other 
EU and non-EU bodies, was inclusive (see also Section 6.4 on coherence). The Commission 
also  respected  the  UN’s  overall  lead,  and  took  steps  to  ensure  a  smooth  transition  into  the 
recovery  phase.  Response  missions  outside  the  Union  were  deemed  effective  also  by 
international actors (UN OCHA, WFP, IOM, UNHCR, etc.) participating in the stakeholders' 
consultation70.  The  importance  of  gaining  a  better/broader  understanding  of  the  context  and 
partners on the ground was highlighted by some of the international stakeholders71. This goes 
in  the  same  direction  as  the  ECA's  recommendation  to  involve  more  systematically  DG 
ECHO personnel in the field during UCPM response missions. 

In order to complement the perception of stakeholders with statistical evidence on the speed 
of  UCPM  response  to  comparable  disasters  inside  Europe  (e.g.  forest  fires),  the  external 
evaluation  looked  at  response  times  for  selected  emergencies  based  on  data  recorded  in 
CECIS72.  While  the  analysis  carried  out  in  the  external  evaluation  report  is  too  limited  to 
allow any solid conclusion on the speed of response, it pointed to a discrepancy between the 
indicators set by the UCPM Implementing Decision and the data that are currently recorded 
(in some cases manually entered) in CECIS. In particular, while the only speed indicator that 
the legislation (Implementing Decision 16 October 2014) refers to is the "availability" of each 
response module,  the parameter monitored by the ERCC is  the "departure time". Therefore, 
there is a need  to  harmonize indicators in  order to  allow comparisons  and  measure whether 
there is any progress in the speed of response. In addition to identifying and harmonizing the 
indicators,  as  already  highlighted  by  a  recommendation  of  the  ECA,  statistics  and  reports 
based on CECIS data should be automatically generated and analysed. 

67 External Evaluation Report, p. 60 
68 External Evaluation Report, p. 39 
69 See footnote 21 
70 External Evaluation Report, p. 41 
71 External Evaluation Report, p. 41 
72 External Evaluation Report, p. 39 

27 

 
 
                                                            
Moreover,  the  lack  of  baseline  and  target  values  (i.e.  what  can  be  considered  as  a  "rapid 
response" to a forest fire/earthquake/flash flood/etc.?) is an obstacle to appreciating effective 
progress  in  making  disaster  response  speedier.  Similar  challenges  exist  for  the  second 
dimension of this indicator ("efficient response"), as explained in section 6.3 on Efficiency.  

In  addition  to  adequately  monitoring  the  speed  of  response,  the  external  evaluation  raised 
other  challenges  to  fully  measuring  and,  possibly,  achieving  the  results  under  this  specific 
objective: 

  Needs  assessments  in  the  Requests  for  Assistance  are  often  not  available  and/or  not 
collected  in  a  standardised  format.  Due  to  this,  the  second  indicator  set  by  the 
Decision  for assessing disaster response,  ("extent to  which the assistance contributes 
to  the  needs  on  the  ground")  can  only  be  measured  based  on  perceptions  and  other 
qualitative input provided by the stakeholders. In this context the external evaluation 
noted a low level of awareness among Participating States of the common template for 
request  for  assistance,  which  was  agreed  in  the  context  of  the  Host  Nations  Support 
Guidelines73. 

  When facing imminent disasters, the pre-deployment of a small UCPM team ahead of 
the  formal  activation  should  be  considered,  as  this  could  prepare  the  ground  for  the 
following deployment and make a significant difference to accelerate UCPM response. 

  Further  simplifying  administrative  and  financial  procedures  for  engaging  response 
capacities  under  the  UCPM  which,  according  to  some  Participating  States,  in  some 
cases is less rapid than bilateral channels. A number of operational 'quick fixes' have 
already  been  identified  by  the  users  (e.g.  integrating  in  CECIS  the  transport  co-
financing request form; revising the minimum threshold for individual transport grant 
applications; etc.) and could be addressed rapidly. 

  Ensuring  the  best  match  between  operational  needs  and  experts  deployed  in  the  EU 
Civil  Protection  Team.  Consulted  civil  protection  authorities  found  the  Terms  of 
Reference produced by the ERCC for selecting experts (their use became increasingly 
systematic  since  2015)  helpful  but  raised  some  concerns  on  the  selection  process  of 
the experts, which could still be improved. 

Specific Objective 4: "To increase public awareness and preparedness for disasters" 

The UCPM has contributed to this objective mainly with the dissemination of good practices 
sharing  and  other  knowledge  management  activities  (e.g.  Civil  Protection  Forum  2015,  3rd 
edition  of  the  World  Reconstruction  Conference,  establishment  of  the  online-based Disaster 
Risk  Management  Knowledge  Centre,  Guidelines  on  Risk  Communication,  etc.).  When 
compared to the other three, stakeholders consulted in the context of the external evaluation 

73 External Evaluation Report, p. 43 

28 

 
 
 
 
 
 
                                                            
feel that progress under this objective has been less advanced. This seems also in line with the 
trend  showed  by  the  2017  Eurobarometer  survey,  whereby  over  the  past  years  a  significant 
part of the surveyed population (42%) is not aware of EU civil protection work. Boosting the 
UCPM visibility (for instance during exercises, as observed by the evaluators during one of 
the  case  studies)  to  European  citizens  would  also  contribute  to  increasing  the  profile  of  the 
public awareness activities supported by the UCPM. 

Finally,  there  is  another  factor  related  to  the  achievement  of  this  specific  objective  which 
should be taken into account in the period leading up to the end of the UCPM implementation 
framework. The activities carried out by the UCPM (mainly complementing the work carried 
out  at  the  national  level  by  civil  protection  authorities)  and  the  UCPM  indicator  for  this 
objective  (level  of  awareness  of  Union  citizens  of  the  risks  in  their  region)  are  on  two 
different  levels.  This  mismatch  can  make  it  difficult  to  find  a  direct  link  between  the 
awareness  raising  and  knowledge  management  activities  financed  by  the  UCPM  and  the 
increased level of awareness among the population at the local level. No baseline or target has 
been set for this risk awareness indicator which, moreover, has not been surveyed by the 2017 
Eurobarometer report on Civil Protection. 

General Objectives: 

  To  strengthen  cooperation  between  the  Union  and  Member  States  and  facilitate 
coordination  in  the  field  of  civil  protection  in  order  to  improve  the  effectiveness  of 
systems  for  preventing,  preparing  for  and  responding  to  natural  and  man-made 
disasters; and 

  To  promote  solidarity  between  the Member  States  and  other  participating  countries 

through practical cooperation and coordination 

Overall, based on the extent to which the specific objectives are being achieved, the UCPM 
has  been  successful  in  progressing  towards  the  general  objectives  of  the  Decision.  National 
civil  protection  authorities  and  other  stakeholders  agree  that  the  UCPM  contributed  to 
improving cooperation in the civil protection sector within Europe, in particular with regards 
to  disaster  response  and  preparedness.  The  creation  of  a  well-established  system  of  civil 
protection  authorities  and  a  network  of  colleagues  who  know  each  other  across  all  the 
Participating  States  was  instrumental  to  enhance  communication  and  information  exchange. 
Survey  results  and  interviews  also  suggest  that  the  UCPM  contributed  to  improving  the 
effectiveness  of  systems  for  preventing,  preparing  for  and  responding  to  disasters74.  The 
legislation does not set specific result indicators for the General Objectives and the external 
evaluation did not suggest potential indicators to use as proxies. 

74 External Evaluation Report, p. 55 

29 

 
 
                                                            
6.3  Efficiency 

Generally  speaking,  the  stakeholders'  perception  of  the  UCPM's  efficiency  is  very  positive, 
particularly with regards to emergency response and, overall, administrative costs associated 
to implementing the UCPM decision are considered acceptable by most Participating States75. 

While  the  efficiency  question  encompasses  the  entire  spectrum  of  UCPM  activities,  it  is 
particularly critical for the response pillar since one of the specific results indicators set by the 
Decision refers to "promoting rapid and efficient response". The external evaluators' analysis 
of  efficiency  focused  on  transport  provision  under  the  UCPM.  In  particular,  the  external 
evaluator attempted to carry out a cost analysis of transport grants requested by Participating 
States  for  comparable  response  operations,  namely  aerial  forest  fire  fighting,  water 
purification  in  the  aftermath  of  floods  and  in-kind  assistance  in  the  context  of  the  refugee 
crisis.  Preliminary  evidence76  collected  in  the  context  of  the  evaluation  points  to  some 
discrepancies in the net contribution requested by Participating States for comparable types of 
services  (cost/mile  ratio).  This  is  the  case  for  Canadair  aircraft  operations  carried  out  by 
different Participating States, where the cost per mile ratio seems to vary significantly (even 
though  the  aircraft  used  is  similar)77.  Similar  discrepancies  also  emerge  from  the  net 
contribution granted to two Participating States providing a water purification module during 
a flood response operation inside Europe and from transport of in-kind resources by truck or 
plane. Although the analysis carried out by the external evaluators remains preliminary due to 
the limited data sample considered, they point to the need to systematically collect cost data 
and run analysis of comparability of transport reimbursement. Similarly, a robust analysis of 
transport costs for in-kind assistance goods and/or capacities via the EC broker or framework 
contractors could help in order to look for any significant variation and determine whether EU 
transport  co-financing  is  always  the  most  cost-efficient  option.  Similarly,  the  external 
evaluation highlights that useful data concerning costs remain locked in paper forms instead 
of  being  recorded  in  CECIS  and  analysed78.  Such  analysis  could  lead  to  determining 
benchmarks  (lower/higher  bounds  or  ranges)  for  cost  of  deploying  certain  assets.  In  some 
cases  this  information  could  lead  to  quicker  approval  of  deployments  and  to  faster 
reimbursement processes. 

As far as efficiency of UCPM-financed activities, the following challenge emerged from the 
external evaluation: 

  The UCPM needs to monitor more systematically and consistently cost and speed of 
response missions. This can help establish baselines/benchmarks (lower/higher bounds 
or  a  range)  allowing  comparisons  between  emergencies  responses  and  produce 
analyses  needed  to  improve  efficiency  of  future  performance.  Moreover,  rigorous 

75 External Evaluation Report, p. 60 
76 These findings need to be considered as preliminary due to the short timeframe of the evaluation and to the fact in many 
cases final invoicing has not yet been processed. 
77 External Evaluation Report, p. 63 
78 External Evaluation Report, p. 65 

30 

 
 
                                                            
monitoring  arrangements  could  also  help  speed  up  the  reimbursement  process  (for 
instance via pre-agreed ceilings). 

6.4  Coherence 

The  UCPM  has  achieved  internal  coherence  between  its  different  components  and  a  good 
level of cooperation with other policy areas. 

Internally, a number of cross-cutting activities (e.g. lessons learned, advisory missions, peer 
reviews,  exchange  of  experts,  etc.)  within  the  UCPM  provide  for  an  integrated  prevention-
preparedness-response approach and therefore help different activities of the UCPM reinforce 
each  other.  This  is  confirmed  by  the  views  collected  through  the  stakeholder  consultations 
with Participating States (see figures 9-10), where a vast majority of the responding national 
civil protection authorities and ministries agreed that the UCPM promoted sufficient linkages 
and synergies across its three pillars79. 

Figure 9: Stakeholder consultations.  

  Source: ICF, Survey of national civil protection authorities from 27 Participating States 

Figure 10: Stakeholder consultations.  

    Source: ICF, Survey of National Ministries (other than civil protection) from 13 Participating States 

Consultations  with  stakeholders  in  the  context  of  the  external  evaluation  also  highlighted  a 
number  of  significant  challenges  for  strengthening  the  internal  coherence  of  the  UCPM 
through its cross-cutting activities80. 

  There  needs  to  be  a  continuous  feedback  loop  between  lessons  learnt  and  the 
implementation of activities (including prevention and preparedness ones, in addition 

79 External Evaluation Report, p. 305 
80 See footnote 45 

31 

 
 
 
 
           
                                                            
to  response).  Participating  States  have  highlighted  that  some  specific  activities  (e.g. 
training and exercises) would benefit from a stronger link with UCPM operations, for 
instance  focusing  on  specific  challenges  arising  from  the  deployment  of  modules  in 
the field (e.g. custom clearance in fragile and conflict-affected states; cooperation with 
host  countries;  integration  in  broader  international  frameworks  for  disaster  response 
and  recovery  planning;  etc.).  To  this  end,  there  is  a  shared  feeling  that  the  lessons 
learnt  process  of  the  UCPM  needs  to  be  more  structured  and  systematic  from 
identification  and 
the  Commission  and 
Participating  States  sides),  with  regular  monitoring  and  transparent  reporting  on  the 
progress. 

implementation  (on  both 

throughout 

  The  UCPM  training  and  exercise  programme  should  promote  a  more  coherent 
approach  between  training  and  deployment.  Training  should  be  focused  on  the 
competences  needed  during  deployment  (including  for  preparedness  and  prevention 
activities). In this respect, as confirmed by one of the case studies81, the link between 
training  and  exercises  is  critical  and  should  be  strengthened.  Consultations  with 
Participating  States  suggested  potential  synergies  between  training  (particularly  the 
most  advanced  courses)  and  ModEx  or  full-scale  exercises,  in  order  to  enhance 
operational  preparedness  of  experts82.  Some  of  these  recommendations  are  already 
being addressed by the Commission. 

  Peer  reviews  should  ensure  a  good  match  between  reviewers  and  the  needs  of 
reviewed countries (i.e. between profile/competences of the reviewer and specificities 
of the country/sector to be reviewed). Additional linkages could be explored between 
training and the deployment of experts on peer review missions. This may include a 
process  to  certify  experts'  technical  and  operational  competence  in  order  to  better 
address the needs of the reviewed countries. 

  Prevention and preparedness ("advisory") missions should build on a well-formulated, 
result-oriented approach. A structured process needs to be put in place to prepare and 
implement these missions, be it in preparation for potential disasters (e.g. assessment 
of  national  emergency  management  capacity)  or,  in  the  aftermath  of  a  disaster,  to 
inform  recovery  and  reconstruction  strategies.  Synergies  with  other  EU-funded 
programmes  (in  particular  the  Prevention  Preparedness  and Response  to  Natural  and 
(IPA) 
man-made  Disasters 
programmes)  and/or  Participating  States  engaged  in  third  countries  should  be 
strategically  considered,  with  a  view  to  provide  additional  resources  (technical  and 
financial) needed for implementing recommendations. 

for  Pre-Accession 

(PPRD)  and 

Instrument 

the 

In  general,  the  consultations  highlighted  the  need  for  a  more  result-oriented  focus  for  the 
cross-cutting  activities  of  the  Mechanism  (e.g.  lessons  learned,  peer  reviews,  advisory 

81 Estonia ModEx exercise, observed in the context of the external evaluation. 
82 See footnote 45 

32 

 
 
 
 
 
                                                            
missions,  exchange  of  experts)83.  This  includes  focusing  on  measurable  policy  and/or 
operational  results  which  should  be  achieved,  in  a  given  timeframe  by  the  beneficiary.  In 
some cases, activities conducted in response to a request from a Participating States or a third 
country (e.g. advisory missions) could be conditioned to the requesting country meeting some 
policy/capacity targets prior to the UCPM fielding a technical assistance mission. 

As  regards  the  external  coherence,  the  external  evaluation  mapped  the  extent  to  which  the 
UCPM  has  been  pursuing  synergies  with  other  policy  areas  and  instruments  in  the  period 
2014-2016. When compared to the final evaluation of the Civil Protection Mechanism (2007-
2013),  the  evaluators  noted  that  the  current  Union  Mechanism  is  cutting  across  a  larger 
number  of  policy  areas.  Although  the  degree  of  coordination  could  only  be  assessed on  the 
basis  of  qualitative  input  provided  during  interviews  and  mostly  at  Commission  level,  a 
higher  degree  of  coordination  with  other  areas84  was  observed,  compared  with  the  previous 
Mechanism.  The  level  of  cooperation  between  the  UCPM  and  other  policy  areas  related  to 
civil  protection,  based  on  the  external  evaluation  (online  survey  of  national  civil  protection 
authorities and interviews with several EC/EU services) is presented in figure 11 below. 

Figure  11:  Current  status  of  coordination  between  UCPM  and  other  policy  areas.  Red  arrows  show  
growing importance of the policy area and need for inter-institutional cooperation. Green arrows 
mean 
the  period  2014-2016.  
Source: 

cooperation  has  grown  over 
(based 
Evaluation 

inter-institutional 

consultation)

stakeholder 

External 

that 

ICF 

on 

83 External Evaluation Report, p. 119 
84 Consultations included 15 DGs and other services of the EU. The complete list is provided in Annex 2. 

33 

 
 
 
 
  
 
                                                            
During  the  first  half  of  its  implementation,  the  UCPM  strengthened  its  linkages  with  key 
policy  areas.  These  include:  research  and  innovation  (including  the  Disaster  Risk 
Management Knowledge Centre  and the Copernicus rapid mapping services); regional policy 
(ex-ante conditionality to cohesion funds on disaster risk management and policy coordination 
with  the  cohesion  policy  instrument);  environment  (integration  of  hydrological  and  civil 
protection early warning systems for floods under the Flood Directive and, to a lesser extent, 
promoting the UCPM to support implementation of the Seveso Directive), climate adaptation 
(sharing  of  knowledge  under  the  Disaster  Risk  Management  Knowledge  Centre  and,  to  a 
lesser  extent,  integration  of  climate  change  in  the  national  risk  assessments);  health 
(cooperation with DG SANTE and the European Centre for Disease Prevention and Control 
(ECDC) on health-related activities, including the European Medical Corps), neighbourhood 
and  enlargement  policy  (in  cooperation  with  DG  NEAR)  as  well  as  coherence  and 
complementarity between CP, humanitarian and development assistance. 

As far as the neighbourhood policy is concerned, further coherence between UCPM and DG 
NEAR activities in Partner countries is needed. In particular, coordination between the yearly 
Prevention  and  Preparedness  Calls  for  Proposals  of  the  UCPM  and  the  disaster  risk 
management  related  activities  of  the  PPRD  was  found  to  be  insufficient  by  the  2017  final 
evaluation of the PPRD-East II85. The evaluation called for enhancing coordination between 
DG  ECHO  and  DG  NEAR  on  their  respective  programmes.  Similarly  the  2017  final 
evaluation of the PPRD-South Programme86 underscored the need to provide to both decision- 
makers  and  practitioners  in  partner  countries  with  more  knowledge  on  the  UCPM  and  the 
opportunities it offers them in prevention/preparedness/response. The IPA programme, would 
benefit from further integration in the UCPM87. This includes, among others, participation in 
UCPM exercises and increased awareness of UCPM procedures; integration of IPA-supported 
multi-countries response modules in the EERC; and joint efforts with UCPM to strengthen the 
governance  of  the  prevention  activities  (e.g.  on  the  Disaster  Risk  Assessment  and  Mapping 
initiative). 

Recent developments affecting Europe's security (mainly related to terrorist attacks) have also 
prompted  the  security  policy  as  a  priority  area  for  cooperation  with  civil  protection.  The 
"Security Union" initiative launched towards the end of 2016 aims at providing a platform for 
cooperation across several EU instruments, including the UCPM and is expected to take on a 
primary  role  in  the  second  half  of  the  UCPM  implementation.  The  approach  taken  by  the 
UCPM-Security Union coordination goes beyond the exchange of information and focuses on 
a common agenda  with specific  milestones.  This result-oriented approach could possibly  be 
replicated in other areas. 

Finally, as far as the response outside of Europe is concerned, the comparative advantage of 
civil protection intervention vis-a-vis humanitarian assistance needs to be carefully considered 
as  the  UCPM  becomes  increasingly  activated  for  dealing  with  humanitarian  and  complex 

85 European Commission, ROM Report, Prevention Preparedness & Response to natural and man-made Disasters in the EaP 
East (PPRD_East II), 03/05/2017 
86 PARTICIP, Evaluation of the PPRD South II Regional Project (ENI South), January 2017 
87 IPA Civil Protection Cooperation Programme II, Final Programme Report, December 2015 

34 

 
 
                                                            
emergencies (see section 6.1). In particular, the UCPM can fill important emergency gaps in 
cases where it is not possible to fully deploy humanitarian teams or where specific technical 
expertise  and/or  assets  are  needed.  In  the  same  vein,  linkages  with  other  prevention, 
preparedness or response activities outside of Europe should be ensured from early stages of 
intervention  under  the  UCPM.  For  example,  in  the  aftermath  of  the  Nepal  earthquake  the 
support  offered  through  the  UCPM  (e.g.  structural  engineering,  water  purification  systems) 
was  followed  by  the  development  of  a  Post  Disaster  Needs  Assessment  (PDNA),  which 
became  the  recovery  and  rehabilitation  framework  for  the  country.  In  such  situations,  the 
assessments  carried  out  by  civil  protection  experts  deployed  by  EU  Member  States  or  the 
Commission  should  feed  into  wider  recovery  frameworks  conducted  by  partner  countries 
governments, with support from the international community. 

6.5  EU Added Value 

The  UCPM  provides  Participating  States  with  cooperation  opportunities  and  increased 
capacity – and predictability – in terms of expertise and resources that would not be possible 
without coordinated effort at EU level. The vast majority of Participating States consulted see 
an added value in EU-level action under the UCPM as opposed to bilateral actions or to the 
action of international organisations or NGOs88. 

Prevention:  Consulted  stakeholders  noted  the  importance  of  the  Decision's  provisions  on 
prevention (e.g. National Risk Assessments) as a "pull factor" to develop throughout all the 
Member  States  policies  and  strategies  addressing  disaster  prevention.  Furthermore,  the 
introduction of European Commission guidelines for risk assessments had a very significant 
impact on this pillar of the UCPM, contributing to the first-ever overview of risks in the EU. 
Prevention  and  preparedness  projects,  which  must  necessarily  involve  three  or  more 
Participating States, deliver an inherent cross-border value by facilitating the establishment of 
informal channels between national authorities having to manage common risks. Most of the 
Participating States consulted by the external evaluators do not believe these projects would 
have been carried out in the absence of UCPM funding or, if they had been, they would have 
involved duplications and fragmentations89. 

88 External Evaluation Report, p. 173 
89 External Evaluation Report, p. 207 

35 

 
 
 
 
                                                            
Figure 12: Stakeholder consultations.  

Source: ICF, Survey of Project Leads of Prevention and Preparedness Projects from 12 Participating 
States. 

A  more robust  UCPM  monitoring framework (as explained in  section 6.2 on Effectiveness) 
that,  for  instance,  allowed  to  keep  better  track  of  outputs  and  outcomes  of  cross-border 
prevention projects (e.g. number of cross-border risk assessments developed by the UCPM, or 
number of macro-regional development plans that take into account findings of risk analysis 
developed  by  the  UCPM)  would  also  help  to  better  appreciate  and  communicate  to 
stakeholders and the public the added value of EU action in the area of civil protection and 
disaster risk management. 

Preparedness:  EU  assistance  to  Participating  States  in  organising  resources  through  the 
creation of the EERC and the intervention modules is considered by stakeholders as the most 
tangible  (and  visible)  added  value  generated  by  the  UCPM90.  The  overall  capacity  and 
deployment capabilities (e.g. training and inter-operability) of the EERC is larger than what 
would  have  been  possible  through  the  sum  of  the  national  capacities.  Voluntary  pooling  of 
Participating States' assets on standby has facilitated the planning of operations and added an 
element  of  predictability,  thus  enhancing  EU-wide  contingency  planning.  In  addition  to 
making response capacities more adaptable to the international context, most of the consulted 
countries agree that promoting common European standards for modules that join the EERC 
has  contributed  to  enhance  their  quality  and  robustness91.  Similar  considerations  apply  to 
training  and  exercises,  which  have  contributed  to  create  a  network  of  EU  civil  protection 
professionals sharing common operating procedures and standards throughout Europe. To this 
end,  the  exchange  of  expert  program  is  also  considered  as  being  a  valuable  instrument  to 
provide  to  a  large  number  of  civil  protection  experts  opportunities  to  share  knowledge  and 
acquire  new  skills  while  belonging  to  an  international  network  of  civil  protection 
professionals. 

90 External Evaluation Report, p. 101-102 
91 External Evaluation Report, p. 102 

36 

 
 
 
                                                            
Response:  The  evolution  of  the  MIC  into  a  single  "hub"  for  information  sharing  and 
operational coordination, the ERCC, was considered by stakeholders to be the most tangible 
added value for European and third countries. Managing multiple bilateral offers at the same 
time  and  without  ERCC  coordination  would  be  more  cumbersome/costly  and  would  bring 
lower  visibility.  Another  significant  value  brought  by  EU-level  action  in  the  area  of  civil 
protection  is  the  possibility  to  leverage  satellite  systems  (e.g.  Copernicus)  for  making 
available  trans-boundary  hazard  early  warning  systems  and  to  inform  disaster  response 
operations (e.g. providing damage assessment maps). 

Preliminary recommendations by the external contractor92 to further increase the added value 
of the UCPM interventions echo those enshrined in the recent ECA report, i.e. shortening the 
process  for  selecting  experts  for  deployment  and  better  capture  UCPM  performance  by 
systematically monitoring quantitative indicators for speed (e.g. time needed to respond to a 
request  for  assistance)  of  response.  Tracking  the  cost-effectiveness  of  EU-level  action  (e.g. 
savings  generated  by  well-coordinated  response  operations)  could  also  demonstrate  another 
significant added value. 

6.6 

Sustainability 

The  sustainability  of  the  UCPM's  results  is  being  ensured  to  a  certain  extent  through  the 
acquisition  of  knowledge  and  skills  by  experts  participating  in  different  UCPM  activities. 
Consultations with Participating States highlighted the importance of training, exercises and 
exchange  of  experts  and  of  fostering  cross-border  prevention  and  preparedness93.  A  critical 
mass  of  trained  European  civil  protection  experts  and  practitioners  enables  the  uptake  of 
knowledge and tools made available by the UCPM. 

Sustainability  of  prevention  and  preparedness  projects  financed  under  the  UCPM  had  been 
flagged  as  a  potential  issue  by  the  ex-post  evaluation  of  the  Civil  Protection  Mechanism 
(2007-2013).  Lack  of  trans-national  coordination  mechanisms  and  insufficient  follow-up 
funding  very  often  limited  the  uptake  of  project  results  at  national  level.  The  survey  of 
stakeholders carried out for the external evaluation of the UCPM shows a better perception of 
sustainability of such projects which, according to the views of the project leads, are likely to 
be continued and/or scaled up with national, regional or other EU funding94. Compared to past 
years,  progress  was  also  helped  by  more  systematic  publication  of  project  results  via  the 
UCPM  website  managed  by  DG  ECHO.  However  statistical  evidence  is  very  limited  and 
uncertainties  remain  on  the  effective  use  of  results  by  end-users  as  well  as  on  the 
public/private sector hand-over after project closure. 

92 External Evaluation Report, p. 117-126 
93 External Evaluation Report, p. 149 
94 External Evaluation Report, p. 288 

37 

 
 
                                                            
The  following  issues  were  highlighted  by  the  stakeholders  consulted  in  the  context  of  the 
external  evaluation  and  during  discussions  with  the  Commission  as  critical  challenges  that 
may affect the sustainability of UCPM activities and their results in the coming years95. 

  A viable financial model to ensure availability of capacities in the EERC. Providing 
realistic  economic  incentives  to  Participating  States  for  pooling  their  assets  and 
developing  new  response  capacities  is  critical  to  ensuring  a  durable  pool  of 
capacities  in  the  EERC.  At  the  same  time,  creating  bigger  incentives  to  pooling 
assets  should  not  come  at  the  detriment  of  Participating  States  '  investments  in 
national response capacities. 

  An  efficient  strategy  to  build  the  European  civil  protection  human  resource 
capacity. New ideas and approaches will be needed to ensure that UCPM foundation 
courses  be  rolled  out  extensively  without  creating  duplications  with  the  training 
program  of  Participating  States,  while  more  specialized/advanced  training  (e.g. 
training for deployment) be delivered to well-identified groups of experts. Matching 
competences  with  needs  is  critical  to  make  sure  the  UCPM  fulfil  its  mission  of 
supporting its Participating States.  

  The  role  of  the  private  sector  in  the  implementation  of  UCPM  activities.  Several 
Participating  States  have  highlighted  the  potential  risks  arising  from  private 
companies  carrying  out  needs  assessments,  advisory  missions,  peer  reviews  and 
other  activities  that  provide  commercial  actors  with  access  to  potential  market 
opportunities96. Measures should be introduced to guarantee that such companies are 
not  in  an  advantageous  position  vis-a-vis  other  private  competitors  or  the  public 
sector  itself  when  participating  in  a  call  for  tenders.  Furthermore,  there  may  be  a 
need to protect confidentiality and secrecy of sensitive information in some sectors 
(e.g. national security threats; critical infrastructure; critical capacity gaps; etc.). 

  Bringing the UCPM closer to the European citizens while guaranteeing support to 
the  national  level.  Subsidiarity  is  one  of  the  core  principles  of  the  UCPM,  whose 
primary  function  is  to  support  Participating  States'  responsibility  to  protect  their 
citizens and assets. Being relevant to the civil protection/disaster risk management 
actors in the front line (regions and other territorial authorities) while strengthening 
the  coordinator/broker  role  of  national  civil  protection  authorities  is  a  critical 
challenge  for  the  sustainability  of  the  UCPM.  Similarly,  the  UCPM  needs  to 
promote  the  bi-lateral,  tripartite  dimension  of  cooperation  between  countries  (e.g. 
neighbouring countries) while ensuring its pan-European approach and values. 

95 Based on dialogue with PS during several meetings occurred in 2016 and first half of 2017 (e.g. Civil Protection 
Committee, CP DG meeting, experts' meetings, etc.). 
96 External Evaluation Report, p. 50 

38 

 
 
 
 
 
 
                                                            
  Maximizing  the  uptake  of  projects  results  (e.g.  prevention  and  preparedness 
projects,  exercises).  UCPM-financed  projects  need  to  be  designed  and  monitored 
with a view to focusing on the uptake of project results by end users. There is a need 
to shift the focus from the output (e.g. the delivery of a risk assessment, a workshop 
or a new training methodology) to the outcome (the actual change generated by that 
product, e.g. a  new infrastructure project that takes into  account  the results of that 
risk assessment or workshop, or the use of that methodology by the trained experts 
in a real-life situation). 

7 

CONCLUSIONS 

Based on the findings of the external evaluation and taking into account the results of recent 
analytical work on the UCPM97 and its own operational knowledge, the Commission's overall 
conclusion  is  that  the  implementation  of  UCPM  activities  is  well  underway  and  on  track  to 
achieve  its  objectives  by  the  end  of  the  legislative  timeframe  (December  2020).  The 
performance  of  the  UCPM  during  the  first  half  of  its  timeframe  has  showed  encouraging 
results  in  each  of  the  three  pillars  (prevention/preparedness/response)  and  overall  positive 
views  from  the  Participating  States  and  the  main  stakeholders.  At  the  same  time,  the 
Evaluation  has  also  highlighted  critical  challenges  that,  if  addressed  promptly,  may  further 
increase the impact of the UCPM. A general area for improvement is the strengthening of the 
UCPM  results  monitoring  framework,  through  the  possible  introduction  of  quantitative 
indicators  which  capture  more  adequately  progress  achieved  under  the  three  pillars  of  the 
UCPM. 

There is sufficient evidence to demonstrate the continued  relevance of UCPM objectives to 
Europe's needs. The results of the 2017 Eurobarometer survey on Civil Protection show that a 
vast majority of European citizens support a common EU policy in the field of civil protection 
and agree that the EU should lead the coordination of disaster response to affected countries, 
both  in  Europe  and  in  the  rest  of  the  world.  Recent  resolutions  of  the  European  Parliament 
have  also  called  on  the  Commission  to  further  support  cooperation  among  European  civil 
protection  authorities.  The  stakeholders  consulted  also  highlighted  the  flexibility  of  the 
UCPM to rapidly adapt and address newly emerging crises which were not "traditional" civil 
protection  emergencies  (e.g.  the  Ebola  outbreak  in  West  Africa  and  the  migrant  crisis  in 
Europe).  The  increasing  number  and  scope  of  UCPM  activations  outside  of  Europe  pose, 
however,  the  challenge  of  finding  the  right  balance  between  being  a  framework  of  mutual 
assistance  between  European  countries  in  the  aftermath  of  natural  and  man-made  disasters 
and evolving towards a more complex instrument capable of addressing crises of a multiple 
nature and with a global reach. 

An analysis of the UCPM effectiveness highlighted encouraging progress under each of the 
specific objectives.  The UCPM is  fully  on track to  achieve objectives pertaining  to  disaster 

97 European Court of Auditors, Special Report no 33/2016; Report from the Commission to the European Parliament and the 
Council on progress made and gaps remaining in the European Emergency Response Capacity, 17.02.2017 

39 

 
 
 
                                                            
prevention  framework  (objective  1)  and  readiness  for  disasters  (objective  2),  based  on  the 
indicators set by the legislation and on feedback received by key stakeholders consulted. The 
UCPM's  response  to  disasters  (objective  3)  has  been  timely  and  effective  according  to  the 
vast majority of stakeholders consulted. Confirming one of the findings of the ECA's review 
of the UCPM, the external evaluation struggled to find robust and consistent quantitative data 
(e.g. speed, needs assessments) to corroborate the stakeholders' very positive views on UCPM 
disaster  response.  Lastly,  public  awareness  and  preparedness  to  disasters  (objective  4)  have 
received  slightly  less  satisfactory  views  from  stakeholders  when  compared  to  the  other 
objectives.  In  this  respect,  it  is  questionable  whether  the  specific  indicator  ("level  of 
awareness of Union citizens of risks in their regions") set by the Decision for this objective 
can be fully attributable to the level of activities carried out by the UCPM. 

The  external  evaluation  also  underscored  several  challenges  which,  if  promptly  addressed, 
could  further  increase  the  effectiveness  of  the  UCPM.  These  include,  among  others: 
establishing a baseline/benchmarks (e.g. speed, cost/effectiveness, etc.) for comparable types 
of emergencies; developing a more evidence-based process to set capacity goals for disaster 
preparedness;  revisiting  the  system  of  incentives  to  pool  response  assets  and  develop  new 
capacities  in  the  EERC.  Disaster  prevention  is  considered  a  critical  area  that  may  require 
additional resources in future (currently only 20% of UCPM budget) although a more robust 
impact evaluation may be required to establish where the UCPM action to date has been most 
effective in assessing/reducing disaster and climate risks. Cooperation and coherence between 
the UCPM and the EU climate change adaptation agenda should be enhanced. Overall, closer 
linkages  between  UCPM  activities  in  prevention  and  preparedness  and  broader  EU-wide 
programmes  (e.g.  Cohesion/European  Regional  Development  Fund,  European  Investment 
Bank, etc.) should be explored. 

Generally  speaking,  the  stakeholders'  perception  of  the  UCPM's  efficiency  is  positive; 
particularly that emergency response and administrative costs associated to implementing the 
UCPM  decision  are  considered  acceptable  by  most  Participating  States.  The  external 
evaluation has attempted to carry out analysis of costs for comparable response missions (e.g. 
aerial  forest  fires  missions;  transport  of  modules  and  in-kind  assistance;  etc.).  Although 
evidence cannot be considered sufficient to draw final conclusions, this confirmed the need to 
systematically  carry  out  cost  analyses  and,  possibly,  determine  cost  benchmarks  (or  ranges) 
for certain type of missions/transport. 

Internal  coherence  is  deemed  satisfactory  by  the  external  evaluation.  Several  cross-cutting 
activities of the Mechanism (e.g. lessons learned, advisory missions, peer reviews, exchange 
of  experts,  etc.)  provide  for  an  integrated  prevention-preparedness-response  approach. 
Externally, linkages  between  the UCPM and other policy areas  have  increased compared  to 
the previous framework (2007-2013). This was confirmed by interviews with a large number 
of EU services. Focusing on a result-based agenda and milestones is seen as conducive to a 
more  effective  cooperation  and  it  is  felt  that  enhanced  coordination  mechanisms  need  to  be 
considered  in  order  to  improve  synergies  between  instruments  funding  prevention  and 

40 

 
 
preparedness  activities  in  Neighbourhood  and  Pre-Accession  countries98.  As  far  as  response 
outside of Europe is concerned, the comparative advantage of civil protection intervention àa-
vis  humanitarian  assistance  needs  to  be  carefully  considered  as  the  UCPM  becomes 
increasingly activated for dealing with humanitarian and complex emergencies. 

The  existence  of  a  single  European  "hub"  for  information  sharing  and  operational 
coordination, the introduction of common European standards for disaster response capacities 
and  common  guidelines  on  risk  assessments,  which  have  contributed  to  the  first-ever 
overview of risks in the EU, are perceived as having delivered high  EU added value in the 
areas of disaster response, preparedness and prevention, respectively. Increased capacity and 
predictability  of  emergency  response  capacity  provided  by the  UCPM  would  not  have  been 
possible without coordinated effort at EU level. Over the next years, introducing indicators to 
monitor  the  effectiveness  and  efficiency  of  the  UCPM  more  adequately  may  provide 
additional  evidence  to  appreciate  (and  communicate)  the  added  value  of  EU-level  action  in 
civil protection. 

Finally,  knowledge-sharing  and  training  activities  are  critical  to  creating  a  community  of 
European  civil  protection  practitioners  and,  ultimately,  to  ensuring  sustainability  of  the 
UCPM's results. However, at a system level, several issues are considered as critical for the 
sustainability  of  the  UCPM  going  forward.  A  viable  financial  model  needs  to  ensure 
availability  of  capacities  in  the  Voluntary  Pool;  an  efficient  strategy  to  build  the  European 
civil protection human resource capacity needs to be rolled out; the role of the private sector 
in  the  implementation  of  UCPM  activities  needs  to  be  clarified;  end-users  needs  need  to 
inform  design  and  monitoring  of  prevention  and  preparedness  projects  financed  by  the 
UCPM,  in  order  to  maximize  their  impact  beyond  the  project  closure  (impact  should  be 
evaluated  one  or  two  years  after  funding  has  ended).  The  UCPM  needs  to  get  closer  to  the 
territorial  actors  while  supporting  the  role  of  the  national  civil  protection  authorities  in 
cementing  the  risk  management  governance  all  along  the  European-national-regional-local 
chain. 

The  results  of  this  Evaluation  provide  a  good  analytical  basis  to  start  reflection  on  a  future 
framework. The Commission invites the European Parliament and the Council to take note of 
these evaluation findings. 

98 This emerges as the main findings of recent evaluations, including: ROM Review of PPRD-East (2017); Evaluation of 
PPRD-South (2017); Evaluation of IPA II (2015).  

41 

 
 
                                                            
ANNEX 1: INTERVENTION LOGIC OF THE UCPM 

42 

 
 
 
 
 
 
 
ANNEX 2: PROCEDURAL INFORMATION 

A) Lead DG: European Civil Protection and Humanitarian Aid Operations (ECHO) 

B) Reference in Decide: 2016/ECHO/007 

C) Organisation: The external evaluation was supervised by an Inter-service Steering Group 
(ISG),  composed  of  representatives  of  the  EEAS  and  the  following  European  Commission 
(EC)  DGs:  CLIMA,  DEVCO,  ECHO,  ENER,  ENV,  HOME,  JRC,  MOVE,  NEAR,  RTD, 
SANTE  and  SG.  The  external  evaluation  contract  had  a  length  of  9  months,  from  October 
2016 to June 2017. Four meetings were held: 

  9 November 2016:  Kick-off meeting 

  8 December 2016: 

Inception report meeting 

  3 March 2017:  

Interim report meeting 

  20 April 2017:  

Draft final report meeting 

D) Quality assurance:  

  The  ISG  assessed  the  external  draft  Final  Report  against  the  following  criteria: 
compliance  with  technical  specifications;  accuracy  and  completeness  of  data; 
relevance  and  soundness  of  analysis;  robustness  of  conclusions;  usefulness  of 
recommendations; general clarity of the report. 

  The ISG provided the following main comments to the external draft Final Report: 

o  Certain  missing  elements  should  be  provided:  a  chapter  explaining  the 
intervention, a chapter on the evaluation methodology, the case study reports, 
a proposal on the dissemination plan…; 

o  The soundness of the arguments should be improved, and triangulation should 

be more evident; 

o  Conclusions  on  cost-effectiveness  should  be  better  supported  on  quantitative 

data; 

o  Conclusions on effectiveness, coherence and EU added value should be better 

elaborated; 

o  Some 

links  between  conclusions  and 

recommendations  should  be 

strengthened; 

o  Recommendations should be further tailored and detailed; 
o  The general clarity of the report should be improved, and the report should be 

shortened. 

These comments were fully taken into account in the final version of the report. 

E)  Evidence:  Several  information  sources  were  used  to  reach  triangulated  conclusions,  i.e. 
document review; analysis of the EC's project database (‘HOPE’); face to face and telephone 
interviews;  four  surveys  (to  national  ministries,  civil  protection  authorities,  Trainers  and 
national  training  coordinators,  civil  protection  experts  and  project  coordinators);  an  Open 
Public Consultation, and case studies. 

43 

 
 
F)  External  expertise:  The  evaluation  was  carried  out  by  an  external  consultant,  ICF 
Consulting  Services  Limited,  who  was  selected  through  a  competitive  process  under  DG 
ECHO's  evaluation  framework  contract.  The  current  Staff  Working  Document  aims  at 
providing  a  self-reflected  summary  of  all  sources  of  evidence,  including  the  evaluation 
carried  out  by  the  external  consultant,  about  the  first  years  of  existence  of  the  current 
Mechanism. The report is available at: http://bookshop.europa.eu. 

LIST OF EVALUATION QUESTIONS 

Effectiveness 

  Question 1. To what extent have the objectives of the Decision been achieved? 

To  what  extent  has  the  Union  Mechanism  contributed  to  achieving  the  general 
objective of the Decision? To what extent has progress been made in relation to the 
four  specific  objectives?  Given  the  current  timeframe,  is  the  Union  Mechanism  on 
track  to  achieve  the  specific  objectives  of  the  Decision?  To  what  extent  have 
external factors influenced the observed progress towards the objectives? 

Efficiency 

  Question 2. To what extent are the costs of the activities performed under the 

UCPM justified when compared to their benefits? 
To what extent did ECHO achieve cost-effectiveness in its actions under the UCPM? 
To  address  this  question  the  evaluator  is  invited  to  propose  a  dedicated,  overall 
approach.  What  main  factors  influenced  cost-effectiveness  of  the  actions?  Were 
there  significant  differences  in  costs  and  objective  achievement  between  the 
Participating  States?  If  so,  what  are  the  underlying  causes?  Which  decisions  could 
contribute to improve cost-effectiveness? 

  Question  3:  To  what  extent  have  the  monitoring  arrangements  contributed  to 

an efficient and effective implementation of the intervention? 
Are the indicators currently set by the Decision adequate and sufficient to capture to 
monitor  successful  implementation  of  the  Decision?  Which,  if  at  all,  additional 
indicators should be considered? Has data been properly collected and monitored? Is 
data generally available? 

Relevance 

  Question  4:  To  what  extent  are  the  Decision's  objectives  still  relevant  to  the 
needs identified and to the European Commission's priorities for 2014-2020? 
To what extent do the general and specific objectives of Decision still correspond to 
current needs within the EU? To what extent have the general and specific objectives 
of  the  Decision  proved  to  be  appropriate  to  address  the  issues  highlighted  in  the 
Impact Assessment (2011) Were the main findings and lessons learnt presented in the 
Ex-Post  Evaluation  of  the  Community  Civil  Protection  Mechanism  (2007-2013) 

44 

 
 
 
taken into account for the implementation of the UCPM? Has the Mechanism been 
flexible enough to address emerging/unanticipated critical issues in the disaster risk 
management  arena?  To  what  extent  has  the  UCPM  integrated  in  its  functioning  as 
well as in its activities scientific and technological research and development that has 
become available since its creation? 

Coherence 

  Question  5.  To  what  extent  is  this  intervention  internally  and  externally 

coherent? 
How  well  are  the  different  activities  of  the  UCPM  articulated,  interlinked,  and 
mutually  supportive?  What  examples  of  synergies  exist  between  different  activities 
and  areas  (i.e.  prevention/preparedness/response)  of  the  UCPM?  Can  further 
synergies  be  sought?  Are  there  any  missed  opportunities?    To  what  extent  are  the 
potential  synergies  between  the  UCPM  and  other  EC  policy  areas  as  well  as 
international frameworks/initiatives being exploited? 

EU added value 

  Question  6.  What  is  the  EU  Added  Value  of  the  UCPM,  including  the  added 

value to other participating states? 
What results of the intervention can be identified that could not have been achieved 
without a coordinated effort at the EU level? Which criteria could be considered to 
demonstrate  EU  added  value  of  the  UCPM-supported  interventions  in  prevention, 
preparedness, and response? 

Sustainability 

  Question 7. To what extent are the results achieved by the UCPM sustainable? 

To  what  extent  are  the  results  achieved  by  the  UCPM  likely  to  last  beyond  the 
timeframe  of  the  intervention?  Are  adequate  arrangements  in  place  to  ensure 
(financial, capacity, technical, etc.) sustainability of current and future interventions? 

45 

 
 
 
 
ANNEX 3: OVERVIEW OF ACTIVITIES CARRIED OUT UNDER THE UCPM 

(2014-2016) 

Prevention 

Preparedness 

Response 

Improving the knowledge base for 
disaster prevention 

Emergency Response Coordination 
Centre (ERCC) 

Grants for transport and related 
logistics, inside the Union 

Grants for procurement, inside the 
Union: 
(1) Dispatching expertise 
(2) Transport and related logistics 

Grants for transport and related 
logistics, outside the Union 
Grants for procurement, inside the 
Union: 
(1) Dispatching expertise 
(2) Transport and related logistics 

EU overview of risks 

Early warning and analysis 

Prevention projects 

Preparedness projects 

Training 

Training Network 
Civil Protection Mechanism 
Exercises 
Exercises on civil protection 
modules, technical assistance and 
support teams and EU Civil 
Protection Teams (EUCPT) 
The European Emergency 
Response Capacity (EERC) 
Voluntary Pool 
Maintenance and further 
development of the Common 
Emergency Communication and 
Information System (CECIS) 
Addressing temporary 
shortcomings 

Exchange of experts Programme 

Lessons Learned Programme 

Advisory missions for prevention and preparedness in and outside 
the Union 

Peer reviews 

Awareness raising and dissemination of good practices 

CECIS hosting, ERCC security and resilience 

= Prevention Pillar 

= Preparedness Pillar 

= Response Pillar 

46 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
  
 
 
 
 
ANNEX 4: SYNOPSIS REPORT ON STAKEHOLDER CONSULTATION 

Table of Content 

1. 

INTRODUCTION ........................................................................................................................ 48 

2.  SCOPE OF THE CONSULTATION ......................................................................................... 48 

3.  STAKEHOLDER GROUPS COVERED BY THE CONSULTATION ACTIVITIES ........ 48 

4.  CONSULTATION METHODS AND TOOLS ......................................................................... 49 

4.1 

4.2 

4.3 

4.4 

Interviews .............................................................................................................................. 49 

Online surveys ...................................................................................................................... 50 

Open Public Consultation ..................................................................................................... 50 

Case Studies .......................................................................................................................... 51 

5.  RESULTS OF EACH OF THE CONSULTATION ACTIVITIES (TARGETED) .............. 51 

5.1 

5.2 

5.3 

5.4 

5.5 

5.6 

Relevance .............................................................................................................................. 51 

Effectiveness ......................................................................................................................... 52 

Efficiency .............................................................................................................................. 52 

Coherence ............................................................................................................................. 52 

EU Added Value ................................................................................................................... 53 

Sustainability ......................................................................................................................... 53 

6.  RESULTS OF THE CONSULTATION IN THE FRAMEWORK OF THE OPC ............... 54 

6.1 

6.2 

6.3 

6.4 

6.5 

6.6 

General Questions (for both questionnaires) ......................................................................... 54 

Relevance (only for in-depth knowledge) ............................................................................. 54 

Effectiveness (only for in-depth knowledge) ........................................................................ 55 

Efficiency (only for in-dept knowledge) ............................................................................... 55 

Coherence (only for in-depth knowledge) ............................................................................ 55 

EU added value (only for in-depth knowledge) .................................................................... 55 

7.  CONCLUSIONS .......................................................................................................................... 56 

47 

 
 
 
 
 
1.  INTRODUCTION 

The general objective of the consultation activities99 carried out in the context of the Interim 
Evaluation of the UCPM was to ensure that all concerned stakeholders had the opportunity to 
express  their  views  on  the  performance  of  the  UCPM  during  the  first  half  of  its 
implementation. In line with this, the following specific objectives were set out: 

a)  Collect  views  and  opinions  of  the  European  citizens  on  the  results  achieved  by  the 

UCPM to date; 

b)  Determine  the  importance  of  different  priorities  for  civil  protection  and  disaster  risk 

management as they are perceived across the EU; 

c)  Obtain  targeted  feedback  and  technical  advice  from  the  primary  stakeholders  of  the 

UCPM; 

d)  Gather evidence to support the interim evaluation of the UCPM for each of the main 

evaluation criteria. 

The  Commission  contracted  an  external  consultant  to  carry  out  the  consultations,  prepare  a 
report, and integrate the main findings into the final evaluation report. 

2.  SCOPE OF THE CONSULTATION 

In  line  with  the  objectives,  a  consultation  strategy  was  set  up  based  on  the  EC  Better 
Regulations Guidelines for conducting evaluations100. The consultation questions were based 
on the five criteria recommended by the Better Regulations guidelines: (1) Effectiveness, (2) 
Efficiency,  (3)  Relevance,  (4)  Coherence,  and  (5)  EU  added  value.  In  addition,  (6) 
Sustainability was also added to the evaluation criteria and assessed via the consultation. 

The  consultations  covered  all  activities  carried  out  under  the  UCPM  in  the  timeframe  of 
January  2014  –  December  2016.  The  evaluation  encompassed  actions  carried  out  under  the 
framework of the UCPM and spanning across the three fields of prevention, preparedness and 
response to natural and man-made disasters. 

3.  STAKEHOLDER GROUPS COVERED BY THE CONSULTATION ACTIVITIES 

The following stakeholders were identified for input collection: 

-  Entities  implementing  the  UCPM:  Member  States  (MS)  Civil  Protection  (CP) 
authorities contributing with modules and/or experts to the Voluntary Pool; Other MS 
CP authorities; non-MS Participating States (PS) CP authorities. 

99 The consultation strategy was published on 24 November 2017 and is available at: http://ec.europa.eu/echo/sites/echo-
site/files/consultation_strategy.pdf 
100 See: http://ec.europa.eu/smart-regulation/guidelines/toc_guide_en.htm 

48 

 
 
                                                            
-  Entities affected by the UCPM: Host countries having received UCPM support (EU, 
Accession, Neighbourhood, and Third Countries), European Parliament, Committee of 
Regions, local authorities. 

-  Entities  with  a  stated  interest  in  the  UCPM:  EU  MS  (authorities  other  than  CP101), 
non-MS  PS  (authorities  other  than  CP),  UN  agencies,  World  Bank,  NGOs,  private 
sector, foundations, etc. 

-  Entities  having  expertise  relevant  to  the  UCPM  (i.e.  disaster  risk  management): 

universities/centres of excellence and think tanks, mainly in PS. 

The Open Public Consultation included views from all of the four groups above whereas the 
targeted  consultation  focused  on  the  following  ones:  entities  implementing  the  UCPM, 
entities affected by the UCPM, and entities with a stated interest in the UCPM. 

4.  CONSULTATION METHODS AND TOOLS 

The  overall  consultation  strategy  aimed  at  gathering  information  and  opinions  on  the 
implementation  of  the  Decision  as  well  as  to  test  analyses  and  findings.  Stakeholders  were 
consulted through the following means: 

(i)  Interviews 
(ii) Online surveys 
(iii)Open Public Consultation 
(iv) Case studies 

4.1 

Interviews 

Interviews were expected to be completed by end of February. In agreement with the ISG, it 
was  decided  to  extend the  consultation  period  until  April  to  finalise  two  interviews  with  an 
EU Agency and an external evaluator that could not be undertaken beforehand. 

A  total  of  40  interviews  with  stakeholders  were  originally  planned  however,  in  the  end,  64 
interviews were carried out. In particular, the external contractor conducted a higher number 
than planned interviews with: DG ECHO officers, other DGs, EU Agencies, among others. 

Telephone and face-to-face interviews were conducted with relevant staff from a broad range 
of  EC  Directorates-General  (DGs)  and  EU  services,  i.e.ECHO  ENV,  CLIMA,  NEAR, 
HOME, SANTE, ENER, EU-Agencies (ECDC, EMSA, EASO, EEA), as well as with ECHO 
international  partners  including  disaster  management-focused  organisations  (e.g.  UN 
agencies)  and  national  authorities  from  Participating  States.  Interviews  with  professional 
organisations  (e.g.  exchange  of  experts  programme  manager,  exercise  provider,  transport 
provider) and with external evaluators (External Evaluation of EMSA, External Evaluation of 
PPRD South II) were also carried out. 

101 This includes other government counterparts whose actions are overlapping with activities carried out by the UPCM. 
Examples include: Ministries of Social Affairs (e.g. protection of people from the impact of disasters), Ministry of National 
Security/Interior (e.g. management of civil protection consequences of security threats), Ministry of Public Works (e.g. 
protection of assets), Ministry of Health (e.g. prevention and management of pandemics). 

49 

 
 
                                                            
The main purposes of these interviews were to: 

1.  Collect  evaluative  evidence  with  regard  to  the  effectiveness,  efficiency,  relevance, 

coherence, EU added value and sustainability of the mechanism; and 

2.  Outline the lessons learned, success factors, gaps and areas for improvement. 

4.2  Online surveys 

In  order  to  reach  the  large  number  of  actors  involved  in  the  implementation  of  the  Union 
Mechanism  over  the  2014-2016,  five  (rather  than  the  initially  planned  four)  online  surveys 
were carried out targeting the following stakeholder (sub-)groups: 

1.  National CP authorities Contact Points; 
2.  Other relevant national authorities (e.g. other line ministries); 
3.  Trainers and National Training Coordinators; 
4.  CP experts participating in the exchange of experts programme 
5.  Leads of prevention and preparedness projects. 

The five online surveys were launched through an online application and a separate web-link 
was distributed to each target group, focused along the lines of the evaluation questions. 

The surveys were  launched three  weeks later than initially  scheduled (mid-December 2016) 
but all remained open for six weeks (as initially envisaged) in order to generate the maximum 
number  of  replies  for  each  of  these  five  surveys.  In  view  of  response  rates  being  in  line  or 
beyond  expectations  the  three-week  delay  is  not  considered  to  have  had  an  impact  on  the 
findings presented in the final report. 

The response rate for the National civil protection authorities Contact Point survey was higher 
than  expected,  as  well  as  for  the  other  relevant  national  authorities  and  experts  in  the 
exchange  of  experts  programme.  Responses  received  from  National  Contact  Points  were 
followed up with phone interviews. The response rate from national training coordinators was 
initially  lower  than  for  civil  protection  authorities  and  project  coordinators  (initially  due  to 
missing  contact  details) but  additional  reminders  were  sent  and  the  survey  closure  date  was 
extended to the end of February 2017. 

4.3  Open Public Consultation 

The  Commission  launched  an  Open  Public  Consultation  (OPC)  on  the  implementation  and 
performance  of  the  Union  Mechanism  for  a  period  of  three  months,  starting  on  the  24 
November 2016 and ending on 23 February 2017. The OPC was divided in two parts  - one 
designed for respondents who indicated to have limited/no knowledge of the Mechanism and 
one for those with in-depth knowledge. In total the OPC led to 67 answers for the survey from 
respondents without in-depth knowledge about the UCPM and 61 answers to the survey from 

50 

 
 
 
respondents  with  in-depth  knowledge  about  the  UCPM,  from  26  countries.102  A  brief 
summary of the main elements raised by respondents is included in section 7 below. 

4.4 

Case Studies 

Three  case  studies  were  also  carried  out  as  part  of  the  stakeholder  consultation  for  this 
evaluation.  The  information  collected  was  analysed  and  used  to  triangulate  information 
received from all consultation tools. The case studies selected were: (1) a case study on forest 
fires in Cyprus; (2) a module exercise (ModEx) in Estonia; and (3) the case study of the 
response to the migration emergency on the Balkan route. During these case studies a total 
of 23 interviews were conducted with relevant stakeholders (members of relevant ministries 
and  national  civil  protection  authorities,  local  experts,  trainers,  etc.).  The  case  studies  also 
served to test and confirm preliminary findings. 

5.  RESULTS OF EACH OF THE CONSULTATION ACTIVITIES (TARGETED) 

This section presents a summary of the views of the main stakeholders responding to each of 
the evaluation questions (i.e. Relevance, Effectiveness, Efficiency, Coherence, Added Value 
and  Sustainability).  Stakeholders  have  been  divided  into  three  main  categories,  namely:  a) 
Participating States; b) EC Services; and c) International Partners. This division corresponds 
to the way the data was presented by the external contractor in the Interim Evaluation. 

For each main question, this section highlights the main elements where broad consensus was 
reached by each stakeholder group, as well as some diverging opinions. 

5.1 

Relevance 

  Participating States agreed on the need to ensure a strategic link between the UCPM 
mandate  and  its  activities.  This  includes  the  need  to  allocate  resources  (e.g.  for 
response) based on needs assessment  and on the comparative advantage of the civil 
protection  vs.  other  actors  (e.g  humanitarian  aid,  police,  etc).  There  was  also  a 
consensus on building the training and exercise programme on a need-based approach 
which is informed by operational lessons learnt in the first place. 
Different  views  emerged  on  what  the  new  training  programme  should  look  like 
(number of people trained for each course, frequency, content, etc.). Views among PS 
diverged also with regards to providing  new/increased incentives for Voluntary  Pool 
assets, due to the possible risk of encouraging PS to be too reliant on EU assistance for 
events which should be addressed by national capacities. 

  EC  services  noted  the  need  to  develop  long-term  strategies  as  opposed  to  ad  hoc 
activities  from  the  UCPM  and  called  for  further  integration  with  EC  programmes 
attaining similar/related objectives. 

102 The report on the Open Public Consultation, summarising the main contributions by stakeholders can be found here: 
http://ec.europa.eu/echo/sites/echo-site/files/ucpm_-_opc_report_0.pdf  

51 

 
 
                                                            
  International  partners  (including  UN  agencies)  called  for  an  increased  use  of  risk 
assessment  maps,  more  emphasis  on  international  partners  in  training  and  exercises. 
Finally,  some  partners  underscored  that  the  UCPM  response  operations  should  be 
more used as an entry point to discuss disaster risk management with third countries. 

5.2 

Effectiveness 

  Consulted Participating States agreed, both in the online survey and in the interviews 
that  the  UCPM  has  been  broadly  effective  during  its  first  3  years.  They  also  agreed 
that the UCPM needs to better take into account existing bilateral agreements between 
PS  (proximity  factor)  and  clarify  its  cooperation/overlaps  with  the  humanitarian 
assistance. 
Diverging views were recorded with regards to the selection of experts for deployment 
(a  number  of  PS  believe  the  process  can  still  be  improved),  the  training  programme 
(quotas allocated by country size vs. by number of assets contributed to the Voluntary 
Pool), and to the effectiveness of peer reviews and their format (e.g. experts selected). 
  Several  EC  services  highlighted  that  improvements  (e.g.  more  strategic  use)  to 
advisory missions could increase their effectiveness. The integration of climate change 
adaptation into the UCPM (prevention and preparedness) as well as energy (e.g. risk 
assessments, preparedness plans) was also highlighted. 

  International  partners  underscored  the  need  for  the  UCPM  to  further  build  on  the 
close  linkages  between  the  humanitarian  aid  actors  and  local  communities/actors. 
They also raised the need to carefully look at how civil protection missions outside EU 
and  in  contexts  of  complex  emergencies  can  be  perceived  as  less  neutral  than 
humanitarian actors. 

5.3 

Efficiency 

  Participating States noted that the efficiency of some activities could be increased by 
better  defining  their  objectives  (e.g.  training  people  for  deployment  vs.  to  simply 
increasing their knowledge) and in some cases, with the streamlining of administrative 
processes. 

  EC  services  tended  to  question  the  efficiency  of  in-kind  assistance,  marking  a 

difference with the view of Participating States. 

  International partners did not express any views on this due to limited information 

available to them. 

5.4 

Coherence 

  Participating  States  agreed  to  a  large  extent  that  the  UCPM  cross-cutting  activities 
(e.g.  lessons  learnt,  peer  reviews,  advisory  missions,  exchange  of  experts)  should  be 
driven  by  a  more  result-oriented  approach  to  reinforce  the  internal  coherence  of  the 
mechanism. Participating States expressed diverging views when assessing the risk of 
duplicating training activities at EU and national level. 

52 

 
 
  At EC Services level, the main point of concern related to the relation of the UCPM 
with  other  policy  areas.  Generally,  although  an  improvement  of  linkages  with  other 
areas was appreciated in relation to the previous mechanism (2007-2013), EC services 
concluded  that  the  level  of  horizontal  cooperation  should  be  reinforced.  These  may 
include  research  and  innovation,  regional  policy,  environment,  climate  adaptation, 
health,  the  neighbourhood  policy,  as  well  as  better  integration  of  the  new  security 
landscape affecting Europe. 

  The  main  point  of  concern  for  International  Partners  is  the  growing  interlinkage 
between civil protection operations and humanitarian aid operations, as the mechanism 
is increasingly activated to deal with humanitarian and complex emergencies. Whereas 
the  UCPM  could  positively  contribute  to  humanitarian  operations,  the  mandate  and 
division of tasks need to be clearly set out to avoid overlapping and duplications. 

5.5 

EU Added Value 

  Participating States agreed to a large extent on the clear added value that the UCPM 
delivers in each of its three main pillars (i.e. prevention, preparedness and response). 
Participating  States  highlighted  specially  the  creation  of  the  ERCC  as  the  most 
tangible  and  visible  expression  of  the  EU  added  value.  In  relation  to  the  prevention 
pillar,  generally  Participating  States  agreed  that  the  actions  foreseen  under  the 
legislation,  (e.g.  requirement  to  produce  National  Risks  Assessments,  conduction  of 
Prevention and Preparedness projects, etc.) represented a clear EU added value as they 
helped  to  establish  a  common  understanding  of  risks  across  Europe  and  facilitated 
cross-border  cooperation,  inherent  when  responding  to  disasters.  The  main  point  of 
concern,  though,  which  also  generated  broad  consensus,  was  the  lack  of  clear  and 
quantifiable indicators that can help monitoring the results to better establish the EU 
added value of these actions. 

  As far as EC Services are concerned, the main element emphasised by respondents is 
the clear added value that the UCPM can generate by becoming the 'coordination hub' 
of  disaster  response  within  the  Union.  As  well  as  Participating  States,  EC  Services 
also agreed that a better tracking system of cost-effectiveness of EU-level action could 
demonstrate further the significant added value. 

  International  Partners  agreed  that  the  creation  of  the  ERCC  represents  a  positive 
contribution  towards  the  development  of  international  networks  of  civil  protection 
experts, as well as to facilitate the cooperation between EU and international actors. 

5.6 

Sustainability 

  Participating States generally agreed on the idea that better linkages between training 
and deployment need to be found, to ensure that knowledge acquired can be translated 
into actual action. On a similar note, Participating States also pointed out that a better 
follow-up  on  recommendations  done  to  a  beneficiary  country  would  be  essential  to 
ensure sustainability of changes. Some Participating States also raised concerns about 

53 

 
 
the  participation  of  private  companies  in  UCPM  activities  (e.g.  needs  assessment, 
advisory missions, peer reviews, etc.) 

  EC  Services  agreed  on  a  general  level  that  a  viable  financial  model  to  ensure 
availability  of  capacities  in  the  EERC  needs  to  be  established  to  ensure  the 
sustainability of the model. 

  International partners did not express views on this due to the limited information 

available to them. 

6.  RESULTS OF THE CONSULTATION IN THE FRAMEWORK OF THE OPC 

As stated above, the OPC was designed following a two-track approach: a questionnaire for 
respondents with limited/no knowledge of the UCPM, and a separate one for respondents with 
in-depth knowledge. A summary of the results is presented here below: 

6.1  General Questions (for both questionnaires) 

Hazard/Emergencies 

  Respondents without in-depth knowledge about the UCPM indicated that the top three 
emergencies  that  the  UCPM  should  deal  with  are:  earthquakes,  floods  and  terrorist 
attacks. 

  On the contrary, respondents with in-depth knowledge established earthquakes, floods 

and forest fires as the three main priorities. 

Source of Information 

  For  respondents  without  in-depth  knowledge  the  preferred  source  of  information 

about the UCPM would be national civil protection authorities. 

  Whereas for respondents with in-depth knowledge, the main source of information 

about the Mechanism would be by far DG ECHO. 

Pillars 

  Respondents  without  in-depth  knowledge  agreed  to  a  large  extent  that  an  EU-
coordinated action in the field of prevention, preparedness and response would be 
more effective than single actions undertaken by Member States. 

  Similarly, respondents with in-depth knowledge also agreed that the effectives of 
an  EU-coordinated  action  would  be  higher  (particularly  in  the  field  of 
preparedness and response) than single undertakings. 

6.2 

Relevance (only for in-depth knowledge) 

  Respondents  without  in-depth  knowledge  about  the  UCPM  agreed  on  the 
continuous relevance of the Mechanism as it addresses the critical challenges faced 
by  the  EU.  Although  still  high,  lesser  consensus  is  reached  when  assessing  the 
capability of the Mechanism to adapt to new risks in the emergency management. 

54 

 
 
6.3 

Effectiveness (only for in-depth knowledge) 

  Respondents without in-depth knowledge stressed that coordination at EU level in 
the  three  pillars  of  the  UCPM  (prevention,  preparedness  and  response)  has 
improved the effectiveness of the disaster response mechanisms at national level. 
Some criticism was raised, though, regarding the response to the migration crisis, 
which was not considered positively handled. 

6.4 

Efficiency (only for in-depth knowledge) 

  Half  of  respondents  without  in-depth  knowledge  noted  that  the  administrative 
procedures  to  prepare  and  submit  an  application  for  the  prevention  and 
preparedness projects were too cumbersome, therefore undermining the efficiency 
of the Mechanism. 

6.5 

Coherence (only for in-depth knowledge) 

  For  respondents  with 

in-depth  knowledge, 

the 
Mechanisms  (i.e.  the  three  pillars)  was  achieved  to  a  large  extent.  Regarding 
external coherence (i.e. coherence with other policy areas), respondents agreed that 
clear  links  were  established  with  the  humanitarian  aid.  However,  a  lack  of 
synergies was mentioned regarding migration and refugees. 

internal  coherence  within 

6.6 

EU added value (only for in-depth knowledge) 

  By  a  large  majority,  respondents  with  in-depth  knowledge  noted  that  individual 
Member  States  benefit  notably  from  a  coordinated  EU  action  through  the 
Mechanism. Consensus was also reached on the fact that EU action also increased 
the international visibility of the EU as a global actor. 

In the framework of the Open Public Consultation, and in addition to the information gathered 
through  the  questionnaires  specifically  designed,  five  other  contributions  (documents)  were 
submitted via the online portal. These are of different nature and scope and therefore difficult 
to categorise. For transparency purposes, they are presented here below: 

-  National Red Cross Societies of EU Member States and the International Federation of 
Red  Cross  and  Red  Crescent  Societies,  Contribution  to  the  Public  Consultation  on 
Evaluation of the Union Civil Protection Mechanism, February 2017. 

-  NN, The Securitisation of European Union Civil Protection. Investigating the logics of 
securitisation  and  the  development  of  European  Union  Civil  Protection,  Thesis, 
School  of  Politics  and  International  Relations  –  University  College  Dublin,  August 
2016. 

-  NN,  Position  Paper  on  the  European  Commission's  'Public  Consultation  on  the 

Interim Evaluation of the Union Civil Protection Mechanism'. 

-  PLATAFORMA,  Forced  displacement  and  development.  Plataforma's  comments  on 

- 

the non-paper for expert consultations. 
International  Amateur  Radio  Union,  Response  to  Public  Consultation  on  the  Interim 
Evaluation of the Union Civil Protection Mechanism, February 2017. 

55 

 
 
 
7.  CONCLUSIONS 

The  stakeholder  consultation  provided  valuable  feedback  on  the  functioning  of  the  UCPM 
during its first three years of implementation, highlighting its achievements as well as some of 
its shortcomings. 

In  general  terms,  stakeholders  present  a  relatively  high  degree  of  satisfaction  with  the 
functioning of the Mechanism. Responses from the consultation exercise seem to indicate that 
the Mechanism is on its way to achieve its general and specific objectives. However, crossing 
all the input collected during the stakeholder consultation, there are certain recurring points of 
concern. 

Stakeholders emphasised the need for a more strategic use of the UCPM activities already in 
place,  that  is,  foster  synergies  within  its  three  pillars  and  reinforce  and  systematise  the 
integration of lessons learnt into the feedback cycle. 

Overall,  stakeholders  also  pointed  out  the  need  to  enhance  the  synergies  with  other  policy 
areas,  actors  and  instruments  with  the  aim  to  strengthen  the  external  coherence  of  the 
Mechanism, and improve international cooperation. 

Likewise, a recurrent point of concern among stakeholders seems to be certain administrative 
burdens that prevent the Mechanism from being speedier and fully effective, particularly for 
disaster response operations. 

As a central piece of the Interim Evaluation of the UCPM, the results of the stakeholder 
consultation have highlighted the positive elements, as well as some shortcomings of the 
Mechanism. Several of the latter may be rectified by possible amendments to the 
implementing decision, i.e. Commission Implementing Decision 2014/762/EU. As for the 
rest, the results of the consultation will certainly be integrated in the reflection process for 
future possible modifications of the UCPM. 

56