EUROPEAN COMMISSION Brussels, 12.6.2018 SWD(2018) 331 final COMMISSION STAFF WORKING DOCUMENT on the ex post evaluation of the " Prevention, Preparedness and Consequence Management of Terrorism and other Security related risks" 2007-2013 Programme (CIPS) Accompanying the document Report from the European Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions ex post evaluation for the period 2007 to 2013 of actions financed by the "Prevention and fight against crime" programme (ISEC) and the " Prevention, Preparedness and Consequence Management of Terrorism and other Security related risks" programme (CIPS)

{COM(2018) 455 final}

{SWD(2018) 332 final}



Contents 1 Introduction 2 Background to the initiative 2.1 Policy context 2.2 Baseline 2.3 The CIPS Programme as funding instrument 3 Evaluation questions 4 Method 5 Limitations 6 Status of the Implementation 7 Answers to the evaluation questions 7.1 Relevance 7.2 Effectiveness 7.3 Efficiency 7.4 Coherence 7.5 EU added value 8 CONCLUSIONS ANNEX 1 — Procedural information

ANNEX 2 — Stakeholders’ consultation ANNEX 3 — Methodology ANNEX 4 — List of evaluation questions ANNEX 5 — List of abbreviations and country codes ANNEX 6 — The role of the Joint Research Centre within CIPS ANNEX 7 — Sectoral analysis of CIPS projects ANNEX 8 — The different dimensions of CIPS ANNEX 9 — Intervention logic of the CIPS Programme 1Introduction The 2007-2013 ‘Prevention, Preparedness and Consequence Management of Terrorism and other Security-Related Risks’ programme (CIPS) was established by Decision No 2007/124/EC and is one of the two financial programmes falling under the framework programme on ‘Security and Safeguarding Liberties’. Its total budget amounted to EUR 126.8 million. The other component of this framework programme is the ‘Prevention and Fight against Crime,’ which is covered in a separate report. The CIPS programme concerns critical infrastructure and other security issues, including operational issues in areas such as crisis management, environment, public health, transport, research and technological development. Its general objective was to support Member States’ efforts to prevent, prepare for and to protect people and critical infrastructure from terrorist attacks and other security-related incidents. The Decision establishing the CIPS programme requires the Commission to submit to the European Parliament and the Council three evaluations reports: -an annual presentation on the implementation of the programme; -an interim evaluation report on the results obtained and the qualitative and quantitative aspects of the implementation of the programme by 31 March 2010; -A communication on the continuation of the programme by 31 December 2010;

-An ex post evaluation report by 31 March 2015. This staff working document reports on the ex post evaluation for the evaluation of the CIPS programme 2007-2013 annual work programmes implemented in the participating Member States, based on the findings of an ex post evaluation study carried out by an external consultant and building on the previous evaluations mentioned above. The ex post evaluation assessed the CIPS programme on the basis of the following criteria: ·relevance (whether its objectives met policy needs); ·effectiveness (whether the objectives had been achieved); ·efficiency (whether the costs were proportionate to the benefits gained); ·coherence (between the actions financed by the instrument); and ·added value (of intervening at EU level). The interim evaluation of the Internal Security Fund (‘the Fund’), its police component being the successor of the ISEC and CIPS programmes 2014-2020, takes into account the conclusions and findings of this evaluation, assessing what has already been implemented under the ISF-Police and what needs to be further addressed in the next Multiannual Financial Framework post-2020. The interim evaluation of the Fund is due in 2018. The Commission will submit an interim evaluation report on the Fund by 30 June 2018 to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions. The results of the interim evaluation together with the results of this ex post evaluation will contribute to shaping future EU migration and home affairs policies, and in particular in preparing the new funding instruments for the Multiannual Financial Framework post-2020. 2Background to the initiative 2.1Policy context This section introduces the EU policy context of the two main focus areas under the CIPS programme, namely: (i) protection of critical infrastructure and (ii) terrorism. On the basis of the conclusions adopted in 1999 by the Tampere European Council in November 2004, the European Council reaffirmed its promotion of freedom, security and justice through the prevention of and fight against crime. This led to the Hague programme (2004-2009) which focused on strengthening security in the EU’s fight against terrorism, recruitment, financing, protecting critical infrastructures and developing a sound consequence management framework. More recently, the 2010-2015 Stockholm programme, complemented by an action plan, led to the development of an EU Internal Security strategy, underlining the need to fight against terrorism. i) Protection of critical infrastructures Infrastructures are considered ‘critical’ when their disruption could have an impact on the functioning of a society (in terms of economy, security and people’s wellbeing). Critical infrastructure protection is linked to terrorism as well as to chemical, biological, radiological, nuclear, and explosives (CBRN-E) threat and cyber security. However, critical infrastructures need to be protected also from man-made and natural disasters. Furthermore, these are highly interdependent: the disruption of one of them might cause what is known as the ‘domino effect’, with potentially broader consequences and multiplied impacts. The EU has been active in critical infrastructure protection since 2004, although protecting critical infrastructures is a Member State competence (while being highly interdependent). On the critical infrastructure protection legal framework, the main pieces of legislation are:

i) the 2006 European Programme for Critical Infrastructure Protection (EPCIP); ii) the Council Directive 2008/114/EC of 8 December 2008 on the identification and designation of European critical infrastructures and the assessment of the need to improve their protection; and iii) Commission Staff Working Document on a new approach to the European Programme for Critical Infrastructure Protection, which is the result of a review of the Critical Infrastructure Protection approach presented in 2013 and heavily focuses on interdependencies of critical infrastructure protection across sectors. The framework established by the EPCIP provides procedures to identify and designate European critical infrastructures, and to carry out a threat assessment, introduced by Directive 2008/114/EC. The Critical Infrastructure Warning Information Network and the critical infrastructure protection expert groups at EU level are also included. The general objective of this framework is to enable the implementation of EPCIP, information exchanges and to protect critical infrastructures. Furthermore, the Council identified two main sectors of critical infrastructure: energy and transport. On the energy sector, European critical infrastructures include those linked to subsectors like electricity (infrastructures and facilities to generate and supply electricity), oil (oil production, refining, treatment, storage and transmission by pipelines) and gas (gas production, refining, treatment, storage and transmission by pipelines, LNG terminals). Within the meaning of transport sector, road, rail, air, waterways as well as ocean and short-sea shipping and ports are included. The EU has also been active in the protection of CBRN materials, in particular those already present in the EU soil, e.g. stored in national plants although not included in the sectors regulated by Directive 2008/114/EC. For this reason, the EU adopted a CBRN action plan in 2009, as a main outcome of the CBRN Task Force that was set up in 2008. The main goal of the action plan is to minimise the threat and damage of CBRN incidents, by promoting a risk-based approach to security. ii) Terrorism Since 2007 Europol registered a decrease in the total number of completed, foiled and failed attacks. That said as from 2013 a slight increase has been observed. Over the evaluation period, the terrorism threat has significantly evolved and included, although in a smaller scale, also left-wing, separatist, anarchist and right wing extremism. More specifically, the EU has witnessed a number of religiously-inspired terrorist attacks as from 2011. Indeed, the last years of the period under evaluation and those immediately subsequent saw a significant change in the terrorism phenomenon, in particular with the continuation of the civil war in Syria and the rise of the Islamic State in Iraq and Syria (ISIS). At the EU level, a Council Framework Decision on combating terrorism was adopted in 2002, followed three years later by the EU Counter-Terrorism strategy that was made up of four parts: (i) prevent, (ii) protect, (iii) pursue and (iv) respond. Furthermore, the EU internal security strategy in action (EU ISS) was adopted for the 2011-2014 period. The EU internal security strategy identified five strategic objectives, including the objective of prevention of terrorism and prevention of radicalisation and recruitment. To prevent terrorism, the EU committed to cut off terrorists’ access to funding and non-conventional materials (CBRN) and strengthen the protection of aviation, land and maritime transport systems. 2.2Baseline In order to fully identify the contributions the funding made to the different CIPS areas it would have been desirable to establish a baseline, reflected in a set of indicators, including data on national spending in the various areas. However, as explained in the limitations section, such baseline is lacking for the CIPS Programme and it has not been possible within the scope of this evaluation to develop a detailed baseline covering the situation before the start of the Programme. 2.3The CIPS Programme as funding instrument As mentioned above, one of the general objectives of the CIPS programme was to support Member States’ effort to prevent, prepare for, and to protect people and critical infrastructure against terrorist attacks and other security-related incidents (prevention and preparedness, general objective 1). Together with prevention and preparedness, the programme also sought to ensure consequence management (post-attack), namely protection in the areas such as the crisis management, environment, public health, transport, research and technological development and economic and social cohesion, in terrorism and other security-related risks in the area of freedom, security and justice (consequence management, general objective 2).

Specifically, the CIPS programme was particularly focused on critical infrastructures which is defined as: ‘those physical resources, services, information technology facilities, networks and infrastructure assets which, if disrupted or destroyed, would have a serious impact on the critical societal functions, including the supply chain, health, safety, security, economic or social wellbeing of people or of the functioning of the Community or its Member States’. To achieve its overarching objectives, the CIPS programme was designed to contribute to achieving seven specific objectives (the first five specific objectives are linked to the general objective 1 whereas the last two are linked to the general objective 2): Implementation through Direct Management EU financial support for CIPS was implemented under the direct management mode through projects supported by Commission grants, contracts for services concluded following calls for tenders published by the Commission or through administrative arrangements with the Joint Research Centre (JRC). Through direct management, the Commission retained full responsibility for implementing and carrying out all the programming and operational work. Political priorities and objectives for projects were defined in the annual work programmes adopted each year, and calls for proposals were launched following the approval of each annual work programme. Member States were involved in approving the annual work programmes and could provide advice on the selection of projects to be funded, but they were not involved in the management of the programme. Procurement contracts, as well as grants to standardisation bodies awarded without a call for proposal, i.e to the European Committee for Standardisation (CEN), could also have been implemented under CIPS. More precisely, the funding available under the CIPS programme was awarded as follows: 1.Action grants through open calls for proposals (76.5 % of total funding). 2.Grants to standardisation bodies awarded without a call for proposals — i.e. to the European Committee for Standardisation (CEN) (0.6 %). 3.Public procurement (8.5 %). 4.The CIPS programme specific actions to be carried out with the JRC (14.4 %). On the types of actors who could have applied for CIPS funding, according to the legal basis, these included but were not limited to public bodies (national, regional and local), private sector, universities and research institutes. The most common coordinating organisations were universities or research organisations followed by private companies, which played a key role in CIP. Financial support under the CIPS programme was provided for the following types of actions: 1.Operational cooperation and coordination (strengthening networking, mutual confidence and understanding, development of contingency plans, exchange and dissemination of information, experience and best practice). 2.Analytical, monitoring, evaluation and audit activities. 3.Development and transfer of technology and methodology, particularly regarding information sharing and inter-operability. 4.Training, exchange of staff and experts. 5.Awareness and dissemination activities. As part of the evaluation, categories of activities were defined to ensure a coherent classification according to the objectives of the action under CIPS. All action grants were subsequently allocated to one of the categories. More specifically, a grant could have been used to implement the activities in two or more categories. However, every grant was classified under the category which represented the dominant element. The categories of activities are the following (including the number of action grants implemented under each category): 1.Development of tools — risk assessment, including projects aimed at developing tools, instruments and methodologies for the assessment or risks and vulnerabilities of a critical infrastructure or of a security-related event (37). 2.Development of tools — Protection, including projects which aimed at developing tools, instruments and methodologies to better protect critical infrastructure or guarantee security at a particular sensitive event (33). 3.Cooperation and partnerships, including projects aimed at improving cooperation and partnerships between international stakeholders, Member States authorities, and between the public and private sectors. This category might also include best practice manuals targeting authorities in the Member States (27). 4.Response, including projects aimed at improving the response to terrorist attacks or security-related incidents thus minimising risks and consequences (14). 5.Training, exercises and simulations including projects which envisaged a ‘live’ component or a live simulation of a security-related incident (15). 3Evaluation questions The evaluation was planned and tendered out before the Better Regulation guidelines were adopted on 19 May 2015, but still covered all the five key evaluation criteria required by the guidelines i.e. effectiveness, efficiency, relevance, coherence and EU added value. To assess these criteria, the evaluation reviewed the 10 questions that are in Annex 4. 4Method The Commission’s evaluation relies on an external study carried out between August 2016 and August 2017 by a consultancy specialised in carrying out evaluations. The study’s methodology combined desk research, and both qualitative and quantitative analysis. It required a systemic synthesis of the evidence on the implementation of the CIPS programme. Information was drawn mainly from various primary and secondary sources, such as: • Primary sources: three online surveys (coordinating organisations, partner organisations and project participants); stakeholder consultations (37 face-to-face and telephone interviews were conducted with five stakeholders types; thematic case studies (2); analysis of public consultation responses (6 responses); expert workshops (2) and expert input. • Secondary sources: quantitative data on all CIPS projects provided by the Commission; in-depth analysis of a sample of projects (29 projects, representing 28 % of all 104 finalised projects); a literature review of EU policy baseline and context and a literature review of national baseline and context. A more detailed overview of the methodology used for the evaluation study is in Annex 3. Information was triangulated when possible to ensure validity and robustness. Findings from the study are presented in this report together with the financial data extracted from ABAC. 5Limitations The evaluation process encountered a number of difficulties that induced methodological limitations and hindered the extent to which findings and conclusions could have been formulated. A summary of these limitations is reported in this section and more details can be found in Annex 3. The lack of a baseline

The first major obstacle encountered in the evaluation was that it has not been possible to establish a baseline (i.e. the situation before the intervention of the programme) for measuring the level of prevention, preparedness and coordination of crisis management and security actions in the aftermath of a terrorist attack or of security incidents across the EU due to a lack of ex ante targets to measure results against. Furthermore, no methodologies were adopted for measuring in a comprehensive manner most of the activities covered by the actions funded through CIPS. Consequently, no robust conclusions could be drawn on the situation before the intervention started (i.e. the baseline) for the different thematic areas covered by the programme and the potential effect the funding has had on these compared to the situation after the programme had been implemented. In addition, differences in the definitions and content of critical infrastructures across the EU, including their interdependencies, made setting a common EU baseline even more problematic. Therefore, effectiveness has been mostly evaluated in terms of outputs or perceived impacts or results– which were more easily measured and compared across Member States — rather than real impacts or results. However, to partly overcome these limitations, where this was relevant, the findings and conclusions in the ex post evaluation have been compared to the findings and conclusions formulated during the interim evaluation of ISEC and CIPS and where it concerned similar evaluation questions. Given the lack of an existing baseline, the evaluation team had developed a set of indicators during the inception phase in order to map out where possible the national and EU context for the different thematic areas covered by CIPS and to identify trends in those data before, during and after the programming period. However, the team encountered significant data issues when trying to populate the indicators and it was agreed with the Steering Group that due to the lack of (comparative) data, this information would only be used for illustrative purposes and to focus on presenting the EU context for the different policy areas (see Section 2). Time lag between programme implementation and the ex post evaluation There was a time lag between the implementation of the Programme (2007-2013) and the ex post evaluation (August 2016-August 2017). This meant that the relevant beneficiaries and national and EU authorities and experts were often no longer working in the same position and were therefore difficult to reach. This is reflected in a low response rate to the online survey for coordinating and partner organisations as well as to the public consultations (see Annex 3). To try to mitigate these difficulties, the external consultancy took some measures, such as sending several email reminders to the people surveyed, searching for alternative contacts when possible, raising the individual partner organisation interviews to two rather than one, and extending the deadline for the online survey carried out as part of the case studies in order to boost the number of responses. Overall, all evaluation questions have been answered to some extent, thanks to the use of triangulation of data where possible. However, where the evidence base has been deemed to be too low to be reliable, this has been indicated. Gaps in financial data The calculations on the absorption rate included all amounts in ABAC until 31 March 2017 although a number of projects were not fully completed by that date. For this reason, absorption rates and trends, in particular for the last programming year, should be interpreted with caution since final amounts may be subject to variations on the basis of final payment for the non-completed projects. To try to mitigate this limitation, the costs have been estimated as accurately as possible, but have to be interpreted with caution. However, the absorption rate was not expected to change significantly. As no estimates were available for JRC data, it has not been possible to analyse the absorption rate for the JRC part of the procurement sector. Limitations on monitoring the results and impacts of the programme Issues were noted when monitoring the progress of the project and the final results hindered the assessment of effectiveness, efficiency and EU added value. The absence of a central repository to store the CIPS findings led to a variety of definitions of results and impacts of the programme depending on the thematic area. Therefore the possibility to compare the results across different thematic areas was restricted. However, to mitigate this limitation to the extent possible, data has been compiled and triangulated from multiple sources (interviews, surveys, sampled projects). 6Status of the Implementation As set out in the annual work programme 2007-2013, a total budget of EUR 126.8 million was allocated to the CIPS programme, of which EUR 74.6 million was committed. The minimum EU co-financing increased from EUR 50 000 (2007-2009) to EUR 100 000 in 2010 and remained at EUR 100 000 from 2011 onwards. The maximum rate of the EU co-financing increased consistently with the raise in minimum EU grant, from 70 % (2007-2009) to 80 % in 2010 and stayed at 90 % from 2011 onwards for the remainder of the programme implementation period. In total, 126 CIPS grants were implemented across the programming period, of which 121 were action grants to open calls for proposals and five were direct grants to European Committee for Standardisation. The number of grants implemented per year remained largely the same throughout 2007-2013 (between 17 and 20), with a rise registered in 2009 (27). More action grants were funded via open calls (76.5 %), followed by 14.4 % allocated to administrative arrangements with the JRC. Less than 8.5 % was allocated to the remaining public procurement and less than 1 % to grants to European Committee for Standardisation. The number of projects implemented varied across the programme years. For instance, at the start of the programme, in 2007, it was quite low (17 projects) but increased in 2009 to 27 projects. The demand-driven design of the annual work programme (see section 7.1) resulted in a significant geographical imbalance in terms of the number of projects funded per coordinating organisation in each Member State. While organisations from Italy and Spain coordinated 48 projects and 23 projects respectively, organisations from 15 Member States (UK, BE, NL, DE, AT, CZ, PO, HU, RO, BG, EL, CY, SE, EE, SI) coordinated less than eight projects, with Slovenia, Cyprus and Estonia at the bottom of the scale. These countries only coordinated one project each. Conversely, a slightly more balanced spread was achieved by the partner organisations, with five or more partner organisations located in 21 Member States. The majority of grants were cross-border (73 % or 88) as national grants represented only 27 % (or 33). Italy coordinated the highest number of national projects (15), followed by Spain (5). From the evaluation analysis it emerged that, overall, only 11 Member States coordinated at least one national project. However, this irregular geographical spread of coordinating organisations was counterbalanced by the location of the partner organisations. Indeed, a total of 86 national partner organisations were involved across these projects; while Italy had the highest number of partner organisations involved overall (47 organisations across 15 projects), Romania had the highest number of partner organisations involved in one grant (9 organisations). Projects were considered as cross-border if they included at least one partner organisation based in a different Member States than the one where the leading organisation was based. In terms of the geographical distribution of cross-border projects, coordinating organisations were based in 16 different Member States. The highest number of cross-border projects was hosted by Italy (33 projects or 36 %), which is consistent with the over-representation of coordinating organisations located in that Member State within the whole programme, followed by Spain (18 projects or 20 %). Only in Romania the number of national projects was higher than the one of cross-border projects. Some organisations in Slovenia and the UK coordinated an equal number of national and cross-border partnerships. Overview of main characteristics of projects under the programme

This subsection presents the descriptive analysis carried out on the sample projects, namely projects which were selected for in-depth evaluation analysis. The documents accompanying the sample projects have been analysed and included the grant agreements, the final narrative reports, the mid-term reports (as from 2010) and the final evaluation reports prepared by the Commission. General description of the sample The sample of 29 projects represents 28 % of all 104 finalised projects from the CIPS programme and was representative of all projects. Indeed, the list of finalised projects broadly matched the characteristics of the complete list. The characteristics of the sample project are further described below. However, given that the sample only contains 29 projects, this list should not be considered as exhaustive. 98 % of CIPS grants were funded through open calls while only 2 % were funded via grants to standardisation bodies. In the sample of projects compiled the proportion is 97 % to 3 %. The sample is representative of the projects awarded per annual work programme per year. It contains a lower proportion of projects from 2013 as at the time of the selection only three projects were eligible and thus included. Partnership characteristics of the sample projects An analysis of the partnership characteristics of the sample projects was carried out and included. For instance, the type of organisations and partnership funded, the size of the partnership and the diversity of partnership. The findings on possible outcomes and results in light of the above-mentioned partnership characteristics are described in the EU Added value section. In terms of types of coordinating organisations funded, the most common were universities or research organisations (10 projects) followed by private sector organisations. The latter were often partners with public organisations to deliver CIPS projects and vice versa. On the size of partnerships, it varied across the sample. Overall, the average number of organisations participating in a CIPS project within the sample was four. On the diversity of partnership, a considerable variation in the number of different types of organisation included within a project partnership was shown by the sample. Among projects implemented by more than one organisation the average number of types of organisation involved was three. Project activities and outputs As explained in Section 2.3, financial support under the CIPS programme was awarded to five types of activities. i) Operational cooperation and coordination; ii) Analysis, monitoring and evaluation activities; iii) Development: development and transfer of technology and methodology (including information sharing and inter-operability); iv) training: exchange of staff and experts; and v) Awareness: awareness and dissemination activities. Activity V ‘Awareness’ was the only mandatory component for projects (as from 2009) which explains the high number of projects implementing this activity (86 %). In terms of outputs, over a third (38 %) of all CIPS projects in the sample database involved an exchange of information or practice, and this was the most common output under activity 1. For activity 2, over half (52 %) of all CIPS projects in the sample conducted a study or evaluation. Other outputs were created by four projects within the sample and included: 1.Creation of a database, classification of information;

2.Identification of relevant geographical information systems; 3.Analysis of technical standards and methods for resistance assessment and analysis of legislative and related documents; and 4.Policy analysis for creation of a handbook. For activity 3, over a third (38 %) of sample projects undertook the development of new equipment/software while one third (31 %) were related to the development of new methodologies. Only 3 % of sample projects concerned a transfer of equipment or software. Three projects in the sample produced outputs that did not fit into the categories provided, namely: 1.Development of a new model for security in critical infrastructures; 2.Multi-level framework for the alignment of regional CIP/R strategies with national and EU strategies; and 3.Development of security standards and elaboration of methodologies for assessment of objects’ security. Only seven projects (representing 24 % of the sample projects) undertook training activities (activity 4). As awareness activities (activity 5) became compulsory after 2009, the vast majority of sample projects reported to have undertaken this activity. The most common output of activity 5 was the organisation of dissemination events (83 % of projects) followed by dissemination materials (69 %). Analysis of fund absorption As stated in the introduction and the implementation section, a total budget of EUR 126.8 million was allocated to the CIPS programme of which EUR 74.6 million was committed and EUR 51.7 was spent. Indeed, not all available funding was awarded due to the fact that some project applications did not meet the eligibility/ quality criteria for being awarded funding. The absorption rates are calculated using the financial data included in the annual work programme, the modifying decisions applied on these amounts and the DG Budget ABAC system. Absorption rate of grants During the evaluation period, a total of EUR 97.8 million was allocated to grants, of which EUR 51.5 million was committed and EUR 43.0 million was spent by 31 March 2017. This translates into an absorption rate of 83 %. The table below presents an overview of the overall spending for the programme. Table 3: Overview of allocated, committed and spent funding, in millions of EUR (rounded) by 31 March 2017 Year Allocated Committed Spent 2007 10.4 6.6 4.5 2008 12.1 8.4 6.9 2009 14.6 9.3 7.8 2010 16.1 5.9 4.6 2011 17.0 6.2 5.9 2012 18.0 6.5 5.7 2013 9.5 8.6 7.6 Total 97.8 51.5 43.0 Source: ICF analysis of data from the annual work programmes (allocated), and DG Budget ABAC data (committed and spent). Over the programming period, CIPS had a higher amount of committed funding (83 %) compared to ISEC (74 %) with regard to overall grants. The reasons of the higher absorption rate under CIPS included the strongest knowledge of beneficiaries in order to properly manage the calls, and the changes adopted by the Commission to simplify the overall implementing process. Furthermore, the enhanced capacity of CIPS stakeholders in planning project’s budget could also be considered as one of the reasons that contributed to this effect. More specifically, as reported in the Figure 1 below, an increase in the fund’s absorption was registered for CIPS as from 2007 (68 %) to 2013 (88 %), with a pick in 2011 (96 %). The average absorption rate after 2011 was 90 %, and this can be explained by the fact that stakeholders became more familiar with the programme and as a consequence they were able to provide more realistic budget projections.

Figure 1 Committed and spent budget for grants, and absorption rate, by year by 31 March 2017 Source: ICF analysis of data on committed and spent budget. Absorption rate in the procurement sector A total of 106 procurement items were funded by the programme. The number of public procurement items in this sector experienced a significant increase during the evaluation period, from only 6 in 2007 to over 20 in 2012 and 2013. On CIPS procurement amounts, EUR 10.80 million was set aside in line with the annual work programme, of which EUR 10.3 million was committed and EUR 8.7 million was spent by 31 March 2017. Figure 2 below shows the absorption rate per year, an average absorption rate of 85 % was registered. Figure 2 Initial procurement amount committed, paid and absorption rate across the period 2007-2013 by 31 March 2017 Source: ICF analysis of DG Budget ABAC data. Absorption rate of JRC actions/ administrative arrangements Seven administrative arrangements or actions carried out with JRC were identified by the evaluators based on the ABAC data received from the Commission. Originally, a budget of EUR 18.2 million was allocated to actions and administrative arrangements with the JRC of which EUR 12.8 million was committed across the programming period. Final data on project expenditure was not available at the time of writing the report. 7Answers to the evaluation questions 7.1 Relevance

EQB1.a: To what extent did the objectives of the CIPS Programme correspond to the needs related to the prevention, preparedness and consequence management of terrorism and other security-related risks? The objectives of the CIPS programme were designed to be in line with the EU policy framework in the area of critical infrastructure protection. The specific objectives were linked to the general objectives, which are preparedness and protection (Art.4 (2)) and consequence management (Art.4 (3)). Further details on the architecture of the fund are reported in Section 2.3. The European programme for critical infrastructures protection (EPCIP), whose implementation was supported by the CIPS programme, recognises that many critical infrastructures have a cross-border dimension, which would affect a number of EU countries. The Commission considered that the objectives of the CIPS Programme corresponded to the needs related to prevention, preparedness and consequence management of terrorism and other security related risks. Furthermore, the Programme responded to a real need for cross-border cooperation and coordination in the areas of prevention, preparedness and consequence management of terrorism and other security-related risks given the increase of terrorist attacks in recent years. It should be pointed out that from the interviews with the stakeholders consulted it emerged that the CIPS programme was of continuous relevance to the prevention, preparedness and consequence management of terrorism and other security-related risks throughout the evaluation period. The continuous relevance of the Programme was also ensured by the process of setting the CIPS priorities and sub-priorities, which were identified by Member States and competent Directorate-Generals of the Commission on a yearly basis and included in the annual work programmes. The majority of Member State authorities representatives interviewed did not mention any issues related to the process of priority-setting. Most CIPS coordinating organisations and partner organisations responding to the online survey considered the priorities set by the Commission in the calls for proposals/annual work programmes as relevant to the specific needs of the target group (9 out of 15), to the specific strategic needs at EU level (9 out of 15) and to the specific strategic need at national level (10 out of 15) (conclusions). Furthermore, around 50 % of coordinating organisation responding to the survey reported that they designed the project on the basis of a needs assessment whereas 20 % highlighted that such assessment was ‘sometimes’ carried out and another 20 % replied negatively to this question. Needs assessments were often carried out through meetings and discussions with relevant stakeholders, including experts, critical infrastructure operators and relevant Ministries and government agencies. EQB1.b: To what extent did the design of the CIPS programme correspond to the needs related to the prevention, preparedness and consequence management of terrorism and other security-related risks? The design of the programme relates to the main features of the CIPS programme, namely its scope, funding mechanisms, types of actions and stakeholders targeted under the programme. The evaluation study assessed to what extent this design corresponded to the needs related to the prevention, preparedness and consequence management of terrorism and other security related risks during the evaluation period. The rationale behind the CIPS programme was to provide funds where demand was highest in line with the priorities in the annual work programmes. For this reason, the CIPs programme was broadly demand-driven (based on open calls for proposals and allowed actors to apply for funding based on a project proposal) rather than policy-driven. Indeed, if no project promoters applied for funding as part of a specific priority in a given year, that particular priority would not be implemented. However, the Commission could add remaining funding from targeted calls to a subsequent general call. The logic behind the annual work programme was to provide funds where demand was highest in line with the priorities outlined in the annual work programme. The demand-driven design of the programme contributed to a significant geographical imbalance in its implementation, especially for the location of the coordinating organisations (see Section 7.1). For this reason, the Commission took measures to boost the geographical spread, particularly through the organisations of information days in the Member States. However, the programme’s implementation depended heavily on the pro-activeness of relevant government representatives and potential applicants from the Member States. On the thematic coverage of the programme, as provided in Section 2.3, this was defined in broad terms and covered: 1.Risks: terrorism and other security-related risks (this could include other criminal activities, natural disasters etc.); 2.Protection of people and critical infrastructure; 3.Pre-attack (preparedness and prevention) and post-attack (consequence management); 4.Unlimited range of sectors: energy, cyber, food, health, space, etc.; 5.Activities: developing tools, methodologies and guidelines, training, exchange of experience, best practice and know-how, etc.

The evidence gathered confirmed that this extensive coverage was perceived as positive by consulted stakeholders as it allowed for flexibility throughout the funding period. Meanwhile, a finding emerging from the various consultation rounds and expert opinions was that the broad coverage of the programme might have contributed to the lack of clarity of its scope, which may have rendered the drafting of applications challenging for a number of organisations. For this reason, the difference between European and national critical infrastructure could be made clearer, as CIPS has a strong link to the EPCIP. The funding mechanism was considered appropriate by the majority of stakeholders consulted. Moreover, the close involvement of the JRC in the implementation of the programme was considered to be positive (further details on the role of the JRC are reported below). However, a minority of stakeholders considered that the programme should have been more flexible, as some of the eligibility criteria for obtaining CIPS funding discouraged some organisations from applying.

7.2 Effectiveness To report on effectiveness of the Fund, the analysis presented below is structured around the two general objectives and the seven specific objectives set out in the legal basis. The assessment is based on nine evaluation questions (EQsB2 to B6). General Objective 1- Support Member States’ effort to prevent, prepare for and to protect people and critical infrastructures General Objective 2 — Crisis Management Specific Objective 1 — Stimulating, promoting, and supporting risk assessments on critical infrastructures: Specific objective 2: Stimulating, promoting, and supporting the development of methodologies for the protection of critical infrastructures (risk assessment methodologies) Specific objective 3: Promoting and supporting shared operational measures to improve security in cross-border supply chains Specific objectives 4: Promoting and supporting the development of security standards, and exchange of know-how and experience on protection of critical infrastructures: Specific Objectives 5: Promoting and supporting Union-wide coordination and cooperation on protection of critical infrastructures; Specific Objective 6: Promoting and supporting exchange of know-how and experience related to the consequence management, in order to establish best practices:

Specific Objective 7: Promoting joint exercises and practical scenarios in consequence management, including security and safety components, in order to enhance coordination and cooperation between relevant actors at the European level: EQ B.2: To what extent did the CIPS programme contribute to supporting Member State efforts to prevent, prepare for and protect people and critical infrastructure against terrorist attacks and other security-related incidents? EQ B3: To what extent did the CIPS programme contribute to protection in areas such as crisis management, environment, public health, transport, research and technological development in terrorism and other security-related risks? Overall, the evaluation findings suggest that the programme broadly achieved its general and specific objectives. This is supported by most coordinating organisations responded that the objectives were fully achieved (13 out of 14 respondents). Nevertheless, these findings should be seen in the light of the methodological limitations presented in Section 5, in particular the fact that the progress was assessed at output level mainly, on a limited sample of projects and based on qualitative evidence. Also, when looking to effectiveness, it should be taken into account that the programme’s first general objective was only to support Member States efforts in protecting people and critical infrastructure without substituting to Member States in this area. In addition, the relatively small allocated budget of EUR 126.8 million — in comparison to other EU programmes as well as national budgets in CIP area — should be also taken into account when assessing the results and impacts of the programme. Finally, the broad scope of the programme, resulting in a high number of projects of relatively small value, should also be taken into consideration when analysing its overall effects. Regarding the results and outcomes of the sample of projects analysed (29), from the evaluation study, it emerged that the majority of these projects (62 %) have led to an improved understanding of methods, tools and practices, 59 % to an increased level of knowledge and skills of practitioners and 52 % to enhanced networking and forms of sharing information and best practice. In terms of impacts, the majority of projects analysed (16) contributed to the increase of prevention capacity to protect people and critical infrastructure against terrorism and other security-related incidents. A high level of projects (15) contributed to increase the level of preparedness whereas only 7 projects contributed to increase risk assessment capacity on critical infrastructures. Conversely, no projects contributed to increase response capacity. Other impacts registered were the exchange of good practice, between Member States on technology and the improvement of participants’ skills and knowledge. On the overall implementation of activities, the majority of coordinating organisations involved in CIPS projects who participated in the survey reported that all project activities (12 out of 14 respondents) or most of them (2 respondents) were delivered. From the analysis of the sample of projects for in-depth analysis it also emerged that 90 % of projects achieved all of their objectives whereas only 10 % of projects partially achieved their objectives. The final assessment carried out by the Commission on the sample projects reported 7 % of projects as being excellent, 59 % as having had positive results and impacts and 21 % were considered acceptable. Furthermore, 26 projects from the sample (out of 29) reached their initial expectations. On the other hand, some of them also implemented activities not originally envisaged such as additional external events and/or higher number of participants to activities. However, the surveyed also reported on obstacles encountered during the project’s implementation and these included, for instance, (i) the low level of participation from project participants (6 projects) and the lack of cooperation from partners (1 project), (ii) technical difficulties, and (iii) language barriers. Moreover, based on the final evaluation carried out by the Commission it emerged that some outputs could have been better designed to gain a better outreach. Indeed, considering that CIP is technically complex, the results were perceived in some cases as being too technical to promote wide use among policymakers. In certain other instances, the Commission highlighted that some projects had a too strong national focus which prevented the broader use of their results. In terms of actions funded under CIPS, a representative sample of 29 projects have been reviewed and showed that most of the projects were funded in the sub-area of protection of critical infrastructure and people (10 projects or 35 % of the sample), followed by risk assessment (eight projects or 28 % of the sample). Most of the examined projects (20 projects or 69 %) had the development of methodologies for the protection of critical infrastructures as an objective. More precisely, 38 % of the sampled projects aimed to contribute to the establishment of frameworks for the exchange of know-how and experience on the protection of critical infrastructures; 21 % aimed at establishing best practices and coordinating response measures; 17 % included coordination and cooperation on protection of critical infrastructures; 14 % focused on joint exercises and practical scenarios and 7 % on operational measures to improve security in cross-border supply chains. Evidence collected through the stakeholder consultation showed that results based on the following two components were perceived as particularly valuable and contributed to the increase of the overall effectiveness of the programme:

a) potential for transferability to other sectors and other Member States; and b) readiness to be used by critical infrastructure owners and operators in contrast to prototypes and models that are not operable or market-ready. In consequence, for the future, similar programmes should be carried out in order to motivate coordinating organisations to work towards the development of demonstrators or prototypes (to be developed as market-ready products) which are based on the results of previous projects. Furthermore, given the relative newness of the Programme and the lack of resources of the Commission at the time to closely follow all CIPS projects from start to end, the evaluation showed that a dedicated technical assistance budget could have been useful. Such a budget could have better supported the development of a state of the art monitoring system to collect and analyse data on financial progress, outputs and results of projects, as well as provided technical expertise throughout the lifecycle of projects and increased the dissemination of results. Therefore, the lack of a dedicated technical assistance budget has negatively affected the effectiveness of the programme. Sectoral analysis of CIPS projects Projects were categorised per specific sectors depending on their aims and purposes. The sectoral dimension has enabled the coverage of sectors with critical importance such as energy, transport and telecommunication. However, some sectors were covered to a much greater extent than others, and the more popular were the information, communication and technology (ICT, 23 %), transports (18 %) and energy (12 %). Most of the projects were ‘multi-sector’ which means that they covered more than one sector. Nearly a third of all projects were granted in 2009, half of them were led by a variety of bodies in Italy (including universities and research institutes, government ministries and regional/city councils). The remaining projects were funded to organisations in Greece (2), Spain (5), France (1), Hungary (1), the Netherlands (1), Romania (2) and UK (2). More details on the sectoral analysis of CIPS projects are reported in Annex 7. Dissemination Dissemination activities on the results were carried out by the coordinating organisations and, depending on the nature of results achieved, took place via dedicated websites, press announcements, printed material and conferences. DG HOME, with the technical support of the JRC, developed the Critical Infrastructure Warning Information Network (CIWIN), which aimed to provide a platform for exchanging CIP-related information, studies and/or good practices across all EU Member States and included also the results of CIPS-funded projects. The CIWIN became operational only in January 2013 and the stakeholders involved appreciated the Commission’s efforts in providing information on the outcome of CIP national contact points meetings and the CIWIN platform. It should be highlighted that despite the absence of a central repository for CIPS, the active involvement of a number of Directorate-Generals in the implementation of the CIPS programme proved that the monitoring and dissemination activities were shared between the Directorate-Generals involved. The results produced by CIPS-funded projects implemented by JRC are stored in their publication repository: PUBSY. EQ B4.a: To what extent did the CIPS programme contribute to the protection of people and critical infrastructure by promoting and supporting risk assessments on critical infrastructure, in order to upgrade security? EQ B4.b: To what extent did the CIPS programme contribute to the protection of people and critical infrastructure by promoting and supporting the development of methodologies for the protection of critical infrastructure, in particular risk assessment methodologies? Overall, the Commission evaluated the results of the CIPS projects on protection methodologies as good on average, and the transferability of the outputs was particularly appreciated, stressing the importance of being able to adapt the outputs to other sectors and to implement them on a large scale.

More than 60 % of all CIPS-funded projects focused on developing frameworks, methodologies, tools and approaches for risk assessment at a sectoral level. Some of the tools and methodologies that resulted from CIPS projects were the following: i) A risk assessment toolbox for assessing the risks of terrorist attacks against specific critical infrastructures; ii) Risk assessment conducted on online social media through the analysis of open source intelligence (SNAPSHOT project); and iii) Methodologies dedicated exclusively to risk assessment of interdependencies and contingency plans of critical infrastructures. As an example of projects focusing on risk assessments at sectoral level, the project Threat-Vulnerability Path Identification for Critical Infrastructures Compilation (THREVI2) was led by a private company (NIER INGEGNERIA S.p.A., Italy) and is a compilation of a comprehensive all-hazard catalogue for critical infrastructures. A literature review of data and historical events to develop a comprehensive catalogue of hazards and threats was produced and disseminated in conferences and workshops. Furthermore, when analysing the risk assessment, a particular attention was given to the ‘cascading effect’ of common infrastructures. Indeed, to ensure the effectiveness of the risk assessment, the cascading effect needs to be taken into account in order to prevent and reduce consequences of security incidents. Critical infrastructures are interdependent within different sectors, not only for the proximity of their geographical location but also because of resource exchange or cyber dependencies. Such interdependencies, if not well assessed and considered, may lead to catastrophic cascading failures and domino effects. For this reason, approximately 20 projects funded by CIPS dealt with interdependencies of critical infrastructures, predominantly with a focus on risk assessment. The focus of a number of CIPS projects on interdependencies and ‘cascading effects’ in case of disruption and destruction of critical infrastructures during a terroristic attack or natural disaster was considered to have contributed to the programme’s objective related to critical infrastructure protection. The JRC also contributed to the promotion and support of risk assessment capacity on critical infrastructures with different specific projects. For instance, one project on studies and support related to threat and vulnerability assessment (2008) and another one on Risk Assessment methodologies and tools for CIP entitled ‘Increasing resilience against all-hazards’ (2013). EQ B4.c: To what extent did the CIPS programme contribute to the protection of people and critical infrastructures by supporting shared operational measures to improve security in cross-border supply chains, in compliance with EU regulations on competition within the internal market? It is not possible to draw clear conclusions on CIPS’s contribution in this area due to the small number of projects implemented in this sector. Only 6 out of 126 projects were dedicated to the security of cross-border supply chains, and focused on drinking water, food, electricity and ports. Only one project did not focus on a specific type of supply chain but was dedicated to the improvement of the management of cross-border disruptions. On the projects’ objectives, these varied and included: (i) identification and assessment of cascading effects in the global ports’ supply chain, and (ii) strengthening the resilience capacities required to manage cross-border critical infrastructure disruptions. Some experts considered the wording ‘supply chain’ as misleading given that what needed to be protected was the cross-border infrastructure distributing the goods and not the ‘supply chain’, which refers to the whole distribution process. EQ B4.d: To what extent did the CIPS programme contribute to the protection of people and critical infrastructures by promoting and supporting the development of security standards, and exchange of know-how and experience on protection of people and critical infrastructures? EQ B5: To what extent did the CIPS programme contribute to promoting and supporting exchange of know-how and experience related to the consequence management, in order to establish best practices so as to coordinate the response measures and to achieve cooperation between various actors in crisis management and security actions? On average, the results of CIPS projects on exchange of know-how and experience were considered as good by the Commission, and the transferability of the outputs as particularly valuable. Overall, 17 projects focused on the exchange of know-how and experiences related to both protection and crisis management. For instance, the ‘MIRACLE’ project aimed to support the coherent development of critical infrastructures protection and/or resilience (CIP/R) strategies at regional level. Its final objective was to improve existing capacities of EU Member States to prevent, prepare and protect people against terrorist attacks and security-related risk by setting up an international network of professionals and institutions for the exchange of know-how in the resilience of critical infrastructures. Following selection and analysis of best practice, a manual dedicated to sharing good practice within Member States was produced. In term of results, it was considered as user-friendly and well adapted to the target audience, represented by policymakers. EQ B4.e: To what extent did the CIPS programme contribute to promoting and supporting Union-wide coordination and cooperation on protection of critical infrastructure? The results of CIPS projects on EU-wide cooperation and coordination on CIP were evaluated by the Commission as not very effective, with a few peaks of excellence. Several projects contributed to promoting and supporting EU-wide coordination and cooperation among Member States.

In prevention, EU-wide cooperation and coordination is linked to the exchange of information on threats among public authorities and stakeholders involved in the protection of critical infrastructures. CIPS funded for example the European Cooperation Network on Critical Infrastructure Protection (EUCONCIP) which brings together experts on CIP from across Europe and set the ground for the creation of a European partnership between critical infrastructures stakeholders, with the aim to strengthen public-private international cooperation, planning, networking and cross-border exchange of information. As another example of project, the ‘security liaison officer’ had the objective to support the creation of the role of liaison officer in charge of security for all European and national critical infrastructures. Although every critical infrastructure has a security manager, the role of security liaison officers is to enable the protection of the European critical infrastructure through better information sharing and cooperation as security managers could also have other roles and responsibilities. EQ B6: To what extent did the CIPS programme contribute to promoting joint exercises and practical scenarios in consequence management, including security and safety components, in order to enhance coordination and cooperation between relevant actors at the European level? Overall, a significant number of projects (19 projects) contributed to promoting joint exercises and practical scenarios between different actors at European level. An example of interesting CIPS project on consequence management is the ‘Disaster 2.0’ project. It aimed at establishing how successful the Web 2.0 platforms were in mitigating damage to critical infrastructure, building resilience in the public, and what were the challenges encountered by their use. The project also explored the potential of next generation Semantic Technologies (Web 3.0) in disaster response. Furthermore, it collected substantial data on behaviour, uptake and barriers to the use of social media in emergency management and by emergency management agencies. As envisaged, it also developed a vision for the application of semantic technologies in this field by emergency management agencies, for which it created a prototype software. 7.3 Efficiency EQ B7.c: What kind of initiatives or approaches were adopted to simplify access to and implementation of the actions funded by the programme (e.g. changed to shared management in ISF-P)? As mentioned in Section 5, due to the lack of an initial baseline and to the difficulties encountered in comparing project costs, the assessment made of efficiency below relies largely on the qualitative data collected through the stakeholder consultations and interviews. Furthermore, the low number of respondents to the surveys (19 %) should be also taken into account. In order to improve the inefficiencies related to accessibility, the interim evaluation of the programme’s implementation completed in 2010 identified certain areas for improvement. Consequently the Commission adopted the following measures: i) Changes related to the application and award process Overall, the changes introduced to the application and award process contributed to the decrease of ineligible project applications. However, some other factors such as the improved know-how of the application system and rules may have also contributed to this decrease. Application and awarding/grant agreement procedure — guidance for applicants The Commission organised some information days in 2009-2011 in Brussels which aimed to explain the application procedures to potential applicants. Some regional information days were also held in seven Member States (AT, IT, RO, HU, SE, SI and the UK) in 2010-2011. A significant increase in the total number of applications was registered from 2011. The Commission also improved the guidance documents for the calls for proposals. However, there was still room for improvements as the volume of accompanying documents could have been seen as burdensome and the information could have been compiled in a single package. Overall, 10 out of 14 respondents to the survey for CIPS coordinating organisations were satisfied with the management of the application process applied by the Commission. Furthermore, 9 out of 13 respondents also reported that they did not encounter any issue related to the signing of grant agreements (conclusions). However, a limited number of respondents (4) highlighted that they experienced certain problems, such as communication issues due to the numerous changes affecting the Commission’s staff and a lack of flexibility in the administrative process. ii) Revision to the process of budget verification in 2010 Between 2007 and 2010, the Commission carried out a detailed analysis of the eligibility of the budgets outlined in the project applications. Consequently, a more efficient approach on the negotiations for the proposed budget with the beneficiaries was adopted by moving the negotiations before the award decision (whereas originally these took place between the award decision and the signing of the agreement). This change led to a sudden decrease of the number of days between the award decision and the grant agreement signature. Furthermore, from 2010 the concept of ‘value for money’ was considered as an award criterion hence cost-effectiveness of projects increased. The ‘value for money’ principle was overall considered as a key driver across all activities by the majority of stakeholders therefore it is recommended to fund similar activities in future programmes. iii) Strengthening the rules of cross-border partnerships The Commission restricted the rules on the eligibility criteria on cross-border projects as, in the first years of the programming period, the Commission was not in a position to easily check the degree of cross-border partnerships. For this reason, for the 2007 call for proposals, applicants had to ensure that cross-border projects involved at least one partner organisation from another Member State or from an acceding/candidate country. However, stricter rules followed from 2008, such as the obligation to sign a partnership declaration.

iv) Maximum length of projects In 2008, the maximum length of a project was decreased from three years to two years. Several reasons led to this decision, for instance, the fact that reducing the project’s length contributes to a better match between the implementation period and the period when the policy formulation was laid down in the annual work plan. Indeed, the two years duration was considered as the best time period reflecting overall the timeline set to achieve national policy priorities. v) Minimum and maximum EU grant From 2010, the minimum EU co-funding amount to be requested increased from EUR 50 000 (2007-2009) to EUR 100 000. The amount was increased mainly because higher amounts were needed for high technological investments. Agreements on budget thresholds were discussed between Member States and the Commission. However, increasing the minimum EU grant was perceived as most problematic in EU-12 Member States. The maximum rate of the EU grant share increased consistently with the increase in minimum EU grant, from 70 % (period 2007-2009) to 80 % in 2010 and remained at 90 % from 2011 onwards for the remainder of the programme implementation period. Pre-financing rules To improve cost-effectiveness, the CIPS programme had generous pre-financing rules, i.e. coordinating organisations received 60 % (until 2010) to 80 % (in 2011) at pre-financing stage. The more generous advance proved to be particularly helpful for the smaller organisations to properly set up the partnership and undertake different actions. Overall, coordinating organisations interviewed considered this as advantageous and no specific issues were mentioned. vi) Changes to reporting requirements and monitoring of results Financial reporting Project partners were requested to provide information on the financial aspects of project implementation. The 2010 calls onwards, financial reporting became more detailed on cost claim requirements. Therefore, further guidance was provided in separate documents. A few grant beneficiaries stressed the fact that reporting project expenses in detail was too burdensome. Monitoring of results and technical reporting As stated above under the effectiveness section, one of the main weaknesses of the management of the programme by the Commission was the absence of a centralised coordinated approach for monitoring project results. Furthermore, as other Directorate-Generals were involved in the programme’s implementation, coordinating the monitoring of results across them was difficult to ensure. For this reason, interim reporting was introduced to strengthen monitoring results through data collection and to allow coordinating organisations to take stock of progress made to date and/or possible needs for adjustments. Although projects were required to consider EU added value as part of project implementation, there were no obligations to report on this. The reason behind this choice was to reduce the burdens on organisations, based on the fact that more detailed reporting alone does not ensure effective monitoring of results, which should consequently be accompanied by the formulation of indicators and quantification of results. Nevertheless, this still represents a shortcoming and it is recommended to include EU added value and a self-assessment of key results and overall impacts on the monitoring report. Administrative burden Despite the changes introduced during the CIPS programming period, some stakeholders raised concerns on the level of administrative burden. Five respondents to the survey for coordinating organisations considered the level of administrative burden as ‘significant’ whereas three of them as ‘very significant’. Similar concerns were raised by partner organisations. Private companies involved in the CIPS implementation and who have implemented in the past projects under Framework Programme 7 reported a higher level of administrative burden under CIPS. From the interviews it emerged that the main obstacles were a lack of flexibility and excessive bureaucracy (e.g. high level of detail required conversely to some other Programmes such as Framework Programme 7 and H2020) in relation to administrative processes and the need for simplification of reporting in terms of length and detail (e.g. financial information requirements which is further explained below under ‘monitoring and reporting’).

A shift towards shared management as a management mode for the successor fund Unlike CIPS, which was implemented 100 % under direct management, the successor fund, ISF-Police, is being implemented on the basis of a combination, with around 60 % of the available funding being allocated to national programmes under shared management and 40 % allocated to EU actions under direct management. This change is expected to have an impact on the efficiency and effectiveness of the Commission’s management of the ISF programme. However, it was not in the scope of this ex post evaluation to determine to what extent the shift to shared management has provided efficiency and effectiveness gains in the context of the current ISF-Police programme. EVQ B7.a & B7.b: To what extent were the results of the CIPS programme achieved at a reasonable cost in terms of financial and human resources deployed? On the cost-efficiency of the financial resources deployed a budget of EUR 126.8 million was allocated to the CIPS programme as laid down in the annual work programme 2007-2013, of which EUR 74.6 million was committed and EUR 51.7 million was spent. There was an increase in the fund’s absorption as from 2007 (68 %) to 2013 (88 %), with a peak in 2011 (96 %). The average absorption rate after 2011 was 90 %, and this can be explained by the fact that stakeholders became more familiar with the programme and in consequence they were able to provide more realistic budget projections. The EU funding provided was perceived by the vast majority of stakeholders consulted as having been sufficient and as having been highly cost-efficient for all activities implemented under CIPS. Overall, the evaluation analysis showed that 88 % of the sample projects estimated budget was spent. However, some differences between the estimated budget included in the grant agreement and what was effectively spent (as declared in the final cost statement) were registered for certain costs categories. This led to a lower absorption of funding than what was expected. It should be highlighted that most underspend related to costs for travel and for conferences and seminars. Variations in expenses from the initially forecasted budget to the final declared project expenditure were due to different reasons. For instance, the fact that costs finally resulted higher or lower than expected or the fact that outputs were not delivered. Sufficiency of EU funding On average, 88 % of projects costs were funded by the EU contribution (for the 28 sample projects analysed). For these projects, on average 74 % of financing was made up by the EU contribution. The projects less dependent on the EU contribution registered 64 % of income made up by the EU contribution, and only 10 out of 28 projects had an EU contribution of 88 % or higher. Overall, coordinating organisations reported that EU funding was sufficient to implement the planned activities. Only a limited number of stakeholders highlighted the need for additional funding and the fact that certain costs were underestimated, i.e. staff’s costs. Value for money Overall, for four out of five activities formulated in the survey, at least half of all respondents to the coordinating organisations and partner organisations considered these to be high to very high value for money. On awareness and dissemination activities, just under half of the coordinating organisations responding to the survey considered these to be high to very high value for money (with the total number of respondents ranging from 13 to 14 per activity). Efficiency of human resources deployed On the cost-efficiency of the human resources deployed, a few key performance indicators were introduced by the evaluation team to measure the efficiency of programme management by the Commission. On human resources, they were considered sufficient for a fairly low number of projects across the whole programming period. However, certain coordinating organisations also highlighted few weaknesses. For instance, the frequent staff changes and internal re-organisations (at their level and at the level of the Commission) led to more time-consuming communications, with valuable knowledge getting lost in the process. More specifically, five coordinating organisations also reported the absence of follow-up/feedbacks from the financial operational officers. Several key performance indicators related to the Commission’s management of the implementation of the programme were stipulated in the Financial Regulation and are reported below. 1.With regards to the time to award (time from the call for proposals’ application deadline until the award decision), the evaluation showed an overall negative trend. However, it should be noted that it remained below the maximum of 6 months anyway. 2.On the time to contract (time from the award decision until the grant agreement signature) the average number of days was 122. A positive trend was registered during the evaluation period, with the average time reduced by more than half (from 181 days in 2007 to 53 days in 2013). 3.On the time to pay (time to pay the pre-financing and the final invoice), the evaluation showed that, overall, 73 % of payment requests were paid on time, with 27 % being late. Significant differences were identified between the pre-financing stage (87 % of requests were paid on time) and the final payment stage (only 59 % of invoices were paid on time). Analysis of project applications

According to the award decisions, a total of 352 applications were received over the period 2007-2013 and 129 grants were awarded (37 %). The success rate increased from 53 % in 2007 to 69 % in 2008. However, from 2009, the success rate started to decrease (24 % in 2013). The two last years of the programme the number of applicants almost doubled, nevertheless, the number of projects awarded remained constant over the period. Between 2007 and 2013, 224 applications were not awarded a grant. 91 were ineligible and 133 were unsuccessful. However, the ineligibility of projects started decreasing as from 2010. This was most probably due to the fact that eligibility criteria were better followed and understood in the later years of the programme and that the changes introduced by the Commission to the guidance and the application procedure helped applicants to produce conform proposals. 7.4 Coherence Coherence with other EU programmes To avoid overlaps between CIPS programme and other similar EU programmes, some internal mechanisms such as consultations and inter-service consultations were established. Moreover, a representative of DG Enterprise for the security research funding in Framework Programme 7 was involved in the evaluation of proposals for the first two calls for CIPS proposals 2008/2009 to the extent needed to avoid duplication of what was done under Security Research in Framework Programme 7. The Framework Programme 7 and the Civil Protection Financial instrument were selected for analysis and comparison with the CIPS programme. The coherence between different funding instruments can be multi-dimensional in terms of: (i) objectives and themes, (ii) programme management and eligible actions, and (iii) target groups. Coherence with Framework Programme 7 Overall, the evaluation study concluded that the activities implemented under Framework Programme 7 in security were coherent with the CIPS programme. Indeed, this latter focused on prevention and protection measures against terrorist attacks as well as crisis management whereas projects under Framework Programme 7 were more specifically oriented to conduct research on development of new technology solutions for civil protection, including risks arising from terrorist attacks. The JRC also ensured synergies between the two programmes. More specifically, the actions carried out with the JRC under the CIPS programme and the activities of the JRC on CIP carried out under FP7/H2020 were merged into two JRC actions (first CIP action, and as of 2013 CIP and ERN-Sys actions). Under the administrative arrangements between DG HOME and JRS, some actions under CIPS were carried out by the JRC and included in the annual work programmes. However, given that the JRC also had its own annual work programmes (based on the Framework Programme 7); actions funded through CIPS were designed in coherence with the JRC’s own priorities. No overlaps were identified in terms of target groups of CIPS and Framework Programme 7. Furthermore, the mapping of CIPS projects revealed that a number of other groups of beneficiaries were targeted, including the private sector, NGOs, the general public, researchers and universities. In terms of eligible actions, the evaluation identified possible incoherencies and overlaps between the eligible actions of CIPS and Framework Programme 7. As stated above, actions funded under the Framework Programme 7 security area included, for instance, ensuring the security of citizens, infrastructure and utilities through technology solutions for civil protection and/or analysing and securing critical infrastructure. There was scope for complementarity with CIPS activities on: ‘development and transfer of technology and methodology’; ‘analytical, monitoring and evaluation activities’ and ‘operational cooperation and coordination’. Coherence with the Civil Protection Financial Instrument On the Civil Protection Financial instrument, its actions specifically related to research on prevention measures and disaster response and management, hence coherence between the two instruments was also ensured. No overlaps were identified in terms of target groups of CIPS and the Civil Protection Financial Instrument. Furthermore, the mapping of CIPS projects revealed that a number of other groups of beneficiaries were targeted, including the private sector, NGOs, the general public, researchers and universities. On actions funded under the Civil Protection Financial instrument, they were set out in Article 4 of Council Decision 2007/162/EC Euratom and included e.g. studies; training, exercises, workshops, exchange of staff and experts; public information, education and awareness raising and associated dissemination actions and monitoring, assessment and evaluation activities. The scope for complementarity between CIPS and the Civil Protection Financial instrument was ensured in all eligible actions under CIPS. Overall, little to no evidence of overlap was observed between the three programmes compared, i.e. CIPS, Framework Programme 7 and the Civil Protection Financial instrument due to their differences in terms of thematic focus, eligible actions and eligible stakeholders and target groups. Furthermore, good coordination between different Directorate-Generals in the Commission (including DG CNECT, DG ECHO, DG MOVE, DG ENER, DG JRC, etc.) through CIPS inter-service group was ensured. Therefore links and complementary were enhanced. Synergies between the results of actions funded under the CIPS programme and the results of similar actions supported under other EU programmes The assessment of this evaluation question was limited by a lack of data collected. It is based on the stakeholders' responses to an online survey. A total number of 20 replies were received. Bearing this limitation in mind, 14 out of 20 respondents reported that they knew about other activities implemented outside the project and covering similar objectives only partially. Activities identified as covering similar objectives were for instance other Framework Programme 7 funded projects and/or other projects under ENISA and were implemented at national, EU and international levels. Examples of links were provided only by six respondents who replied to this question and included, for instance, development of networks and exchange of best practice and links related to the exchange of results and mutual update. Coherence with other related actions supported by other national resources

The assessment of this evaluation question was limited by a lack of available data due to the fact that, according to a number of Commission officials interviewed, national actions were not systematically reviewed on their potential coherence/similarities during the design and implementation of CIPS. From the interviews with some Member States where no coordinating organisations were presents, it emerged that the scope of the CIPS programme, maybe because of its broadness, was lacking clarity. Furthermore, certain of these Member States reported that there was no need for extra funding since CIP programmes were already established at national level, hence no risks of duplication occurred. Although the data collected for this evaluation study showed little evidence of coherence between CIPS and the national frameworks, overall, it can be concluded that Member States were aware of the scope of the programme as they received a draft of the work programme and were invited to provide their comments. In a few cases, the evaluation identified connections between CIPS and national actions. For instance, the German federal cabinet in 2009 approved the national KRITIS-strategy, and this was in line with the CIPS programme thus coherences were ensured. 7.5 EU added value EQ B9: To what extent would the beneficiaries under the CIPS programme (Member States, universities, institutes, associations, etc.) be able to carry out the activities necessary for the implementation of the EU policies in the prevention, preparedness and consequence management of terrorism and other security related risks without the support of the CIPS programme? Overall, stakeholders consultations showed that the added value of the programme is closely linked to the ‘importance’ of EU funding for the organisations involved and to its ability to foster cross-border cooperation which would often not have taken place if projects were supported through national funding alone. Findings suggest that organisations struggled to identify and access national funding opportunities to implement the programme’s activities hence it can be assumed that a significant part of them would have not been developed in the absence of the CIPS funding. Coordinating organisations confirmed that, during the programming period, only few national funding opportunities were available to develop the activities implemented under the various policy areas of the Programme. 9 out of 13 respondents to a dedicated survey stated that the project activities would not have been implemented without EU funding. Only 4 respondents reported that the project could have been fully financed through other funding sources. Nevertheless, these respondents stressed the fact that they preferred to use the CIPS funding anyway, as the latter provided them with the opportunity to work with partners in other countries and to have an impact at EU level. Out of the 9 respondents who reported that they could not have funded their projects through other sources, 8 indicated that it was due to the unavailability or insufficiency of other funds whereas 3 explained that other potential funding sources typically did not support their project’s aims. The CIPS programme had a strong transnational dimension as reflected in its objectives which were in line with the EU objectives on the development of EU policies in the prevention, preparedness and consequence management of terrorism and other security-related risks and on enhancing coordination and cooperation between relevant actors at EU level in protection of critical infrastructure. However, the EU added value may have been reduced by the fact that, as mentioned above, CIPS projects were mostly led by coordinating organisations from a limited number of Member States (mostly Italy, Spain and to a lesser extent the United Kingdom and France). Despite the Commission's efforts in trying to involve applicants from other Member States through regional information days, the geographical spread remained unchanged. The evaluation found different reasons for the significantly varying degrees of uptake across the Member States and included, besides issues such as the ability of potential applicants to draft high-level quality proposals, also the pro-activeness of Member States' Committee representatives promoting CIPS amongst relevant organisations. Conversely, a better geographical balance was reached with partner organisations where 30 or more of them located in 3 Member States (i.e. Italy, Poland and Spain) and 15 or more in 5 Member States (i.e. Czech Republic, France, Germany, Romania and the UK). Overall, partner organisations located in 21 of the 28 Member States participated to five or more CIPS projects, achieving a better geographical spread with regard to partner organisations than coordinating ones. EU added value was enhanced by the fact that the vast majority of CIPS projects were cross-border, with 73 % of projects including at least one organisation from a Member State different than the Member State in which the coordinating organisation was located. From 2009 the eligibility criteria on cross-border projects became stricter to ensure that each cross-border partner had an active role in the activities implemented. Overall, coordinating organisations interviewed assessed positively the partnerships established through projects. From the survey it emerged that the majority of these organisations considered the EU added value of the partnerships as positive a. For instance, it contributed to improved knowledge and expertise on the topic of the project (8 out of 14) or it helped to gain a better understanding on how neighbouring countries work (6 out of 14). Furthermore, coordinating organisations interviewed assessed positively the partnerships established through projects and considered it brought EU added value. Examples of EU added value were the possibility to collaborate in real-life scenarios, in particular when it concerns cross-border cooperation (i.e. joint patrols to reduce terrorism and other security related risks) and the exchanges and training of staff from different countries engaged in protection of CI. The most important EU added value was found in the increase of knowledge and expertise related to a certain topic and the increased knowledge of EU policies related to this topic, according to 13 respondents (out of 14 responding to the online survey for coordinating organisations). Similarly, the analysis of the sample projects showed that, overall, the activities implemented and the outputs provided by both cross-border and national CIPS projects contributed to the EU dimension of the programme. 69 % of the projects were dedicated to the development of common methodologies or tools, for instance on risk assessment. The activities, implemented by national and cross-border partnership, contributed to the EU added value as they produced outputs to be used in other Member States. As an example of EU added value projects, the Threat-Vulnerability Path Identification for Critical Infrastructures (THREVI2) was a national project which contributed to the implementation of Directive 2008/114/EC by promoting the development of risk assessment methodologies. Another national project which developed methodologies to be used at EU level was the ‘Interactive Risk Assessment in critical infrastructure based on the Integrated Geographic Information System’. It developed a practical tool for protection of critical infrastructure and identified interdependencies between sectors in CIP. 8CONCLUSIONS This section summarises the evaluation findings and provides an assessment of the CIPS' overall role in the implementation of prevention, preparedness and consequence management of terrorism and other security-related risks. Relevance

Overall, the evaluation showed that the CIPS programme was of continuous relevance to the prevention, preparedness and consequence management of terrorism and responded to a real need for cross-border cooperation and coordination in the above mentioned areas. The stakeholders that were consulted reported that such cooperation would not have been financed through alternative sources of funding, such as national budgets, due to the financial crisis that occurred during the programme period. The relevance of the CIPS programme was also ensured by the process of priority-settings: these were identified by the Member States and competent Directorate-Generals on a yearly basis and included in the annual work programmes. Therefore an updated and more detailed analysis of needs was ensured. Most CIPS coordinating organisations and partner organisations responding to the online survey considered the priorities set by the Commission in the calls for proposals/annual work programmes as relevant to the specific needs of the target group, at EU level and national level. However, the demand-driven design of the annual work programme resulted in a significant geographical imbalance in terms of the number of projects funded per coordinating organisation in each Member State, although a slightly more balanced spread was achieved for the partner organisations. In order to allow Member States more equal access to funding and to improve their participation, a shift towards shared management mode was introduced for the successor fund, ISF-Police. Unlike CIPS, which was implemented only under direct management, ISF-Police is being implemented on the basis of a combination, with around 60% of the available funding being allocated to national programmes under shared management and 40% allocated to Union actions under direct management. This change is expected to have an impact on the efficiency and effectiveness of the Commission’s management of the ISF Programme. The evaluation also identified a time lag between the formulation of priorities under AWPs and their actual implementation; this was due to the fact that the AWPs reflected the priorities set in the previous year, which meant that there was a one year difference between the setting of priorities and their implementation, which at times hindered the programme’s relevance. This was partly addressed in the successor fund, ISF-Police, through the inclusion of emergency assistance “to address urgent and specific needs”. Effectiveness Bearing in mind the limitations of the evaluation study, it can still be said that the CIPS programme broadly achieved its general and specific objectives. The in-depth analysis of the results, outcomes and impact (where this was evident) of the sample of CIPS projects and on the views gathered from the stakeholders consulted, showed that CIPS contributed positively on critical infrastructure protection. However, when assessing the impacts of the programme, its relatively small budget and wide scope (resulting in a high number of relatively small values of projects) should be taken into account. An important element of a number of CIPS projects was the focus on interdependencies and preventing ‘cascading effects’ in case of disruption and destruction of critical infrastructures during a terroristic attack or other security related risks, thus improving its overall contribution to critical infrastructure protection. However, the results of CIPS projects on EU-wide cooperation and coordination on CIP were evaluated by the Commission as not very effective, with a few peaks of excellence. Certain issues which hindered the assessment of effectiveness (and consequently of efficiency and EU added value) were also identified. Given the relative newness of CIPS and the lack of capacity at the time within the Commission to closely follow all projects from start to end, the absence of a dedicated technical assistance budgethat could have supported a monitoring system to collect and analyse data on (financial) progress, outputs and results, has negatively affected the effectiveness of the Programme. Key elements of this monitoring system could have been, for example, the use of monitoring visits and the use of an IT system to systematically record data on financial progress, outputs and results. This has been addressed to a limited extent under the successor fund, ISF-Police, through the possibility for the financial instrument to contribute annually to technical assistance activities, which could be used for the development of a monitoring system. However, under ISF-Police a problem remains in this area, given that the technical assistance available cannot be used for staff contracting and the development and implementation of a monitoring system requires also additional human resources on both the Member State and the Commission side. This remains to be addressed in the future MFF. Furthermore, the evaluation found that the creation of a central repository containing detailed data on individual CIPS project results and outputs would have greatly enhanced the effectiveness of the programme. For example, methodologies developed to investigate critical infrastructures protection, best practices, training materials, etc. could have been compiled centrally and thus, their potential multiplier effect and continuous usage for other interested stakeholders in other Member States could have been maximized. In consequence, it is recommended that for projects conducted under the successor funding instrument, a repository of the results and outputs of such projects could be created. This has not yet been put into practice under ISF but should be taken into consideration for the next MFF. Moreover, evidence gathered through the stakeholders' consultation showed that results based on potential for transferability to other sectors (and other Member States) and readiness to be used by critical infrastructure owners and operators, in contrast to prototypes and models, were perceived as particularly valuable and contributed to the increase of the overall effectiveness of the programme. In consequence, for the future, similar programmes should be carried out in order to motivate coordinating organisations to work towards the development of demonstrators or prototypes (to be developed as market-ready products) which are based on the results of previous projects. Efficiency In general, CIPS was found to be efficient to a large extent. The vast majority of consulted stakeholders perceived EU funding to have been sufficient and the value for money was perceived as good to very high. No significant differences in the cost-effectiveness of individual projects were found following a comparison of similar financial inputs to their outputs and results. Some private stakeholders mentioned the higher level of administrative burden in comparison with the FP7 rules they were accounted to. A significant upward trend in the absorption of funds spent on grants was observed, rising from 68% in 2007 to 88% in 2013, with 83% across the 2007-2013 period. Most probably, this was due on the one hand, to the changes introduced by the Commission during the CIPS programming period in order to increase the overall efficiency of the Programme and, on the other hand, thanks to the fact that stakeholders became more familiar with the Programme. It should also be pointed out that, overall, the assessment of the data on KPIs showed a satisfactory performance with regard to the average time needed to award, to contract and to make pre-financing payments. However, in terms of timing for processing final payment requests, from the data collected it was evident that, in particular for the earlier years of the Programme, it resulted that the majority of these were not processed on time. Pertaining to the monitoring of project outputs and results, the evaluation found that a more in depth reporting from project beneficiaries would have strengthened the overall efficiency of the Programme. Indeed, in terms of improving the monitoring of project results on the basis of the required reporting, it was concluded also that a qualitative assessment of the results of the Programme was missing due to the absence of a peer review on project results conducted by external experts. Although under the current Fund this has slightly improved as Member States have to submit an Annual Implementation Report under Shared Management and Interim and Final Reports under Direct Management, the absence of a peer review still represents a shortcoming that should be addressed under the next generation of Funds. As already stated under relevance, with regards to dissemination of projects outputs and results, consulted experts and coordinating organisations recognised that the absence of a central repository bringing together the findings of CIPS projects informing stakeholders on available results has decreased the effectiveness and efficiency of the Programme to a certain extent. It is therefore recommended to set up such a central repository bringing together the outputs/results of the projects in future similar programmes. Coherence

The CIPS investments made under the 2007-2013 annual work programme were coherent with activities funded under other EU Funds, namely Framework Programme 7 and the Civil Protection Financial instrument. Indeed, little to no evidence of overlap occurred between funding instruments due to the differences of these Programmes in terms of thematic focus, eligible actions and eligible stakeholders and target groups whilst demonstrating the good coordination between different Directorate-Generals in the Commission (including DG CNECT, DG ECHO, DG MOVE, DG ENER, DG JRC, etc.) through CIPS inter-service group. In this way, links and complementary were enhanced. The JRC acted as a ‘synergy vehicle’ between CIPS and Framework Programme 7. Actions funded through CIPS were designed in coherence with the JRC’s own priorities. On coherence with initiatives at national level, the data available was not sufficient to assess the extent to which this was achieved. EU added value The evaluation showed that due to a lack of national funding for cross-border cooperation in many Member States, a significant part of the activities developed under CIPS would not have been developed at all or would have not achieved the same results, in particular not at a cross-border level. Additionally, the high added value of CIPS was also linked to the high level of co-funding it provided (which could go up to 90 %). As reflected in the objectives of the Programme, CIPS had a strong cross-border dimension. However, as mentioned above, a major weakness identified by the evaluators related to funding distribution that was uneven and concentrated in a few Members States with coordinating organisations that were mainly located in Italy and Spain. Although the Commission took measures to bo