EUROPEAN COMMISSION Brussels, 1.9.2016 SWD(2016) 279 final COMMISSION STAFF WORKING DOCUMENT Final evaluation of the ISA programme Accompanying the document Report from the Commission to the European Parliament and Council on the results of the final evaluation of the ISA programme {COM(2016) 550 final}



Table of Contents Executive summary Introduction 1 Context 1.1 Background of the ISA programme 1.2 Objectives of the ISA programme 1.3 ISA programme governance 1.4 ISA Work Programme 2 Methodology 2.1 Inception (Evaluation structuring) 2.2 Data collection and analytical tools 2.3 Data analysis and synthesis 2.4 Formulation of judgments and reporting 3 Findings and results 3.1 Relevance 3.2 Efficiency 3.3 Effectiveness 3.4 Utility 3.5 Sustainability

3.6 Coherence 3.7 Coordination 4 Recommendations 4.1 Recommendations from previous evaluations 4.2 Recommendations from the final evaluation



Table of tables Table 1 Evaluation questions related to the ISA programme’s performance Table 2 Tabular overview of the actions Table 3 Mapping of evaluation criteria with main evaluation questions Table 4 Key stakeholder groups Table 5 Data collection and analytical tools Table 6 Data analysis and synthesis steps Table 7 Formulation of judgments and reporting Table 8 Alignment of the ISA programme with the DAE Table 9 Alignment of the ISA programme with the European eGovernment Action Plan 2011-2015 Table 10 Alignment of the ISA programme with CEF Table 11 Alignment of the ISA programme with the DSM Strategy Table 12 Funds released and withheld

Table 13 ISA unit resources - turn over in the period January 2010 to mid-2015 Table 14 Operational ISA Solutions Table 15 Common frameworks within the ISA Programme Table 16 Common Services within the ISA Programme Table 17 Reusable generic tools within the ISA Programme Table 18 Utility scores at action level Table 19 ISA and related other EU initiatives, legislation or funding tools Table 20 Level of adoption of ISA solutions by Member States Table 21 Level of adoption of ISA solutions by Commission's services Table 22 ISA website statistics for the three years between June 2012 and June 2015 Table 23 Public consultations launched by ISA Table 24 Validity of recommendations of the final evaluation of the IDABC programme Table 25 Validity of recommendations of the interim evaluation of the ISA programme

Table of figures Figure 1 Hierarchy of objectives of the ISA programme Figure 2 ISA Governance Figure 3 Methodology of the final evaluation Figure 4 Intervention logic of the ISA programme Figure 5 Needs of EU public administrations

Figure 6 Needs addressed by the ISA programme Figure 7 Alignment of ISA objectives with EU initiatives Figure 8 Selection of proposals for new actions in the different Work Programme revisions Figure 9 Overview of ISA action’s budget deviation Figure 10 Overview of the 6 most significant ISA action’s budget deviation Figure 11 Total budget and estimated financial cost for the period 2010 – 2015 Figure 12 ISA unit activities and tasks by types of post Figure 13 Number of HR allocated in 2010 - 2015 Figure 14 ISA actions delayed and ahead of schedule Figure 15 ISA programme efficiency: Schedule Variance Figure 16 ISA programme efficiency: % of delay Figure 17 Most frequently mentioned measures to improve the ISA programme Figure 18 Main ISA results perceived Figure 19 ISA contribution to common frameworks Figure 20 ISA contribution to common services Figure 21 ISA contribution to reusable generic tools Figure 22 ISA contribution to the assessment of ICT implications Figure 23 ISA actions measured for Utility Figure 24 Use of ISA solutions by Member States Figure 25 Use of ISA solutions by Commission's services

Figure 26 Number of views extrapolated to daily averages per year Figure 27 Number of downloads extrapolated to daily averages per year Figure 28 The changes’ potential to improve the benefits of the ISA programme Figure 29 The extent of agreement on the sustainability benefits of the proposed measures Figure 30 ISA Contribution to interoperability in Europe Figure 31 Mapping of the ISA actions to overall approach taken by the programme Figure 32 Summary of the links among ISA actions Figure 33 Synergies between ISA actions Figure 34 Interrelation between ISA actions Figure 35 Clusters of the ISA Programme Figure 36 Synergies between ISA and other EU initiatives Figure 37 Respondent’s agreement levels on the awareness raising of the ISA programme Figure 38 Respondents’ agreement levels on the interaction of their country and ISA Figure 39 Member States with/without ISA event occurrence Figure 40 Respondents’ agreement levels on the ISA Coordination Group coordination with the ISA Working Groups in communication of the ISA results at national level Figure 41 Extent of agreement on ISA’s effectiveness in engaging relevant stakeholders



Acknowledgments This assignment was completed by a Kurt Salmon and KPMG team, the evaluation team, guided by Commission through a steering committee. The team performed its task under the lead of Alessandro Zamboni, and supported by the engagement partner Jean Diederich, along with Ludovic Mayot, Michael Scanlon, Natalia Petrovova and Arianna Volpe. The production of this report would not have been possible without the efforts of the many stakeholders that we have interviewed and surveyed. The authors would like to express their gratitude to all of them. Lastly, the evaluation team would like to thank all the Commission officials and especially the members of the unit DIGIT.B6 and the steering group who have been helpful and co-operative in providing guidance, information and feedback during this assignment.



List of acronyms Acronym Description ARE3NA Reusable Inspire Reference Platform ABB Architectural Building Blocks AD Administrator function group ADMS Asset Description Metadata Schema AST Assistant function group BOMOS Governance, Maintenance and Development Model for Open Standards BRIS Business Registers Interconnection System CAMSS Common Assessment Method for Standards and Specifications CarTool Cartography Tool CEF Connecting Europe Facility (CEF) CF Cohesion Fund CESAR Community of European Semantic Asset Repositories CESI Common Information Sharing Environment CIP Competitiveness and Innovation FrameWork Programme CIPA Common Infrastructure for public administration Sustainability CIRCABC Communication and Information Resource Centre for Administrations, Businesses and Citizens CISE Common Information Sharing Environment CPSV-AP Core Public Service Vocabulary Application Profile CTI Comité Technique Informatique DAE Digital Agenda for Europe DCAT-AP Application Profile for data portals in Europe DG Directorate General DG CNECT Communications Networks, Content and Technology DG (ex DG INFSO) DG ENTR DG Enterprise and Industry DG TAXUD DG Taxation and Customs Union DG DIGIT Directorate-General for Informatics DSI Digital Service Infrastructure DSM Digital Single Market DSS Digital Signature Services EAEC European Atomic Energy Community EAFRD European Agricultural Fund for Rural Development EC European Commission ECAS European Commission Authentication Service ECI European Citizens’ initiatives ECAS European Commission Authentication Service ECLI European Case Law Identifier e-CODEX e-Justice Communication via Online Data Exchange e-Sens Electronic Simple European Networked Services e-TrustEX Trusted Exchange Platform EEC European Economic Community EEP Energy-Europe Programme EESSI Electronic Exchange of Social Security Information EFIR European Federated Interoperability Repository EIA European Interoperability Architecture EIF European Interoperability Framework EIS European Interoperability Strategy EIRA European Interoperability Reference Architecture ELI European Legislation Identifier EMFF European Maritime & Fisheries Fund EPC European Professional Card epSOS European Patients - Smart Open Services EQ Evaluation question ES Earned Schedule ERDF European Regional Development Fund ESF European Social Fund ESIF European Structural and Investment Funds EU European Union Eulf European Union Location Network EVM Earned Value Management FA Focus Area FP7 7th FrameWork Programme GENIS Generic and Interoperable Notification Services GeoDCAT-AP An extension of the DCAT-AP for geospatial data HLCIT High Level Committee on Information Technology HMs Horizontal Measures HoU Head of Unit IA Impact Assessment ICT Information and Communication Technology ICT PSP Information and Communication Technology Policy Support Programme IDA Interchange of Data between Administrations IDABC Interoperable Delivery of Pan-European eGovernment services to public administrations, Business and Citizens IMI Internal Market Information IMM Interoperability Maturity Model INSPIRE Infrastructure for Spatial Information for Europe Directive IPM Interactive Policy Making IRM Information resource Manager IS Information Systems ISA Interoperability Solutions for European public administrations ISA² Interoperability solutions for European public administrations, businesses and citizens ISA Committee Committee on Interoperability Solutions for European public administrations ISA staff HR of the ISA unit ISA unit DG DIGIT Unit B.6 ISPMB Information Systems Project Management Board LEOS Legislation Editing Open Software LSPs Large Scale Pilots MARKT Internal Market and Services M&E Monitoring & Evaluation MFF Multi-annual Financial Framework MT@EC Machine Translation Service by the European Commission NIF National Interoperability Framework NIFO National Interoperability Framework Observatory NIS National Interoperability Strategy NoE Network of Excellence OCS Online Collection Software ODR Online Dispute Resolution OSOR Open Observatory and Repository OSS Open Source Software PCIs Projects of Common Interest PEPPOL Pan-European Public Procurement OnLine PKI Public Key Infrastructures PSC Point of Single Contact PSI Public Sector Information PV Planned Value SEPD Single European Procurement Domain SEMIC Semantic Interoperability Centre Europe SD-DSS Digital Signature Services software SMT Statistical Machine Translation SNE Seconded National Expert SPOCS Simple Procedures Online for Cross-border Services sTESTA Data Communication Network Service STORK Secure idenTity acrOss boRders LinKed SV Schedule Variance TCO Total Cost of Ownership TES Trans-European Services TESTA Trans-European Services for Telematics between Administrations TESTA NG Trans-European Services for Telematics between Administrations New Generation TOR Terms Of Reference TL Manager Trusted Lists management software WP Work Programme YE Your Europe



Executive summary 1. Context & evaluation objectives The legal decision on ‘Interoperability Solutions for Public Administrations’, which led to the creation of the ISA programme, was adopted by the European Parliament and the Council on 16 September 2009 with a budget of €164,100,000 for its implementation from 1 January 2010 to 31 December 2015. Building on the advances made by its predecessor IDA and IDABC programmes, the ISA programme was established against the background of major EU policies, to which it needed to be aligned. These included the Europe 2020 Strategy , developed as a response to the financial and economic crisis, the Digital Agenda for Europe, one of the seven flagship initiatives of the Europe 2020 Strategy and the eGovernment Action Plan 2011-2015, which aimed to achieve a new vision of European public administrations through a more open model of design, production and delivery of online services to deliver greater value to citizens with fewer resources. Although other major policies have been launched since the inception of the ISA programme, notably the Connecting Europe Facility (CEF) and the Digital Single Market (DSM) strategy, the final evaluation finds that it is well aligned with both. As stipulated under Article 1(2) of the legal decision, the programme aims ‘to support cooperation between European public administrations’ with a view to enable ‘the delivery of electronic public services supporting the implementation of Community policies and activities including bodies performing public functions on their behalf’ by ‘facilitating the efficient and effective electronic cross-border and cross-sectoral interaction between such administrations’. To achieve such objectives, the programme has been implemented by means of actions, i.e. studies and projects, as well as by accompanying measures to support the: Creation and improvement of common frameworks in support of interoperability across borders and sectors; Improvement of existing reusable generic tools as well as the establishment of new ones; Operation and improvement of existing common services as well the establishment of new ones; and Assessment of ICT implications of proposed or adopted Community legislation. Currently, 38 actions are being implemented under the ISA Work Programme (2015) and are aligned with the clusters defined in the EIS; Trusted information exchange, Interoperability architecture and Assessment of ICT implications of EU legislation. 12 of these actions correspond to common frameworks, 8 to common services and 13 to generic tools. One action corresponds to the assessment of ICT implications of EU legislation and the remaining four actions concern accompanying measures and monitoring of the ISA programme. The European Commission contracted KURT SALMON and KPMG to conduct an independent, external final evaluation of the ISA programme , based on Article 13(3) of the legal decision with the four main objectives: Objective 1: Evaluate the implementation of the ISA programme from its commencement in January 2010 until the time that this evaluation takes place. Objective 2: Assess the relevance, effectiveness, efficiency, utility, sustainability, coherence and coordination of the ISA actions as well as assess performance, qualitatively and quantitatively, when possible, against its objectives and rolling Work Programme.

Objective 3: Examine the benefits of the ISA actions to the EU for the advancement of common policies. Objective 4: Identify areas for potential improvement and verify synergies with other EU initiatives in the area of cross-border and cross-sectoral interoperability as well as sharing and reuse. 2. Methodology In order to satisfy the objectives of this final evaluation, data was collected using a bespoke methodology comprising of several techniques and quantitative and qualitative analysis. The final evaluation’s methodology conforms to the methodological guidance for evaluations provided by DG Budget and adheres to developments in the Smart Regulation policy . During the data collection phase, the evaluation team used a combination of desk research, surveys (73 ), phone or face-to-face interviews (37) and also case studies of specific actions of the Work Programme (2) in order to fulfil the objectives of this evaluation. Furthermore, written requests were addressed to specific stakeholders to gather additional information on specific sub-issues of the final evaluation. 3. Findings and results 3.1. Relevance The ISA programme’s objective ‘to support cooperation between European public administrations by facilitating the efficient and effective electronic cross-border and cross sectoral interaction between such administrations’ is still pertinent to meet evolving needs at both national and EU levels. The ISA programme facilitates interaction between European public administrations. The modified scope of the follow-on ISA² programme will, most likely, enhance this objective. The follow-on ISA² programme’s scope aims to facilitate electronic cross-border or cross-sector interaction between European public administrations, businesses and citizens as well as between European public administrations (Conclusion No 1). The needs of European public administrations, identified at the time of the ex-ante and interim evaluations of the ISA programme, are still considered as the current needs of European public administrations. In particular, the need to have ISA solutions more reused as well as the need for an increased collaboration between the programme and both the Member States and Commission services are as valid now as they were at the time of the interim evaluation. The evaluation notes that, as a voluntary policy, the ISA programme has no capacity to guarantee greater reuse of its solutions and increased collaboration depends on the willingness of all stakeholders (Conclusion No 2). The programme has already started to take into account new evolving needs of European public administrations on governance and coordination, primarily through the revision of the EIS and through enhanced internal coordination within the European Commission. Finally, the ISA programme’s objectives are well aligned with the Digital Agenda for Europe, the European eGovernment Action Plan 2011-2015, the Connecting Europe Facility and the recently launched Digital Single Market Strategy (Conclusion No 3). 3.2. Efficiency The ISA programme can be considered as efficient, overall. First of all, the evidence from action 5.01 Monitoring & Evaluation found that the programme has been considered on track during the entire period from 2010 to 2015 with an average delay of less than 5%. In addition, 90% of surveyed Member States’ representatives and EC officials affirmed their view that the programme has delivered on time and within its original scope. Furthermore, a majority of these stakeholders (68% of Member States’ representatives and 58% of EC officials) felt fully or somewhat involved in the annual revisions of the ISA programme (Conclusion No 4). The evaluation found that the selection procedure of proposals for new actions and the funds release process can be regarded as efficient. Moreover, the total executed budget for the 2010 – 2014 Work Programmes is equal to EUR 128.2m, only slightly exceeding its planned budget of EUR 127.4m by less than 1 per cent (0.63%), yet still within the overall financial envelope. Given these budgetary findings, the evaluation confirms the programme has been efficient in this regard. The same can be concluded for the allocation of human resources, even though the actual allocation was always lower than what was foreseen in the ISA legislative proposal.(Conclusion No 5). 3.3. Effectiveness The ISA programme has delivered operational solutions, i.e. common frameworks, reusable generic tools and common services, contributing to the achievement of the programme’s objective to facilitate effective collaboration between European public administrations. Member States’ representatives and EC officials affirmed their view that the programme contributes more to the establishment of new common frameworks, common services and reusable generic tools than in improving existing ones. Overall, more interviewees and online survey respondents believed that all the ISA solutions facilitate interoperability between European public administrations (41%) than did not (30%). The remaining respondents said that they did not have sufficient overview of the programme to comment on all solutions. The evaluation found, nevertheless, that the need for the further development of the EUSurvey tool in the ISA² programme has to be appraised given respondents’ specific feedback on this (Conclusion No 6).

The evidence from the evaluation confirms that the programme contributed to sharing and re-use, through the establishment of common frameworks. The programme also contributed to organisational and legal interoperability, although the results from the ISA actions are scattered in this regard, which indicate space for improvement. Now that the assessment of ICT implications of new EU legislation has been included in the Impact Assessment process (Better Regulation Guidelines), the ISA programme has a defined role to promote this exercise and support the DGs and services undertaking these assessments with guidance as part of this internal EC process (Conclusion No 7). 3.4. Utility The evaluation concludes that, overall, the achieved and anticipated results and impacts of the ISA programme largely address the business needs that they intended to. Some ISA actions have led to the production of 23 solutions. Of these 23 solutions, 15 are used by up to all Member States; 14 are used by several Commission services, and 5 are used by a handful of other EU institutions. Nevertheless, other actions have also produced results if not specifically related to solutions. For example, action 3.01 Assessment of ICT implications of new EU legislation has achieved its main objective as it has succeeded in being included in the formal Impact Assessment process even though there were not so many assessments conducted during the ISA programme. Member States’ representatives and EC officials considered the top three benefits delivered by the ISA programme to Member States to be the use of ISA solutions, the existence of the programme itself (to raise awareness about the importance of interoperability and to keep it on the political agenda) and the provision of references, such as the EIF, supported by action 4.2.03 NIFO, networks and solutions, such as action 2.04 sTESTA and action 2.06 CIRCABC. In addition, the evaluation concludes that the ISA programme contributes to the modernisation of the public sector in Europe. The evaluation notes that the ISA programme does not coordinate, in a systematic way, with Commission DGs and services that run sectoral programmes, within which many cross-sector and/or cross-border services are managed and to which interoperability could greatly contribute (Conclusion No 8). The evaluation confirms the existence of low levels of awareness about the programme among its interviewed and surveyed key stakeholders, Member States’ representatives and EC officials. Paradoxically, the additional measures, as suggested by the same interviewees and survey respondents, are all either in place or being developed or expanded. The evaluation, therefore, concludes that the follow-on ISA² programme should raise awareness about these existing measures and improve them by adapting them to the needs of targeted/specific stakeholders (Conclusion No 9). 3.5. Sustainability The evaluation found that the financial, technical and operational sustainability of the developed solutions, maintained and operated through the ISA programme, is ensured. From a technical and operational standpoint, the annual ISA Work Programme summarises the actions that are funded based on numerous criteria, such as the technical need for an action to deliver a solution and the operational role that it fulfils. From a financial standpoint, sustainability of the developed solutions will be guaranteed, primarily, by the follow-on ISA² programme, although not after that. However, this may change subject to an impact assessment or other policy options, which may be pursued, during the period of the follow-on ISA² programme. Some solutions, created by the ISA programme, are now funded, operationally sustained and further developed outside of the framework of the programme by other EU initiatives and programmes. To address the long term sustainability of the solutions, the ISA² programme should explore how to financially re-engineer certain services, which it provides, to ensure that users pay for them. In addition, the importance of focusing on the longer term organisational sustainability of ISA actions cannot be underestimated. Beneficiaries should formalise governance of services supported by these ISA actions to ensure their continuity. The actual use and benefits of the solutions as well as the numbers of users of each ISA solution, which are not currently tracked in a systematic way, should be measured by the programme, to help identify options to ensure their sustainability outside of the programme’s framework or without its funding (Conclusion No 10). 3.6. Coherence The evaluation found that a holistic approach within the framework the ISA programme does exist, even if the programme does not convey this sufficiently. This holistic approach is based on the strategic approach of the ISA programme, created in January 2013, including the mapping of the ISA actions to the various cycles (Inception, Execution, and Operation), the interrelation between the various actions, as well as its global approach as contributor to interoperability in Europe. The evaluation found that the ISA programme houses independent actions, with which stakeholders are often more familiar than the programme itself. Levels of awareness about programme’s synergies are also confirmed to be lacking among the interviewees and survey respondents. In the context of the low levels of awareness, there is a need to provide greater clarity on the documented process about the selection of proposals for new actions (Conclusion No 11). Although a considerable amount of synergies exists between the programme’s actions and other EU activities, and that synergies were achieved between programme actions quite well and other EU activities fairly well, the evaluation concludes that further synergies could have been achieved. The analysis showed that 25 ISA actions have more than five links with other ISA actions; including six ISA actions that are linked to more than ten other ISA actions. 24 actions have between one to five links with other actions. Given the ISA programme’s holistic approach, it is normal that some actions have more synergies with other actions, which does not reduce the value of the latter actions. The analysis showed that the ISA programme has achieved synergies with several activities at EU level, including CEF. The ongoing efforts of the ISA programme and DG DIGIT to work with relevant DGs to increase the EC’s coherence on interoperability and eGovernment services, through enhanced coordination are satisfactory, for example, in the following separate activities; in the Inter-service Group on Public Administration Quality and Innovation and in its publications, such as the NIFO and eGovernment factsheets (Conclusion No 12).

3.7. Coordination The ISA programme coordinates formal activities well between the Member States, including the ISA Committee (Conclusion No 13); as well as with EC officials (Conclusion No 14) to ensure their engagement in the programme. Two-thirds of Member States’ representatives and just over half of EC officials fully or somewhat agreed that the interaction between ISA and its stakeholders was effective to ensure consistent exchange of information, views and best practices. In addition, the ISA programme coordinates activities well with stakeholders outside of the context of European public administrations during the course of its activities to ensure their engagement in the programme; when appropriate (Conclusion No 15). In fact, the evaluation found that the increase in coordination with stakeholders since the interim evaluation has been a major success of the ISA programme although potential to build on this success with more targeted efforts does exist. Between 2010 and 2015, approximately two thirds of all events, which the ISA programme organised and held with Member States’ representatives, and 63% (25) of the 40 events organised by Member States, in which the ISA programme participated, were held since 2013, i.e. after the interim evaluation. With regard to Member States, the evaluation concluded that there is scope for individual national representatives on the ISA Committee, Coordination Group & Working Group to coordinate better with their own national colleagues on interoperability issues to better realise the programme’s achievements. The ISA programme increased its presence at conferences and other events run by the EC, having been involved in over 60 events since its inception, with three-quarters of these events taking place since the interim evaluation. This is a clear indication of the focus of the ISA programme to increase interaction with its stakeholders and their increasing interest in the programme. A majority of the Member States’ representatives and EC officials fully or somewhat agreed that the electronic platforms (i.e. Joinup, the ISA website and dashboard), used by ISA to deliver consistent interaction between stakeholders and with the European Commission, were adequate. However, there was a disappointing lack of knowledge among the stakeholders about the results achieved at national and EU level, i.e. about the reuse of the ISA programme at national level, as well as about the use of ISA results by DGs.



4. Recommendations 4.1. Recommendation 1 The ISA² programme, which serves an EU policy, should continue to align itself with other relevant EU policies (Conclusions No 1, 2 & 3). It should also reflect the current needs of stakeholders to have more emphasis on having solutions operational and ready-for-use (Conclusion No 6). 4.2. Recommendation 2 The ISA² programme should support the review of the EIS on interoperability and to ensure its success. The alignment between the programme and the future strategy should be monitored regularly, as recommended in the EIS implementation review 2012 (Conclusion No 11). 4.3. Recommendation 3 The ISA² programme should continue to focus on the current ISA activities but place more emphasis on achieving legal and organisational interoperability. Now that the assessment of ICT implications of new EU legislation has been included in the Impact Assessment process (Better Regulation Guidelines), the ISA² programme should exploit its defined role to promote this exercise and support the DGs and services undertaking these assessments with guidance as part of this internal EC process (Conclusion No 7). 4.4. Recommendation 4 The ISA² programme should update and implement its communication strategy for the follow-on programme, with a focus on targeted engagement, including sector-specific stakeholders, by building on its substantial efforts since the interim evaluation. This would address the lingering gap between the actual performance of the programme and the stakeholders’ perception of it. This should be supported by the new Inter-service Group on Public Administration Quality and Innovation as the basis for improving the adoption of operational ready-for-use solutions (Conclusions 9, 13 – 15). 4.5. Recommendation 5 The ISA² programme should develop a more systematic business-case approach based on the Monitoring & Evaluation reporting, and the study on the cost-benefits of interoperability, both of which are ongoing. A clear business-case approach that satisfies defined eligibility criteria, as is likely to be the case in the ISA² programme based on its legislative proposal, would provide such an approach and clear guidance for stakeholders (Conclusions 5, 8 & 10).

4.6. Recommendation 6 DIGIT should, within its means and insofar as is possible, respect the targets of the follow-on programme’s envisaged staff levels (Conclusion 5). 4.7. Recommendation 7 DIGIT should build on the improvements in the coordination of activities related to interoperability and eGovernment across the Commission that it has achieved since the interim evaluation (Conclusion N° 12). 4.8. Recommendation 8 The ISA unit should document the existing services and tools established or improved by the ISA programme (Conclusion N°6) within the European Interoperability Cartography (EUCart) and Joinup, by applying the principles and guidelines from the European Interoperability Reference Architecture (EIRA). This would allow the programme to further clarify its benefits and its increasing levels of interaction with stakeholders (Conclusion No 13 – 15). 4.9. Recommendation 9 The ISA unit should develop a more systematic approach to support and monitor the use of common services and generic tools, but also the application and implementation of common frameworks (Conclusion No 6). Introduction Kurt Salmon and KPMG, as part of the KonSulT consortium, were mandated by the European Commission to conduct the final evaluation of the Interoperability Solutions for European public administrations (ISA) programme. This is based on Article 13(3) of the Decision No 922/2009/EC , which states that ‘the ISA programme shall be subject to an interim evaluation and a final evaluation, the results of which shall be communicated to the European Parliament and the Council by 31 December 2012 and 31 December 2015 respectively’. As stated in the same Article 13(3), the final evaluation aims to ‘examine issues such as the relevance, effectiveness, efficiency, utility, sustainability and coherence of the ISA programme's actions’ and to ‘assess performance against the objective of the ISA programme and the rolling Work Programme’. The final evaluation should also specifically ‘examine the extent to which the programme has achieved its objective’. In addition to the six aforementioned issues required by the legal decision, coordination is also considered as an additional issue, also called evaluation criteria, to be examined by the final evaluation. In fact, the evaluation should also, as stated in Article 13 (3) of the ISA legal decision, analyse the ‘benefits of the actions to the EU for the advancement of common policies, identify areas for potential improvement and verify synergies with other EU initiatives in the area of cross-border and cross-sectoral interoperability’. In accordance with these seven evaluation criteria, Table 1 presents the main evaluation questions driving this evaluation. EQ Nr Criteria Evaluation question EQ1 Relevance To what extent were the ISA programme's objective(s) pertinent to meet evolving needs and priorities at both national and EU levels? EQ2 Efficiency How economically were the various inputs converted into outputs? EQ3 Efficiency Which aspects of the programme were the most efficient or inefficient, especially in terms of resources mobilised? EQ4 Effectiveness To what extent did the ISA programme's results and impacts achieve its objectives? EQ5 Effectiveness Are there aspects that were more or less effective than others, and, if so, what lessons can be drawn from this? EQ6 Utility How did the ISA programme's achieved and anticipated results and impacts compare with the business needs they intended to address? EQ7 Utility Which measures could be taken to improve the utility of the next programme’s actions? EQ8 Sustainability To what extent is the financial, technical and operational sustainability of the developed solutions, maintained and operated through the ISA programme, ensured? EQ9 Coherence To what extent did the ISA actions form part of a ‘holistic’ approach within the framework of the programme? EQ10 Coherence How well were synergies achieved between programme actions and with other EU activities? EQ11 Coordination To what extent did coordination of activities between Member States, including the ISA Committee, exist to ensure stakeholders' engagement in the ISA programme? EQ12 Coordination To which extent were activities coordinated or aligned with the needs of other stakeholders with whom the Commission was supposed to interact in the framework of ISA? In order to monitor the progress of the final evaluation, a Steering Committee was specifically appointed. It is composed of the following members: Ms Margarida Abecasis, European Commission, DG DIGIT B.6;

Mr Ioannis Sagias, European Commission, DG DIGIT B.6; Mr Konstantinos Bovalis, European Commission, DG DIGIT B6; Mr Hans Vanderbeke, European Commission, DG DIGIT B6; Mr François Kodeck, European Commission, Former SG – IT Governance task force; Mr Paul-Hervé Theunissen, European Commission, DG TAXUD A5; Ms Natalia Aristimuno Perez, European Commission, DG DIGIT B2; Ms Kate Anderson, European Commission, DG GROW B1; and  Mr Michel Millot, European Commission, DG JRC H6. This report has been structured in the following manner: Chapter 1 is an introduction that provides insights on the background and objectives of the ISA programme as well as its governance and Work Programme; Chapter 2 contains a presentation of the methodological framework for the evaluation; Chapter 3 discusses in detail findings and presents the related conclusions for each evaluation criteria; Chapter 4 Assesses the extent to which the recommendations from previous evaluations have been taken into account and includes, based on conclusions presented in Chapter 3, practical recommendations for the future. Annexes referred to in this report, which contain more detailed evaluation findings, are available in a separate document called ‘Final report – Annexes’.



1Context This chapter presents the background as well as the objectives of the ISA programme, respectively in sections 1.1 and 1.2. The ISA programme governance is described in section 1.3 whereas section 1.4 provides insights on its Work Programme. 1.1Background of the ISA programme In 1995, the Interchange of Data between Administrations (IDA) programme was established to promote the development and operation of trans-European telematic networks for data interchange between Member State administrations and/or EU institutions. It ran from 1995 to 1999. Following the first IDA programme, its successor programme, IDA II, was launched in 1999 and operated until the end of 2004. This programme differentiated from its predecessor by being more directed towards the market and interoperability, with the aim of increasing the efficiency of the delivery of eGovernment services to European businesses and citizens.

Decision No 2004/387/EC of the European Parliament and of the Council, adopted on 21 April 2004, concluded negotiations on the Interoperable Delivery of Pan-European eGovernment Services to public administrations, Business and Citizens (IDABC) programme, which succeeded to IDA II and entered into force on 1 January 2005. The ISA programme aimed to build on the experience gained from the IDABC, IDA II and IDA programmes, which made important contributions to ensure interoperability to support the electronic exchange of information between European public administrations. As mentioned in Article 17(1) of the ISA programme’s legal decision, ‘the financial envelope for the implementation […] for the period from 1 January 2010 to 31 December 2015 shall be EUR 164,100,000, of which EUR 103,500,000 is for the period from 1 January 2010 until 31 December 2013. For the period following 31 December 2013, the amount shall be deemed to be confirmed if it is consistent for this phase with the financial framework in force for the period commencing in 2014.’ As further detailed in section 1.4, the ISA programme is based on a rolling Work Programme that has to be revised and approved by the Commission at least once a year (cf. Article 9(2) of the legal decision).



1.2Objectives of the ISA programme The hierarchy of objectives is the first step to design the different elements of the intervention logic. The identification of these objectives is indispensable for assessing, primarily, the relevance, effectiveness and utility of the programme. As defined in the framework of the interim evaluation, Figure 1 depicts the hierarchy of objectives of the programme. Source: Decision No. 922/2009/EC of the European Parliament and of the Council of 16 September 2009 on interoperability solutions for European public administrations (ISA) (Decision located in OJ L 260, 3.10.2009, p.20). Global objectives provide a basis for assessing the ISA programme in relation to the longer term effects (global impacts). As stipulated under Article 1(2) of the Decision N°922/2009/EC, the objective of the ISA programme is to ‘support cooperation between European public administrations by facilitating the efficient and effective electronic cross-border and cross-sectoral interaction between such administrations, including bodies performing public functions on their behalf, enabling the delivery of electronic public services supporting the implementation of Community policies and activities.’ With reference to Article 1(1) and Article 1(2), European public administrations are defined as local and regional administrations and EU institutions (formerly called Community institutions) and bodies, including bodies performing public functions on their behalf. This definition applies to the remaining parts of this document. As defined in the framework of the interim evaluation of the ISA programme, the global objective of the ISA programme can be expressed as ‘to support cooperation between European public administrations, with a view to enable the delivery of electronic public services supporting the implementation of Community policies and activities.’ Intermediate objectives provide a basis for assessing the ISA programme in relation to the medium term effects regarding both direct (public administrations) and indirect beneficiaries (citizens and enterprises). They are used as the means of attaining the global objective of the ISA programme. The evaluation considers the following statement, ‘to facilitate the efficient and effective electronic cross-border and cross-sectoral interaction between European public administrations’, as the intermediate objective. This is fully in line with the legal decision and with the definition agreed at the time of the interim evaluation. Specific objectives provide a basis for assessing the ISA programme in relation to its short term results for the direct beneficiaries. These objectives are equivalent to the deliverables of the ISA actions. With reference to Article 3 of the legal decision, the ISA programme specific objectives could indeed be grouped into four broad categories, according to the different activities that the ISA programme shall support and promote:

-Common frameworks; -Existing and new Common services; -Existing and new reusable Generic tools; -Assessment of ICT implications. Indeed, the global and intermediate objectives of the ISA programme are realised through six specific objectives, which are constructed based on the above mentioned categories and correspond to the direct effects of the actions. Operational objectives are related to the ISA programme actions aimed at operating the Work Programme. This relates primarily to the resources allocated, i.e. budget and human resources, which are required to accomplish the actions. With reference to Article 4, this is related to the objectives of the actions, which are specified in the Work Programme entries.

1.3ISA programme governance The European Commission's Directorate General for Informatics (DIGIT) is responsible for coordinating the implementation of the ISA programme. As stated in Article 12 of the ISA legal decision, ‘the Commission should be assisted by the Committee on Interoperability Solutions for European public administrations (the ISA Committee)’. Indeed, DIGIT is assisted by a management Committee (ISA Committee), composed of representatives of each EU Member State that participate in the programme. Article 4 and Article 7 of Decision 1999/468/EC [‘Comitology decision’ of 28 June 1999] are used to regulate the role of the ISA Committee and the procedure to be followed. Regulation (EU) No 182/2011 (new comitology Regulation) was adopted on 16 February 2011. It repeals Decision 1999/468/EC (the old ‘Comitology Decision’). There are now two comitology procedures - the advisory procedure and the examination procedure. The examination procedure replaces the management and regulatory procedures. The examination Committee can approve or reject the implementing measure with qualified majority (the same voting rules as those that apply in the Council). In case no decision is taken, depending on the subject matter of the implementing measure, the Commission can either adopt the measure or submit it to an appeal Committee, where a new final voting takes place. Figure 2 illustrates the current governance structure of the ISA programme. Source: Regulation No 182/2011 of the European Parliament and of the Council of 16 February 2011 laying down the rules and general principles concerning mechanisms for control by Member States of the Commission’s exercise of implementing powers - (Regulation located in OJ L 55, 28.02.2011, p.13). The roles and responsibilities of the various stakeholders involved in the management and delivery of the ISA programme can be summarised as follows. The ISA Committee is composed of representatives of Member States. Its role is to assist the Commission in the implementation of the ISA programme by delivering opinions on and amendments to the Commission’s proposals. Therefore, the ISA Committee enables the supervision of the Commission’s executive power. In addition, the Commission’s draft implementing acts are discussed and debated during the ISA Committee meetings.

In accordance with Article 15 of the Decision No 922/2009/EC, ‘the ISA programme shall be open to participation, within the framework of their respective agreements with the Community , by the countries of the European Economic Area and the candidate countries’. Representatives of third parties, as well as experts, can also attend the Committee meetings but cannot be present at or participate in the vote. According to the Article 7 of the Decision 1999/468/EC (old ‘Comitology decision’), ‘each Committee shall adopt its own Rules of Procedure’. In that regard, in order to ensure continuity and consistency at working level, the ISA Committee created the ISA Coordination Group. The ISA Coordination Group, established by the adoption of the Terms of Reference of the ISA Coordination Group, is composed of high level experts, appointed by the ISA Committee and sometimes members of the ISA Committee itself. Its role is to apply the European Interoperability Strategy (EIS) and its related priorities into concrete actions for the ISA Work Programme, based on the expertise of the Member States, so that the ISA Work Programme actions are coordinated and aligned with the initiatives at national level. Indeed, the ISA Coordination Group ensures consistency in Member States’ contributions across the ISA Work Programme’s actions and specialists groups . The ISA Coordination Group reports to the ISA Committee on a regular basis, more specifically upstream to the ISA Committee meetings, in order to prepare their discussions. At the suggestion of the ISA Coordination Group, the ISA Committee can appoint the ISA Working Group Members in order to delegate work on specific scopes to experts. According to the Terms of Reference of the ISA Working Group, the role of the Working Groups is ‘to assist the Commission in implementing concrete actions of the ISA Work Programme and possibly other EU initiatives’, while facilitating the cooperation between the Commission and the Member States at the same time. Thus, they can contribute to ISA actions in different ways: ‘by national information gathering, by providing input to, reviewing and commenting on draft documents and deliverables, by communicating and explaining ISA results, by contributing to the identification of relevant stakeholders and by informing Working Group members about national interoperability initiatives’. Each Working Group is established on the basis of a call for nominations; as a matter of fact, they may include members of the ISA Committee. In addition, the members of the Working Group can also include representatives of other DGs, aside from DG DIGIT, and the appointed Member States high level experts. Four Working Groups have been established within the ISA programme. Three of them follow the EIS clusters, namely one on Trusted Information Exchange, one on Interoperability Architecture and a last one on Exchange of Best Practices. These three groups were established in 2010 following the adoption of the ISA programme. They include stakeholders from different Member States as well as DG DIGIT officials and occasionally have representatives from the JRC or DG CNECT. The fourth one was established more recently - in 2013, in the context of the ISA actions 1.17 Reusable Inspire Reference Platform and action 2.13 Development of a European Union Location Framework. It is an ISA-INSPIRE Working Group on Spatial Information and Services. The reason for establishing it was the importance of interoperability of geospatial information. Creation of such group aimed to facilitate the dialogue between the geospatial and egovernment communities in general; and particularly between the INSPIRE and ISA communities. Its members are a mix of INSPIRE and eGovernment representatives. European Commission’s stakeholders include representatives from DG CNECT, DG DIGIT, DG Environment and the JRC. Each ISA Working Group addresses specific issues and, consequently, prepares the work of the ISA Coordination Group. In addition, members of the ISA Working Groups are meant to report nationally to the ISA Coordination Group members.



1.4ISA Work Programme Article 9 of the ISA legal decision requires that the Commission establishes a rolling Work Programme for the implementation of the ISA programme for its full duration. The Commission must approve, at least once a year, any modifications of the ISA Work Programme, adopted for the first time by the Commission in July 2010. Amendments can be related either to minor or to significant changes of existing ISA actions or to proposals for new ISA actions. Five revisions of the rolling Work Programme have occurred to date. Its main goal is to define the actions which will be implemented throughout the year. The ‘ISA actions’ can be of different kinds; namely studies, projects and accompanying measures. The policy context of those actions corresponds to the vision and goals outlined in the European Interoperability Strategy (EIS) and the European Interoperability Framework (EIF). The programme’s budget finances a series of ISA actions aimed at establishing or operating three different types of solutions, but also assessing the ICT implications (with reference to Art. 3(b)) of proposed or adopted EU legislation and planning for the introduction of ICT systems to support the implementation of such legislation. With reference to the ISA legal decision: Article 2(b): ‘Solutions means common frameworks, common services and generic tools’; Article 2(c): ‘Common Framework means strategies, specifications, methodologies, guidelines and similar approaches and documents’;

Article 2(d): ‘Common Services means operational applications and infrastructures of a generic nature which meet common user requirements on policy areas’; Article 2(e): ‘Generic tools means reference platforms, shared and collaborative platforms, common components and similar building blocks which meet common user requirements across policy areas’. As of today, 38 actions are being implemented under the ISA Work Programme (2015) , out of which 12 correspond to common frameworks, 8 to common services, 13 to generic tools and 1 to the assessment of ICT implications of EU legislation. The rest of the ISA actions (4 actions) refer to ‘Accompanying measures’ (Article 2(g): ‘Accompanying measures means strategic and awareness-raising measures, measures in support of the management of the ISA programme and measures in relation of the sharing of experience and the exchange and promotion of good practices’), ‘Awareness’ and ‘Monitoring’ of the ISA programme. The design of the actions is based on proposals made on an annual basis by the Commission services and Member States through the ISA Committee. Actions relevant to specific policy areas are designed in close coordination with the Commission service(s) responsible for the policy area. The ISA programme is articulated around the following three clusters: 1.Trusted information exchange; 2.Interoperability architecture; and 3.Assessment of ICT implications of EU legislation. A tabular overview of the actions is proposed in Table 2 and includes the action number, the type of action (i.e. study, project or accompanying measure), the type of activity (common services, reusable generic tools, common frameworks and assessment of ICT implications) and the year of inclusion into the rolling Work Programme as well as the year of conclusion, for actions having their funding concluded yet. Nr Action Action type Solution type Inclusion year Conclu-sion year 1.Trusted Information exchange 1.01 Promoting semantic interoperability amongst the EU MS Project Common services 2010 Ongoing 1.02 Access to base registries Study Common frameworks 2010 Ongoing 1.03 Catalogue of Services Study Common frameworks 2010 Ongoing 1.04 ECAS-STORK Integration Project Reusable generic tools 2010 2014 1.05 STORK Sustainability Project Reusable generic tools 2010 2014 1.06 CIPA (including PEPPOL) Project Reusable generic tools 2010 Ongoing 1.07 e-PRIOR Project Reusable generic tools 2010 Ongoing 1.08 Trusted Information Exchange […] Project Reusable generic tools 2010 Ongoing 1.09 Supporting tools for TSL […] Project Reusable generic tools 2010 2014 1.10 IMI system Project Reusable generic tools 2010 2014 1.11 Interoperable and Generic Notification Services Project Reusable generic tools 2010 Ongoing 1.12 OSS platform for online collection of statements of support for ECI Project Reusable generic tools 2011 Ongoing 1.13 Legislation Editing software - LEOS Project Reusable generic tools 2011 Ongoing 1.14 Cross Sector SOLVIT Project Common services 2011 2013 1.15 Open Government Data Study Common frameworks 2011 2011 1.16 CISE Project Reusable generic tools 2012 Ongoing 1.17 ARE3NA Project Reusable generic tools 2012 Ongoing 1.18 Federate Managed Authentication Services for ECAS Project Common services 2012 Ongoing 1.19 PEPPOL Sustainability Project Common services 2013 2013 1.20 Application of EU Law: […] Project Common services 2013 Ongoing 1.21 European Legislation Identifier (ELI) Study Common frameworks 2014 Ongoing 1.22 Big data and open knowledge […] Study Reusable generic tools 2015 Ongoing 2.Interoperability Architecture 2.01 EIA Study Common frameworks 2010 Ongoing 2.02 CAMSS Study Common frameworks 2010 Ongoing 2.03 PKI Services Project Common services 2010 2014 2.04 sTESTA Project Common services 2010 Ongoing 2.05 CIRCABC Project Common services 2010 Ongoing 2.06 EUSurvey Project Common services 2010 Ongoing 2.07 Your Europe - Facilitating reuse of content from national portals Project Common frameworks 2010 2011 2.08 MT@EC Project Common services 2010 2014 2.09 Document repository services […] Project Reusable generic tools 2010 Ongoing 2.10 Multisectorial crisis and business continuity services Project Reusable generic tools 2010 2011 2.11 Promoting consistent EU e-procurement monitoring and performance Study Common frameworks 2010 2014 2.12 eHealth EIF Study Common frameworks 2011 2012 2.13 EULF Study Common frameworks 2012 Ongoing 2.14 Assessment of TES Study Common frameworks 2012 Ongoing 2.15 Interoperability agreements on electronic document and electronic file Study Common frameworks 2013 Ongoing 2.16 European Single Procurement Document Service Project Common services 2014 Ongoing 2.17 eCertis Project Reusable generic tools 2014 Ongoing 2.18 Participatory knowledge for supporting decision making Project Reusable generic tools 2015 Ongoing 3. Assessment of ICT implications of EU legislation 3.01 Assessment of ICT implications of EU legislation Study Assessment of ICT implications of EU legislation 2010 Ongoing 4. Accompanying Measure 4.1 Raising Interoperability awareness 4.1.01 Communication activities Acc. M. Awareness raising 2010 Ongoing 4.1.02 Interoperability Maturity Model Study Common frameworks 2011 Ongoing 4.2 Sharing of best practices 4.2.01 Integrated Collaboration Platform – Joinup Project Common services 2010 Ongoing 4.2.02 Community building and effective use of the collaborative platforms Acc. M. Stakeholders involvement 2010 Ongoing 4.2.03 NIFO Study Common frameworks 2011 Ongoing 4.2.04 EFIR Project Reusable generic tools 2012 Ongoing 4.2.05 Sharing and reuse strategy Study Common frameworks 2012 Ongoing 4.2.06 Interoperable testbed Study Common frameworks 2013 Ongoing 5. Programme Management 5.01 Monitoring and Evaluation Acc. M. Monitoring & Evaluation 2010 Ongoing 5.02 EIS Governance support Acc. M. Management activities 2010 Ongoing 2Methodology The final evaluation’s methodology conforms to the methodological guidance for evaluations provided by DG Budget and adheres to developments in the Smart Regulation policy. It incorporated and expanded on the main evaluation questions, listed in the ‘ Introduction ’ section. It is composed of the following four phases, which are elaborated on in this chapter: Source: European Commission, Evaluating EU Activities: A practical guide for the Commission Services, Office for Official Publications of the European Communities, Luxembourg, 2004. 2.1 Inception (Evaluation structuring)

The inception phase defined the methodology ( Figure 3 ), based on the main evaluation questions and the intervention logic ( Figure 4 , below) from the interim evaluation of the ISA programme. It also identified the relevant stakeholders. A major part of this inception phase was the choice of adequate data collection methods and the design of processes to support their subsequent implementation. In particular, these activities included: The development of specific questions, linked to the main evaluation questions, for the data collection phases, including the identification of related judgement criteria and indicators; and Planning and creation of questionnaires/guides for the interviews and online surveys containing the specific questions. The evaluation team’s significant investment in the organisation of the data collection activities as part of the inception phase was to ensure the reliability of the information on which a sound analysis could be built. The evaluation team drew on its experience and expertise to determine these elements and construct the overall methodology to conduct the final evaluation. 2.1.1Evaluation criteria & main evaluation questions The table below maps the evaluation criteria and their definitions, as contained in the EC’s official guidelines22, to the main evaluation questions of the final evaluation of the ISA programme. Evaluation criterion Evaluation criterion Definition ISA Final Evaluation - evaluation questions (EQ) Relevance The extent to which an intervention’s objectives are pertinent to needs, problems and issues to be addressed. To what extent were the ISA programme's objective(s) pertinent to meet evolving needs and priorities at both national and EU levels? Efficiency The extent the desired effects are achieved at a reasonable cost. How economically were the various inputs converted into outputs? Which aspects of the programme were the most efficient or inefficient, especially in terms of resources mobilised? Effectiveness The extent to which objectives set are achieved. To what extent did the ISA programme's results and impacts achieve its objectives? Are there aspects that were more or less effective than others, and, if so, what lessons can be drawn from this? Utility The extent to which effects correspond with the needs, problems and issues to be addressed How did the ISA programme's achieved and anticipated results and impacts compare with the business needs they intended to address? Which measures could be taken to improve the utility of the next programme’s actions? Sustainability The extent to which positive effects are likely to last after an intervention has terminated. To what extent is the financial, technical and operational sustainability of the developed solutions, maintained and operated through the ISA programme, ensured? Coherence The extent to which the intervention logic is non-contradictory/the intervention does not contradict other interventions with similar objectives. To what extent did the ISA actions form part of a ‘holistic’ approach within the framework of the programme? How well were synergies achieved between programme actions and with other EU activities? Coordination The extent to which interventions are organised to maximise their joint effects by mobilising resources combined with harmonising measures. To what extent did coordination of activities between Member States, including the ISA Committee, exist to ensure stakeholders' engagement in the ISA programme? To which extent were activities coordinated or aligned with the needs of other stakeholders with whom the Commission was supposed to interact in the framework of ISA? Once these main evaluation issues and associated questions were agreed, the evaluation team defined specific sub-questions, linked to each main evaluation question, to be addressed by the principal data collection methods, the interviews and online surveys as well as by desk research. These interview and online survey guides are contained in the Annex, which accompanies this report. 2.1.2Intervention Logic The EC’s official guidelines22 define the intervention logic of a programme as ‘the conceptual link from an intervention's inputs to the production of its outputs and, subsequently, to its impacts on society in terms of results and outcomes.’ As agreed with the European Commission, the intervention logic of the interim evaluation of the ISA programme was reused for the final evaluation. The examination of the programme's intervention logic with data collected during the evaluation is of central importance to identify its success in achieving its objectives. Source: European Commission, Evaluating EU Activities: A practical guide for the Commission Services, Office for Official Publications of the European Communities, Luxembourg, 2004. 2.1.3Stakeholders The ISA unit and the Steering Committee identified the following eight main stakeholder groups as those involved in, or impacted by, the ISA programme. Representatives of these stakeholder groups were involved in the final evaluation’s data collection activities.

Stakeholder Group Description Officials from DG DIGIT – Unit B.6 Interoperability solutions for European public administrations (ISA) The unit that commissioned this evaluation and that has the overall responsibility for the management and delivery of the ISA programme. Members of the ISA Committee, the ISA Coordination Group and ISA Working Groups Actively participate in the programme. Monitoring & Evaluation Action contractors & Project Officers outside DG DIGIT’s Unit B6 Responsible for managing and monitoring the performance of ISA actions. Officials from other Commission services These stakeholders can receive ISA funding to implement ISA actions; work in a DG reusing ISA solution(s); serve as Information Resources Managers (IRMs) or Trans European Systems’ (TES) owners; or be in charge of other EU initiatives linked to the ISA programme (e.g. at DG CNECT or DG TAXUD). Standardisation organisations Can be involved in, or affected by, the ISA programme. Other stakeholders Officials from other EU, local and regional institutions who are involved in, or affected by, the ISA programme in any way. Direct beneficiaries of the ISA actions European public administrations including the Commission services. Indirect beneficiaries Citizens and businesses.



2.2Data collection and analytical tools The data collection methods and analytical tools chosen to answer the evaluation questions are both qualitative and quantitative. The table below provides a summary of these. The Annex accompanying this report contains details on each of these. Method/ tool Description Desk research The main objectives of the desk research for the final evaluation were to: Understand the context. Analyse the objectives, scope, means and instruments of the ISA programme. Gather preliminary inputs on the relevance of the programme as well as on its coherence with other similar EU initiatives. Interviews The major data collection activity of this final evaluation was the series of interviews that the evaluation team conducted with 37 stakeholders. Online surveys A series of online surveys was launched and completed subsequent to the series of interviews. Different stakeholder groups were invited to complete specific surveys. Workshop A specific workshop with officials from the ISA unit to explore the findings from the interviews and the online surveys was held to obtain additional information and feedback, validate the ISA Value Chain and to agree the case studies to be developed. Case Study Two short case studies, based on the State of Play of Interoperability and Semantic Interoperability, were developed in order to provide in-depth analysis and recommendations on how their successes can be further enhanced. 2.3Data analysis and synthesis Data analysis and synthesis is the major activity that assessed the data collected during the research phase. It translates data into understandable findings, from which specific conclusions can be drawn as the basis of final recommendations. The following two steps ensured this. Step Description Triangulation Triangulation of data is defined as ‘the use of data collected using different tools and from different sources, and/or analysis from different theoretical perspectives and by different analysts, and at different time’. Testing of the intervention logic Investigation of the relationship between evaluation issues and expected benefits, such as outputs, outcomes and impacts defined in the intervention logic. 2.4Formulation of judgments and reporting The activity to formulate robust conclusions on the implementation of the ISA programme and provide useful and practical recommendations on future Work Programmes, in particular the ISA² programme, involved the following three activities: Activity Description Outputs Formulation of conclusions and recommendations Conclusions and executable recommendations based on findings from the evaluation’s data collection activities and validated by analysis. List of conclusions linked to findings; and List of recommendations linked to their supporting conclusion(s). Reporting Overview of the objectives, the methodology, analytical tools, key findings, conclusions and recommendations of the final evaluation of the ISA programme Final Report; Executive Summary; and MS PowerPoint presentation. Dissemination of recommendations Results of the final evaluation should be integrated into the Commission’s decision making process to contribute to improvements in policy initiatives, programmes and activities, in particular the ISA² programme, and the instruments used to deliver these. Possible development of an implementation plan to overcome the gaps between the objectives of the ISA programme and its actual achievements. 3Findings and results

This chapter is divided into seven sections in accordance with the seven aforementioned evaluation criteria. 3.1Relevance The evaluation of the relevance of the ISA programme is driven by the following evaluation question (EQ): EQ1: To what extent were the ISA programme's objective(s) pertinent to meet evolving needs and priorities at both national and EU levels? As mentioned in section 1.2, the global objective of the ISA programme is to support cooperation between European public administrations, with a view to enable the delivery of electronic public services supporting the implementation of EU policies and activities. The intermediate objective is to facilitate the efficient and effective electronic cross-border and cross-sectoral interaction between European public administrations. To evaluate whether these objectives were pertinent to meet evolving needs and priorities at both national and EU levels, the evaluation verified the extent to which the needs expressed by European public administrations, at the time of the ex-ante and interim evaluations of the ISA programme, can still be considered as current needs related to interoperability. New evolving needs were also identified in order to assess the extent to which the ISA programme’s objectives have addressed them. On the other hand, the evaluation also verifies the pertinence of the programme’s objectives to main EU priorities such as the Digital Agenda for Europe. The remainder of this section is divided into different subsections in accordance with the different judgement criteria allowing the evaluation to answer EQ1. 3.1.1ISA programme objectives With regard to these aforementioned objectives, there was a clear agreement on their continued validity. In fact, 95% of the interviewees and survey recipients agreed (64) or somewhat agreed (41) that the objectives of the programme were still valid today. Moreover, 11 respondents specifically highlighted that there is still work to be performed to fully reach these objectives. It has also to be noted that some respondents (3) clearly suggested that the objectives of any successor programme should not only focus on public administrations but also on public services targeting citizens and businesses. This recommendation has already been taken into account in the proposal for a decision establishing a programme on interoperability solutions for European public administrations, businesses and citizens (ISA²); the proposed general objective of ISA² is to facilitate interoperability between European public administrations, and also between them and citizens and businesses. After having verified the level of validity of the programme’s objectives, the evaluation also addressed a question particularly focused on the intermediate objective of the programme. In fact, all the respondents were asked about their level of agreement on the extent to which the programme facilitates efficient and effective electronic cross-border and cross-sectoral interaction between European public administrations. 40% fully agreed that the ISA programme facilitated efficient and effective electronic cross-border and cross-sectoral interaction between European public administrations (44), whereas 53% of respondents somewhat agreed (58). The respondents primarily commented that the programme aims to and does facilitate interaction between European public administrations but room for improvement exists to concretely achieve this objective (11). 3.1.2Needs of EU public administrations In order to assess the extent to which the objectives of the ISA programme were pertinent to meet evolving needs at national and EU levels, the evaluation firstly identified the main needs of public administrations at both national and EU levels. In this regard, the evaluation verified the extent to which the needs expressed by European public administrations at the time of the ex-ante and interim evaluations of the ISA programme, shall still be considered as current needs related to interoperability: Existing generic tools, services and infrastructures have to be consolidated and re-used; ICT implications have to be taken into account upfront in the legislative process; Data and information exchange has to be secured; Common frameworks, guidelines and specifications have to be established; The benefits of interoperability have to be more communicated; Coordination between the ISA programme and both the Member States and Commission services has to be increased;

ISA solutions have to be more reused. Figure 5 depicts the views of the interviewees and survey recipients on the extent to which these needs should still be considered as current needs of EU public administrations. Source: Results of primary desk research – interviews and online surveys with key stakeholder groups. As depicted in Figure 5 , a large majority of respondents fully and somewhat agrees that each of the proposed needs should still be considered as current needs of EU public administrations. As a result, the evaluation draws a conclusion that these aforementioned needs have still to be considered as current needs of EU public administrations. In addition to these aforementioned needs, eight additional ones were identified as potential new needs during the interviews. Survey respondents have thus been asked to provide their level of agreement to the question ‘To what extent do you agree that each of the following [eight potential] needs should be considered as a new or evolving need since the adoption of the ISA programme?’ Out of a total of 67 answers, these eight needs are presented in a descending order based on the number of 'Fully agree' answers by EC officials and Member States to the aforementioned question (indicated in brackets following each one): 1.Improvement of the governance and coordination among Commissions DGs and services on programmes and initiatives related to interoperability (40); 2.Improvement of the governance and coordination between Member States and the Commission on programmes and initiatives related to interoperability (32); 3.Adaptation and update of the focus of the ISA programme to incorporate evolutions in technology that could support interoperability (e.g. Big Data and cloud computing) (27); 4.Need to address security and cyber-security of Member States as part of work related to interoperability (26); 5.Adaption and update of the focus of the ISA programme to involve businesses, citizens, civil society as stakeholders for the development and use of solutions (26); 6.Incentives for the use of the ISA programme's solutions (e.g. financial support to Member States for adoption and implementation of solutions until operational) (26); 7.Definition and promotion of key interoperability enablers to foster standardisation at national and cross-border levels (25); 8.Improvement of the coordination and cooperation between Member States on programmes and initiatives related to interoperability (23). Based on this ranking, the evaluation concludes that an improvement of the governance and coordination among Commissions DGs and services but also between the Commission and Member States on programmes and initiatives related to interoperability are perceived as the main two new needs to be considered. The evaluation notes that these two new needs are beyond the scope and control of the existing ISA programme alone and that the need for an ‘increased coordination between the ISA programme and both the Member States and Commission services’ was already identified at the time of the interim evaluation. The establishment of the CEF programme, subsequent to the launch of the ISA programme, was frequently mentioned by interviewees and online survey respondents as a reason to ensure ‘increased coordination between the ISA programme and both the Member States and Commission services’. One should note that the opinions on potential needs leading to the aforementioned ranking were homogenous among both EC officials and Member States for most of the needs, except for two of them. In fact, on the one hand, the ‘Adaptation and update of the focus of the ISA programme to incorporate evolutions in technology that could support interoperability’ was mainly seen as an important need for the EC officials whereas it was the one least cited by Member States. On the other hand, Member States consider the ‘Incentivisation of the use of the ISA programme's solutions’ as one of the most important needs whereas it was the one that was considered as the least important by EC officials.

3.1.3Pertinence of the ISA objectives to respond to needs of EU public administrations The list below presents the extent to which the ISA programme has addressed each of the needs identified at the time of its ex-ante and interim evaluations. These needs are presented in an ascending order based on the number of times that they were recognised as addressed by the ISA programme by the 96 respondents who provided an opinion on this matter as shown in the Figure 6 . Source: Results of primary desk research – interviews and online surveys with key stakeholder groups. On the one hand, it appears that the needs of having ISA solutions more reused as well as an increased collaboration between the programme and both the Member States and Commission services are the two needs that have been the least addressed by the ISA programme. In fact, these needs are not seen as addressed by the ISA programme by more than half of respondents. On the other hand, the following two needs are considered as addressed by the ISA programme by a large majority of respondents (respectively by 72% and 82%): Existing generic tools, services and infrastructures have to be consolidated and re-used (69); Common frameworks, guidelines and specifications have to be established (79). With regard to the needs considered as new ones, i.e. improvement in governance & coordination, since the adoption of the programme (cf. previous subsection 3.1.2), respondents were asked to provide their opinion on how the ISA programme has taken them into account. Based on a total number of 23 answers, the programme has partially taken the aforementioned new needs of European public administrations into account for a majority of respondents (61%) and fully taken them into account for 26% of respondents who provided their opinion on this question. A mere 13% considered that the programme has not taken these new needs of European public administrations into account at all. Nevertheless, 22% of these aforementioned respondents highlighted that the proposed ISA² programme is a good opportunity to take the new evolving needs of European public administrations, i.e. improvement in governance & coordination, more into account. The evaluation confirms that the current programme is anyway currently taking new evolving needs on governance & coordination into account. In fact, the European Commission is in the process of revising, in close cooperation with the Member States, the European Interoperability Strategy (EIS) and European Interoperability Framework (EIF). Indeed, as part of the EIS Revision, the interoperability governance organisation & processes are in the process of being established. This is seen as a fundamental opportunity to address these new evolving needs on governance and coordination.

3.1.4Alignment of the ISA objectives with EU priorities In order to assess the extent to which the objectives of the ISA programme were also pertinent to meet evolving priorities at EU level, the evaluation verified the alignment of the ISA programme with the following EU initiatives: Digital Agenda for Europe (DAE); European eGovernment Action Plan 2011-2015; eCommission 2012-2015; Commission IT Rationalisation Initiative; Connecting Europe Facility (CEF);

Digital Single Market (DSM) Strategy. Of 98 EC officials and Member States’ representatives who responded, most believed that the ISA programme objectives are primarily aligned with the DAE and with the European eGovernment Action Plan 2011-2015 as well as, to a lesser extent, with the CEF programme and DSM strategy, which were established much later than the DAE and the European eGovernment Action Plan, respectively on 11 December 2013 and 6 May 2015. In fact, with regard to the alignment between the ISA programme and the DSM strategy, the majority of the interviews were conducted before the establishment of the DSM strategy. This fact explains the reason why the number of ‘Don’t know/No opinion’ answers is higher for this new strategy compared to the number of ‘Don’t know/No opinion’ for DAE, European eGovernment Action Plan or CEF. 80% of respondents were however able to provide their opinion considering that the draft DSM strategy went through inter-service consultation within the European Commission and was made available to Member States as well. Figure 7 describes the level of alignment of the ISA programme objectives with the aforementioned EU initiatives as perceived by Member States and European Commission officials. These are sorted by the overall number of occurrences of the fully agree answers. Source: Results of primary desk research – interviews and online surveys with key stakeholder groups. The objectives of the programme are perceived as less aligned with the eCommission 2012-2015 and Commission IT rationalisation initiative. However, as highlighted by several respondents, the eCommission 2012-2015 as well as the IT Rationalisation initiative are strategies more focused on internal systems of the European Commission whereas the scope of the programme is broader by nature. In fact, the eCommission 2012-2015 is the European Commission’s IT strategy whereas the Commission IT Rationalisation Initiative is an overall strategy established in October 2010 by the European Commission to rationalise and streamline the IT systems it develops, maintains and operates. The evaluation found that the highest level of disagreement was related to the alignment between ISA and the CEF programme. The majority of respondents in disagreement with the alignment between CEF and ISA singled out a lack of coordination with DG CNECT. This aspect will be further detailed in Section 3.6.3. Nevertheless, the factual evidence that several actions previously funded by the ISA programme are now part of CEF Work Programme, e.g. MT@EC, STORK, implies a good alignment between these two programmes and counterbalances the respondents’ opinions. In addition to the EU initiatives depicted in Figure 7 , respondents were also asked about the alignment of ISA with any other EU initiatives. In that regard, one main additional initiative was identified by five respondents, namely INSPIRE. With regard to the perceived alignment with INSPIRE, the evaluation considers that this can be primarily due to two main reasons. Firstly, in 2012, two new actions linked to the INSPIRE directive, namely AR3NA (1.17) and EULF (2.13) were included in the revised ISA Work Programme. Secondly, as explained in section 1.3, the ISA programme also established, in 2013, in the context of these two ISA actions, a dedicated ISA-INSPIRE Working Group on ‘Spatial Information and Services’. This Working Group plays an important advisory role, helping to define requirements and priorities and to establish the direction of the two aforementioned ISA actions. It aims to help to facilitate dialogue between the e-government and European geospatial communities, in general, and between ISA and the community implementing the INSPIRE directive in particular. The establishment of this Working Group can be seen as a positive result in the light of the new evolving needs identified in subsection 3.1.2, especially the new evolving need for improvement of the governance and coordination among Commissions DGs and services on programmes and initiatives related to interoperability. The remainder of this subsection contains further details on the alignment of the programme with major EU initiatives based on desk research in order to clarify the reality. Specifically, these major EU initiatives are the Digital Agenda for Europe (DAE), the European eGovernment Action Plan 2011-2015, the Connecting Europe Facility (CEF) and the Digital Single Market (DSM) Strategy. Digital Agenda for Europe (DAE) On 19 May 2010, the Commission adopted a Communication proposing a Digital Agenda for Europe, one of the seven flagship initiatives of the Europe 2020 Strategy, set out to define the key enabling role that the use of ICT will have if Europe wants to succeed in its ambitions for 2020. Considering that the Digital Agenda for Europe is composed of 132 actions divided into seven pillars, the evaluation aims to verify the pertinence of the ISA programme to the DAE pillars. In this respect, the ISA programme is primarily aligned with the following three DAE pillars: Pillar I: Digital single market; Pillar II: Interoperability and standards;

Pillar VII: ICT-enabled benefits for EU society. Based on evidences provided by the ISA Work Programme revisions and the interim evaluation of the programme, the evaluation confirms that several ISA actions are pertinent to DAE actions under these pillars. For each identified pillar, the specific DAE action is linked with the relevant ISA action(s) in Table 8 . Further details about this alignment are available in Annex. DAE action Related ISA action(s) Pillar I Digital Single Market Action 3 - Open up public data resources for reuse aims at opening up public data resources for reuse by reviewing the PSI Directive. 1.01 - Promoting semantic interoperability amongst the European Union Member States 1.02 - Access to base registers 1.15 - Open Government Data 1.22 - Big Data and Open Knowledge for public administrations 2.18 – Participatory knowledge for supporting decision making Action 8 - Revision of the eSignature Directive aims at proposing a revision of the eSignature Directive with a view to provide a legal framework for cross-border recognition and interoperability of secure eAuthentication systems. 1.05 - STORK Sustainability 1.09 - Supporting tools for TSL and e-signature creation/verification Action 107 - Proposals to strengthen the data industry in Europe 1.21 - European Legislation Identifier 1.22 - Big Data and Open Knowledge for public administrations Pillar II - Interoperability and standards Action 23 - Provide guidance on ICT standardisation and public procurement aims at issuing a Communication in 2011 to provide guidance on the link between ICT standardisation and public procurement to help public authorities use standards. 2.02 – CAMSS 2.16 - European Single Procurement Document Service 2.17 – eCertis Action 24 - Adopt a European Interoperability Strategy and Framework aims at promoting interoperability by adopting in 2010 a European Interoperability Strategy (EIS) and European Interoperability Framework (EIF). 4.2.03 – NIFO 5.02 - EIS Governance support Action 26 - MS [Member States] to implement European Interoperability Framework suggests Member States to apply the European Interoperability Framework at national level by 2013. 4.2.03 - NIFO Action 27 - Member States to implement Malmö and Granada declarations 1.08 – e-TrustEX 4.2.03 - NIFO Pillar VII - ICT-enabled benefits for EU society Action 76 - Propose a recommendation to define a minimum common set of patient data 1.01 - Promoting semantic interoperability amongst the European Union Member States Action 77 - Foster EU-wide standards, interoperability testing and certification of eHealth aims at fostering EU-wide standards, interoperability testing and certification of eHealth systems by 2015 through stakeholder dialogue 2.12 - eHealth European Interoperability Framework 4.2.06 - Interoperability Testbed Action 84 - Support seamless cross-border eGovernment services in the single market 1.05 - STORK Sustainability 1.06 - Common Infrastructure for public administrations Sustainability (CIPA) 1.14 - Cross Sector SOLVIT 2.04 - sTESTA 4.2.06 - Interoperability Testbed Action 88 - Create and implement an ambitious eCommission 2011-2015 Action Plan. 1.07 - e-PRIOR Action 89 - Member States to make eGovernment services fully interoperable 1.01 - Promoting semantic interoperability amongst the European Union Member States 1.08 – e-TrustEx 2.01 - EIA 4.1.02 - Interoperability Maturity Model 4.2.01 - ISA Integrated Collaborative Platform 4.2.03 - NIFO 4.2.06 - Interoperability Testbed Action 110 - Deploy and roll out digital services in key areas of public interest 1.07 - e-PRIOR 1.10 - Internal Market Information (IMI) System 1.14 - Cross Sector Solvit 2.12 - eHealth European Interoperability Framework The Digital Agenda for Europe was updated at the end of 2012 and additional actions have been included under its umbrella as of 2013. Compared to the alignment that was already observed at the time of the interim evaluation between the ISA programme and the first set of actions of the DAE, it should be noted an alignment between ISA actions and the aforementioned additional DAE actions. In fact, ISA actions included in the first ISA Work Programmes (e.g. ISA action - 1.07) as well as new actions included after 2012 are aligned with the initial (e.g. alignment between ISA action 4.2.06 and the DAE action 77) as well as updated version of the DAE (e.g. ISA action 1.21 and 1.22 with the new DAE action 110). European eGovernment Action Plan 2011-2015 In order to support and complement the Digital Agenda for Europe, the European Commission proposed a European eGovernment Action Plan 2011-2015 aimed to support the provision of a new generation of eGovernment services for businesses and citizens. The Action Plan identifies four political priorities (user empowerment, internal market, efficiency and effectiveness of eGovernment and administrations, pre-conditions for developing eGovernment) based on the Malmö Declaration, agreed on 18 November 2009 at the 5th Ministerial eGovernment Conference in Malmö, Sweden, and proposes concrete actions and measures to be implemented by the Commission and the Member States by 2015. As described in Table 9 , the ISA programme is also aligned with these four priorities. Further details about this alignment are available in Annex. Priority Action Related ISA action(s) User Empowerment 1.2 - Collaborative Production of Services 4 - Exchange of knowledge and expertise, agreement on common targets for the roll out of collaborative services 1.18 Federated Authorisation Across European public administrations 1.5 - Involvement of citizens and businesses in policy-making processes 12 - Development of the electronic service to support ‘citizens initiatives’ 1.12 - Open source software for online collection of statements of support for European citizens' initiatives[1] 2.1 - Seamless Services for Businesses 15 - Outcomes assessments of PEPPOL and SPOCS 1.06 - CIPA Sustainability Internal Market 2.2 - Personal mobility 19 - Exchange of best practice and coordination of the efforts to jointly develop and set up interoperable eDelivery services. 1.03 - Catalogue of services 20 – Provision of cross-border and interoperable eDelivery services for citizens 1.10 - Internal Market Information (IMI) System 1.14 - Cross Sector SOLVIT 2.3 - EU-wide implementation of cross-border services and new services 21 - Study of the demand for cross-border services, assessment of the organisational, legal, technical and semantic barriers 1.01 - Promoting semantic interoperability amongst the European Union Member States Efficiency and Effectiveness of eGovernment and Administrations 3.1 - Improving Organisational Processes 25 - Facilitate exchanges of experience, successful solutions and applications, exploring new approaches improving organisation process 4.2.01 - ISA Integrated Collaborative Platform 4.2.02 - Community building and effective use of the collaborative platforms 4.2.03 – NIFO 3.2 - Reduction of Administrative Burdens 29 - Sharing of experiences on implementation of the ‘once only’ registration principle and on electronics procedures and communications, cost-benefit analysis and roadmap design 1.02 - Access to base registries 1.18 - Federated Authorisation Across European public administrations Pre-conditions for developing eGovernment 4.1 - Open Specifications and Interoperability 32 - Put into the European Interoperability Framework (EIF) and the European Interoperability Strategy (EIS) 4.2.03 – NIFO 5.02 - EIS Governance support 33 - Exchanges of expertises, promotions of the reuse and sharing solution to implement interoperable eGovernment services 1.08 - Trusted Exchange Platform (E-TRUSTEX) 4.2.01 - ISA Integrated Collaborative Platform Joinup 4.2.02 - Community building and effective use of the collaborative platforms 4.2.03 - NIFO 34 - Alignment of national interoperability frameworks to the EIF 2.15 - Interoperability agreements on electronic document and electronic file 4.2.03 - NIFO 5.02 - EIS Governance support Looking forward, it is important to note that the ISA programme is in the process of being aligned with the future eGovernment Action Plan 2016-2020.This is being developed within the framework of the Digital Single Market strategy, which is covered below in this sub-section. As part of this, the ISA programme participated in two workshops on the new EU eGovernment Action Plan in June and September 2015. The eGovernment Action Plan 2016-2020 will include an initiative on the application of the ‘once-only principle’ (OOP), which means that citizens and businesses supply information to public administrations only once. The application of this principle is quite difficult in a cross-border context, considering that it is already challenging for Member States to apply in a national context, which is obviously a pre-requisite. Nevertheless, it can be achieved technically through a pilot with Member States to achieve the interconnection of databases, i.e. national registers, to avoid asking citizens and businesses for information more than once. Evidently, the ISA programme has a role to play here, in particular, given its work under action 1.02 Access to Base Registries. In addition, another intended pillar of the future eGovernment Action Plan 2016-2020 is to accelerate Member States' transition towards full e-procurement and interoperable e-signatures, where the work done under actions 1.07 and 1.09 respectively provides the ISA programme with great potential to contribute. Connecting Europe Facility (CEF) The Connecting Europe Facility (CEF) is the common financing instrument of Trans-European networks for the period 2014-2020. During this period, CEF will help to complete the European single market by making available EUR 33.24 billion in the form of procurement, grants and innovative financial instruments. CEF will finance projects of common interest in three different sectors: transport (EUR 26.25 billion), energy (EUR 5.85 billion) and telecommunications (EUR 1.14 billion). Within the telecommunications area, 170 million euros are earmarked for Broadband activities, while 970 million euros are dedicated to Digital Service Infrastructures (DSIs) delivering networked cross-border services for citizens, businesses and public administrations. These projects are to:

improve the competitiveness of the European economy; promote the interconnection and interoperability of national, regional and local networks; and facilitate access to such networks, thus supporting the development of a Digital Single Market. After careful consideration of the respective Work Programmes, the evaluation confirms that several actions included in ISA’s 2015 Work Programme are pertinent to 7 out of 11 planned CEF’s digital service infrastructures (DSIs) under its 2014 and 2015 Work Programme. Table 10 summarises the alignment of the ISA programme with the old and new DSIs as included in the Work Programmes (WP) for 2014 or 2015 respectively. Further details about this alignment are available in Annex. As shown in this table, most of the link existing between ISA actions and DSIs included in the CEF 2014 WP are due to the fact that several mature ISA solutions have been taken over by the CEF programme as of 2014. DSI Related ISA action(s) ISA solutions taken over by CEF DSIs already supported under CEF WP 2014 Electronic identification and authentication — eIdentification and eSignature (Generic Services) 1.04 - ECAS-STORK integration 1.05 – STORK Sustainability ✓ 1.09 - Supporting tools for TSL and e-signature creation/verification ✓ 1.18 - Federated Managed Authentification Services for ECAS 4.2.06 – Interoperability Testbed eDelivery (Generic Services) 1.06 - CIPA ✓ 2.04 - Data Communication Network Service (sTESTA) 4.2.06 – Interoperability Testbed eInvoicing (Generic Services) 1.07 – e-Prior 4.2.06 – Interoperability Testbed Open Data (Generic Services) 1.01 - Promoting semantic interoperability amongst the European Union Member States 1.02 - Access to base registers 1.15 - Open Government Data Automated Translation (Core Platform) 2.08 - Machine Translation Service by the European Commission ✓ New DSIs in WP 2015 eHealth (Core Platform; Generic Services) 2.12 - eHealth European Interoperability Framework eProcurement (eCertis) (Generic Services) 1.07 - e-Prior 2.16 - European Single Procurement Document Service 2.17 – eCertis Digital Single Market (DSM) Strategy The Digital Single Market Strategy, recently adopted by the Commission on 6 May 2015, aims to open up digital opportunities for people and business and enhance Europe's position as a world leader in the digital economy. The DSM strategy includes a set of 16 targeted actions to be delivered by the end of 2016 and is built on the following three pillars: Pillar I: Better access for consumers and businesses to digital goods and services across Europe; Pillar II: Creating the right conditions and a level playing field for digital networks and innovative services to flourish; and Pillar III: Maximising the growth potential of the digital economy. The evaluation highlights that the ISA programme remains relevant primarily for pillar III as some of its initiatives have links to ISA actions. Table 11 summarises the pertinence of the ISA programme towards this pillar. However, the evaluation would like to raise the following caveat. Given that the DSM strategy was only adopted in May 2015, after the adoption of the latest ISA Work Programme, the alignment of the ISA programme with its actions and priorities is difficult to capture. Further details about this alignment are however available in Annex. Actions Related ISA action(s) Pillar III. Maximising the growth potential of the digital economy 15. Adoption of a Priority ICT Standards Plan and extending the European Interoperability Framework for public services 1.01 - Promoting semantic interoperability amongst the European Union Member States 2.01 – European Interoperability Architecture (EIA) 2.02 - CAMSS - Common Assessment Method Standards and Specifications35 4.2.03 – National Interoperability Framework Observatory (NIFO) 16. New e-Government Action Plan including an initiative on the 'Once-Only' principle and an initiative on building up the interconnection of business registers 1.02 – Access to base registries Overall, the most significant relevance is related to action 15 from the DSM: in the roadmap of the recently published Communication, it is mentioned the need to revise and extend the European Interoperability Framework (EIF). The ISA programme is, at the time of this report, in the process of revising the European Interoperability Strategy and, through the NIFO action, extending the European Interoperability Framework. On the other hand, the results of action 1.02 Access to Base Registries, i.e. best practices and recommendations, are in line with the future initiative on the 'Once-Only' principle, which is part of action 16 of the DSM. 3.1.5Conclusions

The following box summarises the main conclusions related to EQ1: ‘To what extent were the ISA programme's objective(s) pertinent to meet evolving needs and priorities at both national and EU levels?’ Conclusion N°1 As mentioned, the ISA programme aims to support cooperation between European public administrations, with a view to enabling the delivery of electronic public services supporting the implementation of EU policies and activities. The evaluation considers this objective as still valid today. Even if room for improvement exists to concretely achieve this objective (see subsections 3.2 to 3.4), it is confirmed that the programme does facilitate interaction between European public administrations. The modified scope of the follow-on ISA² programme will, most likely, enhance this objective. The follow-on ISA² programme’s scope aims to facilitate electronic cross-border or cross-sector interaction between European public administr