EUROPEAN COMMISSION Brussels, 23.9.2019 SWD(2019) 1615 final COMMISSION STAFF WORKING DOCUMENT Interim evaluation of the ISA² programme Accompanying the document REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL Results of the interim evaluation of the ISA² programme {COM(2019) 615 final}



Table of contents 1.Introduction 1.1. Purpose and scope 2.Background to the intervention 2.1. Description of the intervention and its objectives 2.2. Baseline and points of comparison 3.Implementation / State of Play 3.1. Description of the current situation 4.Method 4.1. Short description of methodology 4.2. Limitations and robustness of findings 5.Analysis and answers to the evaluation questions 5.1. Relevance 5.2. Effectiveness 5.3. Efficiency 5.4. Coherence 5.5. EU added value 5.6. Utility 5.7. Sustainability

6.Conclusions 6.1. Context 6.2. Method 6.3. Findings 6.4. Next steps Annex 1: Procedural information Annex 2: Synopsis report of the consultation activities Annex 3: Intervention Logic Annex 4: Evaluation Framework Annex 5: Sample of actions Annex 6: Supporting evidence from desk research Glossary Term or acronym Meaning or definition ADMS Asset Description Metadata Schema CEA cost-effectiveness analysis CEF Connecting Europe Facility CEN European Committee for Standardisation CENELEC European Committee for Electrotechnical Standardisation CEPS Centre for European Policy Studies CPSV-AP Core Public Service Vocabulary Application Profile (CPSV-AP) DCAT-AP Data Catalogue Vocabulary Application Profile for Data Portals in Europe DG Directorate-General DG DIGIT Directorate-General for Informatics DIGIT.D2 Directorate-General for Informatics, Directorate D Digital Services, Unit D2 Interoperability DSM Digital Single Market EEA European Economic Area EFTA European Free Trade Association EIF European Interoperability Framework EIRA European Interoperability Reference Architecture ELISE European Location Interoperability Solutions for e-Government EQ evaluation question ERDF European Regional Development Fund ESF European Social Fund ESIF European Structural and Investment Funds ESPD European Single Procurement Document EU European Union EVM earned value management Horizon 2020 EU framework programme for research and innovation (2014-2020) IAP Interoperability Action Plan (Annex I to the Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions: European Interoperability Framework — Implementation Strategy. Interoperability Action Plan, Brussels, 23.3.2017, COM(2017) 134 final) ICT information and communication technology IDA Programme on Interchange of Data between Administrations IDABC Programme on Interoperable Delivery of Pan-European eGovernment Services to public administrations, Business and Citizens IMAPS Interoperability Maturity Assessment of a Public Service Interoperability As explained in Article 2(1) of the ISA2 Decision , ‘interoperability’ means the ability of diverse organisations to interact towards mutually beneficial and agreed common goals. It involves the sharing of information and knowledge between the organisations, through their business processes and by means of the exchange of data between their respective ICT systems. ISA Programme on interoperability solutions for European public administrations ISA2 Programme on interoperability solutions and common frameworks for European public administrations, businesses and citizens ISA2 actions webpage https://ec.europa.eu/isa2/actions_en ISA2 dashboard https://ec.europa.eu/isa2/dashboard/ ISA2 Decision L 318/1 Decision (EU) 2015/2240 of the European Parliament and of the Council of 25 November 2015 establishing a programme on interoperability solutions and common frameworks for European public administrations, businesses and citizens (ISA2 programme) as a means for modernising the public sector, Brussels 4.12.2015. ISA2 solutions webpage https://ec.europa.eu/isa2/solutions_en ISA2 website https://ec.europa.eu/isa2/isa2_en ISSG Inter-Service Steering Group IT information technology JC judgment criterion Joinup collaborative platform facilitating the sharing and reuse of IT solutions developed for public administrations NIFO National Interoperability Framework Observatory OECD Organisation for Economic Cooperation and Development PMKI Public Multilingual Knowledge Management Infrastructure for the Digital Single Market REFIT the European Commission's regulatory fitness and performance programme RegDel Inter-institutional Register of Delegated Acts SCM standard cost model SEMIC The ISA2 programme action that promotes semantic interoperability among EU Member States SPI schedule performance index SRSP Structural Reform Support Programme TOOP the ‘once-only’ principle project, http://www.toop.eu/ UNPAN United Nations Public Administrations Network VIES Value Added Tax Information Exchange System W3C World Wide Web Consortium 1.Introduction Purpose and scope ISA2 (interoperability solutions for European public administrations, businesses and citizens) is an EU programme that supports the development of digital solutions and common frameworks that enable public administrations, businesses and citizens in Europe to benefit from interoperable cross-border and cross-sector public services. The programme’s primary beneficiaries are European public administrations at all levels - EU, national, regional and local levels - by reusing the solutions offered by ISA2 they can provide better – more interoperable, user-centric and digital – public services. However, the programme also benefits a wider group of stakeholders, including EU businesses and citizens. ISA2 is open to EU Member States, other countries of the European Economic Area and candidate countries. In addition to the 28 EU Member States, three other countries take part in the programme: Iceland, Norway (since 2016) and Montenegro (since 2018). The programme encourages cooperation with other non-EU countries and with international organisations and bodies.

The ISA2 programme — established by the ISA2 Decision — has a budget of €130.9 million and runs for five years from 1 January 2016 until 31 December 2020. In line with Article 13(3) of the ISA2 decision, the Commission must carry out an interim evaluation of the ISA2 programme by 30 September 2019. Besides fulfilling this legal obligation, the evaluation aims to improve the implementation of ISA2 and any of its successor programmes. The evaluation covers ISA2 activities from the programme’s start until the third quarter of 2018 in all participating countries. The evaluation framework builds on the five evaluation criteria stemming from the Commission’s better regulation requirements: relevance, effectiveness, efficiency, EU added value and coherence. As shown in Box 1 , it also addresses utility and sustainability in line with the ISA2 Decision.



Box 1: Evaluation framework The evaluation was based on an evaluation framework composed of seven evaluation criteria and 10 evaluation questions (EQs). The evaluation criteria and the EQs are summarised below: Relevance refers to the alignment between the programme's objectives and the evolving needs and problems experienced by stakeholders. EQ1: To what extent are the objectives of the ISA² programme still pertinent in relation to the evolving needs and problems at both national and EU levels? Effectiveness focuses on the extent to which the ISA² programme has achieved the objectives it intended to achieve, and generated the results it intended to produce. EQ2: How far are the ISA² programme’s results in the process of achieving the programme’s objectives? EQ3: Are there aspects that are more or less effective than others are, and if so, what lessons can be drawn from this? Efficiency concerns the minimisation of costs borne by various stakeholders in achieving the objectives/results identified under the ‘effectiveness’ criterion. EQ.4: To what extent has the programme been cost-effective? EQ.5: Which aspects of the programme are the most efficient or inefficient, especially in terms of resources mobilised? EQ5.1: How is the programme performing relative to the planned work and budget? Coherence is a measure of the degree to which the actions supported by the ISA² programme are consistent with each other (internal coherence) and with the EU policy framework at large and relevant global initiatives (external coherence). EQ.6: To what extent do the ISA² actions form part of a ‘holistic’ approach within the framework of the programme? (Internal coherence) EQ.7: To what extent is the ISA² programme coherent with other EU interventions, which have similar objectives and with global initiatives in the same field? (External coherence) EU added value captures the programme's impacts additional to those that would be achieved if the issues addressed by ISA2 were left solely in the hands of national and sub-national authorities. EQ.8: What is the additional value resulting from the ISA² programme, compared to what could reasonably have been expected from Member States acting at national, regional and/or local levels? Utility refers to (i) the extent to which the results of ISA2 meet stakeholders’ needs and (ii) the level of stakeholder satisfaction with the solutions offered by ISA2. EQ.9: How do the ISA² programme’s actions and results, achieved and anticipated, compare with the needs they are supposed to address? Sustainability measures the likelihood of the ISA² programme's results lasting beyond its completion. EQ.10: To what extent is the financial, technical and operational sustainability of the developed solutions — maintained and operated through the ISA² programme — ensured? For further details, please see Annex 4: Evaluation Framework . 2.Background to the intervention Description of the intervention and its objectives The ISA2 programme’s ultimate objective is to promote the ICT-based modernisation of the public sector in Europe and to facilitate addressing the needs of businesses and citizens, via improved interoperability of European public administrations. More specifically the programme aims to do the following: Facilitate efficient and effective electronic cross-border or cross-sector interaction between European public administrations, businesses and citizens. Contribute to the development of a more effective, simplified and user-friendly e-administration at the national, regional and local administration levels. Promote a holistic approach to interoperability in the EU, by identifying, creating and operating interoperability solutions and facilitating their reuse by European public administrations. This will support the implementation of various EU policies and activities. By working towards achieving the above objectives, the programme intends to address the problem of existing or emerging ‘electronic barriers that impede the proper functioning of the internal market’. It is important to note that ISA2 is part of a wider policy framework related to the digitalisation of public administrations in the EU. In cooperation with the Member States and the Commission, the ISA2 programme promotes and maintains the European Interoperability Framework (EIF) , which has been in place since 2010 and was revised in 2017. The revision of the EIF was called for in the Communication on a Digital Single Market Strategy for Europe , which recognised the role of interoperability in cross-border and cross-sector connections between communities, public services and public administrations. Other initiatives related to the digitalisation of public administrations to which ISA2 contributes include: The eGovernment Action Plans (the current version of which runs from 2016 to 2020), which aim to make ‘public administrations and institutions in the EU open, efficient and inclusive, providing borderless, personalised, user-friendly, end-to-end digital public services to all citizens and businesses in the EU’. The 2017 Tallinn Declaration on eGovernment , a declaration made by ministers in charge of eGovernment policy across the EU to spell out their commitment to a number of principles, including ‘interoperability by default’, which aims to ensure that public services are designed to work seamlessly within the single market and across organisational silos, relying on the free movement of data and digital services in the EU. The Connecting Europe Facility (CEF) established in 2013 as a funding programme providing support to cross-border interaction between the digital services infrastructures of Member States. Based on interoperability agreements between Member States, CEF Digital provides building blocks that ensure that citizens, businesses and administrations can benefit from seamless digital public services across the EU. The Single Digital Gateway , to provide citizens and businesses with online access to information and procedures and to assistance and problem-solving services. The Structural Reform Support Programme (SRSP) , which provides support to Member States in five main reform areas, including IT capacity building in public administrations. ·Several other EU funding programmes, like Horizon 2020 (dedicated to research and innovation), the European Social Fund (ESF) and the European Regional Development Fund (ERDF).

Chapter 5 provides further details of the links between ISA2 and the above-mentioned programmes and initiatives, under the coherence evaluation criterion (section 5.4). The intervention logic aims to clarify the reasoning followed by EU decision makers when establishing the ISA2 programme. It includes a detailed description of: ·the needs, problems and drivers that the programme intended to address; ·the objectives set out for the programme (following a three-level hierarchy: global objective, specific objectives and operational objectives); ·the inputs or activities of the programme; ·the programme's expected results; and ·the logical links between these components. The intervention logic elements and links provide a benchmark against which the interim evaluation can assess the programme's achievements in the first half of its period of operation. Figure 1 provides an overview of the elements of the intervention logic and the links between them, and further details are provided in Annex 3: Intervention Logic . Figure 1: Intervention logic of the ISA2 programme Baseline and points of comparison ISA2 is the fifth in a series of EU programmes providing and promoting interoperability solutions for public administrations in the EU. ISA2 succeeded the ISA programme, which implemented 51 actions during its 6-year period of operation (from 2010 to 2015) with a total budget of around €164 million. The results of ISA represent the main baseline used for the purpose of this interim evaluation. The end of the ISA programme was marked by a final evaluation, which took stock of the programme’s achievements and put forward three recommendations for the future edition of the programme, focusing on the below areas: communication with public administrations and raising their awareness of interoperability; a holistic approach to and cross-cutting view of interoperability; and ·cooperation with other EU policies and initiatives. Although the present interim evaluation focuses on results achieved under ISA2, it also addresses the above-listed recommendations. It is also worth noting that ISA2 aims to achieve its objectives through a number of actions that continue from the previous edition of the programme, as well as through new actions. Where relevant, the impact of ISA on ISA2 was assessed by comparing the performance of actions continued from ISA with that of actions started during ISA2.



3.Implementation / State of Play Description of the current situation

ISA² has been operational from 1 January 2016 and will continue to run until 31 December 2020. It has a total budget of around €131 million. ISA2 consists of nine groups of actions, called packages, which are defined on a yearly basis through an annual revision of the rolling work programme (rolling work programme): key and generic interoperability enablers; semantic interoperability; access to data / data sharing / open data; geospatial solutions; eProcurement / eInvoicing; decision-making and legislation; EU policies — supporting instruments; supporting instruments for public administrations; and ·accompanying measures. The 2018 rolling work programme consists of 53 actions, categorised into the nine packages mentioned above. ISA2 managed 39 actions in its first year of operation, 43 actions in 2017, 53 actions in 2018, and is currently managing 54 actions under the 2019 rolling work programme. Table 1 provides a breakdown of actions and allocated budget per package and per year. As ISA2 is scheduled to end on 31 December 2020, one additional rolling work programme remains to be adopted. The present evaluation is based on the rolling work programme implemented in 2016, 2017 and 2018, as the 2019 rolling work programme had not yet been adopted when the data collection took place. Table 1: Overview of allocated budget and number of actions per package from 2016 to 2018 Package Allocated budget 2016 (thousand €) Actions 2016 Allocated budget 2017 (thousand €) Actions 2017 Allocated budget 2018 (thousand €) Actions 2018 Key and generic interoperability enablers 4 900 6 5 407 6 4 250 7 Semantic interoperability 2 008 3 1 831 3 1 503 4 Access to data / data sharing / open data package 2 800 5 3 548 5 3 925 7 Geospatial solutions 983 1 2 240 1 1 900 1 eProcurement / eInvoicing 2 400 1 1 445 1 1 063 1 Decision-making and legislation 2 260 6 2 608 9 2 235 10 EU policies — supporting instruments 2 935 3 3 580 3 4 820 5 Supporting instruments for public administrations 4 425 12 3 533 13 5 315 16 Accompanying measures 1 280 2 730 2 1 290 2 Total 23 991 39 25 545 43 26 301 53 Source: ISA2 rolling work programmes (2016, 2017, 2018), indicative planning and financial overview. The programme in its entirety is overseen by the Interoperability Unit of DG DIGIT (DIGIT.D2), while individual actions are implemented by different Commission services, depending on the thematic scope. The Member States are also involved in programme governance through two channels: the ISA² Committee, the programme's high-level governing body, and the ISA² Coordination Group, a technical body mandated to ensure coherence between the programme’s actions. All 28 EU Member States participate in the programme. Beyond the EU, ISA2 has three additional countries — Iceland, Montenegro and Norway — participate in ISA2. In addition, an administrative agreement between DG DIGIT and the Agency for the Development of the Government of Electronic Management and the Information and Knowledge Society of Uruguay has been in place since March 2018. ISA2 actions are selected and implemented through a process consisting of four steps taken each year: Submission: Commission services, Member States, and other countries participating in ISA2 are invited to submit proposals for actions to be included in the rolling work programme via a call for proposals. Evaluation and consultation of the ISA2 Committee: The submitted proposals are analysed by DIGIT.D2, which compiles a list of proposals that qualify to be included in the rolling work programme, based on the selection and prioritisation criteria as defined in the ISA2 Decision. The ISA2 Committee gives an opinion on the proposed action list. Adoption: The annual rolling work programme is adopted by the Commission and the budget is released. ·Implementation: The actions in the adopted annual rolling work programme are implemented by the Commission services in charge.

The programme's overall performance and the progress of each action are frequently recorded through the quarterly and annual monitoring and evaluation reports, which feed into the ISA2 dashboard. The dashboard is an online interactive tool that facilitates the dissemination of information about action activities and achievements, and their efficiency, effectiveness and coherence. The efficiency of actions is measured using the earned value management (EVM) analysis (see section 5.3 Efficiency ), effectiveness is presented in terms of performance indicators (see section 5.2 Effectiveness ), and the coherence of actions is mapped using network analysis (see section 5.4 Coherence ). Figure 2 illustrates the performance of the ISA2 programme until Q1 2019, as captured by the ISA2 dashboard based on data from the EVM mechanism. Figure 2: Earned value analysis at programme level Note: The cumulated planned value is the sum of the planned values of the programme's different actions for which the EVM is used. The cumulated earned value is the sum of the earned values of the programme's different actions for which the EVM is used. The budget at completion (BAC) is the sum of all undergoing and completed specific contracts financed by the ISA2 budget. Source: Monitoring team of the ISA2 programme (see also the Efficiency view of the ISA2 Programme ).



4.Method Short description of methodology As a first step, the Commission outlined and agreed on the draft evaluation design (including the intervention logic and the evaluation questions). It then reached out to external experts and asked them to support the evaluation process (see Annex 1 ). The consultant refined the evaluation design and — with the help and under the close monitoring of the Commission — moved to action. First, it collected data, then validated and analysed them, as explained in the following sections. At the end, the consultant summarised its findings in an independent evaluation study (referred to as the CEPS final study in this report). Data collection The data collection phase involved a mix of quantitative and qualitative data collection methods and aimed to gather the evidence base to respond to the evaluation questions (EQs) specified in the evaluation framework ( Annex 4 ). More specifically, data were drawn from two main sources: consultation activities (primary data) and desk research (secondary data). Primary data Primary data were collected between 29 November 2018 and 1 March 2019 via the following consultation activities: short questionnaire distributed during the ISA2 Mid-Term Conference and the kick-off workshop; public consultation; targeted online surveys; targeted in-depth interviews; and ·kick-off workshop. These activities resulted in a total of 129 responses from stakeholders, out of which 14 replies arrived during the public consultation. The low consultation response rate is mainly explained by two reasons: ISA2 is a technical programme producing solutions addressed mainly to European public administrations. Understanding how the programme works requires specific knowledge in the field. Therefore, direct contact between the average citizen/business and the programme tends to be very limited, and this most probably led to the reduced response rate. In order to increase the participation of citizens/businesses, a short questionnaire with the same questions as those of the public consultation was distributed during the ISA2 Mid-Term Conference, which resulted in 15 additional responses. It is worth noting, however, that businesses and citizens benefit from ISA2 solutions and activities indirectly, through their interactions with public administrations. Thus, a low response rate among these groups of stakeholders does not reflect negatively on the programme.

·Stakeholder groups that are in direct contact with the programme (e.g. solution users, action owners, etc.) were consulted via targeted online surveys and interviews, which covered inter alia the same topics addressed by the public consultation. Therefore, stakeholders belonging to these groups were asked not to respond to the public consultation in order to avoid duplicate answers. Despite the relatively low response rate, the consultation activities allowed for a wide coverage of the different stakeholders of the ISA2 programme, ranging from actors involved in programme governance to citizens, i.e. the indirect beneficiaries of ISA2. Moreover, on average, the respondents were characterised by a high level of expertise, which contributed to the quality and reliability of primary data collected. The in-depth interviews and targeted surveys provided particularly useful input to the interim evaluation. They were tailored to stakeholders closely related to the programme and allowed them to provide extensive details, thus making it possible for the evaluation to get deep insights into the subject areas covered. In general, primary data contributed to the assessment of all evaluation criteria. Overall, the answers collected are consistent across stakeholder groups. For more details, please consult Annex 2: Synopsis report of the consultation activities , which presents a breakdown of responses by consultation activity and stakeholder category along with respondent characteristics. Chapter 5 of this report presents the results of the consultation activities using mainly bar charts with averages. The results of the public and targeted consultation activities are combined and presented in aggregated form throughout the evaluation. For each question, average scores were computed for each stakeholder group that provided feedback on the specific question. Averages do not take into account those respondents who answered ‘don’t know/no opinion’ to the question under analysis. The data labels of the bar charts show the average score first, then the number of respondents in brackets. Secondary data The desk research reviewed the following resources: publicly available documents and data sources, including the ISA2 annual rolling work programmes, the ISA2 dashboard, and the webpages dedicated to actions and solutions; and ·additional operational documents including monitoring and evaluation reports, overviews of communication activities, and lists of participants in the ISA2 Committee and Coordination Group. Annex 6 provides an overview of the supporting evidence collected from desk research. Note that, in order to better guide the data collection activities, a sample of 20 actions was selected from the total of 53 actions funded by the ISA2 programme up to 2018. The criteria used for sampling and the sampled actions are presented in Annex 5: Sample of actions . The desk research, review of the programme's operational documents and targeted in-depth interviews focused on the 20 sampled actions. However, stakeholders involved in all sampled actions had the opportunity to respond to either the targeted online surveys or the public consultation, to participate in the kick-off workshop and to complete the short questionnaire distributed during the ISA2 Mid-Term Conference. The data collection was complemented by an expert assessment of the programme, performed by four technical interoperability experts who were part of the consultant’s evaluation team. Data validation The collected data were validated via triangulation in order to ensure the robustness of evidence. The Commission’s Better Regulation Toolbox defines triangulation as ‘the application and combination of several research methodologies in the study of the same phenomenon’. In fact, for all evaluation criteria and questions, data were collected from multiple sources and via at least two different data collection methods (e.g. interviews, targeted questionnaire, public consultation, desk research). Quantitative methods of data analysis Besides the statistical analysis of data collected via consultation activities, desk research and the qualitative analysis of open responses provided by consulted stakeholders and information contained in documentary evidence, three specific quantitative methods were used in order to evaluate the programme's efficiency: The standard cost model (SCM) is a method of assessing administrative costs imposed by rules or policies inter alia on businesses and public administrations. It is based on the identification of the basic components of a rule, the information obligations, whose costs for the addressees can be measured and quantified. An information obligation is a specific duty to gather, process or submit information to the public authority or a third party. The SCM was used to measure the costs borne by action owners in preparing and submitting proposals for ISA2 actions. Cost-effectiveness analysis (CEA) is a method of assessing the merits of a policy in an interim and ex post evaluation setting. In a nutshell, CEA measures the value-for-money of past policies, i.e. the amount of benefit generated by unitary costs. Costs are measured in monetary terms, whereas effectiveness is measured in ‘natural units’, and the unit of account varies depending on the nature of the problem addressed (e.g. the number of users of key and generic interoperability enablers). The heterogeneity of performance indicators available for ISA2 actions makes it difficult to draw conclusions about the programme's overall cost-effectiveness and thus using CEA was only possible for certain action packages and indicators, as described in Chapter 5. ·In line with the PM2 methodology developed by the Commission, the earned value management (EVM) and earned schedule (ES) methods are currently used to monitor and assess the efficiency of the ISA2 programme. EVM is a project management technique that helps determine work progress against a given baseline, so that costs, time, and scope of a certain activity are constantly tracked. In the context of the ISA2 programme, efficiency is assessed at action and programme levels. The implementation of EVM requires managers to calculate the earned value, i.e. a quantification of the ‘worth’ of the work done to date, and the actual costs, i.e. the executed budget for achieving the work, and to compare them with the planned value of such activity. This allows for a better understanding of the programme's performance. The ES is an extension of the EVM method that deepens the level of analysis to a ‘units of time’ layer. In the framework of the ISA2 programme, tailored versions of the EVM and ES approaches are adopted. Limitations and robustness of findings

The consultation strategy and the data collection plan put forward in the Evaluation Framework ( Annex 4 ) were fully implemented. Therefore, the evaluation was able to draw robust conclusions for all EQs based on both primary and secondary data. Nevertheless, some emphasis needs to be placed on caveats that may have an impact on the main findings of the interim evaluation as well as on future evaluation exercises: The timing of this evaluation is compliant with Article 13 of the ISA2 Decision, which requires the Commission to present an interim evaluation to the European Parliament and the Council by 30 September 2019. Given this constraint, the evaluation confined the analysis to actions funded during the programme's first three years (2016, 2017 and 2018). In particular, some actions under evaluation are still ongoing; hence, their outputs and outcomes can only be recorded to some extent. On a more general note, as discussed in the intervention logic ( Annex 3 ), impacts occur over the long term and it is difficult to capture them in an interim evaluation. In principle, in the coming years ISA2 actions may see improvements in the way they achieve the programme’s objectives and deliver the expected results. Only action owners had access to the contact details of their solution users due to confidentiality and data protection reasons. Therefore, action owners were requested to: i) invite users to complete the relevant targeted online survey; and ii) share with DIGIT.D2 the details of those users who gave their consent to participate in interviews. This two-step approach may have limited the number of answers received from solution users. In addition, some action owners explained that they were unable to contact their users for the purpose of evaluation, as the users’ consent to be involved in consultation activities was never collected. Against this background, ISA2 may consider requesting its users to provide their consent to be contacted by Commission staff and contractors for evaluation purposes in the future.

5.Analysis and answers to the evaluation questions In line with the Evaluation Framework ( Annex 4 ), the interim evaluation focused on seven evaluation criteria: relevance, effectiveness, efficiency, coherence, EU added value, utility and sustainability. The below chapters present the following for each of them: a) the pre-defined evaluation questions (EQ) and related judgment criteria; b) the results of the analysis fed by the collected data; and finally c) the answer to the EQ. Relevance Evaluation framework for 'relevance' Relevance refers to the alignment between the programme's objectives and the evolving needs and problems experienced by stakeholders. As shown in Box 1 , the interim evaluation translated this criterion into the following EQ: EQ1:To what extent are the objectives of the ISA² programme still pertinent in relation to the evolving needs and problems at both national and EU levels? To answer this question, the evaluation assessed two main judgment criteria (JCs), by relying on data collected via consultation activities as well as on expert assessment: JC1:Alignment between needs and problems addressed by the programme and current needs and problems experienced at the EU, national and subnational level; and JC2:Alignment between the objectives of the programme and current needs and problems experienced at the EU, national and subnational level. Analysis JC1: Alignment between original and current needs and problems The rapidly evolving nature of ICT and interoperability begs the question as to whether the needs and problems originally targeted by the programme are the same as the current needs and problems experienced by stakeholders. Figure 3 shows the average scores derived from the answers received on this question, per stakeholder group. Overall, more than 70 % of respondents consider that the three original needs, as presented in Figure 3 , are still relevant to a high extent or to the fullest extent (93 out of 128 respondents for the first need, 99 out of 127 respondents for the second need, and 94 out of 125 respondents for the third need). The persistence of the problem of administrative e-barriers originally identified by the programme is confirmed as well. Two-thirds of the respondents (79 out of 118) consider this problem still relevant to a high extent or the fullest extent, in comparison to almost 13 % (15 out of 118) who believe the problem is relevant only to a limited extent or not at all. Figure 3: Extent to which needs and problems originally addressed by ISA2 are currently experienced by European public administrations, businesses and/or citizens (average score of answers, number of respondents) Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest extent. Note: Averages do not account for respondents answering ‘don’t know/no opinion’ (DK/NO).

Source: CEPS final study There are some additional needs and problems that the programme could aim to address, as pointed out by 39 % of the consulted stakeholders (46 out of 129). More specifically, several respondents converged on the following needs and problems related to interoperability of public services that European public administrations, businesses and/or citizens currently experience: EU level: The need for a more binding legal framework for interoperability, including the need for legally binding standards or the general need for a more binding approach to interoperability in the EU. The need for a more prescriptive approach to designing interoperable public services (‘interoperability by design’). It is worth noting, however, that the revised EIF adopted in 2017 explicitly mentions ‘interoperability by design’ as the conceptual model for interoperable public services. Therefore, this need is taken into account through the wider framework of the EIF. The need to improve the way administrations communicate with one another, given the different forms of organisation and functioning of public administrations in the Member States. The need to share best practices. ·The need to account for new developments such as blockchain, privacy-by-design and self-sovereign identities, which are changing the interoperability landscape. National/subnational level: Resource constraints, such as shortage of qualified IT staff, experienced by national and local public administrations. The different political priorities among Member States hindering a consistent approach to interoperability in the EU. ·The limited awareness of ISA2 and other initiatives related to interoperability, especially at the regional and local levels. JC2: Alignment between needs and problems and the programme's objectives After ensuring that all the original needs and problems are still relevant and identifying additional needs and problems experienced by stakeholders, the evaluation checked the degree to which the programme's objectives address these needs and problems. The majority of consulted stakeholders agree that, through its general objective, ISA2 addresses the main needs and problems identified in the field of interoperability of digital public services at the time the programme was set up, either to a high extent or to the fullest extent ( Figure 4 ). This result holds true across all stakeholder groups. Over 80 % of respondents indicate that, by achieving its general objective, ISA2 can address the original needs to a high extent or to the fullest extent (103 out of 125 respondents for the first need, 103 out of 123 respondents for the second need, and 103 out of 124 respondents for the third need). When it comes to the problem of administrative e-barriers, approximately two-thirds of respondents (77 out of 116) agree that ISA2 can address the problem through its general objective, while only 11 % (13 out of 116) believe that the programme can only address the problem to a limited extent or not at all. These results are also confirmed by the independent assessment carried out by technical experts, who pointed out that ISA2 highlights the topic of interoperability and helps build a community of professionals in this area, thus addressing the needs and problems identified in the intervention logic. Figure 4: Extent to which achieving ISA2 objectives contributes to addressing the needs and problems originally addressed by the programme (average score of answers, number of respondents) Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest extent. Note: Averages do not account for respondents answering ‘don’t know/no opinion’ (DK/NO). Source: CEPS final study

When it comes to the new needs and problems identified by consulted stakeholders, on average, respondents believe that the programme, through its general objective, could address them at least to some extent ( Figure 5 ). However, three out of the six respondents responsible for linked EU initiatives who indicated additional needs and problems consider that the general objective of ISA2 cannot address the additional needs and problems, or can only address them to a limited extent, as some of the issues mentioned fall outside the programme's scope (for instance, Member States' different political priorities). Figure 5: Extent to which achieving ISA2 objectives contributes to addressing additional (current) needs and problems identified by consulted stakeholders (number of respondents by stakeholder category) Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest extent; (DK/NO) do not know/no opinion. Source: CEPS final study Answer to the EQ on 'relevance' EQ1: To what extent are the objectives of the ISA² programme still pertinent in relation to the evolving needs and problems at both national and EU levels? The original needs and problems that the programme intended to address are still fully relevant and the objectives of ISA2 are pertinent to addressing them. In addition, consulted stakeholders have identified a number of new needs and problems related to the interoperability of digital public services. Some of these fall outside the programme's scope (such as resource constraints or different political priorities at national level). Others are already being addressed by existing ISA2 actions (for example, the 'Innovative Public Services' action provides support for identifying the innovation potential and framing conditions of emerging disruptive technologies). Finally, ISA² could (better) address some of these additional needs and problems by considering the below measures for future actions and future editions of the programme: Increasing awareness of interoperability beyond national administrations by fostering more collaboration and exchanges of views with regional and local administrations. The recently launched Interoperability Academy action could help with this by developing training materials tailored to the needs of regional and local governments. Performing studies to assess the costs and benefits of implementing ISA2 solutions. The findings of such studies could feed into promotional activities, targeting the sub-national layer of public administrations as well. Improving the sharing of best practices among public administrations, academia and interested professionals thus raising awareness of interoperability and increasing the take-up of ISA2 solutions across Member States. Relying more on potential influencers (like researchers, teachers or committed public servants) when communicating about interoperability. This could eventually create a community of interoperability ambassadors. Building advisory capabilities around interoperability. These would help interested stakeholders to pick the appropriate interoperability solutions that match their needs, and would at the same time provide support services and technical assistance for effective implementation. ·Assessing the rationale and impacts of a possible binding interoperability instrument. Such assessment should rely on the findings of the final evaluation of the ISA2 programme and on the evidence that will be gathered when the implementation of the EIF is evaluated in 2021. Effectiveness Evaluation framework for 'effectiveness' Effectiveness focuses on the extent to which the programme has achieved its objectives and generated the expected results. Analysing the effectiveness of ISA2 requires answers to two EQs: EQ2:How far are the ISA² programme’s results in the process of achieving the programme’s objectives? EQ3:Are there aspects that are more or less effective than others are, and if so, what lessons can be drawn from this? To answer these questions, the evaluation applied the following judgment criteria (JCs):

JC1:Alignment between the programme's actual results, objectives and expected results; JC2:Impact of external factors on the programme's performance; JC3:Awareness of the programme; JC4:Performance indicators; and JC5:Alignment with principles specified in Article 4(b) of the ISA2 Decision. The analysis relies on data collected via consultation activities, a desk review and an expert assessment. Analysis JC1: Alignment between the actual results, objectives and expected results of ISA2 The focus of this judgment criterion falls on the solutions developed by ISA2, as they represent the most tangible results achieved so far. Figure 6 shows the extent to which the programme's actual results are aligned with its objectives, according to the consulted stakeholders. 45 % of respondents (49 out of 109) agree that ISA2 solutions have contributed to achieving the programme's general objective to a high extent or to the fullest extent. In contrast, only 15 % of respondents (16 out of 109) believe that ISA2 solutions have made either no contribution or a limited contribution. The positive feedback is higher when it comes to the programme's specific objectives. For instance: More than 56 % of respondents (59 out of 106) confirm that the solutions developed have contributed to developing, maintaining and promoting a holistic approach to interoperability in the Union (specific objective 1) to a high extent or to the fullest extent; ·More than 56 % of respondents (60 out of 108) indicate that the solutions developed have contributed to facilitating the reuse of interoperability solutions by European public administrations (specific objective 5) to a high extent or to the fullest extent. However, respondents to the consultation activities express a lower level of agreement regarding the contribution to the development of a more effective, simplified and user-friendly e-administration at the national, regional and local levels of public administration (specific objective 3). Only 34 % of respondents (36 out of 106) consider that ISA2 solutions have contributed to this objective to a high extent or to the fullest extent, while 22 % (23 out of 106) believe that there has been either no contribution or only a limited contribution. This is aligned with the assessment of the EU added value criterion (see Figure 17 in section 5.5), which shows that Member States’ initiatives tend to perform relatively better when it comes to achieving specific objective 3. Figure 6: Extent to which ISA2 solutions contributed to achieving the programme’s objectives (average score of answers, number of respondents) Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent. Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). Source: CEPS final study The desk review of secondary data also shows good progress towards the achievement of ISA2 objectives. The 20 sampled actions have already produced 35 solutions, and a further 21 solutions are under development. Most of the operational solutions are used across the 28 Member States and across different EU institutions, thus contributing to a coherent approach to interoperability in the EU ( Annex 6 : Table 9 , Table 10 and Table 11 ). The Interinstitutional Register for Delegated Acts is a good example that illustrates how the various ISA2 solutions help achieve the programme’ objectives. It was set up under the ‘RegDel’ action in order to respond to the joint commitment of the Commission, the Council and the European Parliament to facilitate access to delegated acts through a dedicated register. By doing so, the register also ‘supports the implementation of Union policies and activities’ as envisaged by specific objective 4 of the ISA2 programme. Launched in December 2017, the solution has amassed 2202 active subscriptions and over 144 000 views (up to 1 February 2019). Another example is the European Interoperability Reference Architecture (EIRA©), which helps classify and organise interoperability building blocks used in the delivery of digital public services. EIRA facilitates the reuse of interoperability solutions by European public administrations (specific 5 of ISA2) and is appreciated by its users, as shown in Box 2 .

Box 2: EIRA user testimonial Source: ISA2 communication team Other examples include the ‘Joinup’ solution and events such as the ‘Sharing and Reuse Conference’ that facilitate access to and the reuse of interoperability solutions, as well as the exchange of best practices. Turning to the operational objectives ( Figure 7 ), the highest achievements of ISA2 solutions pertain to the following ones: Operational objective 7 – 68 % of respondents (42 out of 62) confirmed that the existing solutions have contributed to supporting and promoting the maintenance, updating, promotion and monitoring of the implementation of the EIS, the EIF and the EIRA to a high extent or to the fullest extent, and only 1 % (1 out of 62) believe that there was a limited contribution. One good example is the European Interoperability Architecture (EIA), which has developed a conceptual reference architecture that is already deployed in eight Member States. Moreover, the European Interoperability Reference Architecture (EIRA) solution that is available on the ‘Joinup’ platform has been downloaded 1 364 times so far ( Annex 6 : Table 12 ). In addition, the National Interoperability Framework Observatory contributes to this objective by assessing interoperability developments in Member States and national alignment with the EIF. ·Operational objective 9 — according to 68 % of respondents (50 out of 73), ISA2 solutions have contributed to supporting and promoting the maintenance and publication of a platform allowing access to and collaboration with regard to best practices, functioning as a means of raising awareness and disseminating available solutions to a high extent or to the fullest extent, compared to 11 % (8 out of 73) who believe that the contribution has been limited. The ‘Joinup’ action, by maintaining and further developing the collaborative platform with the same name, is directly related to this operational objective. The platform counts over 13 000 members, namely professionals working in the field of eGovernment. It allows members to work together on over 109 thematic working spaces, called ‘collections’ ( Annex 6 : Table 12 ). Based on the feedback received, there are two areas for potential improvement: ·Operational objective 4 — only 23 % of respondents (15 out of 64) believe that ISA2 solutions have contributed to supporting and promoting the identification of legislation gaps, at Union and national level, that hamper cross-border or cross-sector interoperability between European public administrations to a high extent or to the fullest extent. ·Operational objective 5 — only 25 % of respondents (17 out of 68) indicate that ISA2 solutions have contributed to supporting and promoting the development of mechanisms that measure and quantify the benefits of interoperability solutions including methodologies for assessing cost-savings to a high extent or to the fullest extent. The fact that the ‘Legal Interoperability’ action, responsible for the advancement of these two operational objectives, has until now mainly focused on the Commission’s internal policy-making process, may explain the lower awareness levels among the programme’s stakeholders. Figure 7: Extent to which ISA2 solutions contributed to achieving the programme’s operational objectives (average score of answers, number of respondents) Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent. Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). Source: CEPS final study As the programme has only just passed its midway point, it is reasonable to assume that its actual results might not fully match the expected results, since actions are ongoing and solutions will continue to be developed until the programme ends in 2020. Experts confirmed this assumption, pointing out that ISA2 solutions have contributed to, for instance, ‘a coherent interoperability landscape in the Union’, but more time is needed for the expected results to be fully realised. The feedback received as part of the consultation activities provides further evidence of this ( Figure 8 ). Figure 8: Extent to which ISA2 solutions achieved the programme's expected results (average score of answers, number of respondents) Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). Source: CEPS final study The developed solutions are mostly aligned with the expected result of a coherent interoperability landscape in the EU based on a holistic approach to interoperability. In fact, around 50 % of respondents (37 out of 77) believe that ISA2 solutions have achieved this expected result to a high extent or to the fullest extent, in comparison to 8 % (6 out of 77) who argue that there has been limited achievement or no achievement in this area. The achievement of expected results is further assessed below, when the performance indicators are presented. Turning to areas of potential improvement, only 30 % of respondents (23 out of 76) consider that ISA2 solutions led to improved efficiency and productivity of the European public administrations. Interestingly, during the interviews some respondents pointed out that it is difficult to reply to this specific question, as no specific studies have been carried out on the efficiency and productivity effects of ISA2 solutions, although such effects are expected. Future actions may aim to rectify this and ensure that impacts generated by ISA2 solutions for public administrations are identified and quantified. JC2: Impact of external factors on the programme's performance In line with the Evaluation Framework, assessing the programme's effectiveness also implies considering the positive and negative impacts of external factors on the programme's overall performance. To this end, consulted stakeholders were asked to identify and rank the external factors that on the one hand contribute to and on the other hand jeopardise the programme's performance. There is a clear pattern in the answers of all groups consulted on this matter (programme governance stakeholders, action owners and solution users). The calls for common standards and frameworks from public administrations are considered particularly beneficial to the programme's performance ( Figure 9 ), as they can guide the development of new solutions to address the issues raised by public administrations, i.e. the direct beneficiaries of the solutions. Most respondents, 81 % (63 out of 75), believe that these calls contribute to the performance of ISA2 to a high extent or to the fullest extent. This also shows that interoperability is among the priorities of European public administrations. Figure 9: Extent to which the following external factors contribute to the performance of ISA2 (average score of answers, number of respondents) Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent. Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). Source: CEPS final study On the other hand, institutional complexity is the external factor seen as having the greatest negative impact on the programme's performance ( Figure 10 ). 78 % of respondents (64 out of 82) believe that this factor jeopardises the programme's performance to a high extent or to the fullest extent. Institutional complexity comes in the form of multiple layers of governance and differences between national public administrations in various Member States as well as between local public administrations within Member States, all of which can cause coordination issues. Technical experts confirmed this result by identifying institutional and legal complexity as the factors that most jeopardise the programme’s performance. Figure 10: Extent to which the following external factors jeopardise the performance of ISA2 (average score of answers, number of respondents) Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent. Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). Source: CEPS final study JC3: Awareness of the programme When analysing the feedback collected via the consultation activities, each stakeholder group's level of awareness of interoperability issues and, more specifically, of the ISA2 programme, must be taken into consideration. As illustrated in Annex 2 ( Figure 30 and Figure 31 ), the consulted stakeholders were asked to assess their knowledge of digital public services and interoperability as well as their knowledge of ISA2. On average, the results show that most respondents have expert knowledge of interoperability and of ISA2. 71 % of respondents (91 out of 128) are familiar with digital public services and interoperability to a high extent or to the fullest extent, in comparison to 11 % (14 out of 128) who either are not familiar with the field or are familiar with it only to a limited extent. In addition, 63 % (81 out of 128) of respondents are familiar with the ISA2 programme to a high extent or to the fullest extent, compared to 20 % (25 out of 128) who report only a low degree of familiarity or no familiarity with the programme.

One factor that explains these positive awareness results is the ISA2 programme’s strong commitment to ‘enhance information and increase awareness with regard to the programme and its benefits’, as stated in Article 3 of the ISA2 Decision. In fact, raising public administrations’ awareness of interoperability was one of the three recommendations stemming from the final evaluation of the ISA programme. The ISA2 programme addressed this by adopting its communication strategy and engagement plan in 2017 and by increasing its presence at conferences and other events, as shown in Box 3 . The relatively low participation in the interim evaluation’s consultation activities and the need to better liaise with regional and local administrations, identified under the Relevance criterion (see section 5.1), call for continued and even improved awareness raising efforts. JC4: Performance indicators In line with the Evaluation Framework, the desk research collected data on several performance indicators ( Annex 6 : Table 12 ) in order to assess the extent to which the programme’s operational objectives had been achieved and to complement the data gathered via consultation activities. When talking about performance indicators, it is important to distinguish between different categories of ISA2 actions: actions continued from ISA and actions started during ISA2; and ·actions already completed at the time of the interim evaluation and actions still ongoing. The different stages of an action imply different levels of achievement of objectives and expected results, as some solutions are operational while others are in the early proof-of-concept stage. This should be kept in mind when looking at the performance indicators’ results ( Box 4 ). Data show that the 20 sampled actions have developed 35 solutions so far, out of which 10 emerged under ISA2. As shown in Table 2 , the solutions fall into one of the following four categories: common tools/services, common frameworks, common specifications/standards, and studies. Table 2: Overview of solutions developed under the sampled actions, by type Type of solution Total number of solutions New solutions developed by ISA2 Common tools/services 20 4 Common frameworks 9 2 Common specifications/standards 3 1 Studies 3 3 TOTAL 35 10 Source: CEPS final study Aside from these four solution types associated with ISA2 actions, there is also a particular case represented by the ‘Raising Interoperability Awareness — Communication Activities’ action, which is part of the ‘Accompanying measures’ package. There are no solutions associated with this action, and its outputs are of a specific nature, namely events organised or events to which ISA2 representatives actively contributed. Box 3 presents an overview of these communication activities. Box 3: Communication activities From 2016 to the end of 2018, ten major events were organised as part of the ‘Raising Interoperability Awareness — Communication Activities’ action, with an average of 211 participants per event. The events were organised in six different countries: Belgium, Bulgaria, Greece, Italy, Malta, and Portugal, thus ensuring that a wide pool of professionals was reached. In addition, ISA2 representatives participated in 60 events organised in 18 Member States and one EU candidate country (Montenegro) between 2016 and 2018. The number of participants in these events ranged from 125 to 6 000, thus ensuring that information about the programme was disseminated to numerous stakeholders. Tables 13 and 14 in Annex 6 present an overview of these events. The ISA2 website also lists past and upcoming events. An array of metrics are in use to assess the effectiveness of solutions, for example, the number of users, instances of use, downloads and page views. When these performance indicators are assessed in a more in-depth way, it is clear that they support the feedback received from consulted stakeholders that the programme is on track to achieving the expected results. However, it is also clear that more time is needed to fully achieve these results. This pattern is also confirmed when the performance indicators for actions continued from ISA are compared with those for actions started during ISA2, as detailed in Box 4 . Box 4: Comparing solutions and performance indicators of actions continued from ISA and actions started during ISA2 The pattern that emerges when the performance indicators associated with actions continued from ISA and those started during ISA2 are assessed can be illustrated using the two sampled actions that are part of the semantic interoperability package: ·‘Public Multilingual Knowledge Management Infrastructure’ (PMKI) for the Digital Single Market (initiated during ISA2); and ·‘SEMIC: Promoting semantic interoperability amongst the European Union Member States’ (a continuation from the action with the same name that was in place during ISA). The ‘PMKI’ action has so far provided two solutions: i) a common framework in the form of a core data model for multilingual taxonomies/terminologies; and ii) a common specification in the form of semantic links between different language resources. The most suitable performance indicator associated with the core data model is measured by ‘instances of inclusion in academic journals and international conference’. Since the model was published in 2017, two scientific papers based on it have been published in international journals, as part of the action. In addition, the work carried out on the model has been presented at seven international conferences. With regard to the semantic links, the main performance indicator is measured by instances of collaboration with European public administrations with a view to establishing semantic interoperability between national language resources and EuroVoc. Four collaborations have been set up so far. In comparison, for the ‘SEMIC’ action, which has been in place longer than PMKI, the relevant performance indicators are the number of solution users, in particular the number of public administrations using the solutions. For the four solutions that are part of SEMIC, the numbers of users are as follows: ·the ‘Asset Description Metadata Schema’ (ADMS) is used in eight public administrations in the Member States and the Commission, and by one company. In addition, the solution has been used to describe approximately 4 700 interoperability solutions on Joinup; ·the ‘Core Vocabularies’ are used in 11 public administrations in the Member States and in the Commission; ·the ‘Data Catalogue Vocabulary Application Profile for Data Portals in Europe’ (DCAT-AP) is used by 29 public administrations in the Member States and the Commission, and by associations and universities; and ·‘VocBench3’, which also falls under SEMIC and previous versions of which are used in 14 public administrations in the Member States and the Commission, and by universities, institutes, and international organisations. An analysis of the solutions developed under the PMKI and SEMIC actions suggests that the performance indicators assessing actions continued from ISA tend to show a higher degree of maturity and take-up than those developed under ISA2. Therefore, it is clear that solutions need time to show fully-fledged results. This is a natural conclusion, and since there is still over one year and a half left until the ISA2 programme concludes, there is still time for solutions to be further developed.

JC5: Alignment with principles specified in Article 4(b) of the ISA2 Decision The evaluation of effectiveness also requires an assessment of the programme’s compliance with the principles listed in Article 4(b) of the ISA2 Decision. The rolling work programme provides details on how the ISA2 actions should consider these principles. The overall judgment stemming from the desk analysis of the rolling work programme is positive ( Annex 6 : Table 15 ). Supporting examples are provided by examining the contribution of ISA2 actions to the implementation of the EIF, as the principles in question also fall under the scope of the EIF ( Annex 6 : Table 23 ). Answer to the EQs on ‘effectiveness’ EQ2: How far are the ISA² programme’s results in the process of achieving the programme’s objectives? The results achieved by ISA2 so far are aligned with the programme’s objectives. However, they do not yet fully match the expected results, as most of the actions are still ongoing and solutions are still being developed. More time is needed in order to achieve all the expected results. This is confirmed by the assessment of key performance indicators and a comparison between actions continued from ISA and actions started under ISA2. External factors could improve but also jeopardise the way in which the programme achieves its objectives and delivers its results. The main findings hold true across the different stakeholder groups and are further confirmed by expert assessments. The call for common standards and frameworks from public administrations is an external factor that contributes to the programme’s performance. In contrast, institutional complexity could harm the achievement of cross-border and cross-sectoral interoperability. EQ3: Are there aspects that are more or less effective than others are, and if so, what lessons can be drawn from this? While in general the programme is on track to achieving its objectives, it performs relatively less well when it comes to developing a more effective, simplified and user-friendly e-administration at the national, regional and local levels of public administration. This is an area where Member States can in fact have a greater impact. National initiatives related to the ICT modernisation of the public sector can also improve the overall performance of ISA2 by increasing the capacity for the take-up of ISA2 solutions. Regarding the take-up rate of solutions, the evaluation found that the actions that continued from previous editions of the programme have produced solutions that are now more widely used than solutions resulting from newly established ISA2 actions. As regards the specific request of Article 13(5) of the ISA2 Decision to assess the programme’s compliance with the principles set out in Article 4 of this decision, the evaluation found that the rolling work programme process effectively ensures that the ISA2 actions take into consideration the core principles listed in the ISA2 Decision. When it comes to communication and awareness-raising activities, the evaluation results suggest that ISA2 duly acted upon the recommendation of the final ISA evaluation and increased general awareness of the programme. Still, some aspects of ISA2 could benefit from more promotion. For example, as legal interoperability is specifically mentioned among the operational objectives, the solutions developed for assessing the ICT and interoperability impacts of legislation could be more prominently featured in order to improve awareness among stakeholders. Moreover, ISA2 could better stress the benefits of implementing certain interoperability solutions. To this end, it could run studies that quantify the impact of interoperability solutions on the efficiency and productivity of public administrations. Such studies would improve the conditions for assessing the programme’s effectiveness and could have positive spillover effects on dissemination and communication activities: the monetary benefits deriving from ISA2 solutions would be better emphasised, thus providing a better basis for stakeholder support of future EU efforts promoting interoperability. Finally, as noted under section 5.1 Relevance , ISA2 communication activities should better target the sub-national layers of administrations. Efficiency Evaluation framework for ‘efficiency’ Efficiency relates to the extent to which the programme’s objectives are achieved at a minimum cost. This evaluation criterion includes two EQs and one sub-question: EQ4:To what extent has the programme been cost-effective? EQ5:Which aspects of the programme are the most efficient or inefficient, especially in terms of resources mobilised? EQ5.1: How is the programme performing relative to the planned work and budget? To answer these questions, the evaluation focused on three judgment criteria, relying on data collected from the ISA2 dashboard, the monitoring and evaluation reports and consultation activities: JC1:Efficiency of the selection process of the actions to be included in the rolling work programme; JC2:Cost-effectiveness analysis to assess the ratio between allocated funds and actual results of the programme; and

JC3:Earned value management analysis. Analysis JC1: Efficiency of the action selection process The ISA2 programme is implemented in four phases: i) submission; ii) evaluation; iii) adoption; and iv) implementation (further details are provided in Chapter 3). The process to select ISA2 actions includes the first three phases, i.e. submission, evaluation, and adoption. Note that this section does not assess the selection of contractors who will implement the actions, as this process follows the general European public procurement rules. Several elements of the ‘submission’ phase need to be explained here. First, it is important to note that, in order to prepare and submit a proposal to be included in the ISA2 rolling work programme, the applicant must: i) find out about the call for proposals; ii) study the documents of the call for proposals and understand the rules and procedures; iii) prepare a concise description of the proposed action in compliance with the ‘work programme entry template’; iv) collect the required internal authorisations to submit the proposal; and v) submit the proposal by e-mail. Second, it is important to distinguish between two types of proposal, namely proposals for new actions and proposals for continuing actions that have already started. This distinction is relevant when it comes to assessing the overall workload required for submitting a proposal. The efficiency of the action selection process (which includes the submission, evaluation and adoption phases) is assessed based on feedback received from programme governance representatives and action owners, as they are the stakeholder groups directly involved in the process. Most respondents consider the process fit-for-purpose at least to some extent: 16 out of 19 consulted programme governance stakeholders and 21 out of 23 consulted action owners ( Figure 11 ). However, 13 out of the 42 consulted stakeholders indicated that some measures could still be taken to further streamline the process. The measures mentioned by several stakeholders include: Simplifying the rolling work programme, which can be done by: oMaking the template more flexible to account for action specificities. This would ensure better input of information, which would make the process of drafting the proposals more efficient, and would also make it easier for anyone seeking specific information about ISA2 actions to read and understand the work programme. oExtending the duration of a rolling work programme to cover more than one year, thus allowing for multiannual actions. In principle, actions to be run on a multiannual basis can be proposed, but the budget is allocated on an annual basis. Therefore, action owners must submit an updated proposal every year as part of the rolling work programme in order to access funding. oSetting up an electronic data input system rather than full text editing. ·Simplifying the overall process by launching thematic calls for actions, thus ensuring that the selection is more objective-driven. Figure 11: Extent to which the action selection process is fit-for-purpose (number of respondents by stakeholder category) Source: CEPS final study As regards the submission phase of the selection process, 26 out of the 42 consulted action owners and programme governance representatives indicated that they had prepared and submitted proposals. Out of these 26 respondents, 21 also specified the amount of time they spent on preparing the proposal. The answers provided vary between 1 to 30 person-days, with the average being 9 person-days. The large variation can be primarily explained by the differences in the efforts required to submit a proposal for a new action versus the amount of effort required to renew a proposal for an existing action. As regards the time needed for submission, Figure 12 shows a clear concentration of answers: 14 of the respondents indicated that they finalised their proposal in 1 to 6 person-days. However, four respondents indicated that 20 or more person-days were needed. This distribution of answers and the qualitative inputs provided by some of the respondents during the consultation activities makes it possible to differentiate between preparing a renewed proposal and a brand-new proposal. Renewing a proposal for an action that has already been included in a previous rolling work programme tends to require between 1 and 6 person-days at most, while preparing a new proposal takes significantly longer, between 10 and 30 person-days. In most cases, stakeholders prepare the proposals themselves, without relying on external contractors. Additional time is also required to merge similar proposals, upon request by the Commission team in charge of shortlisting the proposals to be funded. Figure 12: Distribution of the number of days needed to prepare and submit a proposal for an ISA2 action (number of respondents) Note: The histogram was constructed using a bin size of 2 days, which grouped the answers of respondents in intervals of 2 days. This bin size was deemed most appropriate after reviewing the answers provided.

Source: CEPS final study The time spent to prepare and submit a proposal for an action to be included in the rolling work programme, whether it is a new or a renewed proposal, can be used to measure regulatory costs by applying the standard cost model, presented in the Better Regulation Toolbox. First, person-days are converted into person-hours by assuming that each person-day corresponds to eight hours. Then, the overall number of hours is multiplied by a standard tariff, i.e. the 2017 hourly labour cost for the service sector at the Member State level (see Annex 6 : Table 22 ). Two respondents indicated that they relied on external service providers to prepare the proposal. In this case, regulatory costs would also include out-of-pocket expenses incurred for such services. Table 3 presents the average regulatory costs both taking into account the fees related to external service providers and without accounting for these additional fees. The average respondent faced regulatory costs of approximately €1 077 for renewing a proposal and €5 651 for preparing a new proposal, without accounting for fees incurred when relying on external service providers. Considering the two instances when external service providers were used in the preparation of proposals, the average regulatory cost for a renewed proposal amounted to €1 138 and for a new proposal the average regulatory cost was €6 590. At the same time, the average funding allocated to an ISA2 action between 2016 and 2018 amounted to €1 217,566. The costs incurred to prepare a proposal are thus very small, ranging between 0.09 % and 0.5 % of the potential funding that could be allocated to the proposal once accepted. Table 3: Average regulatory costs incurred by respondents who prepared and submitted proposals for ISA2 actions Average regulatory cost, including external services (€) Average regulatory cost without external services (€) Average funding allocated per action (€) Renewed proposal 1 138 1 077 1 217,566 New proposal 6 590 5 651 Note: The calculations are based on a sample of 21 respondents who specified the time spent on preparing proposals. Source: CEPS final study One respondent mentioned that the reason they relied on external service providers in this process was the need to tailor the proposal to the requirements of the rolling work programme template. As such, one potential cost-reduction method could be the simplification of the rolling work programme template, thus further corroborating the measures proposed to streamline the action selection process. JC2: Cost-effectiveness analysis Based on the performance indicators discussed in section 5.2 on effectiveness and on data related to the actual costs of actions, a cost-effectiveness analysis could be performed. However, it is important to note the limitations of this method. First, the heterogeneity of performance indicators makes it difficult to draw conclusions about the programme’s overall cost-effectiveness ( Box 5 shows the variety of ISA2 performance indicators). Consequently, computing an aggregate value that is representative of the programme or action packages is quite difficult. Second, such data are only partially available, as some ISA2 solutions are still under development. In addition, as concluded in section 5.2, it is expected that the values of the performance indicators will improve over time, as solutions tend to require longer periods to fully show their results. Third, data on effectiveness are not yet available for a limited number of solutions developed so far. Against this background, the cost-effectiveness analysis can be exemplified by using some of the ISA2 packages that rely on similar performance indicators. Two packages fit this profile: ‘Key and generic interoperability enablers’ and ‘eProcurement / eInvoicing — supporting instruments’. The sampled actions included in both packages use the number of public administrations (at EU, national and regional levels) that use the solutions generated by the actions as a performance indicator. Table 4 presents the estimated average cost per solution user based on the available data on the number of solution users (public administrations) and the actual costs of the action between 2016 and Q3 2018. For the ‘Key and generic interoperability enablers’ package, the average actual cost per public administration using the solutions that are part of this package amounts to €19 420. For the ‘eProcurement / eInvoicing — supporting instruments’ package, the average actual cost amounts to €25 870. Note that these average costs are calculated per public administration using the solution rather than per single end-user. Each administration serves millions of end-users (citizens/businesses); hence, costs per end-user are estimated to be low. Table 4: Costs per user of ISA2 solutions Package Actual costs of sampled actions in the package (€) Number of public administrations using the solutions of the package Average actual cost (€) Key and generic interoperability enablers 4 117,069.81 212 19 420 eProcurement / eInvoicing — supporting instruments 3 544,135.98 137 25 870 Note: The calculations are based on the sampled actions of the indicated ISA2 packages over the period 2016 and Q3 2018. A full overview of the number of users and other performance indicators is presented in Annex 6 : Table 12 . The overview of costs per package is shown in Figure 13 . Source: CEPS final study The appraisal of the programme’s cost-effectiveness could be made more consistent by ensuring that monitoring and evaluation reports quantify some performance indicators common to all actions. Examples of such indicators include the number of EU public administrations using a solution, the number of public administrations participating in pilot projects, a consistent approach to reporting the number of downloads from the ‘Joinup’ platform for solutions that can be downloaded, and/or the number of single users per solution. Box 5: Summary of the main costs and benefits identified Costs. The main costs associated with the intervention fall into two categories: regulatory costs of the selection process for ISA2 funded actions (see Table 3 ) and the costs per user of ISA2 solutions (see Table 4 ). Benefits. In turn, the benefits generated are expressed in the form of performance indicators attached to each solution developed as part of ISA2 actions, which can be measured using a variety of metrics. The benefits generated by the programme can thus be summarised as follows (the list is not exhaustive): ·280 Commission proposals have been screened for ICT impacts since 2015; ·13 440 professionals working in the field of eGovernment are registered on ‘Joinup’ in order to access interoperability solutions and collaborate with one another; ·over 8.8 million documents have been exchanged so far between the Commission, the Council, the Member States, and companies using the ‘e-TrustEx’ platform; ·the ‘European Single Procurement Document’ (ESPD) service has been visited around 4 million times (until April 2019); and ·the ‘Core Public Service Vocabulary-Application Profile’ (CPSV-AP) solution has been downloaded from ‘Joinup’ over 2 500 times. Additional performance indicators, collected as part of the desk review of secondary data, are included in Annex 6 : Table 12 . JC3: Earned value management (EVM) analysis

EQ5 and EQ5.1 focus on progress made in implementing the programme based on the EVM analysis. This analysis helps determine progress of work against a given baseline, so that costs, time, and scope of a certain activity are constantly tracked. To support this, the ISA2 dashboard provides quarterly data on the earned value, planned value, and actual costs of each monitored action. As regards the sample of 20 actions grouped in the respective packages, Figure 13 presents the data at the end of Q3 2018 for the earned and planned value, as well as for the actual costs recorded. The earned value is close to equalling the planned value, which shows that the work is progressing as expected. Even where a bigger gap exists (e.g. in the ‘Key and generic interoperability enablers’ package), the indicators are expected to converge at the end of the year as the ISA2 budget year closes. Figure 13: Earned value, actual costs and planned value sampled actions, by package (Q3 2018) Source: CEPS final study The data available also make it possible to perform a historical analysis of work progress for each package of actions. Progress can also be shown in the form of a schedule performance index (SPI), which is the ratio of earned value to planned value ( Figure 14 ). An SPI value above one indicates that the item analysed is ahead of schedule, while an SPI value below one suggests that the item analysed is behind schedule. Since the beginning of the programme and up until Q3 of 2018, the SPIs of the sampled actions grouped by packages have shown fluctuations, but now tend to converge on one, although values below one are still reported for almost all packages. These data show that the ‘Semantic interoperability’ and ‘Supporting instruments for public administrations’ packages are on track, while the others are close to achieving the planned level of work. Given that some actions started later on in the programme, as is the case of the ‘Geospatial solutions’ package, data for the EVM analysis are not always available from Q2 of 2016, but only from a later point in time. SPI values above two were reported for the ‘Key and generic interoperability enablers’ package in Q2 and Q3 of 2016. This indicates that the work on the package was ahead of schedule in that period. While this result might seem surprising at first, it can be explained by the fact that some of the actions involved started under ISA and continued under ISA2. For these actions, processes had already been established during the previous programme, so work under ISA2 could start much faster than planned, resulting in a higher SPI value at the beginning of the process. In fact, both of the actions sampled from the ‘Key and generic interoperability enablers’ package (‘Trusted Exchange Platform e-TrustEx’ and ‘Catalogue of Services’) had been launched under the ISA programme. Figure 14: Schedule performance indices of the sampled actions grouped by packages Source: CEPS final study Answer to the ‘efficiency’ EQs EQ.4: To what extent has the programme been cost-effective? The heterogeneity of performance indicators makes it difficult to draw conclusions about the programme’s overall cost-effectiveness. For those packages where it was possible to apply cost-effectiveness techniques, costs per end-user (e.g. business, citizens, etc.) have been estimated as low. In this respect, monitoring and evaluation reports could converge towards some common metrics allowing for a more consistent appraisal of cost-effectiveness. For example, the agreed performance indicators could comprise the number of EU public administrations using a given solution, the number of downloads from the ‘Joinup’ platform for downloadable solutions, and/or the number of single users per solution. EQ5: Which aspects of the programme are the most efficient or inefficient, especially in terms of resources mobilised?

EQ5.1: How is the programme performing relative to the planned work and budget? The process of selecting actions to be funded by ISA2 is considered relatively efficient and fit for purpose. On average, six person-days at most are required to prepare and submit a proposal for actions that had already been included in previous rolling work programmes. On the other hand, between 10 and 30 person-days are required to prepare a proposal for a new action. The costs incurred in preparing a proposal are relatively small, ranging between 0.09 % and 0.5 % of the potential funds that could be allocated to the proposal once accepted. Nonetheless, this process could be further streamlined by simplifying the rolling work programme and launching thematic calls for actions, thus ensuring that the selection process is driven by objectives. When looking at progress made in the programme’s implementation, it is clear that all packages are either on track or close to achieving the planned level of work. As some actions started during ISA and continued under ISA2, some processes had already been established during the previous programme. Therefore, work on these actions under ISA2 could start faster than planned. Coherence Evaluation framework for ‘coherence’ Article 13(4) of the ISA2 Decision requires that the coherence of ISA2 actions is assessed; this is referred to as the programme’s ‘internal coherence’. In addition, Article 13(6) pays specific attention to the programme’s ‘external coherence’ in order to ‘identify potential overlaps, examine coherence with areas for improvement, and verify synergies with other Union initiatives, in particular with the CEF’. The coherence criterion translates into two EQs: EQ6:To what extent do the ISA2 actions form part of a ‘holistic’ approach within the framework of the programme? (Internal coherence)

EQ7:To what extent is the ISA2 programme coherent with other EU interventions, which have similar objectives and with global initiatives in the same field? (External coherence) To assess both the internal and the external coherence of the programme, findings from desk research on links between ISA2 actions and the relationship between ISA2 and other relevant EU programmes/policies/initiatives were used in addition to the information provided by stakeholders. In this context, the following judgment criteria are considered: JC1:Internal coherence: -Degree of coherence among actions funded by the ISA² programme; -Level of reuse of results of a funded action by another action within the ISA² programme; JC2:External coherence: -Degree of coherence between the programme and other EU supported programmes; -Level of reuse of results delivered by ISA2 actions by other EU programmes; -Degree of coherence between the programme and other EU policies; and -Degree of coherence between the programme and global initiatives in the field. Analysis JC1: Internal coherence Given that the ISA2 programme currently manages 53 actions, it is important to ensure that they work together to create benefits and that work is not duplicated. The stakeholder groups consulted on this matter (programme governance, action owners, solution users) tend to agree that ISA2 actions generate synergies ( Figure 15 ). At the same time, most stakeholders indicate only limited overlaps between ISA2 actions. Stakeholder feedback on synergies and overlaps, as two sides of the same coin, paints a positive picture of the internal coherence of the ISA2 programme. Only 4 % of respondents (2 out of 53) identify a high degree of overlaps between ISA2 actions, while the vast majority, 77 % (41 out of 53), consider overlaps as either not existing or being limited. In addition, respondents point to the existence of synergies: 41 % of respondents (27 out of 66) identify synergies as existing to a high extent or to the fullest extent and 17 % (11 out of 66) identify either no synergies or limited synergies. It is relevant to note that a significant number of solution users indicated that they were unaware of either synergies or overlaps between ISA2 actions. This feedback can be explained by stakeholders’ various levels of knowledge about ISA2 packages. As not all stakeholders are familiar with all ISA2 packages, they might not be in a suitable position to identify potential synergies or overlaps. Figure 15: Extent to which there are synergies / overlaps between ISA2 actions (average score of answers, number of respondents) Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent. Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). Source: CEPS final study The review of documentary evidence confirms the results of the consultation activities, particularly regarding the existence of synergies between ISA2 actions. The indicator of the ‘network’ of relationships between the programme’s actions, as shown on the ISA2 dashboard, is particularly relevant here. The quantitative assessment of the data provided online for the sample of 20 actions under assessment shows (see Annex 6 : Table 17 ) that there are multiple instances of internal reuse of ISA2 solutions, with actions like ‘SEMIC’ and ‘Joinup’ playing a particularly important role in the system.

At the same time, some actions tend to be less central in the network. This is the case particularly for the ‘EUSurvey’ and ‘European Citizens’ Initiative and European Parliament Election’ actions. The specificity of these actions makes it more likely that their solutions are reused by other programmes or initiatives, rather than by other ISA2 actions. Throughout the consultation activities, additional inputs were collected from respondents concerning synergies between actions and examples of instances when results delivered by an ISA2 action were used by another action (see Box 6 ). Box 6: Examples of internal coherence between ISA2 actions ·The ‘DCAT-AP’ specification, developed under the ‘SEMIC’ action, has been reused in the development of the ‘CPSV-AP’ under the ‘Catalogue of Services’ action. ‘Access to base registries’ reuses ‘Core Vocabularies’, ‘DCAT-AP’ (both part of the ‘SEMIC’ action) and ‘EIRA’ (part of the ‘European Interoperability Architecture’ action). In addition, ‘SEMIC’ solutions have also been used by the ‘ELISE’ action. ·The ‘Re3gistry’ solution, part of the ‘ELISE’ action, is used to share codes developed for ‘Core Vocabularies’ (part of the ‘SEMIC’ action). ·The platform for secure file exchange ‘e-TrustEx’ (part of the action with the same name) is used in the area of eProcurement by the ‘European Public Procurement Interoperability Initiative’ action, in particular by the ‘e-Prior’ solution developed as part of the action. ‘E-Prior’ is a module built based on ‘e-TrustEx’. ·As regards synergies with the ‘Joinup’ platform, there is a particularly strong relationship between the ‘Sharing and Reuse’ action and ‘Joinup’. The former aims to support public administrations in sharing collaborative development, reusing IT solutions and promoting best practices; the latter is a collaborative platform providing the means to achieve this. ·The ‘National Interoperability Framework Observatory’ (NIFO) action is reusing some of the information from the ‘Access to base registries’ action in order to compile the eGovernment factsheets. JC2: External coherence In order to assess the external coherence of ISA2, stakeholders were asked to assess the level of synergies and overlaps existing between ISA2 and four EU programmes identified as relevant for the field of digitalisation of public administrations and interoperability: i) CEF, ii) ESF, iii) Horizon 2020 and iv) SRSP. The results are presented in Figure 16 . Figure 16: Extent to which there are synergies / overlaps between ISA2 and other relevant EU programmes (average score of answers, number of respondents) Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent. Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). Source: CEPS final study Out of the four programmes, respondents to the consultation activities identify the CEF as having the highest level of synergies with ISA2. According to 78 % of respondents (51 out of 65), ISA2 and CEF have a high level or a very high level of synergies. As an example, both programmes contribute to the Commission’s open data policy: the DCAT-AP solution of ISA2 provides a metadata specification used by open data portals across the EU and CEF ensures funding for the public open data infrastructure that aims to increase the reuse of open public data in the EU. However, 18 % of respondents (10 out of 57) also note the existence of overlaps to a high extent or to the fullest extent, which could be addressed by better delineating the scope of the concerned solutions. When it comes to the relationship between Horizon 2020 and ISA2, half of the respondents (31 out of 63) identify synergies to a high extent or to the fullest extent, while only 12 % (6 out of 51) indicate the existence of overlaps to a high extent or to the fullest extent. The situation is similar for SRSP, with 52 % of respondents (22 out of 42) pointing out that there are synergies between ISA2 and SRSP either to a high extent or the fullest extent, and only 6 % (2 out of 32) indicating a high or very high degree of overlaps. At the opposite end of the spectrum, out of the four programmes listed above, the ESF seems to be the programme with the lowest level of coherence with ISA2. Only 26 % of respondents (6 out of 23) indicate the existence of synergies between the ESF and ISA2 to a high extent or to the fullest extent. Respondents also pointed to synergies and overlaps with other EU initiatives. More specifically, three additional EU programmes and initiatives were mentioned: the European Regional Development Fund (ERDF), corporate IT governance and the Single Digital Gateway. This feedback is discussed in the points below.

While the synergies between the ESF and ISA2 appear to be limited, two respondents pointed to the existence of a high synergy level between the ERDF and ISA2. At the same time, one respondent indicated that there could be overlaps between ISA2 and ERDF, possibly due to a lack of coordination. More specifically, national IT systems developed using ERDF funds could be incompatible with ISA2 solutions, thus preventing the adoption of these solutions. A high degree of complementarity was also identified between the Commission’s corporate IT governance and ISA2, as explained by one respondent. The corporate IT governance team maintains that the Commission’s proposed IT investments take into consideration the reuse of IT solutions that have already been developed. In this sense, the work of this team supports synergies between ISA2 and other initiatives, ensuring that ISA2 solutions and any other IT solutions developed so far are reused in the future. In addition, the team helps mitigate the risk of developing very similar IT solutions under different spending programmes, which would create overlaps. ·One respondent also mentioned the existence of synergies with the Single Digital Gateway (SDG). In fact, this is clear given that ISA2 implemented an action dedicated to the interoperability requirements of the SDG in 2018. ISA2 helped detail the IT architecture of the SDG and identify functional, technical and semantic interoperability challenges in its implementation. Further ISA2 solutions like ‘eForms’ and ‘e-Documents’ also support the SDG’s implementation. Table 18 in Annex 6 shows that ISA2 actions have multiple links outside of the programme, as they rely on other EU programmes / policies / initiatives to deliver their results and their solutions are used by other programmes / policies / initiatives. Beyond examining alignment with the initiatives mentioned above, it is worth analysing the degree of alignment of ISA2 with other initiatives or broader policies, like the Digital Single Market Strategy, the eGovernment Action Plan, ICT standardisation or the Tallinn Declaration on eGovernment. When it comes to the Digital Single Market (DSM) Strategy, the desk review of the rolling work plan reveals that ISA2 contributes to all three of the strategy’s pillars (see Annex 6 : Table 19 ): (i) better access for consumers and businesses to digital goods and services across Europe; (ii) creating the right conditions and a level playing field for digital networks and innovative services to flourish; and (iii) maximising the growth potential of the digital economy. Based on the 20 sampled actions, ISA2 in particular contributes to the strategy’s third pillar through: the revision and implementation of the new European Interoperability Framework, which is explicitly mentioned in the DSM strategy; work on supporting the implementation of the once-only principle, carried out as part of several actions such as ‘SEMIC’ and ‘European Interoperability Architecture’; and ·support for the implementation of the Single Digital Gateway. Together with the launch of the DSM strategy for Europe, a new eGovernment Action Plan was announced for 2016-2020, aiming to ‘remove existing digital barriers to the Digital Single Market and to prevent further fragmentation arising in the context of the modernisation of public administrations’. The ways in which ISA2 contributed to implementing the DSM strategy also directly relate to the eGovernment Action Plan. In particular, the revised version of the EIF and the ISA2 actions monitoring its implementation in the Member States (‘EIF implementation and governance models’ and ‘NIFO’) address Action 4 of the eGovernment Action Plan. Furthermore, ISA2 is also contributing to Actions 1 and 6 through its work in the field of eProcurement. For instance, the ‘European public procurement interoperability initiative’ action facilitates the implementation of eProcurement in European public administrations and sup